Log of the #fcrepo channel on chat.freenode.net

Using timezone: Eastern Standard Time
* westgard leaves01:18
* westgard joins02:01
* westgard leaves02:38
* dbernstein joins02:56
[Import/Export Standup]02:59
Finished yesterday:
https://jira.duraspace.org/browse/FCREPO-2352
Add support for user-supplied Bag metadata
In review - should be ready to go at this point.
https://jira.duraspace.org/browse/FCREPO-2356
Ready to drop a pull request: Just waiting for fcrepo-2352 to be
merged.
Working on today:
Nothing - in transit. Will drop the pull the request between flights once escowles or ruebot merges fcrepo-2352.
Blockers:03:00
None
* thomz joins03:26
* dbernstein leaves03:31
* dbernstein joins03:32
* westgard joins04:35
* westgard leaves04:39
* escowles leaves05:39
* escowles joins05:40
ruebot: based on some quick testing, i think there's just one small issue with https://github.com/fcrepo4-labs/fcrepo-import-export/pull/65 — sounds like dbernstein may not have a lot of time to respond today, so i might just add a commit to fix it06:03
* escowles leaves06:27
* escowles joins07:14
<ruebot>escowles: cool. i'll hold off.07:43
escowles: also https://github.com/LibraryOfCongress/bagit-java/issues/72#issuecomment-26718801007:44
<escowles>ruebot: https://github.com/LibraryOfCongress/bagit-java/issues/72#issuecomment-26731953307:48
<ruebot>escowles++ #thanks!
<escowles>i really wish every ticket we filed wasn't met with hostility
<ruebot>escowles: well, chris was cool. or, at least i've always had a good rapport with him on twitter.07:49
<escowles>i'm tempted to file an issue entitled "why is your code even on github if you don't want constructive criticism?"07:50
<ruebot>escowles: ...just reading this one https://github.com/LibraryOfCongress/bagit-java/issues/69
* ruebot snorts
<escowles>yeah, chris has been very reasonable, but john's been pretty hostile
* dhlamb joins
<escowles>and since our app has to calculate its own checksums, handle the whole checksum/file mapping, and now we're going to have to write our own aptrust-info.txt from scratch, i'm wondering why we're using bagit-java at all07:51
<ruebot>"Bagit is designed to be used by Bagit, not the other way around."
that gives me a sad
<escowles>i feel like i could write an easier-to-use bagit-writer by friday...07:52
<ruebot>escowles++
acdha++08:12
* dwilcox joins08:21
<ruebot>[Import/Export Standup]08:26
Finished yesterday:
- Tested all four datasets for -- Initial BagIt Import implementation -- https://jira.duraspace.org/browse/FCREPO-2350
- Worked with Library of Congress on a couple issues with bagit-java and bagit-python
- https://github.com/LibraryOfCongress/bagit-java/issues/72
- https://github.com/LibraryOfCongress/bagit-python/issues/83
- Worked on implementing -- Use newer version of LoC bagit library in import-export -- https://jira.duraspace.org/browse/FCREPO-2360
- Resolved -- Create a config for the default BagIt Profile -- https://jira.duraspace.org/browse/FCREPO-2228
Working on today:
- Sprint leader duties; Reviewing all open tickets
Blockers:
- LoC bagit-java 5.0.0-RC10; requires a non-standard library, and conversations with the maintainer are awkward
* youn joins08:28
* coblej joins08:29
* informatician joins08:33
* westgard joins08:36
* westgard leaves08:41
* informatician leaves08:45
* informatician joins09:00
* informatician leaves
* peichman joins09:04
* whikloj joins09:06
* github-ff joins09:11
[fcrepo-import-export] escowles created fcrepo-2352-escowles (+3 new commits): https://git.io/v1SsU
fcrepo-import-export/fcrepo-2352-escowles 9b67dee Daniel Bernstein: Adds support for user-supplied metadata...
fcrepo-import-export/fcrepo-2352-escowles 2ec026b Daniel Bernstein: * Adds validation test for Access field....
fcrepo-import-export/fcrepo-2352-escowles fb9e847 Esmé Cowles: Making sure tech metadata doesn't overwrite user-supplied metadata
* github-ff leaves
* github-ff joins09:13
[fcrepo-import-export] escowles opened pull request #66: Adds support for user-supplied metadata (master...fcrepo-2352-escowles) https://git.io/v1SsY
* github-ff leaves
* acoburn joins09:23
<escowles>[Import/Export Standup]09:35
* yesterday: code review, documenting BagIt metadata requirements, tracking down bagit-java bugs
* today: mostly meetings
* blockers: ^^^
* bseeger joins09:39
* justinsimpson joins09:42
<dhlamb>acoburn, wow. looking at fcrepo-camel-toolbox now. that is an amazing refactor. that level of modularity is absolutely mind-blowing.09:51
<acoburn>dhlamb: there's one more piece to that refactor, but I'm hoping to get a release out next week.09:52
dhlamb: https://jira.duraspace.org/browse/FCREPO-2363
dhlamb: at this point, we _could_ cut a release for fcrepo-camel/4.5.0. Any thoughts on that?09:53
<dhlamb>acoburn, so that would allow deployment using declarative services?
<acoburn>dhlamb: no, not quite yet09:54
dhlamb: it would allow you to provision the JAVA impl code independent of a particular blueprint config
dhlamb: so, for example, I'd like to extend fcrepo-ldpath so that it works on particular (local) endpoints09:55
<dhlamb>acoburn, or publish it under a different jndi name?
<acoburn>dhlamb: for that, I'll need everything _except_ the existing blueprint.xml code
dhlamb: well, the camel code itself isn't registered as an OSGi service09:56
dhlamb: I'm not entirely sure how one would do that
<escowles>ruebot: only had about an hour to write code this morning, but: https://gist.github.com/escowles/07e32ba2a260a36c468225d53c3d7e93
just sayin'
<acoburn>dhlamb: but maybe you know?09:57
<dhlamb>acoburn, sry, bad example. but assuming you had a service impl, this would allow people to drop in their own blueprint xml to expose it however they wish?
<acoburn>dhlamb: yes, exactly
<whikloj>escowles++
<acoburn>dhlamb: that is precisely what I'd like to do
* coblej leaves09:58
<acoburn>dhlamb: or, one could have multiple deployments that re-use the same impl code
dhlamb: all in the same container
<justinsimpson>[Import/Export Standup]09:59
yesterday: nothing, travel day
today: testing and looking at updating docs, e.g., https://github.com/fcrepo4-labs/fcrepo-import-export/pull/65
blockers: none
<dhlamb>acoburn, i'm sure you could publish a route builder as a service, but it would require some boilerplate. i'm looking it up in OSGi in Action right now.
* coblej joins10:00
* coblej leaves10:01
* coblej joins10:02
<dhlamb>acoburn, i _think_ it could be done using a BundleActivator
<ruebot>escowles: LOL
<dhlamb>acoburn, and iirc, the thought was to hold off on fcrepo-camel til fcrepo-camel-toolbox was ok with you removing the namespace constants?10:03
<escowles>ruebot: i resisted the temptation to use a more colorful name...
<dhlamb>acoburn, so if that holds true, i'd cut the release.
<ruebot>escowles: LoL-BagIt, that's what I'd call it.
<acoburn>dhlamb: ok, I can plan to cut the release tomorrow (unless bseeger wants to do it?)
dhlamb: I've already written the release notes :-)10:04
<ruebot>escowles: did you see whikloj's comment on #66? should i hold off on testing?
<bseeger>acoburn: sure, I can do that.
<whikloj>ruebot: no go ahead, I have revert my objection
<acoburn>bseeger++10:05
<dhlamb>acoburn, I'm trying to get my sea-legs with cutting Alpaca releases. Once I know what I'm doing, I'll gladly help you with them for fcrepo-camel or fcrepo-camel-toolbox.
acoburn, I know it's not a big deal, but I've never done it. I'd like to at least be sandboxed to my own organization in case i botch the job on my first try :)10:06
<whikloj>ruebot/bseeger/escowles: I had a weird experience last night, https://jira.duraspace.org/browse/FCREPO-2362
<ruebot>whikloj: oh, heh. i should have refreshed that page.
<whikloj>ruebot++
<ruebot>^^ westgard - https://jira.duraspace.org/browse/FCREPO-2362
oh. josh isn't here :-(10:07
<whikloj>ruebot: yeah or I would have included him
<ruebot>whikloj++
* coblej leaves10:08
<bseeger>whikloj: I'll see if I can reproduce that. What a strange problem.
<ruebot>dbernstein: are you cool with #66 over #65?10:09
<whikloj>bseeger: Yeah, I wondered if I did something wrong? Or is it the JSON-LD library
[Import/Export Standup]
Yesterday:
Finished https://jira.duraspace.org/browse/FCREPO-2350, https://jira.duraspace.org/browse/FCREPO-235910:10
<dbernstein>ruebot: yeah that looks good.
<whikloj> Created and fixed, https://jira.duraspace.org/browse/FCREPO-2361
Today:
Not sure what is a priority
Blockers:10:11
None
<ruebot>dbernstein: cooooooool. i'll close #65 then.
<dbernstein>ruebot : if you can merge that I can push 2356 into review.
it’s all ready to go.10:12
<ruebot>whikloj: yeah, we're kinda at the end w/r/t phase 2 on import-export https://jira.duraspace.org/issues/?filter=13801 -- unless westgard and bseeger have some python work for you
dbernstein: awesome. let me run a quick sanity test locally, and we should be good to go.
<dbernstein>I’m going to be on vacation starting today so if you guys want to wrap up by tomorrow, feel free to tweak my next PR to your liking.
<ruebot>bseeger++10:14
dbernstein: oh, right!10:15
* ruebot doubles efforts
* coblej joins
* awead joins10:17
[Import/Export Standup]10:18
Finished yesterday:
Nothing
Working on today:
* coblej leaves
<awead> Nothing :(
Blockers:
Traveling, dodging snow.
* coblej joins10:24
<ruebot>dbernstein, escowles: question - i should be getting an aptrust-info.txt tag file if i specify aptrust, right?10:25
ava -jar target/fcrepo-import-export-0.0.1-SNAPSHOT.jar --mode export --resource http://localhost:8080/rest --dir /home/nruest/tmp/import-export-sprint-03/test/bags/hydra --binaries --bag-profile aptrust --bag-config /tmp/bagit-config.yml10:26
<escowles>ruebot: i don't think that's implemented yet
<ruebot>escowles: oh!
* coblej leaves
<ruebot>escowles: if that's the case, then we're good on #6610:27
<escowles>ruebot: but i think dbernstein was going to do that as part of FCREPO-2356
<dbernstein>yeah - that’s right.
* peichman leaves
<ruebot>escowles: ah, that makes sense.
escowles++ dbernstein++
escowles: merge commit on that one, right? make sure we get your and dbernstein's commits10:28
<escowles>ruebot: yep, that sounds good to me
* github-ff joins
[fcrepo-import-export] ruebot closed pull request #65: Adds support for user-supplied metadata (master...fcrepo-2352) https://git.io/v1D9x
* github-ff leaves
* github-ff joins
[fcrepo-import-export] ruebot deleted fcrepo-2352-escowles at fb9e847: https://git.io/v1SRC
* github-ff leaves
* coblej joins10:29
* github-ff joins10:33
[fcrepo-import-export] dbernstein opened pull request #67: Enforces BagIt profile requirements for user-supplied metadata. (master...fcrepo4-2356) https://git.io/v1S0I
* github-ff leaves
* dwilcox leaves10:34
* coblej leaves
* coblej joins10:35
<dbernstein>heads up on that pull request: I just rebased and pushed. I don’t have time to test it/fix it. So forgive me if it doesn’t compile.
It should work though.
<ruebot>dbernstein: compiling it now
<dbernstein>I tested it before I rebased and all was well.
<ruebot>[INFO] BUILD SUCCESS10:36
:-)
<dbernstein>yayuh.
Happy holidays folks!
* dbernstein leaves10:37
<ruebot>escowles, whikloj: who wants clean-up on #67? https://github.com/fcrepo4-labs/fcrepo-import-export/pull/67#issuecomment-26735835710:38
<whikloj>ruebot: Is that a problem?
<escowles>ruebot: i don't think i'll get to it today, so i think whikloj should do it
whikloj: i think we've lost dbernstein10:39
<whikloj>escowles: Yeah, I mean is the fact that our tool enforces the restriction a problem. I have less confidence in the bagit-java tool now than before
<escowles>whikloj: same here10:40
<whikloj>ruebot??
<ruebot>whikloj: abandon bagit-java?10:41
<whikloj>ruebot: no, just don't fix the fact that we are checking the profile restrictions before they do10:42
ala your issue comment above
<ruebot>whikloj: hrm. but, what i have in my sample yml is basically what is in the test fixture
whikloj: https://gist.github.com/3cb4ec7b44efcfa334a4c686c4c7284c10:43
<whikloj>ruebot: So it this failing now?
<ruebot>whikloj: it passes tests, and builds. but, when i run it as in my comment, i get that error.
<whikloj>ruebot: Ohhhhhh, my apologizes. I read that as "We don't need to do this" not "This is failing to work"10:44
<ruebot>whikloj++
whikloj: no worries :-)
<escowles>ruebot: whikloj: so the profile enforcer needs to skip Payload-Oxum/Bagging-Date/Bag-Size because it knows it will generate those later10:45
<ruebot>escowles++ #exactly!
<escowles>we should be able to have a set of terms we'll generate and then exclude those from the fields we iterate over somehow
<whikloj>escowles: right, so should we specify a second part of the profile, like required-at-start versus required-at-end?
escowles: Or are you thinking a static set of tags that are generated so can be ignored at the beginning10:46
<escowles>whikloj: no, i think it's correct for them to be there in the profile — i think the tool should literally remove them from the required fields list when checking them
whikloj: yes, the latter
<whikloj>escowles: ok, will do
<escowles>whikloj++
<ruebot>whikloj++10:47
<escowles>whikloj: also, feel free to pick up https://gist.github.com/escowles/07e32ba2a260a36c468225d53c3d7e93#file-bag2-java and finish that off if you have time
* kefo joins
<escowles>(if you're so inclined)10:48
<whikloj>escowles: What are you using that for?
escowles: A wrapper around a bagit-java bag?
<escowles>whikloj: nothing yet — but that's a first draft at replacing bagit-java with our own class
<whikloj>escowles: ok, I'll see what I can do10:49
<ruebot>whikloj++
* awead leaves10:55
<ajwagner>https://wiki.duraspace.org/display/FF/2016-12-15+-+Fedora+Tech+Meeting11:00
<bseeger>*is here*
* ajwagner is here
<acoburn>*is here*
* informatician joins11:01
<acoburn>ruebot++
<bseeger>ruebot++
* escowles is here11:02
* coblej leaves11:03
<escowles>bseeger++
* awead joins
<ajwagner>bseeger++
* whikloj is here too
<ruebot>bseeger: do you have a link to the testing page?11:04
<bseeger>https://wiki.duraspace.org/display/FF/Release+Testing+-+4.7.1
<ruebot>bseeger++11:05
<dhlamb>if no one's using it... shuffle it out of rotation11:06
<kefo>It was Ben Cail from Brown writing to fedora-tech who encountered this, fwiw.11:07
* awead leaves
* coblej joins
<whikloj>acoburn++11:08
* coblej is here
<ruebot>https://github.com/LibraryOfCongress/bagit-java/issues/7211:09
2.1811:16
<whikloj>acoburn++11:17
<escowles>*crickets*11:20
<ajwagner>https://jira.duraspace.org/projects/FCREPO/issues/FCREPO-232311:21
<ruebot>https://jira.duraspace.org/browse/FCREPO-2323
<dhlamb>love the accordion11:22
is someone at a french restaurant?
<ruebot>dhlamb: i think it is kefo :-)11:23
<kefo>Yeah, we have backfground music in the office.11:25
* ad96 joins11:26
* westgard joins
<dhlamb>what happens if you force yourself to go through the proxy instead of skirting around it?11:28
<westgard>whilkoj bseeger ruebot: which version of the verification tool were you using when you noticed these false reports?11:32
<whikloj>westgard: I used your PR and added the rdflib-jsonld and corrected the extension to .jsonld
westgard: corrected the extension in the EXTS variable I mean11:33
<westgard>I had a similar experience this morning where random resources that hadn't changed were showing up as failing verification.
I was using my current feature branch where I have been refactoring the classes, so I thought I had broken something.11:34
But it sounds like the problem may be deeper than that.
<whikloj>kefo++11:37
<dhlamb>y11:38
<whikloj>westgard: yes11:39
<ruebot>ajwagner: let me know when you're good with the notes11:40
<ajwagner>ruebot: I think we're good. I'll give them a quick read for typos.
The one thing I missed was who was talking about he and Mike Durbin being the last people to touch the file connector.11:41
<acoburn>kefo: I just tried running one of those scripts and was not able to reproduce this
<whikloj>ajwagner: escowles
or Esme in non-irc
<ruebot>ajwagner++11:42
whikloj++
<ajwagner>whikloj: I thought it was Esmé in non-irc =P
ruebot: I don't notice anything glaring so should be all done.11:45
<ruebot>ajwagner: thanks!
<westgard>[Import/Export Standup]11:46
Finished yesterday:
first version of refactored main classes which also handles external binaries
Working on today:
I had hoped to add better error handling to that and make the PR, but FCREPO-2362 has now come up...
I think probably the best way forward might be for me to do a PR right away with the refactored classes, but we don't merge it yet. Then others can troubleshoot FCREPO-2362 against the PR?
Blockers:
None.
<bseeger>westgard, whikloj: I'm going to try it w/o that one remaining PR and see what it does.11:48
westgard, whikloj: though that may not work… hmm… any objections to merging that PR and figuring it out from there?11:50
* peichman joins11:52
* coblej leaves
<bseeger>westgard, whikloj: sorry for my confusion on that PR regarding 'lang'. I think I'm less confused now and have deleted the comments…11:54
<westgard>bseeger whikloj: I don't know what the best procedure is regarding the github repository, but I do have a version of the refactored main classes that changes a lot of the code. We should probably do troubleshooting against that if we cannot locate a point in the past versions where these errors fir appeared.11:55
first appeared
<whikloj>bseeger: which PR?11:56
<westgard>It is strange that we never saw these issues before, and I've run the verifciation tool a lot of times against a variety of data sets
<bseeger>whikloj, westgard: https://github.com/fcrepo4-labs/fcrepo-import-export-verify/pull/13
<whikloj>westgard: It worked great against a good export, it was when I started trying to get an notification that the problem appeared
bseeger: oh okay, no problem11:57
<westgard>whikloj: good point, I've run it against problem data only a fraction of the times compared to good data11:58
* thomz leaves11:59
<westgard>bseeger whikloj : this branch has the refactored classes: https://github.com/jwestgard/fcrepo-import-export-verify/tree/FCREPO-2327
<whikloj>westgard/bseeger: does the verify script compare checksums currently?12:03
<bseeger>westgard/bseeger — good question - the code is in there and looks like it should be correct.12:04
whikloj: ^^ (I'm apparently starting to talk to myself… must be lunch time)
<whikloj>bseeger: hahaha12:05
<bseeger>going to go eat lunch…12:06
<westgard>whikloj/bseeger: the checksum comparison works like this: for resources in fedora, pulls the metadata and extracts the SHA1, then reads the file on disk through the hash algorithm and compares them. The results get recorded in the CSV
It does not pull the binaries down from Fedora, and does not support other hash algorithms12:07
bseeger: bon apetit
sorry appetit someone else will need to supply the accent12:08
<ajwagner>^ ´12:09
<westgard>ajwagner++
* github-ff joins12:13
[fcrepo-camel-toolbox] acoburn opened pull request #126: Add provisioning features independent of blueprint-based configurations (master...fcrepo-2363) https://git.io/v1S5t
* github-ff leaves
<whikloj>escowles: What is the difference between the metadataFields and the APTrustFields in BagProfile, are they just all the fields needed for a normal bag and the extended fields for a profile?12:14
<escowles>whikloj: the APTrustFields are the ones that go in aptrust-info.txt instead of bag-info.txt12:15
* youn leaves
<kefo>acoburn: FCREPO issue with links to gists: https://jira.duraspace.org/browse/FCREPO-2364
<whikloj>escowles: So if you are using a different profile than APTrust will we need to extend BagProfile to hold those fields too? Just wondering if it works to generalize it to the active profile12:16
<escowles>whikloj: other profiles might require other tag files (i think perseids requires a json file somewhere too), but i don't think those are implemented yet
<whikloj>escowles: sorry for the basic questions, but currently if I add ap-trust.txt to my tag config will it be created for every bag or does it wait till I choose the profile "aptrust"12:18
<escowles>i think the best thing would be to create any files mentioned in the bag-config, but i'm not sure that's what happens now
<acoburn>kefo: I'm not sure that I understand the test code — it seems that you are sending a curl command with localhost:8080 but retrieving responses with a FQDN. Is your servlet container rewriting the Host header?12:19
kefo: from what I can tell, what you're seeing is expected behavior
* coblej joins12:29
* escowles_ joins
* escowles leaves12:31
* bseeger leaves12:32
* westgard1 joins12:38
* coblej leaves12:42
* westgard1 leaves12:43
<kefo>acoburn: "Yes" to the rewriting of the host header. My gut was saying this was expected behavior, but how Tomcat functions in this way is not something I'm fully comfortable with. That said, as a test, I set up a separate Connector in server.xml - for localhost on a different port and not using the proxy - and sure enough Tomcat allows localhost host access but the URIs come out the other end all good.12:48
* bseeger joins12:49
* coblej joins12:51
<bseeger>whikloj, westgard: fwiw - I'm getting random verification errors on the repo even with out having purposely messed up the data.13:01
whikloj, westgard - two errors specifically: 1) for internal: WARN: Resource Mismatch '/fcrepo/rest/ad/75/aa/2d/ad75aa2d-e4cd-4e5f-85cb-9f6b70f06d59/internal'13:02
whikloj, westgard - 2) for external: Error communicating with repository. Response: 307 for node http://localhost:8080/fcrepo/rest/ad/75/aa/2d/ad75aa2d-e4cd-4e5f-85cb-9f6b70f06d59/external
whikloj, westgard: the 2nd one makes sense since the code doesn't handle the HTTP 307. First is confusing.
<westgard>bseeger: this is the sort of thing I was seeing earlier today too. I don't know what happened since we never had these issues before.13:05
The 307 I would expect, as you said (that one, BTW, is handled in the refactored classes)13:06
<bseeger>westgard, whikloj - it's definitely odd. The random makes sense since you don't know what order you'll get resources in, but the fact that internal is failing is strange.
<westgard>But the other issue :-(
Is it possible there's a real problem in the latest exporter?
We should try to verify the checksum of the resources that are failing first.13:07
<bseeger>westgard: ohh…. hmmmm….
<whikloj>westgard++
<westgard>I'm in the process of testing against a clean export of fresh data with the latest import/export tool.13:08
And will report back the results here.
* mikeAtUVa joins13:12
* github-ff joins13:16
[fcrepo-camel-toolbox] acoburn opened pull request #127: Fix OPTIONS handling in fcrepo-ldpath (master...fcrepo-2358) https://git.io/v19ep
* github-ff leaves
<bseeger>westgard, whikloj: I think it's somewhere in this code: https://github.com/fcrepo4-labs/fcrepo-import-export-verify/blob/master/verify.py#L197-L20013:17
<whikloj>bseeger: there is no check that the response.status_code == 200 there13:18
bseeger: could be unrelated, but should be there13:19
<bseeger>whikloj, westgard: code tweaks I made and the results: https://gist.github.com/bseeger/015e67c56807dd06476bdb3323570f1913:20
<whikloj>bseeger: you're code doesn't quite match the output but the gist is we're getting nothing back from Fedora for the fixity result match.13:22
<bseeger>whikloj: I know what you mean, I think there are two regex search groups, which is why it's odd.
whikloj, westgard: but the gist of it is that we're not getting the response for resources in that plant dataset that we think we should be getting.13:23
whikloj: yeah, and it's true that I didn't include the code that displays the DISK SHA1 info - that part seems to be functioning fine.13:24
<whikloj>bseeger: maybe we should be calling .../fcr:fixity instead of .../fcr:metadata
bseeger: hahahaha13:25
premis:hasMessageDigest <urn:sha1:f68320af6a9df71338bbd9dda32a78e9574f3453> ;13:26
there are 2 spaces now
That regex should just use \s+
r'premis:hasMessageDigest\s+<urn:sha1:(.+?)>' or some such
<westgard>bseeger whikloj: you both may know this, but if you run verify.py with the csv option, and the tool can't read a checksum from Fedora, it will record the empty checksum value in the CSV13:29
<bseeger>whikloj, westgard: That's odd. I have to sign off for the day now, but can help with this some more tomorrow.13:30
<westgard>in the verification column it will say [blank] != local checksum
* bseeger leaves
<westgard>Thanks bseeger!
whikloj: I ran an export of about 2700 resources (rdf only), ran the verification tool, and 45 of them came up as having different triples (but all the same number of triples). It is possible that the rdf comparison is not working correctly. That part is a fairly recent addition.13:34
I will continue controlled testing and add more logging to document where the differences are showing up. For example it should be fairly easy to report back the different triples where there is a mismatch.13:35
* coblej leaves13:36
<whikloj>westgard: I'll have to try again and make sure to check the csv and log files. I was looking for output on the command line and got either nothing (which I assumed was good) or errors (which are bad)13:37
<westgard>whikloj++13:38
<whikloj>westgard: My python is not super strong, but we need some tests to run this tool against.13:39
<westgard>whikloj: the logs are right now pretty sparse except where there are errors, but should probably be made much more verbose.
+1 to tests
* cmmills joins13:42
<westgard>whikloj: some encouraging news: the mismatches in my recent test appear to be real mismatches (unicode-escaped characters in the file on disk that are not coming back that way in the rdf coming out of Fedora)13:51
So at least the tool is detecting a real difference ...13:53
* mikeAtUVa leaves13:54
* mikeAtUVa joins13:55
* coblej joins13:58
* ad96 leaves14:00
* github-ff joins14:09
[fcrepo-import-export] escowles pushed 1 new commit to master: https://git.io/v19mt
fcrepo-import-export/master 5c01abf Esmé Cowles: Local BagIt implementation instead of using LoC bagit-java
* github-ff leaves
<escowles_>ruebot: i am so sorry — i just pushed a commit to master that i really, really shouldn't have...
just pushed a revert14:10
* github-ff joins14:11
[fcrepo-import-export] escowles pushed 1 new commit to master: https://git.io/v19mg
fcrepo-import-export/master 32a2889 Esmé Cowles: Revert "Local BagIt implementation instead of using LoC bagit-java"...
* github-ff leaves
* github-ff joins
[fcrepo-import-export] escowles created local-bagit at 5c01abf (+0 new commits): https://git.io/v19mr
* github-ff leaves
* acoburn leaves14:14
<ruebot>escowles_++14:16
escowles_: i once deleted a whole repo by accident :-)
escowles_: i can step back two commits and force push too if you want.14:17
escowles_: whikloj ran out to get a present, so we should be safe if need be.14:18
* westgard leaves14:37
<justinsimpson>escowles - should I be able to use fcrepo-import-export to import any arbitrary bag? If i import a bag, that has no rdf in it, just a binary in the data directory, should a binary resource get created in Fedora?14:57
or ruebot, not sure who is the best person to ask that question
<ruebot>justinsimpson: right now, it's only going to import a bag that was exported from fedora14:59
<justinsimpson>kk,15:00
<ruebot>justinsimpson: i don't believe we have scope for any arbitrary bag coming into fedora
<justinsimpson>ok, I thought I heard esme say on the call yesterday that it might work, but I think I misunderstood.15:01
<ruebot>justinsimpson: yeah, i think i remember. i think i was that you could point the utility at the data directory, and something might happen15:08
* dwilcox joins15:09
* dwilcox leaves15:10
* acoburn joins15:13
* dwilcox joins
<acoburn>ruebot: the github releases page has a lot of old RC and SNAPSHOT items: https://github.com/fcrepo4/fcrepo4/releases15:14
ruebot: is there any reason not to remove those?
escowles_: whikloj: ^^^15:15
<whikloj>acoburn: you mean anything label Pre-release?
<acoburn>whikloj: with the exception of the current RC, yes15:16
whikloj: like 4.5.1-SNAPSHOT from last January
<whikloj>acoburn: I can't think of a reason, I image awoods might. But as long as we keep all the actual release artifacts I don't see a problem
<ruebot>acoburn: i was thinking the same a while back. yeah, i think we should definitely remove them.
<acoburn>whikloj: I remember discussing this some time ago, and the old pre-releases were removed15:17
I think we just haven't kept up with that
<whikloj>acoburn: ok then, if this is just procrastination then I say turf 'em
* github-ff joins15:18
[fcrepo4] acoburn deleted fcrepo-4.5.1-SNAPSHOT at b73f270: https://git.io/v1983
* github-ff leaves
* github-ff joins15:19
[fcrepo4] acoburn deleted 4.3.1-SNAPSHOT at 59defcb: https://git.io/vVPJo
* github-ff leaves
* github-ff joins15:21
[fcrepo4] acoburn deleted fcrepo-4.5.0-RC-1 at f6a1adc: https://git.io/vVPJ2
* github-ff leaves
* github-ff joins
[fcrepo4] acoburn deleted list at c5a9b37: https://git.io/v194e
* github-ff leaves
<whikloj>escowles_/ruebot: Playing with bag profiles I think we should provide a template config file. I made one with just bag-info stuff (no APTrust) and the BagConfig now blows up
* github-ff joins15:22
[fcrepo4] acoburn deleted fcrepo-4.1.1-audit.0 at c79ee8e: https://git.io/vVPJQ
* github-ff leaves
<ruebot>escowles_, whikloj, justinsimpson, informatician: https://jira.duraspace.org/browse/FCREPO-2299 -- let me know what you think of the outline
<whikloj>ruebot: Are we worrying about the verify tooling?
<ruebot>whikloj: oh? so something like this: https://gist.github.com/e48e37817be3bc09683d53f78c43e05b15:23
whikloj: good question!
whikloj: i would say no for right now. but, depending on how things land, maybe a video devote to it.15:24
<whikloj>ruebot: that doesn't work for me
* escowles_ is back
<ruebot>whikloj: interesting. are you using default or aptrust?
<whikloj>ruebot: Ahhhh I see, dbernstein is validating the APTrust info right in the constructor
<ruebot>whikloj: ah, i was just going to ask if that is getting validated somewhere15:25
<justinsimpson>ruebot that outline looks good to me, doing any more would definitely make the video longer than 3 minutes
* kestlund joins
* github-ff joins
[fcrepo4] acoburn deleted fcrepo-4.3.1-SNAPSHOT at 59defcb: https://git.io/vVPJo
* github-ff leaves
<ruebot>justinsimpson: yeah. awoods wants to do a couple of them. so, we could definitely do more in other ones.15:26
* github-ff joins
[fcrepo4] acoburn deleted fcrepo-4.2.1-SNAPSHOT at 2df258d: https://git.io/vVPJM
* github-ff leaves
* github-ff joins
[fcrepo4] acoburn deleted 4.2.1-SNAPSHOT at 2df258d: https://git.io/vVPJM
* github-ff leaves
<ruebot>kestlund: a stakeholder! :-) -- if you have time, let me know what you think of the outline for a first video here: https://jira.duraspace.org/browse/FCREPO-2299
<kestlund>ruebot: I have time this afternoon15:27
<ruebot>kestlund++
* github-ff joins
[fcrepo4] acoburn deleted fcrepo-4.1.1-audit.1 at 2c8929d: https://git.io/vVPJ9
* github-ff leaves
* github-ff joins15:28
[fcrepo4] acoburn deleted fcrepo-4.1.1-audit.2 at 4dc1661: https://git.io/vVPJD
* github-ff leaves
* github-ff joins
[fcrepo4] acoburn deleted fcrepo-4.5.1-RC-1 at 33ea9fc: https://git.io/v19BT
* github-ff leaves
* github-ff joins15:29
[fcrepo4] acoburn deleted fcrepo4-4.6.0-RC-3 at 1fc2954: https://git.io/v19Bs
* github-ff leaves
<justinsimpson>ruebot whikloj I think I am seeing the same problem - if I use a bag-info.yml that has an aptrust-info.txt section, the export works, whether I have -g default or -g aptrust15:30
* github-ff joins
[fcrepo4] acoburn deleted fcrepo-4.6.0-RC-2 at 9745fc5: https://git.io/v19BZ
* github-ff leaves
* github-ff joins
[fcrepo4] acoburn deleted fcrepo-4.5.2-RC-1 at b08dc4f: https://git.io/v19BE
* github-ff leaves
<justinsimpson>but if I use a bag-info.yml without an aptrust-info.txt section, I get an error The specified bag config file could not be parsed: null - even when -g default
<ruebot>escowles, whikloj: let's just stick with rolling our own. https://github.com/LibraryOfCongress/bagit-java/issues/72#issuecomment-267434887 -- i don't want to deal with that guy anymore, and he's behaved similarly on the digital-curation list -- https://groups.google.com/d/msg/digital-curation/SoESFTxCj_Q/BkCC0dW9JQAJ15:31
<whikloj>justinsimpson: yeah I'm trying to change how we define a profile.15:32
<f4jenkins>Project fcrepo-module-auth-rbacl build #1237: FAILURE in 27 sec: http://jenkins.fcrepo.org/job/fcrepo-module-auth-rbacl/1237/
* westgard1 joins15:38
* github-ff joins15:40
[fcrepo-import-export] escowles pushed 1 new commit to local-bagit: https://git.io/v19RH
fcrepo-import-export/local-bagit 9840230 Esmé Cowles: Local BagIt implementation instead of using LoC bagit-java
* github-ff leaves
* github-ff joins15:42
[fcrepo-camel-toolbox] acoburn deleted fcrepo-camel-toolbox-4.3.1-SNAPSHOT at 2075359: https://git.io/v190s
* github-ff leaves
* github-ff joins
[fcrepo-camel-toolbox] acoburn deleted fcrepo-camel-toolbox-4.2.1-SNAPSHOT at e866946: https://git.io/v190W
* github-ff leaves
* github-ff joins15:43
[fcrepo-camel-toolbox] acoburn deleted 4.2.1-SNAPSHOT at e866946: https://git.io/v190W
* github-ff leaves
<f4jenkins>Project fcrepo-module-auth-rbacl build #1238: NOW UNSTABLE in 3 min 0 sec: http://jenkins.fcrepo.org/job/fcrepo-module-auth-rbacl/1238/15:44
* dwilcox leaves
<ruebot>escowles: so, really it was a 100 lines to get rid of it. nice!15:45
<escowles>ruebot: yep, it wasn't that bad — got it done in a day working around 3 hours of calls and my son's school play15:46
though travis is failing (tests pass for me locally)
<ruebot>escowles++ #impressive!15:47
<escowles>and it will certainly conflict with https://github.com/fcrepo4-labs/fcrepo-import-export/pull/67
so, i think we should get #67 merged (whikloj is working on that, right?) and then see where we are
<ruebot>escowles: did you squash and force push? i see two travis tests15:48
https://travis-ci.org/fcrepo4-labs/fcrepo-import-export/builds/184358630
https://travis-ci.org/fcrepo4-labs/fcrepo-import-export/builds/184358206
<escowles>yeah, i moved the commit to a new branch, made a couple of small changes and squashed
<whikloj>escowles: Sorry I was tearing it all apart. He's got APTrust as a required part of anything you do...I'll save what I have and go back and try a faster but uglier fix.
<escowles>whikloj: it's ok — better to get it right15:49
we don't want to require aptrust all the time...
(i wonder if the test failure is related to what dbernstein was talking about)15:50
<ruebot>escowles: restarting them both. that's a weird fail.15:51
escowles: oh wait, i read that wrong. i thought it was a javadoc build, not mvn install -B -V15:52
<escowles>ruebot: yeah, i don't know what the issue is, but i think it's a real failure — maybe related to cleaning out the export dir thing that dbernstein mentioned in #6715:53
* dwilcox joins
<ruebot>escowles: yeah, that could be it.
<escowles>i figure i can work on some ITs, and make sure each export is using its own clean dir (my tool creates directories if it needs them unlike some tools i could mention)15:54
</shade>
* ruebot snorts
<f4jenkins>Yippee, build fixed!15:57
Project fcrepo-module-auth-rbacl build #1239: FIXED in 3 min 7 sec: http://jenkins.fcrepo.org/job/fcrepo-module-auth-rbacl/1239/
* acoburn leaves16:03
<kestlund>ruebot: hah! I'm a bad stakeholder for this. I hate learning from videos.16:11
<ruebot>kestlund++ #good feedback!16:12
...and thanks!16:13
kestlund: have you see the API-X video they are working on? https://www.youtube.com/watch?v=aKoH9ilJBok
* justinsimpson leaves16:19
* dwilcox leaves16:21
* dwilcox joins16:27
* github-ff joins16:29
[fcrepo-import-export] escowles pushed 1 new commit to local-bagit: https://git.io/v19wk
fcrepo-import-export/local-bagit 17ca9e3 Esmé Cowles: Isolating tests from each other
* github-ff leaves
* github-ff joins16:33
[fcrepo-import-export] escowles pushed 1 new commit to local-bagit: https://git.io/v19wA
fcrepo-import-export/local-bagit 841c4fb Esmé Cowles: Fixing case-sensitive checksum filename
* github-ff leaves
* github-ff joins16:38
[fcrepo-import-export] escowles pushed 1 new commit to local-bagit: https://git.io/v19rD
fcrepo-import-export/local-bagit bc3890b Esmé Cowles: Fixing case-sensitive manifest filename issue in BagIT
* github-ff leaves
<escowles>took me a while to figure it out, but the test failures were because I'd fixed the manifest filenames to use lowercase algorithm names, and the tests expecting the old uppercase names worked on my case-insensitive filesystem
* coblej leaves16:39
* coblej joins16:40
* justinsimpson joins16:41
<kestlund>ruebot: 5 min! Hell, no. I will try to make it through.
* westgard1 leaves16:46
* informatician leaves16:47
* dwilcox leaves16:48
<kestlund>ruebot: I really like the animations! 52-56 sec. is my attention span.16:51
* justinsimpson leaves16:54
* kestlund leaves16:59
<whikloj>escowles: I think I may have gone too far and left reality behind, my rewrite might have gone off the rails17:00
<ruebot>kestlund++17:03
<whikloj>escowles/ruebot: So I want to keep what I have. But I'm afraid I'm just holding you all up now
<ruebot>might not have been the day for justinsimpson to come and visit me17:04
* mikeAtUVa leaves17:06
<ruebot>whikloj: can you push up to a branch, and we can see what's going on?17:07
<whikloj>ruebot: sure, https://github.com/whikloj/fcrepo-import-export/tree/refactor-profile17:11
<ruebot>whikloj++
<whikloj>ruebot: HAHAHAHAHA
* coblej leaves
<ruebot>whikloj: laughing at my toronto snow vs winterpeg snow?
* kestlund joins17:12
<whikloj>ruebot: laughing that you gave me karma for pushing up what might be a clusterf&*k
* kestlund leaves17:14
<ruebot>whikloj+++++++++++++++++++
* coblej joins
<ruebot>whikloj: https://github.com/fcrepo4-labs/fcrepo-import-export/compare/master...whikloj:refactor-profile?expand=1 -- that's against LoC bagit, and not escowles bagit, right?17:15
<whikloj>ruebot: yeah I hadn't even gotten to escowles Bag replacement17:16
* coblej leaves17:20
<ruebot>whikloj: it's looking ok to me. it's going to be complicated because of the nested ifs. i don't think we can get around that.
whikloj: but, i'll leave final word to escowles on that :-)17:21
<escowles>whikloj: ruebot: i will take a look — at first blush, i'm not sure i like the changes to the profiles (i think having the tag files as the top-level keys makes sense)17:23
* westgard1 joins17:24
<whikloj>escowles: two things I was trying was to 1) not force people to repeat information in every profile, 2) Not hardcode profile information in the functions
escowles: I may have gone sideways on implementation
<escowles>whikloj: my main concern is whether we have that much flexibility in the profiles — ruebot will know more about that, but i don't know if we can change the format like that or not17:25
but it's not clear to me how much those profiles are just something ruebot made up, or whether other people (and other tools) need to be taken into account too
<whikloj>escowles: I can dial that back17:26
it was easier to perform the check for required fields once and have that list change depending on the profile you loadded
<ruebot>something ruebot made up /me cackles loudly17:27
<escowles>whikloj: yeah, i agree it makes it much cleaner to do it that way — otherwise we'd need to loop over the profile top-level elements (or hard-code support for aptrust - which is gross)17:28
well, if ruebot just made up the bag profile thing, then maybe i'm ok with just changing it...
<ruebot>made up which part?17:29
<whikloj>escowles: yeah, I actually thought you wrote those profiles JSON docs
ruebot: https://github.com/fcrepo4-labs/fcrepo-import-export/blob/master/src/main/resources/profiles/default.json
<ruebot>the spec, or the actual profiles in the tickets?
<whikloj>ruebot: is the profile structure defined in the SPEC?
<escowles>i'm not talking about the profiles in our repo — i did make those up mostly17:30
ruebot: did you create the idea of bagit profiles? or is it a bigger thing that we need to take into account?
because if we can change the structure of what a bagit profile should look like, then whikloj's change is worth thinking about
but if there are other impls and users, then maybe we can't just change how profiles are structured17:31
<ruebot>escowles: mark jordan is the man that had the idea, and a group of us hashed out it out at the 2012 Access. Then, we worked on it more after, and I basically maintain it now.
so....... if we have to change stuff, we can. I'm realizing how problematic things are between the profile and the bag-config yaml file.
...and i'm really starting to hate bags :-)17:32
<escowles>because whikloj is doing things like changing the top-level key from "Bag-Info" to "Metadata-Fields": https://github.com/fcrepo4-labs/fcrepo-import-export/commit/8b4e04fd04d8c13f45c6b81b593785ea66ff1a1f#diff-f57a70eda810fb82193598e7fb2d7c7bR10
ruebot: yeah, they are stupidly simple, but just complicated enough to make working with them a pain
<whikloj>escowles: But that doesn't have to affect what we write out to the filesystem17:33
<ruebot>escowles: oh, i glanced over that too fast
<whikloj>escowles: as I said to ruebot I may have gotten in a hole here.17:34
<ruebot>https://github.com/ruebot/bagit-profiles/#examples
you can have metadata values in a profile
well, allowed values
shit. i have to walk up and meet justinsimpson here in a few.17:35
<escowles>ruebot: ok, we can pick this up in the morning — i think we can wrap this up tomorrow either way
<ruebot>escowles: yeah, i have to think more about https://github.com/fcrepo4-labs/fcrepo-import-export/compare/master...whikloj:refactor-profile?expand=1#diff-f57a70eda810fb82193598e7fb2d7c7bL717:36
and down
i'll be online early though, if y'all want to pick it up early
all that said, it's kinda cool to hang out with multiple sprinters during a sprint in different cities :-)17:37
* westgard1 leaves17:39
<whikloj>escowles/ruebot: My apologizes I didn't realize that was an "official" structure for BagIt profiles, I'll change back and we'll have to do some hard coding17:47
<escowles>whikloj: i think we can get the list of files to process from the "Tag-Files-Required" field, so we should be able to avoid hard-coding stuff17:59
but it'll definitely require another level of looping over the various tag files
<whikloj>escowles: ok better than nothing
escowles: I'll see if I can get anymore done tonight, have a good one18:00
* whikloj leaves
* peichman leaves18:11
* kefo leaves18:13
* dwilcox joins18:41
* dwilcox leaves18:45
* awead joins20:54
* awead leaves

Generated by Sualtam