Log of the #fcrepo channel on chat.freenode.net

Using timezone: Eastern Standard Time
* Helge joins05:52
* Helge leaves05:54
* youn joins07:39
* awoods joins08:08
* benpennell leaves08:14
* peichman joins08:15
* coblej joins08:16
* peichman leaves08:23
* dwilcox joins08:28
<coblej>[Import/Export Standup]08:31
Finished yesterday:
Replicated verification tool Unicode-escaped characters failure reported in https://jira.duraspace.org/browse/FCREPO-2369 with exports from Fedora 4.6.0.
Working on today:
Document import structural expectations — https://jira.duraspace.org/browse/FCREPO-245508:32
Blockers:
None
* umgrosscol joins08:43
* benpennell joins09:11
* mikeAtUVa joins09:15
[import/export standup]09:20
Finished last Friday:
- last modifications and changes to https://github.com/fcrepo4/fcrepo4/pull/1193 (relaxed server managed triples)
Today:
- mostly unrelated meetings
- https://jira.duraspace.org/browse/FCREPO-2464 (documenting relaxed server managed triples)
- https://jira.duraspace.org/browse/FCREPO-2461 (updating importer to support relaxed server managed triples)
Blockers:
- probably can't fully finish the importer updates without a version of 4.7.4 available for download for integration tests
<escowles>[Import/Export Standup]09:25
Last Friday:
* Debugged parallel build failures https://jira.duraspace.org/browse/FCREPO-2466
Today:
* Exporting members based on inbound links https://jira.duraspace.org/browse/FCREPO-2453
Blockers:
* None
* coblej leaves09:26
<awoods>mikeAtUVa: can you put the integration tests in place, but @Ignore until a 4.7.4 release comes out?
* coblej joins09:27
<awoods>escowles: did you come to conclusions or fixes re:parallel build failures?
<benpennell>Finished yesterday:09:28
* Second proposal for structure of version exports for https://jira.duraspace.org/browse/FCREPO-2446
Working on today:
* Finish up implementation and unit tests for version export, create PR https://jira.duraspace.org/browse/FCREPO-2458
* Start thinking about implementation of versions importing
Blockers:
* none
* yamil joins
<escowles>awoods: not really — i spent most of my time thinking it was related to my PR to enable user-supplied namespace prefixes — so my only conclusion is that it's related to the recent parallel RDF generation09:30
<awoods>escowles: I was hoping to look more closely at the failures to see if they are in the tests or in the functional code.09:31
* coblej leaves
<awoods><awoods> [Import/Export Standup]
<awoods> Finished last Friday:
<awoods> - Reviewed tickets
<awoods> Working on today:
<awoods> - Reviewing tickets
<awoods> Blockers:
<awoods> - None
<escowles>awoods: i'll write up the behavior that i saw in the ticket09:32
<awoods>escowles: thanks
* lsitu joins09:34
* coblej joins09:35
<lsitu>[Import/Export Standup]09:37
Finished last Friday:
Rebase PR and refactor codes to fix conflicts for Re-import of hierarchy: https://jira.duraspace.org/browse/FCREPO-2428
Updated PR for round-tripping with binaries excluded: https://jira.duraspace.org/browse/FCREPO-2426
Working on today:
Import of Bags should verify binary digest: https://jira.duraspace.org/browse/FCREPO-2418
Imporve PR for round-tripping with binaries excluded: https://jira.duraspace.org/browse/FCREPO-2426
Blockers:
None
awoods: I see two of my standups at night for the next day were not collected somehow. I am changing to do it in the morning instead.09:40
<awoods>lsitu: ok... I must have missed them at night. sorry09:41
<lsitu>That’s Okay. Thanks.09:43
<youn>[Import/Export Standup]09:45
Finished yesterday:
- https://jira.duraspace.org/browse/FCREPO-2369 (Import-export verification tool's rdf comparison fails on unicode-escaped characters) - added note on Unicode errors to README
- tested bag export and verification with bagit-python09:46
Working on today:
- follow up on comments on FCREPO-2369
- get feedback on (and work on) ideas for videos
Blockers:09:47
- none
<awoods>youn++ for the video outline planning
<youn>awoods: Could we use recorded conversations on Google Hangouts in the form of conversations between developers and stakeholders on requirements and use cases, and how they are addressed?09:57
<awoods>youn: sure... nothing wrong with being creative.09:58
youn: it is probably worth clearly defining the target audience and expected outcome of the video(s)
<youn>awoods: I'll try to come up with a more formal proposal for the Google Hangout conversation idea ...09:59
* coblej leaves10:06
* coblej joins10:07
* westgard joins10:08
[Import/Export Standup]
Finished Friday:
https://jira.duraspace.org/browse/FCREPO-2460 (update readme (help text) and inline comments in newly refactored codebase)
https://jira.duraspace.org/browse/FCREPO-2463 (minor formatting fixes in verification tool)
Continued troubleshooting of https://jira.duraspace.org/browse/FCREPO-2369 (Import-export verification tool's rdf comparison fails on unicode-escaped characters)
Working on today:
https://jira.duraspace.org/browse/FCREPO-2457 (handle failed connection gracefully)
Continued troubleshooting of https://jira.duraspace.org/browse/FCREPO-2369 (Import-export verification tool's rdf comparison fails on unicode-escaped characters)10:09
Blockers:
None.
<awoods>westgard/youn/coblej: we will probably want to make sure that investigations of the unicode issue do not prevent us from addressing other issues. We should probably land that topic soon.10:16
* dbernstein joins10:29
* coblej leaves10:36
<dbernstein>[Import/Export Standup]
Finished yesterday:
* Nothing finished (out most of the day)
Working on today:
* https://jira.duraspace.org/browse/FCREPO-2408
(make verification tool work with BagIt Bags)
Blockers:
None.
* coblej joins
<westgard>awoods: well at least one flavor of that problem is 'landed' but I'm not convinced that we won't see it again.10:38
<awoods>westgard: Is it clear what is required to tie up that issue?10:39
<westgard>There was a different set of circumstances where the issue was originally observed. It remains to be determined whether the earlier case is also limited to v4.6 or whether it will be present in later versions.10:41
<dbernstein>youn and westgard: just a head’s up: I plan to be out during the day today. I’ll be back on this evening.10:42
I’m around for a couple of hours this morning though in case there are any issues that need to be discussed.
<westgard>If we can demonstrate that the other flavor of error is seen in 4.6 and not in 4.7.2, then I would be content to call this resolved.
dbernstein: thanks, sounds good10:43
dbernstein: I don't think we need to discuss anything10:44
<dbernstein>okay cool.
<westgard>youn or coblej: anything you think we need to consult on with dbernstein while he's available?10:45
<youn>westgard, dbernstein: no, thanks for asking!
<coblej>westgard: , dbernstein: nothing comes to mind right now10:46
<westgard>meeting adjourned!10:48
youn++, coblej++, dbernstein++
<youn>westgard: I'm with you on the Unicode issues. They're a big deal at my institution, too. Do you want to split up testing various releases? I have a Book1.ttl file that I'm using for testing that I could upload to the team drive ... I think we're basically looking for Unicode escapes to come out in the export that would trigger verification errors ...10:54
* coblej leaves10:56
* coblej joins10:59
* coblej_ joins11:00
* coblej leaves
<youn>What is a transaction test under manual tests for the one-click on https://wiki.duraspace.org/display/FF/Release+Testing+-+4.7.3 ?11:01
<awoods>youn: a plan like you are suggesting sounds good: across Fedora versions, systematically determine where the unicode verification error arises. It is easier to come to closer with a plan.
youn: transaction test: using transactions, i.e. the dropdown that says start transaction.11:02
youn: then, within a transaction perform some basic tasks, such as creating/updating/deleting resources
youn: then, either committing or aborting the transaction11:03
* coblej_ leaves
* coblej joins
<youn>awoods: Thanks. I'll try it.11:04
<awoods>youn: that's wonderful. Please update the testing wiki page with your results.11:11
westgard: are you waiting on dbernstein to review some tickets?11:12
https://jira.duraspace.org/browse/FCREPO-2463
https://jira.duraspace.org/browse/FCREPO-2460
* github-ff joins11:13
[fcrepo-import-export] escowles opened pull request #84: Adding support for exporting inbound references (master...inbound) https://git.io/vHkK9
* github-ff leaves
<westgard>awoods: I thought I saw those being reviewed/merged this morning11:30
<awoods>westgard: hmm... not seeing any change in the JIRA tickets11:31
westgard: is there a process disconnect?11:32
<westgard>awoods: does the pull requester update those or the reviewer?11:33
<awoods>westgard: the reviewer/merger updates the tickets with a comment along the lines of: "Resolved by <link-to-commit>"
dbernstein: are you around?11:35
lsitu: I am looking at the following ticket: "Add support for round-tripping with binaries excluded": https://jira.duraspace.org/browse/FCREPO-2426 ...and still seeing round-tripping issues.11:40
<lsitu>What error do you see?11:41
<awoods>lsitu: can you please verify that the testing scenario works for you: https://wiki.duraspace.org/display/FF/Sprint+3+Feedback+-+A+Woods#Sprint3Feedback-AWoods-Test0-Export/Importexcludingbinaries
lsitu: The export works fine, but the import fails with 404s11:42
<lsitu>I’ll test import to see what’s still wrong. Thanks.11:43
<awoods>lsitu: after starting two servers and loading the data with the ingest.sh script, I run the following commands:11:44
<lsitu>I don’t see the command you use. Could you post it again?11:45
<awoods>java -jar target/fcrepo-import-export-0.1.1-SNAPSHOT.jar -a -d data -m export -r http://localhost:8686/f4/rest/
mv data/f4/ data/fcrepo
java -Dfcrepo.log.importexport=DEBUG -jar target/fcrepo-import-export-0.1.1-SNAPSHOT.jar -a -d data -m import -M http://localhost:8686/f4/rest/,http://localhost:8080/fcrepo/rest/ -r http://localhost:8080/fcrepo/rest/
lsitu: The issue may have to do with the subject of the triples in the RDF... still referencing the source context.11:55
* coblej leaves12:05
* coblej joins12:07
<awoods>lsitu: are you seeing the same 404s?12:10
<lsitu>I think there are a couple of issues:12:11
1. we should not rename it with “mv data/f4/ data/fcrepo”.
2. The hasMember heirarchy wasn’t handled yet and the resource refernce is wrong.
3. There’s a typo in the command for the source/resource map.
<awoods>lsitu: it sounds like 1 and 3 are on my side... which is good.12:12
<lsitu>It’ll eork with PR for import: https://github.com/fcrepo4-labs/fcrepo-import-export/pull/79
<awoods>lsitu: thanks, I will start with giving PR-79 another review12:13
<lsitu>And there is a typo in the source/resource map, which should be “-M http://localhost:8686/f4/rest,http://localhost:8080/fcrepo/rest” without the slash / at the end. Wouuld you like me to create a PR to update readme?12:14
<awoods>lsitu: hmm... I think that needs more than an README update. We should either allow for maps that include or exclude a trailing slash, or we should respond with an informative error message.12:17
<lsitu>Sounds good!
<awoods>lsitu: could you please create a ticket for that?12:18
<lsitu>Sure. I can create a ticket and work on it.
* dbernstein leaves12:28
<westgard>awoods: Another process question: should https://github.com/fcrepo4-labs/fcrepo-import-export-verify/pull/30 have been rebased after https://github.com/fcrepo4-labs/fcrepo-import-export-verify/pull/29?12:39
There are duplicate commits in PR30.
<awoods>westgard: I see two commits: https://github.com/fcrepo4-labs/fcrepo-import-export-verify/commits/master12:41
https://github.com/fcrepo4-labs/fcrepo-import-export-verify/commit/62a2e0c295dfe9be8eec8c5b8b5b2163f9910b5e
and
https://github.com/fcrepo4-labs/fcrepo-import-export-verify/commit/b87744cb3868987017437a8af4e2f3f966ca3d18
westgard... is there overlap in those two commits?12:42
<westgard>awoods: It seems the answer is no, but the description of the commit for PR30 refers to changes that had gone into PR29.12:47
awoods: can the description still be edited?12:48
<awoods>westgard: not without a force-push to master... which we don't want.12:49
westgard: yes, the final commit descriptions should be updated (at the time of merge) to be more meaningful, i.e. include the actual link to the JIRA ticket, remove comments from interim commits, provide a clear description of what the commit is doing.
westgard: there is a "point of process" related to commits12:50
westgard: and commit message associated with merging those commits.12:53
<westgard>awoods: OK. Sorry. I think it's OK like this, but the description is somewhat misleading (the bit in blue at the top of the PR page). It appears that this is auto-generated from the summaries of the commits that make up the PR. And while github is smart enough (it seems) to omit commits already applied at time of merge, the description still reflects the complete list.12:57
<awoods>westgard: agreed. When the person who is performing the "merge/commit" clicks the green button, they have the ability to change the merge/commit message that GitHub auto-generates. They should change the auto-generated message per the suggestions above. That fact is not documented anywhere. :(12:59
* coblej leaves13:09
* peichman joins13:18
* coblej joins13:23
* peichman leaves13:24
* lsitu leaves13:42
* lsitu joins
<benpennell>awoods mikeAtUVa: I created a PR for the export versions ticket (https://jira.duraspace.org/browse/FCREPO-2458) and was thinking of moving on to working on the import, unless either of you think I should look at something else in the meantime?13:58
<awoods>benpennell: import of versions sounds good13:59
<benpennell>awoods: The first step would be addressing getting fedora:created timestamps in roundtripped versions. My inclination would be to add a header to POST requests that allowed for setting the date, possibly borrowing the memento one. I listed a few options in the planning ticket though14:00
* awoods on a call
* peichman joins14:03
* peichman leaves14:23
* benpennell leaves14:53
* coblej leaves15:00
* coblej joins15:02
* coblej leaves15:10
* coblej joins
<awoods>benpennell: as discussed during last Thursday's Fedora Tech call, there was consensus that the creation date of the version may not be that important to retain. What is more important, is the modification date of the resource being versioned.15:11
* coblej leaves15:18
* youn leaves15:23
* coblej joins15:26
* github-ff joins
[fcrepo-import-export] escowles created bag-config-file (+1 new commit): https://git.io/vHIZe
fcrepo-import-export/bag-config-file 2ee5ac0 Esmé Cowles: Write unmodified data directory to config file even when exporting Bags
* github-ff leaves
* github-ff joins15:28
[fcrepo-import-export] escowles opened pull request #86: Write unmodified data directory to config file even when exporting Bags (master...bag-config-file) https://git.io/vHIZC
* github-ff leaves
* coblej leaves15:30
* coblej joins15:36
* benpennell joins
* github-ff joins15:44
[fcrepo-import-export] awoods closed pull request #79: Support re-importing of hierarchy with customized predicates. (master...feature/import_hierarchy) https://git.io/v9Ag6
* github-ff leaves
* coblej leaves15:47
* coblej joins15:48
* benpennell leaves15:50
* dwilcox leaves15:51
* benpennell joins15:55
<mikeAtUVa>benpennell: is the plan to update fcrepo 4.x to support storing version timestamps?15:56
<benpennell>mikeAtUVa: that is what I had been thinking. I'm still trying to work out if modeshape would actually allow for it15:58
<mikeAtUVa>benpennell: oh, we can make it work... we'd likely have to store that value in some other property on the versioned resource...15:59
benpennel: I'd be happy to work such a ticket since I'm already haunted by the ghosts of having recently worked in the fcrepo-kernel-modeshape package.16:01
<awoods>lsitu: ping
<lsitu>Hi
<awoods>lsitu: round-tripping hierarchies works well, thanks16:02
<benpennell>mikeAtUVa: yeah… I was getting the impression it would have to go somewhere other then in the version itself, since i'm not clear on if there is actually anything you could set a property on in the version, although it is a jcrNode, so maybe
<awoods>lsitu: now I am testing round-tripping without binaries
lsitu: ...and seeing an issue
<lsitu>What do you see?
<awoods>lsitu: using the same setup of two servers and the ingest.sh script...
lsitu: I am exporting the collection0 resource without binaries16:03
lsitu: and get an error on reimport
<lsitu>Hmm, what error message you got?16:04
<awoods>lsitu: ERROR 16:04:47.197 (Importer) Error while importing /home/awoods/programming/java/apps/fedora/futures/fcrepo-import-export/data/f4/rest/collection1.ttl (400): javax.jcr.PathNotFoundException: No node exists at path '/bookA' in workspace "default"
javax.jcr.PathNotFoundException: No node exists at path '/bookB' in workspace "default"
javax.jcr.PathNotFoundException: No node exists at path '/bookC' in workspace "default"
<benpennell>mikeAtUVa: setting a property on the versioned resources feels a little weird though since that would be changing the object16:05
<awoods>lsitu: it looks like the referenced resource placeholders are not being created.
lsitu: here are my commands:
java -jar target/fcrepo-import-export-0.1.1-SNAPSHOT.jar -a -d data -m export -r http://localhost:8686/f4/rest/collection0
java -jar target/fcrepo-import-export-0.1.1-SNAPSHOT.jar -a -d data -m import -M http://localhost:8686/f4/rest,http://localhost:8080/fcrepo/rest -r http://localhost:8080/fcrepo/rest/collection0
<lsitu>awoods: Okay, I’ll ao a test and look into it.16:06
<awoods>lsitu: thanks
<mikeAtUVa>benpennell: properties show up when you create a version (the first at least) and the property on the resource would be hidden behind the scenes. It would likely change the modification date... I don't know if that's new behavior or undesirable.16:12
<benpennell>mikeAtUVa: not sure if that's a separate date from the fedora:lastModified (and whatever jcr property its masking) but I don't see that value changing on versioning16:14
mikeAtUVa: seems like it would be undesirable for the etag to change because of versioning though, but maybe that's just a personal issue16:15
<mikeAtUVa>benpennell: but from when there's no versions to when there's at least one, the etag should change because a triple shows up (fedora:hasVersions) linking to the versions.16:16
<benpennell>mikeAtUVa: oh i missed that thing, i probably should have used that in my export PR16:17
mikeAtUVa: last modified still seems unchanged, which is interesting16:21
<mikeAtUVa>benpennell: While I'm not 100% sure what the best behavior would be we can easily support either approach (having it change or not), but should probably endeavor to leave the current behavior so it's less likely to break people's applications.16:23
benpennell: in any event, if you're implementing an import of versions right now, I'd say leave out the version date, we'll make a new ticket for making the version import lossless and it will depend on updating the repository to accept the version date as part of version creation.16:24
<benpennell>mikeAtUVa: yeah i would agree. in terms what setting the property, do you mean it would be a property of the original versioned resource, or on each instance of the versioned resource within versions?16:25
mikeAtUVa: i hadn't started implementing import yet because whether we tried to set dates or not affected which parts of the export needed to be used, but i could definitely start on it16:26
<mikeAtUVa>benpennell: off the top of my head, my first pass at implementing this feature for versioning would be at version creation time to a) add a new versionDate property to the resource b) create the version snapshot, and update the version display to use that property (when present on the snapshot) to override the jcr:versionDate (or whatever it's called) that's used to show the version date.16:28
benpennell: but we don't need to get into implementation details... I think for maximum productivity in this sprint you have to implement version import without support for roundtripping version dates first.16:30
<benpennell>mikeAtUVa: that sounds deceptively straightforward :) but starting from the lossy roundtrip approach seems like a reasonable approach16:31
<mikeAtUVa>benpennell: I know, right... but I'm sure I'll find out why it won't work after nearly fully implementing it.16:34
<benpennell>mikeAtUVa: i'm going to stop thinking about this for the time being, but i'm not clear that the fedora:hasVersions property is really present on the object, it kind of looks like its being added into the response here https://github.com/fcrepo4/fcrepo4/blob/master/fcrepo-http-api/src/main/java/org/fcrepo/http/api/url/HttpApiResources.java#L87-L9116:40
<mikeAtUVa>benpennell: the fact that the etag doesn't change would imply that nothing is added to the object. But the fact that the representation changes for the user seems a pretty clear indicating that the etag *SHOULD* change.16:41
<benpennell>mikeAtUVa: yeah maybe i can blame that and my browser cache for why i didn't notice the fedora:hasVersions property before now, but probably not :)16:45
* yamil leaves16:51
* mikeAtUVa leaves16:56
* coblej leaves16:57
* coblej joins16:58
<lsitu>awoods: Yes, the placeholders for those “hasMember”s reference resources are not created which trigger the error when we import resource with custom predicates while we failed to provide the predicated to export and import it.17:02
It may not be meaningfult to create placeholders and import those resources in this case but we can do that to make it works. What do you think?
<awoods>lsitu: Thanks for digging into the issue. Maybe the behavior is as expected.17:03
lsitu: Leaving it for now is probably appropriate.17:04
<lsitu>awoods: Sure. It sounds good to me.17:05
<awoods>lsitu: thanks again
* umgrosscol leaves17:06
* coblej leaves17:07
* github-ff joins17:23
[fcrepo-import-export] awoods closed pull request #86: Write unmodified data directory to config file even when exporting Bags (master...bag-config-file) https://git.io/vHIZC
* github-ff leaves
<awoods>benpennell: I would say the fact that the etag does not change when the representation of a resource that is versioned changes (with a new hasVersions triple)... is a bug. If that bug is getting in the way for you, please feel free to create a ticket. If it is not, and since we will be significantly changing the versioning approach when we move to the Specified API, we can leave it be for now.17:31
<benpennell>awoods: I don't think there is a rush17:32
* benpennell leaves17:34
<westgard>import/export folks -- quick question:17:49
does the import/export tool support cerificate authentication and if not, are there plans to do so?17:50
awoods escowles -- ping re: question above17:52
<awoods>westgard: it supports basic auth only... at this point
westgard: I gather you have a use case?
<westgard>awoods thanks -- well, if we were going to use it on our production server that would be our preferred auth method17:53
We don't like putting user/password into config files
<awoods>westgard: for good reason17:54
westgard: would you please put in a ticket?
<westgard>I am working on the place in the code of the verifcation tool where this would belong and that got me thinking
I'll put in the ticket for this
<awoods>westgard++17:56
* westgard leaves18:49
* github-ff joins19:17
[fcrepo-import-export] lsitu opened pull request #87: Verify binary digests are used during import of Bags. (master...feature/verify_digest) https://git.io/vHIdu
* github-ff leaves
<lsitu>awoods/escowels: I am running out of tickets now. Please let me know what you would like me to work on next. Thanks.19:23
* mikeAtUVa joins21:11
* umgrosscol joins21:58
* mikeAtUVa leaves22:11
* mcritchlow joins22:40
* awoods leaves22:53
* dbernstein joins23:13
* umgrosscol leaves23:23
* lsitu leaves23:27
* mcritchlow leaves00:04