Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* ksclarke leaves01:47
* dwilcox joins08:24
* mikeAtUVa joins08:26
* dwilcox leaves08:54
* dwilcox joins08:59
* awead joins09:01
* acoburn joins09:05
* ksclarke joins09:09
* whikloj joins09:16
mikeAtUVa: What is the expected output of running migration-utils, do I aim it a F4 instance or does it output something I can use for importing?09:26
<mikeAtUVa>whikloj: right now it doesn't do anything except parse the foxml source...
<whikloj>mikeAtUVa: well that would explain it, thanks :)09:27
<mikeAtUVa>whikloj: I've got code pending that does actual content migration to F4 that I'll roll in when FCREPO-1386 is merged.
<whikloj>mikeAtUVa: no problem, I am trying to assist with dhlamb's "modification" of your utils. https://github.com/daniel-dgi/migration-utils/tree/camel-service09:28
and I wasn't clear on the current working state
<mikeAtUVa>whikloj: awesome! Yeah, right now it's just a framework with no actual migration code present... I figured it would be easier to restructure it before adding too much...09:29
whikloj: do you have an idea how it'll work in a camel context?
<ajs6f>barmintor:awoods: Versioning design might take into account: https://jira.duraspace.org/browse/FCREPO-127509:30
<awoods>mikeAtUVa: I believe Nianli is looking for review of https://jira.duraspace.org/browse/FCREPO-1386
<whikloj>mikeAtUVa: no, but my camel knowledge is less than my java knowledge. That's why I am assisting
<awoods>mikeAtUVa: do you have any time today?
<mikeAtUVa>awoods: I do... from about noon onward...09:31
<awoods>mikeAtUVa: that would be great.
ajs6f: that, too, would be great. Are you thinking along these lines, barmintor?09:32
<mikeAtUVa>awoods: what would be great?09:33
<awoods>mikeAtUVa: you giving https://github.com/fcrepo4-labs/migration-utils/pull/1 a review09:34
mikeAtUVa: did you miss my first message above?
mikeAtUVa: I believe Nianli is looking for review of https://jira.duraspace.org/browse/FCREPO-1386
<mikeAtUVa>awoods: yeah, I did miss that... sorry... but sure, I'll review it and pull it in... I'm not a partisan on spring, so I'll likely miss big-picture stuff, but I'll make sure it works.09:36
* osmandin joins09:37
<awoods>mikeAtUVa: I was planning on giving it a look as well... we can bounce ideas.
* jgpawletko joins09:42
* dwilcox leaves10:21
* dwilcox joins10:28
<barmintor>awoods: I’m not to that point yet, I’m trying to sort out how we should communicate that versioned updates are POSTs to another container10:43
fedora infers containment on a PUT via JCR, right?10:52
<f4jenkins>Project fcrepo4-T2 build #177: UNSTABLE in 5 min 4 sec: http://jenkins.fcrepo.org/job/fcrepo4-T2/177/10:53
<awoods>barmintor: via JCR?
<barmintor>awoods: I think the inference is b/c of the JCR backend
<awoods>barmintor: you mean fedora inspects the JCR node.children for contained resources?
<ajs6f>barmintor: Isn't that entirely at the "discretion" of JAX-RS?10:54
barmintor: Meaning our JAX-RS code.
<barmintor>awoods: No, I mean a PUT to /foo/bar/baz, if it doesn’t exist, creates a resource and gives it a container of /foo/bar
afaict, this is inherited from JCR, not a thing we explicitly do10:55
<ajs6f>barmintor: YOu mean the "create the parents if needed" thing?10:56
<barmintor>ajs6f: well, yes *and*
ajs6f: the implicit notion of parentage there, which you might remember Tom J arguing against in a mega-thread10:57
ajs6f: I AM NOT ARGUING FOR OR AGAINST
ajs6f: I am just documenting
<ajs6f>barmintor: The "Create the parents" thing is definitely ours, not JCR default behavior. I don't know what you mean by the "implicit notion of parentage there". Parentage in the JCR is extremely explicit...
<barmintor>ajs6f: JCR(/foo/bar) is a child of JCR(/foo)10:58
you don’t assert that, it’s just true
right?10:59
<ajs6f>barmintor: Not true. We publish clear child-parent and parent-child triples.
* barmintor sighs
<ajs6f>barmintor: Or do you mean that the client doesn't assert that?
<awoods>barmintor: "curl -i PUT localhost:8080/rest/a/b/c/d" returns a 404 if the parents do not already exist.11:00
<barmintor>ajs6f: is there any way that Fedora 4 creates a node at /foo/bar that is not a child of a node at /foo
<ajs6f>barmintor: No. That we inherit from HTTP/LDP. That's got nothing to do with JCR, to my understanding. We could be persisting to a SQL database and that would still have to be the case.11:01
<barmintor>awoods: Ah! that is contra these tests, but good to know
<awoods>barmintor: what tests?11:02
<barmintor>ajs6f: I don’t believe LDP specifies that containment, but your instinct is a thing Rob and Tom were arguing about
ajs6f: s/believe/think/, this is not an article of faith for me11:03
<ajs6f>barmintor: If LDP does not spec that containment, fine with me— then we should indeed stop publishing those triples. I happily defer to Rob's reading on that.
<barmintor>ajs6f: you have the roles reversed :)
<ajs6f>barmintior: Eh?11:04
<barmintor>Rob was “http suggests this containment”, Tom was “URIs are identifiers”11:05
<ajs6f>barmintor: Then I'm indeed with Rob on this. HTTP does suggest the containment. In fact, I think doing anything else would very much violate "least surprise".11:08
<barmintor>ajs6f: I am looking at some graph-backed LDP impls, and the containment has to be explicated. I am trying to figure out of I need to add this to my list of Fedora-isms Hydra expects.11:10
<ajs6f>barmintor: "explicated"? Not sure what you mean. Do you mean that triples have to be published?11:11
<barmintor>ajs6f: You have to POST to a container to have the relationship inserted
<ajs6f>barmintor: Okay? As opposed to doing what?11:12
<barmintor>awoods: if you have a chance, does the PUT work is there’s only 1 new child node
ajs6f: PUT /foo/bar as opposed to POST /foo, Slug: bar11:13
<ajs6f>barmintor: So you're saying maybe PUT is out of kilter/
?
<awoods>barmintor: "curl -i PUT localhost:8080/rest/a/b" returns a 404 if the parent "a" does not already exist.
<barmintor>awoods++ thanks
<awoods>barmintor: do you not have a running F4?
barmintor: "mvn jetty:run -pl fcrepo-webapp"
<barmintor>awoods: I have not enough memory to also have it running right now, I’m just reading the source11:14
<ajs6f>awoods: Don't we have a demo instance?
<awoods>ajs6f/barmintor: http://futures6.fcrepo.org:8080/fcrepo11:15
<barmintor>awoods: ok, I’m running in jetty11:26
<awoods>barmintor: go, jetty, go11:27
* awoods jumping on first meeting of the day
<barmintor>curl -I is a no-no,
curl -X
Ok, demonstrated: PUT to /a/b, when /a exists, creates /a/b and also the triple <a> ldp:contains <a/b>11:33
<ajs6f>barmintor: So PUT is at least consistent?11:34
<barmintor>ajs6f: I’m not sure what you mean, but it acts the way Hydra assumes Fedora acts11:35
ajs6f: in the graph backed impls, I have to POST to /a with a Slug of ‘b’ to get both
<ajs6f>barmintor: I meant that you can make the same assumptions about containment as you can with POST, but it's good to hear that in any event, it's not requiring our attention.
barmintor: OOh, wait? Really? That doesn't sound right.11:36
barmintor: Which impls?
<barmintor>ajs6f: this is what I was getting at
<ajs6f>barmintor: I mean _they_ don't sound right.
<barmintor>Lyo and Marmotta
<ajs6f>Hm.
<barmintor>Oh, sneaky.11:38
fedora *does* use conneg to figure out if your PUT is for an RDFSource or a binary, but it respects a client-side Content-Disposition=attachment if you want a ttl binary or something11:39
* barmintor ’s mind explodes11:40
<ajs6f>barmintor: That actually sounds useful. Some people (Aaron Brikland) have offered needs that actually touch on those kinds of things (whether/when a bitstream should become transpararent in the repository).
<barmintor>ajs6f: I would prefer the example to have been explicitly indicating the interction model, but w/e as long as I can document it11:42
<ajs6f>documentation++
<osmandin>uml++11:58
not really. afk11:59
<ajs6f>Doing UML builds character.12:02
<osmandin>lumbergh character, sometimes12:05
<ajs6f>Lumbergh?12:20
<osmandin>ajs6f: it's a fictional character (in a movie). but i have found some types of uml diagrams to be useful when communicating something complex.12:24
<ajs6f>osmandin: True that. Especially when you're dealing with mixtures of inheritance and other kinds of relationship.12:25
osmanin: Or swim lanes for tricky interactions. I would love to see swim lanes for LDP interactions.12:26
* dshalvi joins
* osmandin leaves12:34
* jgpawletko leaves12:54
<awoods>ajs6f/barmintor/osmandin: what do you think of the possibility of Audit Service working off of an external triplestore?13:02
<ajs6f>awoods: I think that it's an implementation decision, but I've certainly got nothing against the idea. If I was doing the impl, I'd probably do that.13:03
awoods: Keep in mind that just becase the audit-related triples go into a triplestore to support query, that doesn't mean that the triplestore has to be the authoritative durable store for that info.13:04
awoods: It could be thought of as merely an index.
<awoods>ajs6f: agreed, re:implementation.
<ajs6f>awoods: But without a triplestore, how is audit going to support SPARQL? (Which seems to have been requested.)13:05
<awoods>ajs6f: looking at requirements, a standard triplestore can support import and export
ajs6f: and there are no requirements around durable storage... yet.
<ajs6f>awoods: Yep. Many of the other requirements seem to have to do with the data model, which can be whatever, in or out of a triplestore.
It's orthagonal.13:06
ogonal.
<awoods>onal.
<ajs6f>al.
<awoods>.
<barmintor>awoods: I agree that the data modeling storage can be usefully separated from the querying, and it seems reasonable to me to run the queries against a triple store13:07
<ajs6f>awoods/barmintor: So the LDP "surface"— how does that arise? Another JAX-RS construction added to the repo?
<awoods>ajs6f: are you still referring to the audit service?13:08
<ajs6f>awoods: Yes.
<barmintor>I don’t understand the question, but my answer is usually not the addition of a new jaxrs endpoint13:09
<ajs6f>barmintor: You asked for an LDP-compliant API for audit. A triplestore won't do that. So what will?
<barmintor>Your audit information is LDPRs stored in a container that describes the audited resource13:10
<ajs6f>barmintor: So you're saying, as I suggested, that the repository does that?
<barmintor>some if it, sure13:11
in this case I would advocate wiring something in that does it
not the kernel
<ajs6f>barmintor: Wiring in something is exactly what I suggested. But this is all why I commented:13:12
v
orthagonal
Damn it.
https://wiki.duraspace.org/display/FF/Design+-+Audit+Service?focusedCommentId=68064897#comment-68064897
There.
Putting the info in the repo is apparently not okay with some people.
<barmintor>b/c it would be purge-able?13:14
<ajs6f>barmintor: I don't know. I didn't offer that requirement. awoods?13:15
<barmintor>that’s not a general requirement, that’s a behavior of an audit service implementation they have the prints log files13:16
<ajs6f>barmintor: I agree that it's not a good general requirement, that it's an impl concern— that's what I commented. In fact, I'm trying to build a case here for s to take that "requirement" off the table.13:17
* dwilcox leaves13:18
<barmintor>or, alternately read: that’s evidence that this whole Audit Service thing is not a function of the kernel but an application concern
<ajs6f>barmintor: Yeah, I can see that case. But I do have some sympathy with the notion that to be ultimately trustworthy, audit must have access to the lowest levels of a system.13:19
<barmintor>or, yet another read: maybe that’s a requirement of something Fedora 4 does, it’s for the steering group to work out, I don’t care
If I were on the SG, I would suggest that is a parochial concern that should be able to be accommodated in local installations, but is not part of what the product is13:23
<ajs6f>barmintor: Well, I have no intention of impling audit, so I don't care that much, but I would like to prevent its implementation from sucking up resources that could be used for other things.
barmintor: Sure, but if it _is_ accepted as a requirement, it's going to cost time and effort that could be better spent elsewhere (IMO). That's why I'd like to strike it.
awoods: Would you think this would be a good thing to bring up at the phone meeting on Thursday?13:25
* dwilcox joins13:26
<ajs6f>afk bbs13:36
<awoods>barmintor/ajs6f: re:audit, we have read and write requirements. Some of which comes from the repository and some from external sources.13:43
<ajs6f>awoods: Does that have anything to do with the question about where stuff gets stored?13:44
<awoods>barmintor/ajs6f: I can not help but ponder a model where external events write directly to an external triplestore, and repository events feed into camel which ends up in the triplestore.
ajs6f: what if audit information were stored in the triplestore?13:45
<ajs6f>awoods: I have no problem with that, inherently, but again, how does the LDP surface arise?
<awoods>ajs6f: no LDP13:46
<ajs6f>awoods: I thought that was one of our core principles? (To be sure, I, again, have no problem with that.)
<awoods>ajs6f: just SPARQL-Update and SPARQL-Query
<ajs6f>awoods: Fine by me.
<awoods>ajs6f: I think the core principle is: Use LDP where we can, otherwise use existing standards, otherwise integrate with standard external components.13:47
<ajs6f>awoods: Very sensible and politic.
<barmintor>what do you mean “no LDP”? Is the audit information not a LDPR that you PATCH?
<awoods>barmintor: what if the repo sent out JMS messages that were then fed into camel/triplestore13:48
<ajs6f>Audit is a state of mind. At least that's what I told the IRS.
<barmintor>awoods: fine but orthogonal to my question13:49
<awoods>barmintor: what if no audit information were stored in the repo?
<barmintor>awoods: from the client perspective, is the audit information a LDPR that you PATCH with new events?
<awoods>barmintor: which client?
<barmintor>any client
<awoods>barmintor: a client that is querying audit events?13:50
barmintor: a client that is posting external audit events?
<barmintor>I don’t know about querying
but certainly posting
<awoods>barmintor: are you talking about the case where an external process runs fixity on a binary?
<ajs6f>barmintor/awoods: Maybe we just slice out "external" stuff period. Audit feeds repo events into a triplestore. If you want to overlay "external" info into that, rock on, but it's not the repo's job.13:51
<barmintor>sure
<awoods>barmintor: and that external process wants to ensure that the event is associated with the resource's audit log?
<barmintor>awoods: this is an evergreen use case
<awoods>ajs6f: exactly... but maybe replace "rock on, but it's not the repo's job" with "we will document a pattern and format for you to interact with the external triplestore"13:52
<ajs6f>awoods: Fair enough. Practices are a lot cheaper to maintain than codebases.13:53
<awoods>ajs6f/barmintor: the audit service could collapse into ensuring that the repo publishes good JMS events...
ajs6f/barmintor: and some integration documentation13:54
<ajs6f>awoods: Events, period. JMS is impl.
<awoods>ajs6f: sure
<barmintor>well, again: perhaps the best approach here is not doing it. You can wire in a camel bit. You can have your camel bit ignore system updates to audit containers and log everything else in a LDPR if you want. Then if someone wants to add events externally, they can patch the LDPR
<ajs6f>barmintor++
<barmintor>write a document, exercise left to the reader.
but then the term “requirements” is meaningless.13:55
<ajs6f>One kind of place to store audit stuff just happens to be the repo. Do that and you get LDP for free. Use a triplestore and you get SPARQL for free. Or you can do both, if you like.
barmintor: Documented best practices?
<acoburn>awoods: are you thinking the audit jms feed would be part of the existing fedora topic or part of a separate feed?
<ajs6f>acobrn: Can you think of any kind of audit event that isn't already appearing in the current feed?13:56
<acoburn>ajs6f: validate a checksum
<awoods>acoburn: we could tease out audit events if that made sense
<ajs6f>acoburn: Or add some if they are needed.
<awoods>ajs6f/barmintor: I think we need to offer a working recipe that meets the requirements13:57
ajs6f/barmintor: but that does not imply the need for new code
<barmintor>that is true.
<ajs6f>awoods: It might imply some camel examples.
<awoods>ajs6f: yes
<ajs6f>Or maybe acoburn has already done that. It wouldn't surprise me.13:58
<acoburn>ajs6f: no, I haven't
<ajs6f>acoburn: I'm disappointed. That's not the acoburn I know.13:59
<awoods>ajs6f/barmintor/acoburn: let's talk more on Thurs (tech meeting and audit meeting)... I like the idea, however, of letting external components do the heavy lifting.
* Nianli joins
* awoods letting next door neighbor's dog out and getting lunch
<ajs6f>Hm. A framework for managing and routing events. We cold build one of those. Or we could just use Camel.14:00
afk bbl14:04
* ksclarke leaves14:17
* ksclarke joins14:32
* dshalvi leaves14:58
* osmandin joins15:25
<ajs6f>awoods: Will you put that on the agenda?15:29
<awoods>ajs6f: audit on the tech agenda?15:49
<ajs6f>awoods: Both
awoods: both agendas, that is15:51
<awoods>ajs6f: I am jumping on a call shortly. Feel free to update the agendas... otherwise, I will later.
<ajs6f>awoods: I'll just cut all the other items out. That should give us enough time.15:52
* dwilcox leaves16:13
<whikloj>acoburn: do you a second for a camel question?16:44
<awoods>mikeAtUVa: ping
<acoburn>whikloj: sure
<mikeAtUVa>awoods, yes?
<awoods>mikeAtUVa: I am looking at migration-utils16:45
<mikeAtUVa>awoods: any suggestions? questions?
<awoods>mikeAtUVa: there are probably some things to be updated (logging instead of System.out, more tests, format rules, etc)
<whikloj>acoburn: working on camel migration of Fedora 3 objects to Fedora 4, can we "query" the F4 using some property (ie. old PID) to get the new object. In a camel route of course
<mikeAtUVa>awoods: indeed16:46
<awoods>mikeAtUVa: I am wondering where that project stands, who is collaborating on it, and how much clean-up of the nature I mentioned we should begin.
<mikeAtUVa>awoods: would a chat or hangout be easier, or should I sum it up on IRC?
<awoods>mikeAtUVa: and if you thing Nianli's updates are complete16:47
Nianli: care to join a hangout?
<acoburn>whikloj: you mean is there a way to retrieve (or check for existence) for a resource in F4?
whikloj: if so, you just set the CamelFcrepoIdentifier header to the F4 path16:48
<awoods>mikeAtUVa/Nianli: https://plus.google.com/hangouts/_/event/c1glu6soq43r1rr6ou17qtobug8
<acoburn>whikloj: and then call: to("fcrepo:localhost:8080/rest")
<whikloj>acoburn: retrieve I would think, as we migrate objects we want to maintain relationships. So we will need to have a way to get the "new" id of some objects
acoburn: in the case where CamelFcrepoIdentifier is our subject, I need to get the new object of the triple16:49
<acoburn>whikloj: are you creating the object from inside camel?
<whikloj>acoburn: yes
<acoburn>whikloj: if so, then the response headers will have the new path16:50
whikloj: maybe not in the current release, but that's how it works in the current snapshot
<whikloj>acoburn: right, but once that object is created. If I later have an object that has an old <fedora-rels-ext:isMemberOf> relationship. Can I retrieve the other objects ID?16:51
<acoburn>whikloj: I see. for that you will probably want an external database that persists that mapping16:52
<whikloj>acoburn: So if we use a triplestore during our migration. Could we query that in the route?16:53
<acoburn>whikloj: yes that would work, but it would be harder to implement than just using a simple database16:54
<whikloj>acoburn: ok, thanks for the info16:56
<awoods>mikeAtUVa: http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22fcrepo4-client%22
<acoburn>whikloj: personally, I'd probably use camel-sql, but if you go the triplestore route, you might find the processor classes useful (the ones in o.f.camel.processor)16:57
whikloj: those classes, on their own, won't get you very far, but they are examples of how to translate a message into a Sparql-update query that you can then send to a camel-http4 endpoint16:58
<whikloj>acoburn: I think you are right about using camel-sql, but I wanted to keep my options open.16:59
* whikloj leaves
* osmandin leaves17:02
* acoburn leaves17:57
* ksclarke leaves18:08
* github-ff joins18:24
[fcrepo4] awoods pushed 1 new commit to master: http://git.io/hGd1
fcrepo4/master a78707c osmandin: Remove equals() for a private utility class with no fields...
* github-ff leaves
* github-ff joins18:25
[fcrepo4] awoods closed pull request #749: Remove equals() for a private utility class with no fields (master...1404) http://git.io/hUjm
* github-ff leaves
* travis-ci joins18:45
fcrepo4/fcrepo4#3493 (master - a78707c : osmandin): The build passed.
Change view : https://github.com/fcrepo4/fcrepo4/compare/7c3deba14d23...a78707c0768e
Build details : http://travis-ci.org/fcrepo4/fcrepo4/builds/54792016
* travis-ci leaves
* fcrepo-bot joins19:21
* fcrepo-bot leaves20:28
* ksclarke joins21:29
* Nianli leaves21:53
* ksclarke leaves22:41
* ksclarke joins22:57
* awead_ joins23:10
* awead leaves

Generated by Sualtam