Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* ksclarke leaves02:26
* barmintor leaves03:24
* kaarefc joins03:33
* kaarefc leaves03:46
* kaarefc joins03:47
* kaarefc leaves03:59
* kaarefc joins04:07
* kaarefc leaves04:18
* kaarefc joins04:26
* kaarefc leaves04:32
* kaarefc joins05:11
<pivotal-bot>Frank Asseg added comment: "It seems Fedora produces 404 for all fixity-object requests not just for projected datastreams.05:28
This is repr..." https://www.pivotaltracker.com/story/show/56002916
<fasseg>Anyone already awake, that can verify that all fixity result requests produce a 404? So just ingest a new datastream and try to get it's fixity result node...05:29
* kaarefc leaves06:20
<pivotal-bot>Frank Asseg added comment: "@awoods: Sorry I meant the modeshape release will probably come in "December of 2013" not December 13th. At ..." https://www.pivotaltracker.com/story/show/6141839806:27
Frank Asseg added comment: "@awoods: I agree with @cbeer here, I think postponing the holiday release if possible is woth it. Otherwise..." https://www.pivotaltracker.com/story/show/6141839806:30
Frank Asseg added comment: "And im around after the 13th, so I can certainly help out with the release preparation" https://www.pivotaltracker.com/story/show/6141839806:32
<fasseg>awoods: cbeer: On the ticket https://www.pivotaltracker.com/story/show/61418398 I meant "This release will hopefully happen in Dec 2013" when I wrote:"This release will hopefully happen in Dec 13", I didn't want to say "Dec 13th" !!06:37
<pivotal-bot>bug: Work out the cause of the large file ingest test failure with rhauch (started) / owner: Frank Asseg
Frank Asseg added "Adding a node to a federated read-only Filesystem is enabled in the Frontend and creates a NPE when used" https://www.pivotaltracker.com/story/show/6196991006:47
Frank Asseg edited "Adding a node to a federated read-only file system is enabled in the frontend and creates a NPE when used" https://www.pivotaltracker.com/story/show/61969910
* kaarefc joins06:48
* kaarefc leaves07:03
* tecoripa joins07:35
* tecoripa leaves07:39
<pivotal-bot>Esme Cowles added comment: "I started a script yesterday to generate and roundtrip progressively larger random files. Files up to 256GB..." https://www.pivotaltracker.com/story/show/6177343007:42
* kaarefc joins07:53
<pivotal-bot>Esme Cowles edited "Provide some example SPARQL queries / help text for updating properties in the HTML UI" https://www.pivotaltracker.com/story/show/6150047608:12
* mikeAtUVa joins08:52
<pivotal-bot>Mike Durbin started "Write a velocity template for the fcr:versions endpoint." https://www.pivotaltracker.com/story/show/6147423609:06
Mike Durbin finished "Write a velocity template for the fcr:versions endpoint." https://www.pivotaltracker.com/story/show/61474236
Mike Durbin added comment: "Fair enough. I've amended the pull request." https://www.pivotaltracker.com/story/show/6147423609:11
* tecoripa joins09:26
<awoods>thanks for the clarification, fasseg.09:31
<fasseg>awoods: sry for the ambiguity
<awoods>fasseg: Let't talk on the committers call about release scheduling.09:32
<fasseg>kk
<pivotal-bot>Andrew Woods added comment: "@escowles, please add your configuration as well to the wiki doc." https://www.pivotaltracker.com/story/show/6177343009:34
<awoods>fasseg: Any news on the fixity/federation issue?
<fasseg>awoods: yes, but I guess I'll need some help there, since it seems to me it's not only happening in a federation config09:35
<awoods>fasseg: meaning, you are seeing fixity errors in single-node configurations?
<fasseg>awoods: you can try and request the fixity result node (not the one ending with fcr:fixity but the one below that)09:36
for me it's e.g. http://localhost:8080/fcrepo/rest/0d/76/0b/1f/0d760b1f-09ff-48fb-a3d0-44dc5178247b/fixity/138625422324209:37
which is a fixity result of a datastream in a non federated environment09:38
and Im not able to determine the place where those fixity results are saved in Mode at the moment, it seems to me that they are transient.....
* ksclarke joins
<fasseg>so Im updateing the large files wiki with my federation results atm and will ask Ben and Adam later how the storing of the fixity result is working09:39
<awoods>fasseg: It looks like ajs6f is out today.09:42
<fasseg>awoods: I started documentation at https://wiki.duraspace.org/display/FF/How+to+handle+large+files . Should this rather go into: https://wiki.duraspace.org/display/FF/Design+-+Large+Files ? Or is a link there fine? Since you requested that I add my stuff to the Design page....09:48
* nbanks joins
<awoods>fasseg: Now that escowles is also using your "How to handle large files" page, might as well stick with it for now.09:49
* osmandin joins09:53
afk09:54
* ermadmix joins
<tecoripa>general question for RDF developers: there's FORMAT_URI attribute in the Fedora 3 content models that defines the format uri of the datastream. It's an attribute on the <form> element of the <dsTypeModel>09:55
<dsTypeModel ID="BIB0" optional="false">    <form FORMAT_URI=" MIME="text/xml"></form>  </dsTypeModel>
my question is: does anyone have an opinion or idea about how that should be mapped to a node property?
I have three alternatives:09:56
1) find a nice defined RDF property is some established schema out in the world, and use that. That's the preferable choice, but I spent some time last night looking around, and couldn't come up with anything.
2) Define our own fedora:formatURI property, extension of jcr:content?09:57
3) Do nothing -- ignore that attribute when migrating a fedora 3 content model
<mikeAtUVa>tecoripa: I think it's dependent on why people used that field (if they did at all)...09:59
<awoods>tecoripa: It seems like we will also want to help avoid inconsistencies between this new property and the existing jcr:mimeType property.10:00
<tecoripa>mikeAtUVa: well, it was never validated, internally. people might have used it to perform external validations or bookkeeping
awoods: jcr:mimeType is a different property. I've already put that property in the proposed content model migration doc.10:01
<awoods>tecoripa: Yes, it is a different property.10:02
tecoripa: I am suggesting that we try to help avoid inconsistencies between the two properties.
<tecoripa>I'm thinking a format URI property would be useful to have indexed, so that you could do some sort of external validation of contnet, or find all the PREMIS 1.0 datastreams, for example, and update them to PREMIS 2.0
awoods: inconsistenices, in that the mime type may not go together with the format uri?10:04
<awoods>tecoripa: I take it that the formatURI and the mimeType are related, no?
<tecoripa>I'm thinking format URI would be optional -- not sure it really makes sense for anything that's not xml
awoods: yes, they're related insofar as the mime type tells you what kinds of bytes are in the datastream, and the format URI tells you how those bytes are structured and used (at least in the case of XML)10:05
* escowles joins10:08
<awoods>tecoripa: When I think of formatURI, I think of Pronom PUIDs: http://info-uri.info/registry/OAIHandler?verb=GetRecord&metadataPrefix=reg&identifier=info:pronom/
<tecoripa>looks like the copy I did in the chat window got munged. should look like this : <form FORMAT_URI="some:uri/thingy" MIME="text/xml">10:10
<awoods>mikeAtUVa, All: ReadyTalk for today's committer call?10:11
<mikeAtUVa>awoods: I doubt we'll beat 15, but sure, I think that's what people are used to for the Thursday calls.10:13
<tecoripa>awoods: is file format in the PUID a mime type kind of format, or a file format in the Fedora 3 dsTypeModel sense (i.e., closer to a schema definition)?
<pivotal-bot>Esme Cowles added comment: "I think there are two pieces to this: ""
1. Add some more examples to the SPARQL examples page (https://wiki.d..." https://www.pivotaltracker.com/story/show/61500476
<awoods>tecoripa: PUID uniquely specifies a format, or version of a format.10:16
<pivotal-bot>Eric James added comment: "Tried to reproduce and did not get an error. I created an object, created a datastream with a random file, c..." https://www.pivotaltracker.com/story/show/56002916
* tom____ joins
<awoods>tecoripa: http://www.nationalarchives.gov.uk/PRONOM/Format/proFormatSearch.aspx?status=new10:17
<fasseg>barmintor: It seems that fixity result nodes are not available using a normal binary store either. I wanted to investigate but hit a wall when I was trying to find the method responsible for storing fixity results...Can you point me in the right direction?10:21
<barmintor>Fixity result nodes?10:22
<tecoripa>awoods: thanks. This seems to be close, but maybe not specific enough? I see there are entries in the registry for a few different flavors of XML, but not any for MODS, DC, or other metadata standards. But that opens the question, is FORMAT_URI in Fedora 3 not intended for those kinds of URIs, but more for formats of the kind covered by the pronom registry?
<barmintor>I don't recall them.
<fasseg>sth like: http://localhost:8080/fcrepo/rest/.../fixity/1386254223242
<barmintor>fasseg: that part of fixity I worked on was the result reporting, not the storage
<fasseg>which is pointed to by the RDF produced by the fcr:fixity endpoint10:23
<tecoripa>awoods: in which case, yes, it seems like there is considerable overlap between FORMAT_URI and the file's mime type
<fasseg>barmintor: I see...I'll ask cbeer than, since Adam is off for today...
<awoods>tecoripa: It probably comes back to mikeAtUVa's comment, that how we do (or do not) handle formatURIs will depend on how people need to use it. I do not think we should blindly port the F3 concepts to F4 without usecase justification.10:24
<pivotal-bot>Frank Asseg added comment: "hmm...lemme recheck" https://www.pivotaltracker.com/story/show/5600291610:25
<cbeer>fasseg: i see results.. are you talking about something different? https://gist.github.com/cbeer/28e3063f064cbfdc06d010:26
<fasseg>cbeer: just rechecking my test since ermadmix was also unable to reproduce the issue....so ignore this for a bit....10:27
hmm it seems i still have this issue10:28
ill write up a gist...
<barmintor>awoods: These Fedora events- I presume what we want are created/changed/removed for objects and datastreams. Do I have latitude about getting to those 6 events?10:29
<tecoripa>awoods: agreed. I can see that knowing the kind of ML a datastream contains without having to look at the datastream content itself would be useful: for quick identification of content of a particular flavor of XML, for content validation purposes... but then it opens the door to properties and data getting out of sync, which was always a danger in Fedora 3, too; there's nothing that would prevent me from sticking a JPEG bitstream in a conte10:30
that's a danger with externally-declared mime type properties, too for that matter.
<awoods>barmintor: yes, those are the events of interest. Are you suggesting something sneaky?
<pivotal-bot>Esme Cowles added comment: "I've linked my platform/repository/workflow profiles in the wiki page and added my times for a 512GB file. ..." https://www.pivotaltracker.com/story/show/61773430
<barmintor>awoods: Just trying to synthesize what I've learned looking at our code and at MODE's
<fasseg>cbeer: try this URL to get the result node (It's from your gist): http://localhost:8080/fcrepo/rest/18/6e/7f/09/186e7f09-d3cb-4458-8054-d10c8a38366a/fixity/1386257114927
cbeer: this gives me a 40410:31
<cbeer>fasseg: it doesn't exist. it's in the fcr:fixity response.
those fixity results as transient. we're just giving you a nice URI so you can store it and refer to it later if you want
<fasseg>Hmmm...so this is not an issue with federated nodes as well....10:32
<cbeer>(and because ajs6f couldn't be bothered to figure out rdf streaming with blank nodes)
<fasseg>2 days -> drain :)
<awoods>fasseg: drain stopped. Does that mean fixity is also fine in the federation context?
<fasseg>if those results are supposed to be transient, then yes....10:33
<cbeer>tecoripa: " for quick identification of content of a particular flavor of XML, for content validation purposes."
<fasseg>because I can request fcr:fixity just fine for federated content
<cbeer>tecoripa: isn't that what modeshape sequencers are for?
<awoods>fasseg: ermadmix originally created this bug-ticket. Is it a non-issue? https://www.pivotaltracker.com/story/show/5600291610:34
<tecoripa>cbeer: I dunno. Are they? that would be great.
<pivotal-bot>bug: fcr:fixity bug using the FileSystemConnector (started) / owner: Frank Asseg
<fasseg>awoods: but we're still producing a link to a nonexisting resource in the HTML response, we should probably fix that then...
<cbeer>fasseg: yeah, i'm not sure why those links aren't e.g. .../fcr:fixity#1386257114927
but that's probably a question for ajs6f
<awoods>cbeer/fasseg/tecoripa: We probably need to start a list of ajs6f questions.10:35
<tecoripa>cbeer: yes, sequencers look like they're the ticket
awoods: it would be good to have ajs6f and maybe cbeer give a quick review of the content model migration page, as it currently exists, just to make sure that it makes sense logically and conceptually. nbanks and I came up with what seems (to us) a reasonable approach to content modelling with node types; but it would be good to get people with more experience of the ModeShape way of life to approve.10:38
<awoods>tecoripa: Sounds like a reasonable committers call topic.10:39
<fasseg>awoods: I only see the ERROR 16:38:11.842 (DatastreamService) ALL COPIES OF /projection/sample.100m HAVE FAILED FIXITY CHECKS msg only wehn using projection though, so there still is an issue...
<tecoripa>awoods: this is aside from the technical implementation of our model... it may turn out that what we propose encounters problems when we actually try to make it real, but at least we'll know that conceptually we're on the right track.10:40
<pivotal-bot>Frank Asseg added comment: "Ok so the nodes /fixity/.... are supposed to be transient. But there still is the issue in federation with t..." https://www.pivotaltracker.com/story/show/56002916
Mike Durbin added comment: "This documentation is complete for what will likely be included in the pre-alpha release.10:52
https://wiki.duras..." https://www.pivotaltracker.com/story/show/60798758
Mike Durbin accepted "Add versioning documentation/walkthrough to the wiki" https://www.pivotaltracker.com/story/show/60798758
<barmintor>awoods: NodeMoved has no corresponding FedoraEvent. Ignore, or treat as NodeAdded?10:53
<pivotal-bot>Andrew Woods added comment: "Testing up to 1-TB seems adequate. It is interesting (and good) that no timeouts are coming into play. " https://www.pivotaltracker.com/story/show/61773430
<awoods>barmintor: is NodeMoved a combination of "add" and "delete"?10:54
<cbeer>bah. i was trying to figure out why replication wasn't working with my leveldb config.
turns out if you don't tell it to fetchPersistentState.. it doesn't.
<pivotal-bot>Scott Prater added comment: "https://wiki.duraspace.org/display/FF/How+to+Migrate+a+Fedora+3+Content+Model+to+a+Fedora+4+Node+Type" https://www.pivotaltracker.com/story/show/6096348210:55
<barmintor>awoods: it is
<awoods>cbeer: You are talking about leveldb in the clustered scenario?10:56
<cbeer>awoods: yes. i forgot that actually-usable clustering needed configuration at the loader too
<awoods>barmintor: but the corresponding "created" and "removed" events are not emitted?
<barmintor>awoods: I'll have to write a completely different listener for those events, but I can synthesize the delete/create
<cbeer>barmintor: did you ever run across http://docs.jboss.org/infinispan/5.3/configdocs/infinispan-cachestore-leveldb-config-5.3.html?10:57
<barmintor>awoods: No, but you can get the former path from the event info
cbeer: Nope, but looks like I should have.
cbeer: wait- a similar doc is where I got clued in about singletonStore10:58
<pivotal-bot>Mike Durbin added comment: "This documentation is complete for what will likely be included in the pre-beta release.
https://wiki.durasp..." https://www.pivotaltracker.com/story/show/60798758
<awoods>cbeer: That reminds me... I believe there is a ticket for organizing/cleaning the modeshape and ispn configuration files in our codebase. Can you give that a shot?10:59
<cbeer>awoods: not any time soon. if tcramer wants favorable numbers for next week, i need to work on that first.11:00
<osmandin>readytalk or hangout?11:01
<mikeAtUVa>readytalk11:02
866-740-1260, participant code: 2257295
<osmandin>thx mikeAtUVa
<mikeAtUVa>https://wiki.duraspace.org/display/FCREPO/2013-12-05+-+Fedora+Committer+Meeting
<osmandin>here11:05
there
<mikeAtUVa>https://wiki.duraspace.org/display/FF/Sprints+B8+and+B9+Planning
<awoods>cbeer/barmintor: are you on the call?
<barmintor>AWOODS: TOTALLY11:06
<cbeer>awoods: no, i'm not yet.. trying to get this cluster going11:07
i'll drop in late, hopefully
ok, here.11:13
<mikeAtUVa>https://wiki.duraspace.org/display/FF/Versioning11:16
<pivotal-bot>Scott Prater added comment: "https://wiki.duraspace.org/display/FF/Versioning" https://www.pivotaltracker.com/story/show/6096709011:17
Chris Beer added comment: "what API call did you make? using LDPath or SPARQL? What query?" https://www.pivotaltracker.com/story/show/61942638
<barmintor>mikeAtUVa: Inasmuch as it does affect it, the work I'm doing on events is related to single node performance11:20
I probably need to reboot11:23
<cbeer>mikeAtUVa: you forgot. we should have just volunteered him for something.
ls -la
<barmintor>awoods: yes, the event processing and related node lookups are YK's biggest remaining source of complaints/hotspots
<pivotal-bot>Osman Din added comment: "I wrote a simple program (based on Modeshape classes) to get clues from the FS (see wiki doc for the layout de..." https://www.pivotaltracker.com/story/show/4901279911:29
<tecoripa>https://wiki.duraspace.org/display/FF/Test+-+Platform+Profile%3A+Cluster+at+University+of+Wisconsin (details are still being worked out)11:31
<cbeer>minimally inviable products.11:57
dropping.12:07
<barmintor>awoods: I am told that I am to be availble for that pre-con12:09
<escowles>dropping off -- out to lunch
<osmandin>afk (meeting)12:10
<tecoripa>https://wiki.duraspace.org/display/FF/How+to+Migrate+a+Fedora+3+Content+Model+to+a+Fedora+4+Node+Type
<awoods>barmintor: That is great. Would you be interested in taking the lead on the F4 pre-con and replying to the list thread with subject "[fedora-tech] RE: [ff-tech] Code4Lib/Fedora4 Proposal: Now"?12:11
<barmintor>awoods: I guess so.
<nbanks>tecoripa: https://docs.jboss.org/author/display/MODE/Sequencing#Sequencing-Configuringasequencer
<fasseg>gregjansen: Have we got this fuctionality implemented: "As of ModeShape 3.4.0.Final, the file system connector is pageable, which means it can efficiently expose folders that contain large numbers of items"?12:23
<gregjansen>fasseg: where did I write that?
<fasseg>gregjansen: you didn't its from: https://docs.jboss.org/author/display/MODE/File+system+connector12:24
gregjansen: but you did the connector work, right?
<gregjansen>fasseg: yes at one stage.
fasseg: we had our own connector for a while, but now I think we use the supplied one from MODE12:25
fasseg: we also had one for bagit at one point, but I think it is more of a proof of concept
<awoods>nbanks: Are you planning on posting your minutes here: https://wiki.duraspace.org/display/FCREPO/2013-12-05+-+Fedora+Committer+Meeting12:28
<pivotal-bot>Andrew Woods added comment: "@osmandin, The third item in the ticket description is asking for documentation of a dissection of the seri..." https://www.pivotaltracker.com/story/show/4901279912:33
<awoods>nbanks: got it12:39
* tecoripa leaves12:41
<nbanks>awoods: are they not there?12:42
awoods: oh ok
<awoods>nbanks: they are there, thanks
* tecoripa joins12:43
re-running benchtool with my platform and repo profiles...
times are pretty much the same as they were after the first batch of changes I made to benchtool, which is good.12:44
<awoods>osmandin: Did you catch the action to rerun your single-node tests?12:46
* tecoripa leaves12:47
* ermadmix leaves12:48
<pivotal-bot>Frank Asseg added comment: "problem seems to be that https://github.com/futures/fcrepo4/blob/b843b2c2a5ad4bdc5a85fdf0ee1606f2e7b0b4a0/fc..." https://www.pivotaltracker.com/story/show/5600291612:52
<osmandin>awoods: yes, I'll run the test. sorry i was in a meeting12:54
afk 3012:55
<awoods>osmandin: thanks, np.
<fasseg>barmintor: cbeer: I traced the issue, and the problem is caused by projected binaries not being available in the InfinispanBinaryStore, and therefore the LowLevelCacheEntry for https://github.com/futures/fcrepo4/blob/b843b2c2a5ad4bdc5a85fdf0ee1606f2e7b0b4a0/fcrepo-kernel/src/main/java/org/fcrepo/kernel/services/functions/CheckCacheEntryFixity.java#L68 is empty and a fixity result will be generated from an empty 12:57
since I seem to be unable to get the projected binary from the store itself...12:59
* tecoripa joins13:05
@fasseg: I have a question about benchtool results when running threaded
* ermadmix joins13:06
<tecoripa>it looks like you sum the run time of all the threads, and report that as the duration
fasseg: https://github.com/futures/benchtool/blob/master/src/main/java/org/fcrepo/bench/FCRepoBenchRunner.java#L141
fasseg: but the threads run in parallel13:07
fasseg: so the duration is actually much shorter
<fasseg>tecoripa: that's just for per thread tp isnt it?
and the actual is calculated by tp/thread * numthreads13:08
<tecoripa>fasseg: would it be more accurate to report the average duration for each thread? duration/numThreads
<fasseg>https://github.com/futures/benchtool/blob/master/src/main/java/org/fcrepo/bench/FCRepoBenchRunner.java#L159
tecoripa: but then you're measuring all the overhead...13:09
<tecoripa>fasseg: yes, that's the average throughput per thread
fasseg: but that's not the duration
<fasseg>no, you're measuring the pure throughput per request to fcrepo, which I thought was requested..13:10
<tecoripa>fasseg: is the duration all the overhead? or do you start, stop the clock before and after sending the request and receiving the response?
<fasseg>duration has no overhead...13:11
just the HttClient.execute() methid is wrapped by the duration counter
<tecoripa>fasseg: okay, good. but the final duration at the end that is reported is the *sum* of all those requests timings...13:12
<fasseg>see e.g. https://github.com/futures/benchtool/blob/master/src/main/java/org/fcrepo/bench/Fedora4RestClient.java#L52
<tecoripa>which, when threads are run in parallel, is misleading
https://github.com/futures/benchtool/blob/master/src/main/java/org/fcrepo/bench/FCRepoBenchRunner.java#L148
<fasseg>tecoripa: that's true, that should be duration/numThreads, or I can measure the whole process duration into a separate value...13:14
which we could call runtime or something so it'13:15
s clear it's not the duration of all the requetss
<tecoripa>fasseg: yes, maybe both, so we can compare, what overhead is vs. the actual time it takes to do the requests13:16
<fasseg>will do...
<tecoripa>escowles: have you been doing multithreaded tests with the new benchtool? if so, what numbers have you been recording?13:17
<awoods>mikeAtUVa: I am only seeing the "Root Version" in testing your PR13:35
* ermadmix leaves13:38
<awoods>mikeAtUVa: nevermind... updated the cnd.13:41
<fasseg>tecoripa: I created a PR for the added functionality for you to review: https://github.com/futures/benchtool/pull/813:42
<pivotal-bot>Andrew Woods delivered "Write a velocity template for the fcr:versions endpoint." https://www.pivotaltracker.com/story/show/61474236
* github-ff joins
[fcrepo4] awoods pushed 2 new commits to master: http://git.io/YpT9lA
fcrepo4/master cca858a Michael Durbin: Added Velocity template for HTML view of fcr:versions.
fcrepo4/master bf1d4b1 Andrew Woods: Merge pull request #184 from mikedurbin/versioning-4...
* github-ff leaves
<tecoripa>fasseg: looks good. do you think it would be clearer if you changed "Completed {} {} action(s) executed in {} ms" to "Completed {} {} action(s) executed in {} ms per thread, avg."?13:46
fasseg:https://github.com/futures/benchtool/commit/6b4f509a55aa65e4beb60defa6746e8933573da4#diff-d85d38bee7811c662ffdb1e1e5983735R151
<fasseg>since it was redundant ;)
tecoripa: can you put any comments on the PR? I'm at dinner now...13:47
<tecoripa>fasseg: will do. thanks. and enjoy. and quit typing while eating. you'll get the keyboard all greasy.13:49
<escowles>tecoripa: i've been doing single-threaded tests with the new benchtool. i did some threaded tests early on with the old benchtool, but none with the new one13:56
* travis-ci joins
[travis-ci] futures/fcrepo4#1341 (master - bf1d4b1 : Andrew Woods): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/fa517f255803...bf1d4b111fc4
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14998487
* travis-ci leaves
<tecoripa>escowles: ok, just checking. the numbers reported when running multithreaded were off... but fasseg just fixed it.13:57
<barmintor>gregjansen: you don;t happen to have call info for this APT thing going on right now?14:04
* ermadmix joins14:06
<gregjansen>barmintor: just got back I can try and find it if you still need it14:07
<barmintor>gregjansen: that would help, thanks!
<pivotal-bot>Esme Cowles started "Provide some example SPARQL queries / help text for updating properties in the HTML UI" https://www.pivotaltracker.com/story/show/6150047614:11
<gregjansen>awoods: when adding basic roles impl to fcrepo-webapp it seems like it would involve pulling fcrepo-auths into fcrepo, at least as a submodule.. what do you think?14:33
what do any of you think?14:37
* ermadmix leaves14:38
* ermadmix joins14:41
<gregjansen>awoods: I guess the fcrepo-authz projects are in maven central..14:45
<fasseg>tecoripa: I updated https://github.com/futures/benchtool/pull/8 again, so you can check it out....14:48
<tecoripa>fasseg: looks good to me.14:55
<pivotal-bot>Gregory Jansen added comment: "Thanks @cbeer. I've added one and am testing it in fcrepo-webapp. Trying to decide how to manage the depe..." https://www.pivotaltracker.com/story/show/6185048214:59
* github-ff joins15:07
[fcrepo4] escowles created managed-property-tagging (+1 new commit): http://git.io/QMRF6Q
fcrepo4/managed-property-tagging 80f8cf0 Esmé Cowles: Label managed properties in HTML UI (Addresses https://www.pivotaltracker.com/story/show/61500476)
* github-ff leaves
<pivotal-bot>feature: Provide some example SPARQL queries / help text for updating properties in the HTML UI (started) / owner: Esme Cowles
* github-ff joins15:09
[fcrepo4] escowles opened pull request #187: Label managed properties in HTML UI (master...managed-property-tagging) http://git.io/l1R0QQ
* github-ff leaves
<pivotal-bot>Esme Cowles added comment: "I've updated the Velocity template to label system-managed properties: ""
https://github.com/futures/fcrepo4/p..." https://www.pivotaltracker.com/story/show/61500476
* travis-ci joins15:23
[travis-ci] futures/fcrepo4#1342 (managed-property-tagging - 80f8cf0 : Esmé Cowles): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/commit/80f8cf0ce961
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/15002930
* travis-ci leaves
<pivotal-bot>Nigel Banks added comment: "https://wiki.duraspace.org/pages/viewpage.action?pageId=40076555" https://www.pivotaltracker.com/story/show/6183027415:29
Nigel Banks finished "UI glossary documentation" https://www.pivotaltracker.com/story/show/61830274
Nigel Banks started "Single Click Launcher can't be restarted" https://www.pivotaltracker.com/story/show/61824094
Eric James added comment: "Maybe the getFixity() methods in DatastreamService should be disabled for external nodes, as what is the chec..." https://www.pivotaltracker.com/story/show/5600291615:30
* tecoripa leaves15:40
* tecoripa joins15:42
* tecoripa leaves
<pivotal-bot>Mike Durbin started "Default Storage Parameter Directory: tmp to cwd" https://www.pivotaltracker.com/story/show/6100433615:49
<barmintor>ok awoods: I'm at the end of the rope. I can't limit node removal events to fedora:object and fedora:datastream without hacking MODE15:51
MODE v4 will have an event class that should let us do it, but 3.x cannot15:52
so, I can fairly efficiently do NodeAdded for the fedora types, and likewise PropertyXYZ15:53
but NodeRemoved has to be basically a naked event
the other thing I could do is have fcrepo.kernel.NodeService emit the NodeRemoved events15:55
<pivotal-bot>Mike Durbin started "Single system property to specify directory for all fcrepo content" https://www.pivotaltracker.com/story/show/6078849215:56
* nbanks leaves15:58
* nbanks joins
* nbanks leaves15:59
<barmintor>does anyone else have any thoughts about this eventing business? I'd really like to wrap it up this week.16:01
<escowles>is there any downside to having NodeService emit the NodeRemoved events?
<barmintor>escowles: only inasmuch as FCR is integratable into "pure" MODE contexts16:02
personally, I think the special FCR events should all be emitted by NodeService
excepting perhaps the propertyChanged event16:03
<escowles>integrating FCR into pure MODE contexts seems like a pretty hypothetical benefit, so having all the object-linked events emitted by NodeService seems like a good approach to me16:04
<barmintor>I'll email list so that awoods and ajs6f can complain to me
<escowles>barmintor: no way -- just submit the PR at 5pm and then it can be your turn to be sick tomorrow...16:05
<barmintor>:P
<pivotal-bot>Eric James added comment: "LDPath I think. http://localhost:8080/rest/8f/cc/10/7a/8fcc107a-9e99-4de8-bfde-ab6a54a73e19/fcr:transform/de..." https://www.pivotaltracker.com/story/show/6194263816:06
Mike Durbin added comment: "To minimize impact and maximize configurability I propose we make a new optional property "fcrepo.home" that..." https://www.pivotaltracker.com/story/show/6078849216:35
<awoods>barmintor/gregjansen: back17:02
<gregjansen>awoods: so I was asking about fcrepo-authz projects and whether they need to reside within fcrepo now
<awoods>gregjansen: yes
<gregjansen>k, that's the answer I needed17:03
<awoods>sorry... that "yes" was a hello yes
<gregjansen>awoods: thought it was rather fast
<awoods>gregjansen: why exactly do they need to be in fcrepo4?
<gregjansen>fcrepo-authz depends on fcrepo stuff... if fcrepo-webapp is going to depend on fcrepo-authz stuff...
circular dependencies17:04
<awoods>interfaces are nice
how does moving the authz project into fcrepo4 help?17:05
<gregjansen>awoods: well fcrepo-webapp is under fcrepo and will depend upon fcrepo-authz directly if we want to include basic and access roles api. at the same time fcrepo-authz depends upon interfaces in fcrepo/fcrepo-auth-common17:06
awoods: so interfaces don't completely solve this problem, unless I am missing something
<awoods>ok. It is still not exactly clear to me where authz lives mattering... is it a build issue?17:07
<gregjansen>here is the chain fcrepo-webapp => fcrepo-auth-roles-common => fcrepo-auth-common
<awoods>got it
<gregjansen>awoods: first and last are in fcrepo, middle is not
<awoods>got it
<gregjansen>I mean, it will pull it from maven central for now, but it will be hard to build from scratch without skipping fcrepo-webapp17:08
<awoods>gregjansen: It looks like we just need to make sure the snapshot builds of authz are being deployed to sonatype... like many of our other artifacts.17:09
gregjansen: that is easy
gregjansen: would that help? Maybe there is a philosophical question of where we want authz to live, but it does not sound like a technical issue.17:10
<gregjansen>awoods: we can do that. but when we bump the version number, someone is going to have to bootstrap this back into working condition, i.e. partially build fcrepo (omitting fcrepo-webapp), then build authz, then finish fcrepo.17:11
awoods: maybe there is a standard pattern for this. my inclination would be to say that fcrepo-webapp should be separate if it is going to pull in modules considered outside the core.17:12
awoods: (sorry, these are not the things you wanted to consider this late in the release cycle)17:13
<awoods>gregjansen: It is no big deal... I am just pondering if we want to move authz into fcrepo or not.
<gregjansen>gotcha
<awoods>gregjansen: The question I have is, "do we expect users to rely on this authz module, or is it likely to be swapped with another impl"?17:14
<pivotal-bot>Mike Durbin added comment: "https://github.com/futures/fcrepo4/pull/188" https://www.pivotaltracker.com/story/show/61004336
* github-ff joins
[fcrepo4] mikedurbin opened pull request #188: Changed default data directory from temp to curent working directory and made that more easily configurable. (master...working-dir-fix) http://git.io/F9FbEw
* github-ff leaves
<pivotal-bot>Mike Durbin finished "Default Storage Parameter Directory: tmp to cwd" https://www.pivotaltracker.com/story/show/61004336
Mike Durbin added comment: "https://github.com/futures/fcrepo4/pull/188" https://www.pivotaltracker.com/story/show/6078849217:15
Mike Durbin finished "Single system property to specify directory for all fcrepo content" https://www.pivotaltracker.com/story/show/60788492
<gregjansen>awoods: I think the basic impl is too limited for real use right now, useful for demo. roles api is workable tho.
<awoods>mikeAtUVa getting it done
<gregjansen>awoods: roles api needs more features to hit all known use cases, but it works
<awoods>gregjansen: and roles api is in authz?17:16
<gregjansen>it is a sub-project within the parent pom
<mikeAtUVa>awoods: sorry for the flood... two single point tickets with pull requests and comment updates makes me *look* busy;)
<gregjansen>awoods: I could bring it in separately as a single new sub-project within fcrepo
you guys will appreciate this: we at UNC are currently trying to ingest a 100GB file to managed DS in fcrepo 317:17
<awoods>gregjansen: I appreciate that17:18
<pivotal-bot>Mike Durbin added "FedoraHtmlResponsesIT#testVersionCreationAndNavigation fails sometimes." https://www.pivotaltracker.com/story/show/62016220
<awoods>gregjansen: Ideally, we would have some auth interfaces in core, with impls as external projects.17:19
<mikeAtUVa>gregjansen: have you tried ingesting a smaller file then hacking the data store? ;)
<gregjansen>awoods: Is it working? Nobody knows. Is IRODS receiving it? We don't know that either. We've gotten to the point where we check for open file handles..
* ksclarke leaves
<pivotal-bot>Mike Durbin edited "FedoraHtmlResponsesIT#testVersionCreationAndNavigation fails sometimes." https://www.pivotaltracker.com/story/show/6201622017:20
<awoods>gregjansen: how far away are we from that model?
<gregjansen>awoods: yep. basic is an impl, but I thought that you wanted it bundled into fcrepo-webapp. roles api is kind of a separate beast, not really an impl of any interface.
<awoods>gregjansen: yes, I want our impl deployed with webapp.17:21
<gregjansen>awoods: the mysterious 100GB file model? We can aspire. I think in f4 the file would be federated first, linked with the object, then copied to ispn.
<awoods>gregjansen: no, the "auth interfaces in core, with impls as external projects" model
gregjansen: Is not the dependency webapp has on authz a runtime dependency?17:22
<gregjansen>awoods: yes, it is17:23
hmm
<awoods>gregjansen: If the issue is dependencies and releases, would it not work like changing the version of fcrepo (leaving version of authz as before) and then build fcrepo. Then change the version of authz, release it. Then change the version of authz within fcrepo, and release fcrepo.17:26
* kaarefc leaves
<awoods>gregjansen: I assume that one implication of having external projects is that they may have their own version numbers.
<gregjansen>awoods: 2 suggestions.. (1) move access roles api into auth-common, consider it part of fcrepo api. (2) let users add basic jar if they want to configure basic pep
awoods: seems right17:27
<awoods>gregjansen: as a side, yet related note, fcrepo-auth-common has the wrong package name.17:28
<gregjansen>awoods: ok, will fix
<awoods>gregjansen: meaning, currently it is "org/fcrepo/auth", as opposed to "org/fcrepo/auth/common"17:29
<gregjansen>awoods: y, got it
<awoods>gregjansen: anyways, how do you feel about your plan? (1) and (2)?17:30
gregjansen: would we wire basic-pep into the default webapp?
<pivotal-bot>Andrew Woods edited "Fix fcrepo-auth-common package name" https://www.pivotaltracker.com/story/show/6151470617:31
<gregjansen>awoods: not if (2) b/c it would not be there. we could ship a configuration tho
<pivotal-bot>Andrew Woods edited "Fix fcrepo-auth-common package name" https://www.pivotaltracker.com/story/show/61514706
<gregjansen>awoods: thx for tix17:32
I feel like we are constrained in our config by the repository.json file.. Ideally we could separate the PEP into a swappable authz spring context.17:34
<awoods>exactly
that seems to be the issue
<gregjansen>awoods: fwiw, there is probably a way to do it by making something bean container aware..
perhaps our mode auth provider can be container aware and try to load the pep at runtime17:35
not load, get
<awoods>gregjansen: We may want to keep it simple for now.17:36
<gregjansen>I mean, initial requests are all likely to be admin accounts
<awoods>gregjansen: authz-xacml is not used currently, right?
<gregjansen>awoods: since I need to do this tomorrow, yeah
right
I have some code for that in my repo, but nothing that is ready17:37
PEP is an interface tho and implementations might make sense as separate standalone projects.
one day we will be able to ship them with their own configurations17:38
i.e. package spring files in the jar?
<awoods>gregjansen: Where does "access roles api" currently live?
<gregjansen>it is currently outside of fcrepo, in fcrepo-authz project as a sub-project
<awoods>I am not seeing it
which class?17:39
<gregjansen>I will link
<awoods>I have the code in front of me
<gregjansen>https://github.com/futures/fcrepo-authz/tree/master/fcrepo-auth-roles-common
oh, AccessRoles.java
org.fcrepo.auth.roles.common.AccessRoles17:40
<awoods>gregjansen: that is the REST endpoint... it would be good to have that in core
<gregjansen>awoods: snds gd to me
<awoods>gregjansen: but you are saying that BasicRolesPEP can not be wired in unless it is also pulled into core?17:41
<gregjansen>awoods: shall I make myself a ticket for that?
<awoods>we are getting there
<gregjansen>awoods: It can be wired in. However, it will need to be resolved outside of our reactor build, which it (basic) in turn also depends upon.17:42
IOW, the full rebuild of fcrepo will depend upon this one thing being in maven central, which in turn depends upon fcrepo when it is built17:43
(or local repo)
<awoods>right
unless they had different version numbers.
<gregjansen>well, basic will still depend upon fcrepo of some version17:44
and fcrepo-webapp will still depend upon basic of some version
a bit weird, wouldn't you say
awoods: when it become impossible to say who is downstream of who between two git repositories, you are in trouble17:45
<awoods>gregjansen: maybe it is simplest to just pull in the two projects to core: fcrepo-auth-roles-basic and fcrepo-auth-roles-common
<gregjansen>awoods: that'll work
awoods: the parent project for those two doesn't really do anything17:46
awoods: fcrepo-authz
<awoods>gregjansen: I would like to find the right line where auth impls are external, but modeshape is making that hard for us.17:47
<gregjansen>I think we can probably fix that with a bit more spring magic, in next release.
<awoods>gregjansen: Should the classes in fcrepo-auth-roles-common roll into fcrepo-auth-common?17:48
or stay in their own package?
<gregjansen>I like them in a package
<awoods>ok17:49
<gregjansen>but will admit that it is a bit different from other, i.e. a bunch of diff classes to support 1 function
<awoods>?
gregjansen: so we would end up with four auth packages? auth-roles-common, auth-roles-basic, auth-common, and auth-oauth?17:50
<gregjansen>I guess so
we could put them in a sub-parent project17:51
but f3 went down the rabbit hole that way
<awoods>gregjansen: I am happy leaving them at the top-level for now.
<gregjansen>I would be fine with consolidating access roles and auth common, but it will take more time to do that17:52
mostly b/c of the tests I think
<awoods>no need now
gregjansen: to be clear, fcrepo-authz-xacml will be left in the cold for now.17:53
<gregjansen>I think that is appropriate
so cold
<awoods>so lonely17:54
<gregjansen>yes, how fitting for xacml
<awoods>gregjansen: How do you feel about the auth-common package path refactoring?
<gregjansen>awoods: ambivalent, happy to clean it up17:55
<awoods>gregjansen: Can you do it swiftly and cleanly? Otherwise, we may want to push it out.
<gregjansen>we have our tests. I should be able to do it tomorrow.
<awoods>gregjansen: If you are going to do the package repathing, maybe commit that ticket first.17:56
<gregjansen>(1) move two projects into fcrepo (2) rename package (3) configure fcrepo-webapp with accessroles API (4) ITs (5) PR
you want 2 PRs?17:57
<awoods>The ITs are for the new default webapp functionality?17:58
<gregjansen>awoods: just running existing ones is all
auth-roles-basic has ITs
<awoods>I would say 3 PRs: (1), (2), and (3)
<gregjansen>awoods: no problem17:59
<awoods>gregjansen: can you create the tickets?
<gregjansen>yep
awoods: thx, sorry for keeping you late
<awoods>gregjansen: It is only 4pm18:00
<pivotal-bot>Gregory Jansen started "Expose AuthZ Service/API at Root of F4 webapp" https://www.pivotaltracker.com/story/show/61850482
<gregjansen>awoods: no, it is 6pm Andrew
<awoods>gregjansen: time18:01
barmintor: Where are you landing the eventing issue?18:03
<barmintor>awoods: did you see my messages back there?
<awoods>barmintor: I did...
<pivotal-bot>Gregory Jansen added "rename org.fcrepo.auth to match auth-common" https://www.pivotaltracker.com/story/show/6201948618:04
Gregory Jansen started "rename org.fcrepo.auth to match auth-common" https://www.pivotaltracker.com/story/show/62019486
<awoods>barmintor: The issue is the NodeRemoved event?
<barmintor>ok, I'm running a separate eventlistener that propagates removals "naked"- and they'll be ignored just like they were before
<pivotal-bot>Gregory Jansen edited "rename org.fcrepo.auth to match auth-common" https://www.pivotaltracker.com/story/show/62019486
<barmintor>awoods: yes
I'm running the profiler now
<awoods>and the suggestion that NodeService could potentially be the emitter of our events, to help with the NodeRemoved issue?18:05
<barmintor>awoods: yes18:06
though I would like to argue with ajs6f before I go down that route
<pivotal-bot>Gregory Jansen added "move fcrepo-auth-roles-common into fcrepo" https://www.pivotaltracker.com/story/show/62019586
<awoods>barmintor: obviously, leveraging Mode's/JCR's machinery for events is nice.
<barmintor>awoods: It is, but we want to do something MODE/JCR does not support18:07
<pivotal-bot>Gregory Jansen deleted "rename org.fcrepo.auth to match auth-common" https://www.pivotaltracker.com/story/show/62019486
Gregory Jansen started "move fcrepo-auth-roles-common into fcrepo" https://www.pivotaltracker.com/story/show/62019586
Gregory Jansen started "Fix fcrepo-auth-common package name" https://www.pivotaltracker.com/story/show/61514706
<awoods>barmintor: what would be the "naked NodeRemoved event"? that would be an event from NodeService?18:08
<barmintor>awoods: I f I move the NodeRemoved events to NodeService, I can emit them correctly typed18:09
<pivotal-bot>Gregory Jansen added "move fcrepo-auth-roles-basic into fcrepo" https://www.pivotaltracker.com/story/show/62019734
<awoods>barmintor: Is that what you were referring to as "naked events"?
barmintor: It sounds like the two options are: hack Modeshape or emit from NodeService.18:10
...or wait for Modeshape 4.0?
<pivotal-bot>Gregory Jansen added "Add access roles API to fcrepo-webapp" https://www.pivotaltracker.com/story/show/6201990218:12
<barmintor>awoods: you've got it18:14
<awoods>barmintor: We do not want to wait for Mode4.0 to support delete events... so maybe the solution is to have an impl now that has NodeService only emit the NodeRemoved events, but make the code clearly commented with supporting pivotal tickets to change it out when Mode4.0 arrives. But I agree, it would be good to get ajs6f's opinion.18:16
<barmintor>ok18:18
* osmandin leaves18:19
<cbeer>barmintor: +1 to emitting our own events.18:34
<barmintor>cbeer: noted on the way to the train
* ermadmix leaves18:41
<pivotal-bot>Andrew Woods rejected "Write a velocity template for the fcr:versions endpoint." with comments: "It looks like the new IT is failing: ht..." https://www.pivotaltracker.com/story/show/6147423618:52
Andrew Woods added comment: "It looks like the new IT is failing:18:53
http://ci.fcrepo.org/jenkins/view/FF/job/fcrepo4/1493/ , and
http://c..." https://www.pivotaltracker.com/story/show/61474236
Andrew Woods added comment: "Looks good, @nigelgbanks. Could you please alphabetize the terms?" https://www.pivotaltracker.com/story/show/6183027419:15
Andrew Woods rejected "UI glossary documentation" https://www.pivotaltracker.com/story/show/61830274
Andrew Woods added comment: "Is there an IT that would be more reliable going directly against the REST API?" https://www.pivotaltracker.com/story/show/6147423619:16
Andrew Woods added comment: "Sounds like a good approach." https://www.pivotaltracker.com/story/show/6078849219:33
Andrew Woods accepted "Single system property to specify directory for all fcrepo content" https://www.pivotaltracker.com/story/show/6078849219:34
Andrew Woods added comment: "Looks good, @nigelgbanks. Could you please alphabetize the terms? then put the ticket back in the "Finish"e..." https://www.pivotaltracker.com/story/show/6183027419:36
* fasseg leaves21:17
* ksclarke joins21:22
* fasseg joins21:24
<pivotal-bot>Andrew Woods delivered "Default Storage Parameter Directory: tmp to cwd" https://www.pivotaltracker.com/story/show/6100433623:28
* github-ff joins
[fcrepo4] awoods closed pull request #188: Changed default data directory from temp to curent working directory and made that more easily configurable. (master...working-dir-fix) http://git.io/F9FbEw
* github-ff leaves
<pivotal-bot>Andrew Woods added "Remove junk temp files that leak into the container directory - again" https://www.pivotaltracker.com/story/show/6203175023:30
Andrew Woods started "Remove junk temp files that leak into the container directory - again" https://www.pivotaltracker.com/story/show/62031750
* github-ff joins23:34
[fcrepo4] awoods pushed 1 new commit to master: http://git.io/sfnQyg
fcrepo4/master 0f3c594 Andrew Woods: Remove temp files that leak into container directory - again...
* github-ff leaves
<pivotal-bot>Andrew Woods added comment: "Resolved with: https://github.com/futures/fcrepo4/commit/0f3c594758ecb8391eeca20b52bfa99162e9858b" https://www.pivotaltracker.com/story/show/62031750
Andrew Woods accepted "Remove junk temp files that leak into the container directory - again" https://www.pivotaltracker.com/story/show/62031750
* travis-ci joins23:42
[travis-ci] futures/fcrepo4#1345 (master - 125e2b8 : Andrew Woods): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/bf1d4b111fc4...125e2b8fdc67
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/15022471
* travis-ci leaves
* travis-ci joins23:46
[travis-ci] futures/fcrepo4#1346 (master - 0f3c594 : Andrew Woods): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/125e2b8fdc67...0f3c594758ec
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/15022572
* travis-ci leaves
<pivotal-bot>Andrew Woods started "Fix kitchen-sink build" https://www.pivotaltracker.com/story/show/6187462423:58
* github-ff joins23:59
[fcrepo-kitchen-sink] awoods pushed 1 new commit to master: http://git.io/WZEVNQ
fcrepo-kitchen-sink/master 8bcd421 Andrew Woods: Fix kitchen-sink build...
* github-ff leaves
<pivotal-bot>Andrew Woods added comment: "Resolved with: https://github.com/futures/fcrepo-kitchen-sink/commit/8bcd4213e04a3aba8da2fe7f00bcdc8ccbd70454" https://www.pivotaltracker.com/story/show/6187462400:00
Andrew Woods delivered "Fix kitchen-sink build" https://www.pivotaltracker.com/story/show/61874624
<bljenkins>Yippie, build fixed!00:12
Project fcrepo-kitchen-sink build #697: FIXED in 12 min: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/697/
* ksclarke leaves00:14

Generated by Sualtam