Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* github-ff joins03:38
[fcrepo4] eddies pushed 2 new commits to master: http://git.io/8K86hg
fcrepo4/master 222670d Edwin Shin: Removes repeated httpclient dependency
fcrepo4/master d9feaaf Edwin Shin: Removes obsolete resteasy version property and htmleasy repository
* github-ff leaves
* github-ff joins03:52
[fcrepo4] eddies pushed 2 new commits to master: http://git.io/rLsJOw
fcrepo4/master dde5799 Edwin Shin: minor: adds type information to nodeIterator
fcrepo4/master 4de76cc Edwin Shin: Cleans up unused imports
* github-ff leaves
<bljenkins>Yippie, build fixed!03:55
Project fcrepo4 build #730: FIXED in 17 min: http://ci.projectblacklight.org/jenkins/job/fcrepo4/730/
* Edwin Shin: Removes repeated httpclient dependency
* Edwin Shin: Removes obsolete resteasy version property and htmleasy repository
* travis-ci joins03:56
[travis-ci] futures/fcrepo4#621 (master - d9feaaf : Edwin Shin): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/abcd5ef7d195...d9feaaf7bc0d
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/7656370
* travis-ci leaves
<bljenkins>Project fcrepo-fixity-corrupter build #42: STILL FAILING in 55 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-fixity-corrupter/42/03:57
* github-ff joins04:06
[fcrepo4] eddies pushed 1 new commit to master: http://git.io/Lfvffg
fcrepo4/master de17c08 Edwin Shin: minor: README updates
* github-ff leaves
* travis-ci joins04:10
[travis-ci] futures/fcrepo4#622 (master - 4de76cc : Edwin Shin): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/d9feaaf7bc0d...4de76cc71b02
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/7656566
* travis-ci leaves
<bljenkins>Project fcrepo-fixity-corrupter build #43: STILL FAILING in 1 min 5 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-fixity-corrupter/43/04:20
* travis-ci joins04:26
[travis-ci] futures/fcrepo4#623 (master - de17c08 : Edwin Shin): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/4de76cc71b02...de17c08e93d4
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/7656721
* travis-ci leaves
* kaarefc joins04:33
<bljenkins>Project fcrepo-fixity-corrupter build #44: STILL FAILING in 36 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-fixity-corrupter/44/04:34
* fasseg joins04:46
Oh NICE!!!! you can append ?w=1 to a git page and it will ignore whitespaces in diffs!!06:03
<pivotal-bot>Frank Asseg started "fcrepo4 stress test" https://www.pivotaltracker.com/story/show/5068254707:07
Frank Asseg added comment: "I updated the pul request to the modeshape formatting for easier review by rhauch" https://www.pivotaltracker.com/story/show/48804727
Frank Asseg added "Fixity webapp cleanup" https://www.pivotaltracker.com/story/show/5087913907:10
<fasseg>eddies: can I make you the requester on https://www.pivotaltracker.com/story/show/50879139 "Fixity webapp cleanup"?07:11
<pivotal-bot>feature: Fixity webapp cleanup (unscheduled) / owner:
<eddies>fasseg: sure
<pivotal-bot>Frank Asseg edited "Fixity webapp cleanup" https://www.pivotaltracker.com/story/show/50879139
<bljenkins>Yippie, build fixed!07:40
Project fcrepo-legacy-api build #22: FIXED in 3 min 19 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-legacy-api/22/
Edwin Shin: Fixes compile error resulting from the following update in fcrepo-kernel's RepositoryService:
Project fcrepo-kitchen-sink build #336: SUCCESS in 4 min 11 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-kitchen-sink/336/07:44
* github-ff joins07:52
[fcrepo-bagit-object-serialization] eddies pushed 1 new commit to master: http://git.io/ZVCj9A
fcrepo-bagit-object-serialization/master 337f731 Edwin Shin: Cleanup: adds explicit exclusions in pom, removes unused imports
* github-ff leaves
<bljenkins>Project fcrepo-bagit-object-serialization build #12: SUCCESS in 1 min 58 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-bagit-object-serialization/12/07:54
Project fcrepo-kitchen-sink build #337: SUCCESS in 3 min 30 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-kitchen-sink/337/08:00
* github-ff joins08:25
[fcrepo-bagit-modeshape-federation-connector] eddies pushed 1 new commit to master: http://git.io/yPabKw
fcrepo-bagit-modeshape-federation-connector/master e31b4c2 Edwin Shin: Applies misc. cleanup as part of a vain effort to get the ITs to pass.
* github-ff leaves
* github-ff joins08:28
[fcrepo-bagit-object-serialization] eddies pushed 1 new commit to master: http://git.io/KHqyew
fcrepo-bagit-object-serialization/master 353a78d Edwin Shin: trivial formatting changes
* github-ff leaves
<bljenkins>Project fcrepo-bagit-modeshape-federation-connector build #17: STILL UNSTABLE in 2 min 52 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-bagit-modeshape-federation-connector/17/
Project fcrepo-bagit-object-serialization build #13: SUCCESS in 1 min 38 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-bagit-object-serialization/13/08:29
Project fcrepo-kitchen-sink build #338: SUCCESS in 3 min 48 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-kitchen-sink/338/08:33
Project fcrepo-sequencer-archives build #9: SUCCESS in 1 min 45 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-sequencer-archives/9/08:52
* kaarefc leaves08:57
<bljenkins>Project fcrepo-kitchen-sink build #339: SUCCESS in 7 min 11 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-kitchen-sink/339/09:00
* gregjansen joins
* ajs6f joins09:09
* github-ff joins10:07
[fcrepo-kitchen-sink] eddies pushed 1 new commit to master: http://git.io/R_k_sg
fcrepo-kitchen-sink/master 08c186e Edwin Shin: Changes the default graphite host
* github-ff leaves
* awoods joins10:20
Here are some results of the latest jmeter runs... looks like we may have a bottleneck: https://wiki.duraspace.org/display/FF/Fedora+4+Sample+Results10:22
<bljenkins>Project fcrepo-kitchen-sink build #340: SUCCESS in 15 min: http://ci.projectblacklight.org/jenkins/job/fcrepo-kitchen-sink/340/10:23
<pivotal-bot>Andrew Woods delivered "Update JMeter test plan to work against current REST API" https://www.pivotaltracker.com/story/show/5000812710:28
* barmintor joins10:33
<ajs6f>awoods: Some really slow results in there...10:47
<awoods>ajs6f: particular as you dial up the thread count10:48
<ajs6f>awoods: we're only trying one and five threads at a time? maybe we should try ten or fifteen just to see what the pattern is…10:49
<cbeer>awoods: what was this run against?10:50
<awoods>cbeer: my local install
The main variables are: thread count, datastream size, loops
<cbeer>of -webapp? or kitchen sink?
in tomcat10:51
<gregjansen>have any of you seen exception on repository creation?
java.lang.ClassCastException: org.modeshape.jcr.JcrSystemNode cannot be cast to org.modeshape.jcr.JcrVersionNode
<awoods>although, jetty seemed to produce similar results
gregjansen: no, but I tend to wipe my datastore clean when restarting the repo... particularly if I have updated the codebase between runs.10:52
<ajs6f>all: my headset is kerplunk, so can some dial me in again for standup? 434-243-864410:56
<barmintor>https://plus.google.com/hangouts/_/46ed3fe3f2d8606cbc4c4056cc8bb4cce9176ebc?authuser=0&hl=en ?10:58
I'm all alone?
in a parallel Google?10:59
in +, no one can hear me scream.
<eddies>cbeer: you coming?11:01
fasseg too
<cbeer>oh, and awoods, what was your logging levels set at?11:04
* github-ff joins
[fcrepo4] ajs6f pushed 1 new commit to SpringOAuth: http://git.io/93XTwg
fcrepo4/SpringOAuth 6fec6fe ajs6f: Pausing to wait on Spring advice
* github-ff leaves
<awoods>cbeer: DEBUG11:08
<cbeer>awoods: might want to make that info for performance tests too
we're not exactly light loggers.
<awoods>good point
<ajs6f>Is there a simple way to get a collection or iterator of Datastreams from a FedoraObject, without going through the *Services and fooling with String paths?11:10
<fasseg>ajsf6: https://github.com/ModeShape/modeshape/pull/83911:13
<awoods>barmintor: Do you need to "start" your ticket? https://www.pivotaltracker.com/s/projects/684825/stories/4773057511:14
<barmintor>awoods: Err… probably
<pivotal-bot>Benjamin Armintor started "Every node should not fire EventBus events, etc, etc when a change is made on a different node." https://www.pivotaltracker.com/story/show/47730575
<ajs6f>fasseg: Looks to me like we're waiting on rhauch: "I'll figure out how we can best implement that logic"...11:23
<fasseg>yeah but i still dont understand the issue :/
<bljenkins>Project fcrepo-fixity-corrupter build #45: STILL FAILING in 41 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-fixity-corrupter/45/11:24
<ajs6f>It's the diff between external paths (e.g. paths in a filesystem) and internal paths (i.e. in JCR). Events normally feature an internal path, but we're emitting external paths. The question (that rhauch intends to answer for us) is how to translate.
They haven't had to do that before now because connectors haven't emitted events.11:25
<pivotal-bot>Chris Beer added "Use good predicates and document them in futures/ontology" https://www.pivotaltracker.com/story/show/5089596911:26
<cbeer>awoods: while we were on the call, i was playing with a performance integration test11:39
1000x create, read, delete
if i'm reading this right, we're spending a lot of time in locks.11:40
<pivotal-bot>Andrew Woods added "Setup stand-alone client and repo for JMeter on futures2 and 3" https://www.pivotaltracker.com/story/show/50897055
<cbeer>awoods: 1 and 2, no?11:41
1 has junit already, at least
<awoods>cbeer: you are proposing futures1 for the jmeter client?
<cbeer>that's what we've done before
<eddies>cbeer, awoods: one of the things to note in any benchmark tests (and i think i remember nbanks taking this into consideration when he re-factored the jmeter test plan) is that MODE only stores a single copy of a binary (as determined by its checksum). so you in any synthetic tests, you want to make sure you're using unique objects
awoods: yes, futures1 for the client. again, see: https://wiki.duraspace.org/display/FF/Test+Platform11:42
s/unique objects/unique binary content/
<awoods>eddies: yes the test generates unique datastreams for each request.
<eddies>barmintor: free for a call now?11:44
<cbeer>awoods: futures1:/opt/ff-jmeter-madness11:46
awoods: /etc/default/tomcat711:48
awoods: http://ci.fcrepo.org/jenkins/view/FF/job/jmeter-testing-deploy/11:50
* gregjansen leaves11:52
todo: findOrCreateNode(Session session, String path, String defaultNodeType, String finalNodeType)12:06
* ff-github joins12:12
[fcrepo-fixity] fasseg created cleanup (+1 new commit): http://git.io/rCPWtA
fcrepo-fixity/cleanup 708c156 fasseg: added test container for faster integration tests
* ff-github leaves
<bljenkins>Project fcrepo-fixity build #208: STILL FAILING in 2 min 8 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-fixity/208/12:14
* github-ff joins12:27
[fcrepo4] cbeer pushed 2 new commits to master: http://git.io/DwckPg
fcrepo4/master ed2dbc9 Chris Beer: add jcr folder as the default type for creating nodes
fcrepo4/master 4aa2394 Chris Beer: add better logging to the graph streaming output
* github-ff leaves
<cbeer>hm http://docs.jboss.org/modeshape/2.8.2.Final/api-full/org/modeshape/jcr/JcrTools.html#getReadable(javax.jcr.Node)12:40
<bljenkins>Project fcrepo-fixity-corrupter build #46: STILL FAILING in 37 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-fixity-corrupter/46/12:41
<cbeer>awoods: and in a great display of multitasking, ^^^ should fix the constraint problems for creating random datastreams12:44
<awoods>cbeer: I will give it a spin12:47
<cbeer>barmintor: supposedly i have a ride to NEFUG, so I'll ask you some tough questions about FF and why we didn't embark on a year-long planning process instead.
<barmintor>cbeer: and an admin client12:48
<cbeer>well, right. it'd be fine if we kept an admin client.
maybe enunciate will help with that.
* travis-ci joins
[travis-ci] futures/fcrepo4#624 (master - 4aa2394 : Chris Beer): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/de17c08e93d4...4aa239483f21
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/7669617
* travis-ci leaves
<awoods>cbeer: I still see an error when making this request: curl -X POSdatastream.txt http://localhost:8080/rest/objects/test_20/datastreams/DS_1/fcr:content12:52
assuming /test_20/datastreams does not already exist12:53
<cbeer>guess that's test-worthy then.
i'll take a look in a sec12:54
<awoods>cbeer: this the correct call: curl -X POST --upload-file datastream.txt http://localhost:8080/rest/objects/test_20/datastreams/DS_1/fcr:content
<cbeer>ok, and i reran the yourkit thing i was doing and changed the cpu tracing stuff and got more realistic results this time
<awoods>anything interesting?12:55
<cbeer>not surprisingly, a lot of upstream stuff in MODE and ISPN12:56
<barmintor>cbeer: you know what they say about patches...12:58
<cbeer>test em?
hm. i thought you could upload images to gists13:00
i'd say nothing too surprising
i wonder if TX would help some of that, actually.13:03
we do spend a lot of time writing indexes and stuff back to ISPN
awoods: hm. i just wrote a passing test to do what i thought you were doing.13:10
i guess that's why you write the failing test before writing code.13:11
<cbeer>ok awoods. what error ARE you seeing?
<awoods>It is a shame that the index update is a synchronous part of the content creation flow.13:15
cbeer: WARN 12:53:32.998 (RepositoryExceptionMapper) Caught repository exception: {}
javax.jcr.nodetype.ConstraintViolationException: The values for the '{http://www.jcp.org/jcr/1.0}content' property on node '/objects/test_20/datastreams' no longer satisfy the type and/or constraints on the 'jcr:content' property definition on the 'nt:file' node type definition
<cbeer>yeah, that's the thing the patch fixes.
are you sure you have my commit? i see github is being flaky again13:16
<awoods>I did a pull, and tested... hmm
cbeer: executing the curl request above works for you?13:17
<cbeer>i didn't try curl. i wrote a test that i'm trying to push13:18
<awoods>cbeer: I just double-checked, by baseline is up-to-date.
* ff-github joins13:20
[fcrepo-fixity] fasseg pushed 1 new commit to cleanup: http://git.io/Su8kpw
fcrepo-fixity/cleanup 6f2ee6c fasseg: added jms queue handling
* ff-github leaves
<ajs6f> awoods: A lot of what goes on in indexing is config-able. Some stuff, like the creation of indexes to support path traversal, are not, but we can get in there and tweak, and it might make a big diff. I really don't know much about what the defaults are, but I think they include full-text, for example.
<bljenkins>Project fcrepo-fixity build #209: STILL FAILING in 42 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-fixity/209/13:21
<awoods>ajs6f: sounds like we have some dials to play with, which is good news.13:22
* github-ff joins13:26
[fcrepo4] cbeer pushed 1 new commit to master: http://git.io/DnAhsQ
fcrepo4/master 15f552e Chris Beer: add a test for adding a datastream somewhere in the jcr tree
* github-ff leaves
<ajs6f>awoods: There are even buttons on the dials, and there are switches on the buttons, but we only know what the dials do right now. And we're still building the gauges.13:28
Now _that's_ how you run a metaphor into the ground!
* bshambaugh joins13:29
<cbeer>awoods: that curl works for me.
<bljenkins>Project fcrepo-fixity-corrupter build #47: STILL FAILING in 28 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-fixity-corrupter/47/13:37
* travis-ci joins13:46
[travis-ci] futures/fcrepo4#625 (master - 15f552e : Chris Beer): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/4aa239483f21...15f552ee2dd0
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/7671275
* travis-ci leaves
* gregjansen joins13:54
eddies: I think these projects may be missing fedora-node-types.cnd. Unless they are coming in from somewhere else..13:55
<eddies>cbeer: what's the patch you referred to earlier? ("that's the thing the patch fixes.")
<cbeer>eddies: " add jcr folder as the default type for creating nodes"13:56
<eddies>cbeer: are you talking about a recent commit to fcrepo4 or something in MODE?13:57
<cbeer>eddies: what?
<eddies>gregjansen: i thought that was coming in from one of the fcrepo4 maven dependencies13:58
<gregjansen>eddies: okay, I guess it's in the classpath then.
<eddies>cbeer: i didn't follow what "the patch" that deals w/ index updates being synchronous
<cbeer>eddies: i'm lost. what about index updates?13:59
<eddies>cbeer: awoods said it's a shame that index updates are synchronous, then you said that's what the patch deals with
but maybe you were just responding to an earlier comment?
<cbeer>no. "awoods: javax.jcr.nodetype.ConstraintViolationException: The values for the '{http://www.jcp.org/jcr/1.0}content' property on node '/objects/test_20/datastreams' no longer satisfy the type and/or constraints on the 'jcr:content' property definition on the 'nt:file' node type definition"
<eddies>ah =)14:00
all clearer
<cbeer>ajs6f claims to know something about indexing
<eddies>gregjansen: worth double-checking though
stick a copy of the cnd in src/main/resources
<cbeer>(but please don't.)14:01
<gregjansen>eddies: yeah, at least in my project the fcrepo-kernel is a transitive dependency via the bagit federation
<eddies>cbeer: why not, just as a sanity check
<gregjansen>cbeer: I will remove it again..14:02
<cbeer>gregjansen: why do you need the CND at all?
<ajs6f>no, I claim specifically only to know that I know very little about indexing.
but I do know that there's more we could learn.
<cbeer>ajs6f: which is a lot more than the rest of us
<eddies>it should be coming in via maven dependency. but at this point, i'm at a loss as to what's causing those test failures
<gregjansen>cbeer: I really don't maybe b/c I run our repo factory, but I could use the jcr factory instead and just remove all fcrepo dependencies.. sound right?14:03
<ajs6f>are we going to seriously try to tune indexing for speed?
<cbeer>wait, eddies, gregjansen: are we talking about the identity sequencer, or something else?
<gregjansen>cbeer: ID sequencer and the bagit federation module have the same exception14:04
<eddies>the ITs are failing
<gregjansen>cbeer: it thinks a node is versionable, but then it isn't
cbeer: org.modeshape.jcr.JcrSystemNode cannot be cast to org.modeshape.jcr.JcrVersionNode
<awoods>ajs6f: if the indexing is making ingest unbearably slow, do we have a choice?
<cbeer>is indexing making it slow, really?14:05
(and, are we actually calling it slow?)
<ajs6f>I missed the evidence on that: it's indexing that's kicking us in the crotch on indexing?
on ingest, I mean.
<barmintor>indexing is definitely the chokepoint on indexing.
<gregjansen>eddies: I think I will make this sequencer pure modeshape code.. it'll copy whatever the federated nodes show, which might be fedora types, but it doesn't need to know that..14:06
<cbeer>gregjansen: i think that's smart.
i'll take a look at the bagit sequencer error
i don't know why it'd care about versioning
<eddies>gregjansen: that sounds good. and cbeer, another set of eyes on that project would be much appreciated14:07
<ajs6f>Then we got to choke indexing, I guess. It's config-able in the JSON for the JCR repo, and ought to be minimize-able..
We could even turn off persistent indexes by default.
<cbeer>ajs6f: i think you should read barmintor's sentence again.
<ajs6f>Oh. Right.14:08
<awoods>ajs6f, cbeer, barmintor: That is the first task: determine what is kicking us in the crotch... but the gist cbeer posted hinted towards indexing.
* barmintor is a little punchy
<cbeer>awoods: did it?
cbeer: in the first two charts, the top lines are lucene.14:09
<ajs6f>What do the columns in those charts mean?
<ajs6f>Ah, details. Yes.
Perhaps I should ask: what do those charts mean.14:10
<eddies>11% to lucene
<ajs6f>11% of execution time per request?
<eddies>11% of anything is a kick in the crotch? =)
<ajs6f>Taking 11% off the current awful times is not going to be a magic bullet.
<awoods>I read it as 11% of the crotch-kicks come from lucene.14:11
<cbeer>eddies: i can't even get fcrepo-bagit-modeshape-federation-connector to build.
cannot find symbol
[ERROR] symbol: class ConnectorChangedSet
<ajs6f>So our crotch quotient is too high, and we're wanting to lower Lucene's contribution.
<awoods>cbeer may have more details... since the screen shots came from him.
<eddies>actually, another 5% is LuceneBackendQueueTask14:12
<ajs6f>cbeer: How hard was it to generate this charts?
<eddies>cbeer: that sounds like part of MODE-1710
<ajs6f>Can we try again with indexing minimized?
eddies:cbeer: Yes, it does.14:13
<cbeer>ajs6f: i just ran that attached junit test with yourkit profiling on.
<ajs6f>That's a type from 1710.
Okay, so if I write an index-light config, we can run it pretty easy, eh?
<eddies>barmintor, cbeer: are either of you actually away next week for hydra-partners?14:14
<cbeer>eddies: yes, i am. (which is why i put it on the calendar and the wiki.)
<cbeer>but i'm going to make jcoyne help me get hydra on fcrepo4 working14:15
that would be good.
<cbeer>eddies: what does the bagit federation thing have a dependency on the http api?14:16
<eddies>i dunno. i didn't actually go through the project's dependencies14:17
<cbeer>(still trying to figure out what happened to the ConnectorChangedSet)
<eddies>cbeer: fyi, i just successfully ran mvn clean compile -DskipTests14:20
i wonder if it's not actually using the fcrepo-snapshot
maybe it's using my local build.14:21
<eddies>ah, do you have a more recent build of mode from the 1710 branch?
<gregjansen>1710 seems to be building 3.3-SNAPSHOT
of mode
<cbeer>do i just kill the directory in .m2?
<eddies>yeah, the artifact is the same
oh. wait, you mean the specific mode directory. yes that's ok14:22
don't delete all of ~/.m2 =)
<awoods>cbeer: datastream curl request does indeed work. Thanks.
ok: Caused by: javax.jcr.RepositoryException: java.lang.ClassCastException: org.modeshape.jcr.JcrSystemNode cannot be cast to org.modeshape.jcr.JcrVersionNode14:23
that's where i should be, right?
<barmintor>eddies: I won't be at Hy. partners14:24
<cbeer>yeah. i don't see why this needs an fcrepo-http-api dependency anyway.
<eddies>kill it with fire
<cbeer>is there any reason we need to pre-create /objects and /federation anyway?
<gregjansen>cbeer:eddies: I got past exception by removing our factory from the loop.. Is it pulling an old cnd?
<barmintor>cbeer: probably not federation; arguably not objects if we put in some logic to create intermediate folders on POST14:26
<cbeer>yep. ok. we'll see what the tests think.
<eddies>gregjansen: what factory from what loop?
<gregjansen>eddies: our modeshape repository factory in fcrepo-kernel..14:27
<barmintor>cbeer: I would honestly love to see those nodes not be bootstrapped, and have a more straightforward startup
ok, and i don't see any reason for this bagit http test.. maybe it belongs in some webapp sanity check14:28
but probably not here
<gregjansen>seems like /objects could be created in a migration scenario
<cbeer>gregjansen: you said you could swap our repo factory with something from MODE?14:30
<gregjansen>cbeer: org.modeshape.jcr.JcrRepositoryFactory
cbeer: I did a factory-method to create repo bean14:31
<ajs6f>I'm no fan of preemptive node creation (i.e. objects/ and so forth) but making POST create intermediating resources seems a little heavy. Maybe a switch that can be thrown like mkdir's "-p" switch?
* github-ff joins
[fcrepo4] cbeer pushed 1 new commit to master: http://git.io/g86PHg
fcrepo4/master b4a418c Chris Beer: don't auto-create nodes when starting the repository.
* github-ff leaves
<cbeer>ajs6f: that's behavior we inherit from jcrtools14:32
<ajs6f>What is behavior from JcrTools?
<cbeer>JcrTools.findOrCreateNode /some/path/will/be/created/for/me
we just never told the repository what primary type the intermediate nodes should be14:33
<ajs6f>Oh, yeah, sure, no prob. It's connecting that behavior to POST that makes me feel a little weird.
To any HTTP verb, that is.
o we do that now?
<cbeer>yes. and always have.
gregjansen: can you gist that bit? i'm feeling lazy14:34
<ajs6f>Ooh. Well, if no one else cares, I won't make a stink about it.
<gregjansen>cbeer: yah
<cbeer>(of course, i'm now wondering why we need spring mixed in here..)
<ajs6f>I guess I'm thinking of someone creating an object in a known collection who POSTs:
<gregjansen>cbeers: https://gist.github.com/gregjan/568698714:35
<ajs6f>to a collection that doesn't exist (like a typo kind of error) and successfully creates an object in the wrong place and then has to go and find it… you get the kind of annoyance I mean? Not a big deal. Just can be annoying.14:36
<gregjansen>are the federated bags fedora types yet?
<ajs6f>Personally, I'd rather get a 404.
<awoods>ajs6f: I could go either way on the 404 issue here. Part of it depends on/how/the/path/of/the/object is perceived. If the path is just an abstract location, the fact that it contains '/' is arbitrary.14:41
<cbeer>if the tests pass, ship it, right?14:43
* github-ff joins14:44
[fcrepo-bagit-modeshape-federation-connector] cbeer pushed 1 new commit to master: http://git.io/HiVN1w
fcrepo-bagit-modeshape-federation-connector/master eb565ab Chris Beer: streamline tests by removing fcrepo4 dependencies, spring, etc and just use modeshape gear
* github-ff leaves
<bljenkins>Project fcrepo-fixity-corrupter build #48: STILL FAILING in 28 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-fixity-corrupter/48/14:45
<awoods>eddies: On futures2: WARN 19:45:22.307 (GraphiteReporter) Unable to report to Graphite14:46
java.net.ConnectException: Connection refused
<gregjansen>gregjansen: no, the federated bags are not typed as fedora objects
<ajs6f>awoods: True. We're intentionally conflating paths and identifiers. I dimly suspect we may need to un-conflate them, sooner or later.
<eddies>cbeer: i'm getting weird behavior trying to rebase modeshape/fcrepo-snapshot against master (i already pulled upstream on master)14:47
<bljenkins>Yippie, build fixed!
Project fcrepo-bagit-modeshape-federation-connector build #18: FIXED in 2 min 22 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo-bagit-modeshape-federation-connector/18/
<eddies>would you mind trying a rebase of fcrepo-snapshot against master for me?
awoods: what ip are you using?14:48
i think you need to use an internal address
<cbeer>eddies: ok. you want the upstream master with 1710 and 1908 applied, right?
<awoods>I have not set anything. The message I posted just showed up in the tomcat log on futures2.
eddies ^^^
<eddies>well, what i really want is fcrepo-snapshot to have the latest commits from rhauch on the master14:49
but yeah
* bshambaugh leaves14:50
<eddies>awoods: you're deploying kitchen-sink on futures2?14:51
* travis-ci joins14:52
[travis-ci] futures/fcrepo4#626 (master - b4a418c : Chris Beer): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/15f552ee2dd0...b4a418cf5b21
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/7673259
* travis-ci leaves
<awoods>eddies: yes
<eddies>awoods: i guess you must be if you're see that
so kitchen-sink defaults to localhost as the graphite server
<awoods>futures1=jmeter client, futures2=load-server
<eddies>you need to set -Dfcrepo.metrics.host=<ip for futures6>14:53
<awoods>eddies: thanks, will do
<eddies>who fixed fcrepo-bagit-modeshape-federation-connector?14:54
<cbeer>i did.
<gregjansen>cbeer: did you switch repo factories?14:55
<cbeer>and i've pushed a new snapshot
gregjansen: yes. there's no fcrepo4 dependencies in there at all
<gregjansen>cbeer: so I noticed
<cbeer>eddies: for future reference (i leave it to you to stick in some appropriate place if you think it's worth it..)
$ git checkout fcrepo-snapshot14:56
$ git reset --hard origin/master
$ git merge --squash futures/mode-1710; git commit -m "squash commit of mode-1710"
$ git merge --squash futures/mode-1908; git commit -m "squash commit of mode-1908"
git push futures fcrepo-snapshot -f
for me, origin is modeshape/modeshape
and futures is futures/modeshape
oh, and git fetch origin; git fetch futures first..
<eddies>ok. so that's different from how i have usually managed feature branches on forks…14:57
i would normally have done:14:58
git checkout master; git pull upstream
git checkout fcrepo-snapshot; git rebase master
(where upstream is modeshape/modeshape)14:59
<cbeer>might work.15:01
i'm just too lazy when squashing the current branches is just as easy
<eddies>cbeer: does the squash not make pull requests to upstream messier? (from the perspective of the reviewer of the PR)
<cbeer>eddies: this is just for the snapshot build.. not going upstream15:05
we don't care about the commits
(and squashing them makes it easier to cherry-pick them around later. like when i was updating one or the other branches)15:06
<eddies>ok. works for me15:07
<cbeer>barmintor: can you help me put my linked open data into glacier with fcrepo4?15:08
<awoods>jmeter on futures1 is working against fcrepo on futures2. Next is to define the variations of thread-count, loops, and datastream-size for a series of tests.
<barmintor>cbeer: Fedora 4 fixes linked open data with bitstreams.15:09
<pivotal-bot>Chris Beer edited "Fix ITs for fcrepo-bagit-modeshape-federation-connector" https://www.pivotaltracker.com/story/show/50749285
Chris Beer started "Fix ITs for fcrepo-bagit-modeshape-federation-connector" https://www.pivotaltracker.com/story/show/50749285
Chris Beer accepted "Fix ITs for fcrepo-bagit-modeshape-federation-connector" https://www.pivotaltracker.com/story/show/50749285
<gregjansen>eddies: looks like I will modify the bagit connector such that it monitors the folder and publishes changesets (at last at the top/bag level)15:11
<eddies>gregjansen: as opposed to monitoring for changes to the manifest only? (which is what i thought we were doing, but i don't remember for sure)15:12
<cbeer>ajs6f: huh. we don't have any indexing configuration defined15:13
<gregjansen>eddies: oh hey, how did I miss that monitor..15:14
eddies: aha, that monitor creates a changeset, but it is always empty15:16
* jcoyne joins15:22
<barmintor>HOW IS IT 3:30
I spent too much time on that email
that guy was, at least briefly, in the IRC channel. Did he ask any questions?15:23
<jcoyne>bamintor: I was hoping esmé would tackle that one.
barmintor: have you seen what UCSD is doing with Hydra?15:24
<barmintor>I have only heard the legends
<jcoyne>It's either brilliant or insane
Is UCSD contributing to FF?15:27
<cbeer>yes, we have esme for a couple sprints.15:28
<eddies>jcoyne: yes, esme, on alternating sprints
<jcoyne>They are looking at adopting it then?
<cbeer>or just want to know what the cool kids are doing15:29
<eddies>gregjansen: as i recall, only on updates to the manifest file
barmintor++ good email15:30
<ajs6f>cbeer: We did, at least for some things, because we wanted to have disk-persistent indexes, and the default is in-memory.15:31
Maybe we lost it at some point?
<cbeer>oh, maybe it's just not in the http api json
anyway, i'm going to try the async indexing option15:32
yeah, i can't think of why we'd care about sync indexing.15:34
i assume the node traversal just uses the jcr properties it finds
<ajs6f>We had some reason to care about persistent indexes, but I don't think we ever discussed (a)synchrony.15:35
strangely, when you make something that took about 10% of your total time to do
<ajs6f>I think the query machinery may depend on indexes to some extent, but I really don't know for sure...
<awoods>cbeer: not unless we wanted to fail an ingest if the indexing chucked... which I do not believe is the case.
<cbeer>when you make it async, it takes 10% less time.
<ajs6f>BUT LUCENE WAS TAKING 11%! Where is that !% going!?!?!15:36
<cbeer>itunes :)
<ajs6f>What a CPU remora.
<awoods>cbeer: gistfile1 is sync, and gistfile2 is async?15:37
<cbeer>awoods: i labeled them.
i'm thinking about creating an fcrepo-performance-tests project
with this thing15:38
and a more realistic config in it
and figure out how to exclude it from our builds, but run it on jenkins
<awoods>It seems like jmeter does this for us
<cbeer>is that silly/redundant with jmeter?
<awoods>I would suggest getting a suite of jmeter tests to run with ci builds.15:39
<pivotal-bot>Chris Beer added "Create a suite of jmeter tests to run with CI builds" https://www.pivotaltracker.com/story/show/50913615
<cbeer>good. assigned!
because i'm reading http://docs.codehaus.org/display/SONAR/JMeter+Plugin
<awoods>another plus of jmeter (besides the threads, response checking, timers, etc) is the fact that we can easily turn the output into graphs.15:40
<cbeer>and now following at all.
anyway. we /do/ need a place to collect various configurations, righT?
<awoods>yes. which is currently ff-jmeter-madness15:41
<cbeer>there are fcrepo4 configs in there?
i thought ff-jmeter-madness was pretty much black-box
<awoods>no, jmeter configs
<cbeer>oh, we're talking about different things then.15:42
<cbeer>i think we need a place to collect various fcrepo4/modeshape/ispn configs
so we can run tests against them
<awoods>the fcrepo configs can be anywhere on the filesystem, in relation to the actual webapp?15:43
<awoods>I am just wondering how we may want to swap out different configs in an automated way.
for testing
and if it makes sense to collect those configs along with the ff-jmeter-madness project.15:45
<cbeer>i think it'd be better to keep them closer to fcrepo4... if only because they're also a set of example configurations people might use15:46
<ajs6f>I'm heading out of the office— see y'all soon.
<awoods>goodbye, ajs6f15:47
* ajs6f leaves
<awoods>cbeer: true
<cbeer>(except the ones that suck.)
<awoods>yes, I suspect the good configs will be more attractive examples than the sucky ones.15:48
* nbanks leaves15:49
<cbeer>ok. let me see if i can pull together some examples15:53
* nbanks joins16:00
* nbanks leaves16:12
<gregjansen>on mode-1710, trying to make federated events, looks like the Connector needs to know the node key to publish a newNode event. I'm not sure how you would get that without knowing the federated node and querying for it first, which seems broken..16:17
* nbanks joins16:40
<gregjansen>^^^ aha, c.f. Connectors class16:47
<pivotal-bot>Chris Beer estimated "Backup a running repository" as 2 points https://www.pivotaltracker.com/story/show/4610822916:48
Chris Beer started "Backup a running repository" https://www.pivotaltracker.com/story/show/46108229
* gregjansen leaves16:52
* nbanks leaves17:01
<pivotal-bot>Chris Beer added comment: "Turns out it doesn't work with ISPN binary store :/" https://www.pivotaltracker.com/story/show/4610822917:08
Chris Beer unstarted "Backup a running repository" https://www.pivotaltracker.com/story/show/46108229
* jcoyne leaves17:22
<eddies>ok. i'm officially done w/ calls for tonight. and time for bed17:26
<pivotal-bot>Chris Beer added comment: "blocked by MODE-1952" https://www.pivotaltracker.com/story/show/46108229
<barmintor>Ok, rejected17:30
jcr Events don't know what session they originated from
no can do17:31
sorry cbeer
<cbeer>let me peek at MODE and see how hard it'd be to add that
(i wonder how sequencers know what to do?)17:32
i guess sequencers run within a session?
<barmintor>cbeer: they don't care about sessions, right? They just login and do their business
<cbeer>hm.. but they can't be triggered by the changebus can they?
<barmintor>sequencers actually get a node, so they can pull that node's session
but event listeners just get an event type and a path to a node
are you thinking sequencers might be local to the cluster member?17:38
<cbeer>i don't know what i'm thinking.
* jcoyne joins
<barmintor>what was that other ticket eddies wanted me to move on to?17:41
I need to email him
* jcoyne leaves17:47
<cbeer>barmintor: performance related?17:48
(i don't remember.)
<barmintor>me either. I'll ask him tonight17:51
Time to head out
* barmintor leaves
* bljenkins leaves18:03
<pivotal-bot>Chris Beer added "MODE-1952" https://www.pivotaltracker.com/story/show/5092315518:18
Chris Beer started "MODE-1952" https://www.pivotaltracker.com/story/show/50923155
* bljenkins joins18:19
* jcoyne joins18:48
* jcoyne leaves19:00
* fasseg leaves19:19
<pivotal-bot>Chris Beer added comment: "https://github.com/ModeShape/modeshape/pull/845" https://www.pivotaltracker.com/story/show/5092315519:59
Chris Beer edited "MODE-1952" https://www.pivotaltracker.com/story/show/50923155
Chris Beer started "Backup a running repository" https://www.pivotaltracker.com/story/show/4610822920:00
* github-ff joins
[fcrepo4] cbeer created backup (+1 new commit): http://git.io/R9JXIw
fcrepo4/backup cf5aefb Chris Beer: expose endpoints to backup a running repository (and restore from a backup). For now, send backups to a temp directory
* github-ff leaves
<pivotal-bot>Chris Beer added comment: "https://github.com/futures/fcrepo4/pull/70" https://www.pivotaltracker.com/story/show/4610822920:01
* github-ff joins
[fcrepo4] cbeer opened pull request #70: expose endpoints to backup a running repository (and restore from a back... (master...backup) http://git.io/b6uvOA
* github-ff leaves
<bljenkins>Project fcrepo4 build #737: UNSTABLE in 9 min 35 sec: http://ci.projectblacklight.org/jenkins/job/fcrepo4/737/20:10
* nbanks joins20:28
* nbanks leaves20:33
* nbanks joins22:29
* nbanks leaves22:34