Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* fasseg joins06:52
awoods: so SCC hasnt' chnaged and I can not reach serguei, the admin....07:07
So I'll checkout your AWS stuff
oh the sed foo is strong in this one07:09
never got used to that, was always awking
even for simple s//
* eddies leaves07:12
* eddies joins07:18
* eddies leaves
* eddies joins
<fasseg>According to osmandin's results he gets about 59mb/s from the Yale cluster with 4 nodes: https://wiki.duraspace.org/pages/viewpage.action?pageId=3466802707:19
awoods: Do we have a fedora account for AWS?07:25
awoods: these scripts for AWS depend on a Puppet installation?07:32
--platform dependency :)
* nbanks joins07:45
* ajs6f joins08:39
<awoods>fasseg: The script installs puppet on the vanilla AWS instance.08:43
<fasseg>i see
<awoods>fasseg: We do have an account...
give me a moment, fasseg, and I will get you set up.08:47
<fasseg>awoods: no rush enjoying a coffee break atm anyway..
<pivotal-bot_____>Gregory Jansen edited "Integrate an XACML engine with a role-based PEP" https://www.pivotaltracker.com/story/show/5419683409:04
* gregjansen joins09:09
* nbanks leaves09:16
* osmandin joins09:20
* nbanks joins09:27
* ermadmix joins09:56
* ksclarke joins09:59
* tecoripa joins
<pivotal-bot_____>Andrew Woods added comment: "Looks good. Pending @cbeer review." https://www.pivotaltracker.com/story/show/5944113010:08
<fasseg>awoods: Im having trouble to get your aws scripts to run, in the readme it says:10:16
"In your AWS EC2 account, you will need to create a security group and an ssh keypair"
but I dont know how to login to the fcrepo4 AWS account...Is their already a ssh pubkey stored for me?
*there
and if yes can you post the id? ;)10:17
<osmandin>fasseg: regd benchmark took, the ingest time seems to be off.. it's saying 10 minutes for 100 objects, whereas it took only a couple...10:18
<fasseg>osmandin: Oh did I introduce a bug there? lemme check
<awoods>fasseg: you should be able to log into the AWS account with "fasseg" and your password: https://console.aws.amazon.com/10:21
<fasseg>awoods: ah ok I didnt' try console.aws... i tried logging in straight to the AWS page10:22
* github-ff joins10:23
[fcrepo4] awoods closed pull request #126: ModeShape upgrade to 3.6, including necessary change to test PEP (master...pivotal-59504832) http://git.io/rzLlGw
* github-ff leaves
<pivotal-bot_____>Andrew Woods added comment: "Resolved with: https://github.com/futures/fcrepo4/commit/d9b2b2f6ca8785b1aee95ba02a5cda275afe7a3e" https://www.pivotaltracker.com/story/show/59504832
Andrew Woods delivered "Upgrade Modeshape Version to: 3.6.0.Final" https://www.pivotaltracker.com/story/show/59504832
* github-ff joins10:25
[fcrepo-jms-indexer-pluggable] yecao opened pull request #10: changes that can reproduce httpmime lib missing error of indexer web app (master...httpmime) http://git.io/jZjPTQ
* github-ff leaves
<pivotal-bot_____>Ye Cao added comment: "Hi Andrew, I tried it but it seems not working. I make a pull request, so you could reproduce the error above: ht..." https://www.pivotaltracker.com/story/show/5745328010:29
<fasseg>osmandin: can you checkout the project a new and try again, but it seems for me it worked fine before, but i changed tha handling of the shared variables a bit....10:31
<osmandin>fasseg: sure thanks.10:32
<fasseg>osmandin: and remeber that it only counts the time needed for the actual requests, so it won't include the time needed for the datastream generation, but I doubt that this will slow down the tests that significantly...
<osmandin>fasseg: for re-running the benchmark do you normally wipe out old data from node tomcats
<fasseg>for a rerun I normally wipe everything but for just small tests, i just overwrite10:33
* travis-ci joins10:37
[travis-ci] futures/fcrepo4#1054 (master - d9b2b2f : Andrew Woods): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/69ffe6958b68...d9b2b2f6ca87
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/13041352
* travis-ci leaves
<fasseg>awoods: Im not able to login there using fasseg as a username, I can only login to my private account..Can oyu tell me which email has been used for this account?10:40
* nbanks leaves10:41
* nbanks joins10:45
<awoods>fasseg: There is no email address associated with your user. Let me look into what is needed to get you into the console. ajs6f or cbeer may have some insight in the meantime.10:46
* eddies leaves10:47
<ajs6f>I remember it being confusing, and I was glad when I didn't have to do it anymore.10:48
* tecoripa leaves
<bljenkins>Project fcrepo-fixity-corrupter build #301: SUCCESS in 2 min 7 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/301/10:49
* eddies joins10:50
* eddies leaves
* eddies joins
<awoods>fasseg: https://643699039425.signin.aws.amazon.com/console/10:51
<fasseg>nice thanks!10:52
that works
<awoods>fasseg: glad to hear it
<bljenkins>Project fcrepo-kitchen-sink build #551: FAILURE in 3 min 10 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/551/
<pivotal-bot_____>Gregory Jansen added comment: "Okay, filtering of children, at least for our RDF graph responses, happens behind the scenes in ModeShape..." https://www.pivotaltracker.com/story/show/5505351010:55
Gregory Jansen added comment: "If we want to change this and filter the children ourselves, then we will need to use an internal session..." https://www.pivotaltracker.com/story/show/5505351010:56
Frank Asseg finished "Add configuration files used to the wiki SCC benchmark results for better reproducability" https://www.pivotaltracker.com/story/show/5941536010:57
Frank Asseg added comment: "I unstarted this ticket until the time the SCC cluster becomes more usable. FIZ is currently investigating a..." https://www.pivotaltracker.com/story/show/5940135010:58
Frank Asseg unstarted "Test different confguration settings of infinispan/jgroups in order to raise performance on the SCC cluster" https://www.pivotaltracker.com/story/show/59401350
Frank Asseg added "Investigate issues encountered on the AWS cluster instances" https://www.pivotaltracker.com/story/show/5956958011:00
Frank Asseg started "Investigate issues encountered on the AWS cluster instances" https://www.pivotaltracker.com/story/show/59569580
Gregory Jansen started "Extend PEP to support enforcement of access policies for graph representations. (filtering of child nodes)" https://www.pivotaltracker.com/story/show/5505351011:02
Gregory Jansen added comment: "To answer for search engine support (outside of the core), say in Solr. A practical strategy will be to r..." https://www.pivotaltracker.com/story/show/5505351011:09
<ajs6f>https://wiki.duraspace.org/display/FF/2013-10-21+to+11-01+FF+Stand-up+Mtgs#id-2013-10-21to11-01FFStand-upMtgs-Friday2013-10-2511:11
<osmandin>fasseg: similar results. the duration is reported as 750472..whereas it took around 90 seconds I think.11:12
<fasseg>so it tell you it's 750 secs but it was 90?!
wtf? im going back to programming class
I thought it was the other way around11:13
<ajs6f>Maybe you accidentally turned on the tenths of a ms switch.
<osmandin>It says "Complete ingest of 100 files . .. "
and says 'through was 000.00mb/s"11:14
'throughput was 000.00mb/s"
<fasseg>hmm for me it produces the right numbers, and the same numbers as a "time java -cp target/bench-tool-0.0.1-SNAPSHOT-jar-with-dependencies.jar org.fcrepo.bench.BenchToolFC4 http://localhost:8080/ 100 1000000 1"11:15
ill look more into this
oh jesus me fool11:16
I just added up the ingest times of the individual threads making it of course larger than the runtime
yeah sry osmandin the stupid hat is on me today11:17
just go 2 commits back and all will be fine ;)
lemme update the repo
<osmandin>fasseg: but the graphs are ok right
<fasseg>for the indivdiual requests yes, but the throughput calc is all *** up
<osmandin>btw i pasted a new D w/ mod_jk11:18
confluence ate the old one (but it might be in history) -- and confluence seems to be pretty slow today11:19
* eddies leaves11:37
* kaarefc joins11:38
* kaarefc leaves11:40
<fasseg>osmandin: Can you try again with a fresh build from master?11:41
* barmintor joins11:49
<osmandin>fasseg: thahnks, the time now seems to be ok12:00
* github-ff joins12:01
[fcrepo-kitchen-sink] awoods pushed 1 new commit to master: http://git.io/9FTR0A
fcrepo-kitchen-sink/master b40408b Andrew Woods: Update dependencies to fix build
* github-ff leaves
<pivotal-bot_____>Andrew Woods accepted "Upgrade Modeshape Version to: 3.6.0.Final" https://www.pivotaltracker.com/story/show/5950483212:05
<bljenkins>Yippie, build fixed!12:06
Project fcrepo-kitchen-sink build #552: FIXED in 5 min 2 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/552/
* tecoripa joins12:09
<pivotal-bot_____>Andrew Woods added comment: "The native capability provided by ModeShape to simply not return inaccessible children seems like a reasona..." https://www.pivotaltracker.com/story/show/5505351012:15
<barmintor>awoods: tecoripa signed off on branch last night; will squash/merge/release 3.7.1 tonight12:18
<awoods>barmintor: That is good news. Let me know if I can help.
* github-ff joins12:19
[fcrepo4] awoods pushed 1 new commit to master: http://git.io/F2xYbQ
fcrepo4/master e18ddb8 ajs6f: Add capability to map node-types to RDF triples...
* github-ff leaves
<pivotal-bot_____>Andrew Woods added comment: "Resolved with: https://github.com/futures/fcrepo4/commit/e18ddb8cefb1d75f61df6a52d3be6028782d0ea3" https://www.pivotaltracker.com/story/show/59441130
Andrew Woods delivered "Replace functionality to translate between rdf:type and JCR mixins" https://www.pivotaltracker.com/story/show/5944113012:20
Gregory Jansen added comment: "Yes, so currently the effective roles are returned as JSON for a GET request, when the "effective" parame..." https://www.pivotaltracker.com/story/show/5505351012:21
A. "Senphitic" Soroka added comment: "It would be more "Fedora 4"-ish to return this info as RDF." https://www.pivotaltracker.com/story/show/5505351012:24
A. "Senphitic" Soroka added comment: "And you could reuse the existing response machinery." https://www.pivotaltracker.com/story/show/55053510
Scott Prater started "Create integration tests for Fedora objects with ACLs" https://www.pivotaltracker.com/story/show/59162556
* travis-ci joins12:34
[travis-ci] futures/fcrepo4#1055 (master - e18ddb8 : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/d9b2b2f6ca87...e18ddb8cefb1
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/13047346
* travis-ci leaves
* eddies joins
* eddies leaves
* eddies joins
<pivotal-bot_____>Andrew Woods accepted "Replace functionality to translate between rdf:type and JCR mixins" https://www.pivotaltracker.com/story/show/5944113012:35
* github-ff joins
[fcrepo4] awoods closed pull request #125: Fix rdf type jcr mixin translation (master...FixRDFTypeJCRMixinTranslation) http://git.io/Wek1Dg
* github-ff leaves
* github-ff joins12:37
[fcrepo4] ajs6f force-pushed RDFIterationForMutation from 6813bdd to 6c93d97: http://git.io/9OyXSg
fcrepo4/RDFIterationForMutation 466bc2e ajs6f: Distinguishing between managed and unmanaged RDF
fcrepo4/RDFIterationForMutation a315594 ajs6f: Developing machinery to compare new and old material for updating
fcrepo4/RDFIterationForMutation 6fe7529 ajs6f: Building new iterator-driven persistence machinery
* github-ff leaves
<bljenkins>Project fcrepo-fixity-corrupter build #302: SUCCESS in 1 min 7 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/302/12:38
* github-ff joins12:47
[fcrepo4] ajs6f pushed 1 new commit to RDFIterationForMutation: http://git.io/_APFgw
fcrepo4/RDFIterationForMutation eb488cf ajs6f: More unit tests
* github-ff leaves
<bljenkins>Project fcrepo4 build #1238: UNSTABLE in 14 min: http://ci.fcrepo.org/jenkins/job/fcrepo4/1238/12:50
* ermadmix leaves13:06
<pivotal-bot_____>Osman Din added comment: "Test results are at : https://wiki.duraspace.org/display/FF/Test+-+Cluster+Setup+-+Yale" https://www.pivotaltracker.com/story/show/59252840
<bljenkins>Yippie, build fixed!
Project fcrepo4 build #1239: FIXED in 15 min: http://ci.fcrepo.org/jenkins/job/fcrepo4/1239/
<pivotal-bot_____>Osman Din finished "Create fcrepo4 single node benchmarking result on the Yale cluster" https://www.pivotaltracker.com/story/show/59252840
<bljenkins>Project fcrepo-fixity-corrupter build #303: SUCCESS in 1 min 8 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/303/13:07
Project fcrepo-kitchen-sink build #553: UNSTABLE in 3 min 4 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/553/13:10
* osmandin leaves
<pivotal-bot_____>Andrew Woods added comment: "Very true, @ajs6f, very true. RDF-all-the-things!" https://www.pivotaltracker.com/story/show/5505351013:19
A. "Senphitic" Soroka added comment: "We're nothing if we aren't consistent on our adoption on Internet comic memes." https://www.pivotaltracker.com/story/show/5505351013:21
<cbeer>ajs6f: nicely done on the RdfStream stuff
i'm porting over the CND => RDFS stuff and it's really nice.
* ermadmix joins13:22
<ajs6f>cbeer: I'm knee-deep in what the iterator-driven thing means for persistence. Thank goodness barmintor finally beat the unit test gospel into me...
cbeer: also, I've stopped assuming that we can invert prefix-namespace maps, because we can't stop people from registering the same namespace under several prefixes. So I can't use your nifty Guava BiMap trick any more… :'(13:28
* kaarefc joins13:30
<cbeer>ajs6f: is the goal to expose the rdf stream up to the http api, or is there clever logic in some of those jena classes to handle the streaming?13:34
<ajs6f>cbeer: Turtles all the way up, because I couldn't see my way to impl'ing Jena's Model over a stream. They want to be able to do too many things that aren't streamy. We _could_ do it, but there's the other issue:13:35
cbeer: awoods agreed with me that it doesn't feel right to pass Jena's abstractions out from our kernel.13:36
cbeer: Keep in mind that there's RdfStream.asModel() and RdfStream.fromModel() for when you want to quickly pass between the two world.
(e.g. for easy serialization).
<cbeer>ajs6f: right. that's where I thought we landed. so, in -http-api land, what am I supposed to do to render some triples? maybe i haven't found the right thing yet13:37
<ajs6f>cbeer: Given an RdfStream?13:38
<cbeer>ajs6f: yeah, i have one of those.13:39
<ajs6f>Model model = myStream.asModel(); rolls it up into a Jena Model, and then model.write(myWriter) or the like, just like we did before.
<cbeer>ok, cool. thanks13:40
<ajs6f>Obviously, you want to have filtered down your stream to something that makes sense to serialize first.
If you use Guava's Iterators.filter (love that little fellow!), keep in mind that it's _not_ quasi-self-bounded—13:41
<cbeer>ok
<ajs6f>filter(myStream, myPredicate) returns an Iterator<Triple>, not an RdfStream.
the quick but slightly ugly is:
new RdfStream(filter(myStream, myPredicate)).addNamespaces(mystream.namespaces()),13:42
which returns a stream with the namespaces back, which is ready have asModel() called on it.
addNamespaces, like almost everything else in RdfStream, acts as a quasi-fluent method, returning the stream for further use.13:43
I thikn I should probably just add Iterators.transform and Iterators.filter to RdfStream as convenience methods that repack the namespaces. They're really useful in this kind of work.13:44
Feel free to do that if you get tired of repacking the maps yourself.
* osmandin joins13:47
* github-ff joins
[fcrepo4] ajs6f pushed 1 new commit to RDFIterationForMutation: http://git.io/Q7YCPA
fcrepo4/RDFIterationForMutation 6c5f91a ajs6f: Completed positive persistence
* github-ff leaves
* ermadmix leaves13:54
* kaarefc leaves14:01
<pivotal-bot_____>Gregory Jansen added comment: "This thread has de-jena-rated." https://www.pivotaltracker.com/story/show/5505351014:03
<bljenkins>Project fcrepo-fixity-corrupter build #304: SUCCESS in 55 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/304/14:04
Project fcrepo-kitchen-sink build #554: STILL UNSTABLE in 2 min 40 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/554/14:07
* ermadmix joins14:10
* kaarefc joins14:17
* github-ff joins14:33
[fcrepo4] ajs6f pushed 1 new commit to RDFIterationForMutation: http://git.io/HB-d7w
fcrepo4/RDFIterationForMutation f63bf3c ajs6f: Unit tests for negative persistence
* github-ff leaves
<pivotal-bot_____>Andrew Woods added comment: "Give this a try, @yec: ""14:39
https://github.com/yecao/fcrepo-jms-indexer-pluggable/pull/4" https://www.pivotaltracker.com/story/show/57453280
Osman Din started "Create fcrepo3 benchmarking result on the Yale cluster" https://www.pivotaltracker.com/story/show/59252564
Andrew Woods accepted "Add configuration files used to the wiki SCC benchmark results for better reproducability" https://www.pivotaltracker.com/story/show/5941536014:41
Andrew Woods added comment: "Please add "throughput" for easy comparison with @frankasseg 's page." https://www.pivotaltracker.com/story/show/5925284014:43
<bljenkins>Project fcrepo-fixity-corrupter build #305: SUCCESS in 55 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/305/14:49
Project fcrepo-kitchen-sink build #555: STILL UNSTABLE in 2 min 21 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/555/14:52
* gregjansen leaves15:04
<pivotal-bot_____>Gregory Jansen started "Integrate an XACML engine with a role-based PEP" https://www.pivotaltracker.com/story/show/5419683415:05
* gregjansen joins15:06
* gregjansen leaves15:07
* nbanks leaves15:12
* kaarefc leaves15:23
* fasseg leaves15:24
* tecoripa leaves15:28
<ajs6f>cbeer: It occurs to me that the DC generator stuff was written before you did fcrepo-transform. I wonder if we could rejigger it to use that good new gear… get people away from the idea of a well-known datastream towards that of a "house transform" which pulls from whatever props are there to build DC...?15:48
<cbeer>ajs6f: i thought we totally dropped the DC generator
shows what i knw15:49
<ajs6f>I just watched it build.
Maybe we _should_ just drop it in favor of a sample module which shows how to provide DC from a transform
?
* tecoripa joins
<cbeer>+1 from me15:50
* github-ff joins15:51
[fcrepo4] ajs6f pushed 1 new commit to RDFIterationForMutation: http://git.io/89GBVQ
fcrepo4/RDFIterationForMutation 6a22fd7 ajs6f: Moved iterating RDF generation to the outer layer of JcrRdfTools
* github-ff leaves
<ajs6f>Kill 'em all. The best coverage is over the smallest code.
<cbeer>"fcrepo4: really, more of an architecture than an application."
<ajs6f>That's a Thorny quote, right?15:52
* github-ff joins
[fcrepo4] ajs6f opened pull request #127: Rdf iteration for mutation (master...RDFIterationForMutation) http://git.io/8D9mxg
* github-ff leaves
<bljenkins>Project fcrepo-fixity-corrupter build #306: SUCCESS in 1 min 14 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/306/16:09
Project fcrepo-kitchen-sink build #556: STILL UNSTABLE in 2 min 54 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/556/16:12
* github-ff joins16:13
[fcrepo4] ajs6f pushed 1 new commit to RDFIterationForMutation: http://git.io/hWlaqA
fcrepo4/RDFIterationForMutation ce5f246 ajs6f: Moving iterated generation of versioning-related triples through to the domain models
* github-ff leaves
<ajs6f>awoods: ping?
<awoods>ajs6f: hi
<ajs6f>awoods: That last commit was the first time a source of RDF has been exposed all the way through (versioning info)16:14
I'd like to get an intermediate merge.
I've got a lot of work still to do, but this is good stuff already.
I'd like other people to ciriticise,
criticize,
and I'd like to avoid getting too far down a branch.16:15
Can we do a PR?
<awoods>ajs6f: regarding DC generator?
<barmintor>don't you have enough people to criticize?
<ajs6f>awoods: Nothing to do with DC at all. This is about the iteration stuff.
<awoods>ajs6f: sorry, my client had not scolled.16:16
scrolled
<ajs6f>barmintor: I'm _much_ more supportive than other people. Compared to my incredibly supportive attitude, everyone else is _crap_.
<awoods>ajs6f: You are just asking for an interim merge to master of your branch?16:17
<ajs6f>awoods; Right.
Let me dump stack and keep going with less divergence.16:18
<awoods>ajs6f: That is fine. Is cbeer, or others, working in the same classes?
<ajs6f>awoods: I'm thinking that cbeer is going to be turning out good stuff, but I'd hate to run into merge hell between us.
awoods: Exactly.
<awoods>ajs6f: np, one moment16:19
<ajs6f>cbeer: Ready to rebase a jot?
<cbeer>ajs6f: i'm happy to rebase.
<ajs6f>Shouldn't be much.
awoods: I've got a PR, but it's marked as "Don't merge". I'll clear that.16:20
Yikes!16:21
awoods: I just looked at our Travis logs, I'm seeing:
$ JAVA_OPTS=-Xmx512m mvn install -q -DskipTests=true
* osmandin leaves
<ajs6f>Are we running w/o tests?
<awoods>in travis, yes. in jenkins tests are enabled.16:22
<ajs6f>Oh, okay. As long as our CI is sane. {wheew!}
Does Jenkins run branches?16:23
<awoods>no
<ajs6f>Is that a big deal? (Asked of anyone who knows anything about Jenkins: not me.)
Meaning could we turn that on?
* osmandin joins16:24
<awoods>ajs6f: I suspect we could turn on branches. That machine is a dog, however. You like CI to run your branches so you do not have to build locally?
<ajs6f>I do build locally, of course. I like belts, but I also like suspenders.16:25
<awoods>ajs6f: I am not sure what the rationale was to turn off tests in travis, but I would rather we enable them there than on jenky.16:26
<pivotal-bot_____>Osman Din added comment: "Installed F3.7 (quick setup, single load). Using mod_jk from another VM, got results mentioned in Results sect..." https://www.pivotaltracker.com/story/show/59252564
<ajs6f>awoods: Fine by me. I'd just like to have as much coverage of coverage as possible.
Do we need to find some more CI resources?16:27
We run Jenkins here, although I haven't fooled with it in years. For all I know, it may run on a hulking box.
<awoods>ajs6f: If we had a replacement for gluck, that would probably be good.
ajs6f: I just pulled down your PR. It does not build... are you working that?
<ajs6f>I'll ask around.
It just built for me. What failed?16:28
<awoods>ajs6f: same error as in travis
<ajs6f>Hm. I didn't see that. That's really not helpful: "The forked VM terminated without saying properly goodbye."
No tests failed.
I'll try a local build again and see if I can replicate.
<pivotal-bot_____>Osman Din added comment: "Added this info. The tool is not generating this anymore." https://www.pivotaltracker.com/story/show/59252840
Osman Din finished "Create fcrepo3 benchmarking result on the Yale cluster" https://www.pivotaltracker.com/story/show/5925256416:29
* kaarefc joins
<pivotal-bot_____>Osman Din started "Check how much the "write-behind" setting in infinispan impacts the ingest performance" https://www.pivotaltracker.com/story/show/5925225016:30
<bljenkins>Project fcrepo-fixity-corrupter build #307: SUCCESS in 1 min 6 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/307/16:31
<ajs6f>awoods: mvn clean install ran straight through, no problem, although my laptop is burning hot. I was running with -Xmx2048M -XX:MaxPermSize=256M, although that shouldn't really matter.16:34
<bljenkins>Project fcrepo-jms-indexer-pluggable build #87: FAILURE in 3 min 47 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-jms-indexer-pluggable/87/
<ajs6f>Sure does take a while to build, tho'.
I wish we could pull rss and jms and webhooks and some of these things out of there.
<tecoripa>anyone: gregjansen isn't around, so I'll ask here: what's the REST API to add, delete ACLs?16:35
<ajs6f>I'm using jdk1.7.0_12
<tecoripa>poking around on the wiki, not finding it, tho
<bljenkins>Yippie, build fixed!
Project fcrepo-kitchen-sink build #557: FIXED in 4 min 0 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/557/
<awoods>tecoripa? I thought gregjansen was creating that REST API?
<ajs6f>awoods: Not much I can do if I can't replicate. Maybe someone else can pull the branch and try?16:36
<awoods>tecoripa: ...in the mystery git repository.
<tecoripa>awoods: yes, and I thought he had finished it... maybe not, though
<awoods>ajs6f: -Xmx1024m -XX:MaxPermSize=512m
<tecoripa>don't make me look at source...
<ajs6f>The mystery git repo sound a bit like Scooby Doo's mystery van.
awoods: I'll try with tighter heap16:37
<awoods>tecoripa: he may have finished it, but I am still waiting on this ticket: https://www.pivotaltracker.com/story/show/59359584
<pivotal-bot_____>feature: Move Auth PEP project into Futures (unstarted) / owner: Gregory Jansen
<ajs6f>awwods: and more permgen, like you. But it seems a bit odd. The travis build failed after _unit tests_
<tecoripa>awoods: yeah, that's waiting on me to finish the integration tests.16:39
<ajs6f>awoods: No, kernel ran through unit tests fine and kept going (still going).
* osmandin leaves
<awoods>ajs6f: you can always bisect through your commits.
<tecoripa>awoods: I sense a race condition here...
sigh. I'll look at source.
<ajs6f>awoods: bisect?
<awoods>git bisect
ajs6f: It is a little gem cbeer shared.
<ajs6f>awoods: I don't see how that helps if it doesn't fail for me...
<awoods>ajs6f: true16:40
<ajs6f>Replication is the root of all fixes.
<awoods>ajs6f: I can do it if you like.
<ajs6f>Find the bad commit?
<awoods>ajs6f: yes
<ajs6f>awoods: If you can replicate, please do.
<awoods>ajs6f: I get the error... here we go.16:41
<ajs6f>I'd sure like to know what causes the JVM to crack out with no message.
<awoods>it says: goodbye
no it does not
<ajs6f>awoods: Are you joking?
{sigh}
* kaarefc leaves
<tecoripa>awoods: duh! <slaps forehead>. I have the code right in front of me. In a method I wrote last night.16:43
it's Friday.
<awoods>tecoripa: a gift for yourself16:44
<tecoripa>awoods: those are the best kind. just what I always wanted.
<awoods>tecoripa: speaking of things you always wanted: eggnog has already hit the shelves.16:45
<ajs6f>awoods: My build with your tighter params went fine, too. {urg}16:48
<awoods>ajs6f: It would appear that your first commit is the culprit: 466bc2e98bf5bdf92a19a094861c76d64b57272316:50
<ajs6f>!
That was nothing more than the addition of some constants, a predicate or two that uses them...16:51
Are you getting any kind of error message?16:52
<awoods>ajs6f: One moment, I am making sure master builds...
<ajs6f>thnx
<awoods>ajs6f: Then I will cherry-pick your first commit to see if things choke.16:53
<ajs6f>thnx. I appreciate the x check. I'm surprised that anything is funky, and particularly so that that commit has anything odd about it. It was the simplest of all.16:54
<awoods>ajs6f: Hopefully that means it will be the simplest to fix.
<ajs6f>awoods: It better be. it's how we distinguish between RDF that the repo generates and RDF that the repo only curates.16:55
<awoods>ajs6f: still building...16:59
<ajs6f>awoods: Cripes. Are you getting my feeling about taking things out of that repo / out of the core? I feel the neeeed… the need for— modularization!17:00
<awoods>ajs6f: success
<ajs6f>From that "bad" commit?
<awoods>from master17:01
now for the baddie
<ajs6f>Oh, okay.
It's a good commit, it's just misunderstood.
<awoods>ajs6f: boom!17:04
<ajs6f>with a message?
<awoods>permgen17:05
<ajs6f>?!?
There's no static mocking in there, I don't think, or any of the usual suspects.
hm
* kaarefc joins17:06
<ajs6f>Seriously? PermGen?
<awoods>ajs6f: which ticket is this?17:08
ajs6f: I will attach the build log.
<ajs6f>https://www.pivotaltracker.com/s/projects/684825/stories/59243236
I am at a loss.
I cannot imagine what would be in anyway conducive to a PermGen blowout...
<pivotal-bot_____>Andrew Woods added comment: "" https://www.pivotaltracker.com/story/show/5924323617:09
Andrew Woods started "Alter outward-facing contracts in fcrepo-kernel to expose new iterator-based machinery for RDF." https://www.pivotaltracker.com/story/show/59243236
<ajs6f>And more interestingly, why I didn't get it with the same memory settings as you used.
<awoods>different jvms
different platforms
<ajs6f>What's you JVM?
I'm on Oracle HotSpot jdk1.7.0_1217:10
<awoods>java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)
<ajs6f>Doesn't help much. I just can't see a potential culprit. I guess I'll have to get a 1.7.0_09 JVM and start where you left off.17:11
Write once, run anywhere. {bitter grimace}17:14
<awoods>ajs6f: I like the idea
<ajs6f>awood: I like the ideas of community, fecundity, and peace. Don't seem like Oracle much cares about any of the four.17:15
awoods: I'm going to stop proselytizing for Scala, since it runs on the JVM. It's time for us to step up to ML. Instead of writing code to impl a repo architecture, we can write code that simply proves that such an impl exists.17:17
All: anyone know where to get old versions of HotSpot?17:18
awoods: Can you do me one last favor? Can you run it with -X, to get every possible detail?17:21
<awoods>ajs6f: sure
<ajs6f>awoods: This could be something really subtle with Maven.
awoods: I'm weirded out by the fact that it _completes the unit tests_ and _then_ blows PermGen.17:22
<pivotal-bot_____>Andrew Woods added comment: "" https://www.pivotaltracker.com/story/show/5924323617:24
<awoods>ajs6f: there you go
<ajs6f>awoods: thnx
<awoods>ajs6f: fyi, I can not find a listing anywhere, but this old link downloads a jdk. You may be able to hack the URL for other versions.17:27
sudo wget --no-check-certificate --no-cookies --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com" http://download.oracle.com/otn-pub/java/jdk/7u9-b05/jdk-7u9-linux-x64.tar.gz
<ajs6f>awoods: thnx17:28
What a way to end to week.
<awoods>ajs6f: Fridays can be dangerous
<ajs6f>awoods: Better set some velocity for next week aside to the account of "Java bullshit".17:29
* github-ff joins17:35
[fcrepo4] ajs6f pushed 1 new commit to RDFIterationForMutation: http://git.io/2c1w-A
fcrepo4/RDFIterationForMutation 86f1e6b ajs6f: Bumping Maven Surefire version
* github-ff leaves
<ajs6f>awoods: Up for one last assist?17:40
<awoods>ajs6f: sure... I was just whipping up a late lunch17:44
ajs6f: What do you have?
<ajs6f>awoods: I thought you were on the East Coast?
awoods: Can you go to fcrepo4/pom.xml, and in the pluginManagement,17:45
<awoods>ajs6f: where do you think I am?
<ajs6f>The East Coast. Generically.
<awoods>I am Mountain Time
<ajs6f>Well, that's more comfortable than being a mountain man.17:46
<awoods>pluginManagement
<ajs6f>Change,
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.15</version>
</plugin>
to
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<configuration>
<argLine>-Xms512m -Xmx1024m -XX:MaxPermSize=512m</argLine>
</configuration>
</plugin>
essentially, I think we may have been running with much less permgen for unit tests than we thought.
Because we're forking from surefire.17:47
(We could also try not forking from surefire, but that might get hairy.)
<awoods>building
* ermadmix leaves17:48
<ajs6f>awoods: also, I bumped that surefire version in my branch, and while it didn't make Travis happy in and of itself, it did improve the output.
Now I get a message:
Exception in thread "main" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
which is better than stony silence.
<awoods>We made it past kernel17:49
<ajs6f>awoods: now I suspect that I pushed just a jot past a limit we didn't know we were approaching.
We're currently running all unit tests with default memory settings.17:50
<awoods>let that be a lesson to us all
<ajs6f>Ideally, that should be fine.
But we are doing a boatload of static and final mocking (h/t to MODE and ISPN).
awoods: Is your build going faster than usual?
<awoods>ajs6f: it has not finished yet17:51
<ajs6f>awoods: my unit tests are going like lightning.
I think we were suffering a lot of GC in unit tests or some other such JVM esoterica.
Dang! That's the fastest build I've seen in months!
<bljenkins>Project fcrepo-fixity-corrupter build #308: SUCCESS in 58 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/308/17:52
<ajs6f>awoods: So there's a decision or two to be made here, and it's all about Maven Surefire, forking, and memory management.
awoods: I think.17:53
<awoods>ajs6f: what do you have in mind?
either tweaking the pom or refining our code?
<ajs6f>Not sure yet. I'm checking right now what the default heap size is (since obviously, permgen < heap)
awoods: Maybe a little of both. I'm sure barmintor will agree that the less we use PowerMock the happier we will all be.17:54
PM is dark wizardry.
Rings of Power -ish.
<awoods>ajs6f: I have a little pet peeve regarding static methods and testing.17:55
<ajs6f>But that will take time, and some of the static and final mocking is unavoidable.
<bljenkins>Project fcrepo-kitchen-sink build #558: UNSTABLE in 2 min 33 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/558/
<ajs6f>awoods: We can at least stop mocking our _own_ statics.
awoods: We do that. We really do.
<awoods>ajs6f: yes. modeshape and ispn are a different matter.
<ajs6f>awoods: I'm going to commit this fix-attempt to my PR and see if Travis buys this.17:56
<awoods>ajs6f: It looks like your tweak did the trick.
build passed
<ajs6f>Well, then my hypothesis seems a little more likely.
And we know what the real targets are.
<awoods>ajs6f: In the meantime, might as well push the tweak... Or I can in the process of squashing your PR.
<ajs6f>I'll push it. I'd like to shove it in Travis' face, that snarky SOB.
<bljenkins>Yippie, build fixed!17:57
Project fcrepo-jms-indexer-pluggable build #88: FIXED in 5 min 42 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-jms-indexer-pluggable/88/
<ajs6f>awoods: new ticket?
awoods: Make memory big, and tests small. (Chore)
<awoods>ajs6f: new ticket is in order.
you or me?
<ajs6f>me.
<awoods>hit it
ajs6f: ...at the moment, boiling water on the stove is calling for broccoli.17:58
<ajs6f>HIt it: http://www.youtube.com/watch?v=WZ3zZ-aWi6U
* github-ff joins17:59
[fcrepo4] ajs6f pushed 1 new commit to RDFIterationForMutation: http://git.io/O3otIQ
fcrepo4/RDFIterationForMutation 7d7f27d ajs6f: Bumping available heap for forked unit tests
* github-ff leaves
<awoods>ajs6f: I would have thought you would push the pom-tweak into a separate commit.18:02
<ajs6f>I did. ?18:03
<awoods>as opposed to adding it to your PR
<ajs6f>I don't get it. Do you mean a separate PR?
<awoods>yes
<ajs6f>You said a separate commit.
<awoods>yes
<ajs6f>Okay, I can send a PR, np.
<awoods>or just push it right in
<ajs6f>let me finish with the tix.
<awoods>no PR required.
<ajs6f>Okay, I push it. I'm extremely confident it is a very good thing.
<awoods>afk18:04
broccoli
<pivotal-bot_____>A. "Senphitic" Soroka added "Heap management for forked unit tests" https://www.pivotaltracker.com/story/show/59601514
* github-ff joins18:06
[fcrepo4] ajs6f pushed 1 new commit to master: http://git.io/G5HtBg
fcrepo4/master 4fb55c1 ajs6f: Bumping Surefire version and adding config to provide more heap during unit test exercise
* github-ff leaves
<pivotal-bot_____>A. "Senphitic" Soroka edited "Heap management for forked unit tests" https://www.pivotaltracker.com/story/show/5960151418:07
A. "Senphitic" Soroka added this label to 1 story https://www.pivotaltracker.com/story/show/59601514
A. "Senphitic" Soroka added "Die static mocking to death!" https://www.pivotaltracker.com/story/show/5960169018:09
<ajs6f>awoods: The PR went through Travis okey-dokey.18:10
<bljenkins>Project fcrepo-fixity-corrupter build #309: SUCCESS in 53 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/309/18:14
* travis-ci joins18:18
[travis-ci] futures/fcrepo4#1060 (master - 4fb55c1 : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/e18ddb8cefb1...4fb55c115012
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/13063452
* travis-ci leaves
<bljenkins>Project fcrepo-fixity-corrupter build #310: SUCCESS in 53 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/310/18:30
<awoods>ajs6f: nice job18:31
<ajs6f>awoods: If it's okay by you, I'm going to bump the Failsafe version too. They should move together, and I checked the build.18:32
<pivotal-bot_____>A. "Senphitic" Soroka added "Use Maven parallelism in CI" https://www.pivotaltracker.com/story/show/59603190
<awoods>ajs6f: hit it
<ajs6f>awoods: Also, have you looked at Failsafe's parallel abilities? We should take advantage of them, not only because they speed things up massively, but because they show you where you're going to have race conditions under load.18:33
<awoods>ajs6f: no harm18:34
ajs6f: I will play around with it locally
<ajs6f>awoods: Well, yes harm, because if you turn them on now, our itests go to hell in a handbasket. We aren't very well equipped for that sort of thing.
<awoods>ajs6f: we do not really know the hardware env of travis.18:35
<bljenkins>Yippie, build fixed!
Project fcrepo-kitchen-sink build #559: FIXED in 4 min 25 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/559/
<awoods>ajs6f: so it does not currently work?
<ajs6f>awoods: Ok, but the other one, Jenkins? Using two thread instead of one can't hurt.
<awoods>if it works.
* ksclarke leaves18:36
<ajs6f>Not failsafe parallel. I guess it works in the sense of showing us all kinds of weaknesses in our tests.
—threads works sweetly for me.
On a two-core laptop.
* github-ff joins18:37
[fcrepo4] ajs6f pushed 1 new commit to master: http://git.io/sGOJhQ
fcrepo4/master d6c271c ajs6f: Bumping Failsafe version, to keep it in step with Surefire
* github-ff leaves
<ajs6f>Okay, I feel better about Friday. Time for me to go make supper.
See y'all soon.
<awoods>thanks, ajs6f
* ajs6f leaves18:38
* travis-ci joins18:48
[travis-ci] futures/fcrepo4#1061 (master - d6c271c : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/4fb55c115012...d6c271cd68c1
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/13064315
* travis-ci leaves
<bljenkins>Project fcrepo-fixity-corrupter build #311: SUCCESS in 1 min 0 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/311/18:55
Project fcrepo-kitchen-sink build #560: UNSTABLE in 3 min 26 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/560/18:58
* kaarefc leaves19:11
* kaarefc joins19:22
* kaarefc leaves
* tecoripa leaves19:25
* ksclarke joins19:28