Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* eddies joins07:36
* eddies leaves
* eddies joins
* JasonDGI joins08:42
* ajs6f joins09:08
* avessey joins
<eddies>ajs6f: is the readme for ff-modeshape-prototype up to date?09:46
that is, are the first two steps supposed to work:09:47
$ mvn clean jetty:run
$ curl "http://localhost:8080/rest/fedora/describe"
because I get a 503: service unavailable error trying to hit describe09:48
<cbeer>lose the fedora.09:58
my fault
* github-ff joins
[ff-modeshape-prototype] eddies pushed 3 new commits to master: http://git.io/4glOHA
ff-modeshape-prototype/master c9869ff Edwin Shin: Adds lifecycle-mapping plugin configuration for Eclipse m2e to quiet m2e errors
ff-modeshape-prototype/master b034b68 Edwin Shin: Adds trivial DTD to quiet Eclipse warnings for logback. See: http://stackoverflow.com/questions/5731162/xml-schema-or-dtd-for-logback-xml
ff-modeshape-prototype/master d24c248 Edwin Shin: Updates .gitignore to include ActiveMQ/, FedoraRepository/, indexes/ and ObjectStore/
* github-ff leaves
<cbeer>eddies: could you also send out the yourkit license info?10:00
<eddies>cbeer: sent10:03
ajs6f: i think that issue i asked you about was, at least in part, related to swapping. i forgot those machines only have 1G of RAM
Yeah, that makes some sense.
<cbeer>so, maybe that's the problem.. and i'll try to crank it down in the MAVEN_OPTS
but we may have chosen a product that really likes its RAM
<ajs6f>I was going to suggest today that we start by getting at least two other people to run the test at two other sites.
And see if the barrier appeared, and if so in the same place.
<cbeer>yeah, i was going to try AWS today
<ajs6f>Yeah, the RAM may be an iissue.
I think we need more info before bringing it to ISN.10:06
* barmintor joins
* github-ff joins
[ff-modeshape-prototype] eddies pushed 1 new commit to master: http://git.io/SXiupw
ff-modeshape-prototype/master ad5d1c8 Edwin Shin: typo correction
* github-ff leaves
<barmintor>eddies: am I renting the car tomorrow?10:07
<cbeer>ajs6f: agreed. and i've started lurking in #modeshape10:12
<eddies>barmintor: yes10:13
<ajs6f>Gee, i didn't even realize that was there. I'm going to lurk there too.
cbeer— when we turned on eviction for the store, did we set a limit on the number of objects in RAM?
Maybe we need to.
<cbeer>yeah, i think we said 10010:14
<ajs6f>Why 100?
<cbeer>(and, i think it's working fine, because we aren't running into JVM memory errors)
<ajs6f>Mm. Good point.10:15
<cbeer>gah. and this license is out of date too
<ajs6f>Can you put the exact invocation of the JMEtere script and the exact commit on here when you get a chnce? I will happily run the test.
The YourKit license?10:16
<cbeer>yes. time to figure out how to download an older yourkit version
<ajs6f>Well, if you can't, I will zip and ftp you mine. (Which is an older version with the DuraSpace license working in it.)
Or you could try using the Eclipse plugin.10:17
<cbeer>ajs6f: jmeter/bin/jmeter.sh -n -t plans/fedora.jmx -Jfedora_server=futures5 -Jfedora_port=9999 -Jfedora_context=rest -Jnumthreads=5 -Jloopcount=1000 -Jfilesize_mean=1024 -Jfilesize_stddev=1 -Jtest_label=modeshape
on 14e9184dd023be002c5ff187002a8f1617e518f7
<ajs6f>Thanks— that's "ff-jmeter-madness"?10:18
<cbeer>oh. yes. and ff-modeshape-prototype is...
be044557154fb75e551ba03dc9ff126ed5e33b84 with logging turned down to INFO and clustering disabled10:19
but clustering seems to be irrelevant
<ajs6f>Thanks— I'm going to wait until I'm inside and plugged in. I know this is going to grind my laptop and drain the battery.10:20
<cbeer>k. i'll try it locally too10:21
* ajs6f1 joins10:22
* anusha joins10:24
* ajs6f leaves
* anusha leaves10:33
<ajs6f1>Find anything?10:35
<cbeer>nope. just impressed how much more friendly it was than jconsole/jvisualvm
<ajs6f1>Oh, it's worlds better. Just have to have your trigger finger ready to take a snapshot. :)10:36
Eddie, can we ask DuraSpace to update that icense?10:37
<cbeer>i forgot that RH bought JBoss10:38
(and never knew that adobe bought Day (/jackrabbit devs))
<ajs6f1>Yeah, Jckrabbit is at the heart of their CMS product.10:39
Makes you wonder what kind of scaling techniques they know about.
<barmintor>we run the Day CMS here, actually10:40
it's a hog. I think it scales with lots of caching and sysadmin tears.10:41
<cbeer>eddies: i saw from the last SG minutes we're (you're?) writing something about our choice of modeshape.. have you seen http://modeshape.wordpress.com/2012/10/18/when-is-modeshape-a-good-fit/ yet?10:42
<ajs6f1>Ah. Sysadmin tears: the magic fluid that dissolves all substances.
<cbeer>(and http://modeshape.wordpress.com/2012/01/28/modeshape-3-0-alpha1-is-here-and-it-rocks/ and http://modeshape.wordpress.com/2011/06/20/what-distinguishes-modeshape/)
ajs6f1: i think we just hit swapping issues on futures510:45
<ajs6f1>Obliquely relevant : http://stackoverflow.com/questions/6392815/what-does-modeshape-offer-that-jackrabbit-doesnt.
<cbeer>or, i was about to run out of disk space
<ajs6f1>Grr. This disk space thing is getting annoying. Maybe we need to sophisticate the JMeter script to leave less on disk?10:47
<cbeer>this was locally.10:48
* anusha joins
<ajs6f1>Oh. You should clear off all those pirated movies, then.10:49
<cbeer>more like finally repartition my HD.. our IT group decided I really needed a 500G scratch disk and 100G of everything else.10:50
<cbeer>#modeshape seems friendly enough10:52
<ajs6f1>Yeah, I can see at least a few of the core devs there, which is always a good sign.
<JasonDGI>meeting today?10:58
<ajs6f1>I can't get to a phone for a bit, but I'll be here for questions.10:59
* anusha leaves
<cbeer>ajs6f1: before i jump in, i wasn't talking about the binary values themselves, it's the temp files modeshape is using to figure out the sha-1 key to store the binary values11:01
unless you're trying to talk about something else entirely
<ajs6f1>I was, but I just introduced that question.
I was also curious about the binary GC generally.11:02
Under high load, that could be an interesting question for us.
<cbeer>dialing in now.11:03
<ajs6f1>You're stil on dial-up!?
cbeer: I just learnt that we should be using JcrConstants and not string values to create nodes.11:04
* jonathangee joins11:05
* ajs6f1 leaves11:07
* anusha joins
<cbeer>JasonDGI: i added this ticket in pivotal and just assigned it to you: https://www.pivotaltracker.com/story/show/43679887
<cbeer>JasonDGI: i've got gsearch installed and somewhere along the way got discouraged and gave up11:08
<JasonDGI>everyone feels that pain at some point
<cbeer>i assume you're immune by now
<JasonDGI>immune it a bit much, maybe resistant
<cbeer>anusha++ # nice graphs11:09
* ajs6f joins
<cbeer>confluence-- though. what a pain
<ajs6f>Just called in.11:12
<barmintor>anusha: those are great, it's too bad confuence's fancy image popup thing gets in the way
<cbeer>anusha: i think you've done this before, but: futures5:/opt/ff-modeshape-prototype and mvn -Djetty.port=9999 clean jetty:run11:13
<anusha>cool, thanks.
<cbeer>anusha: e.g.:11:14
jmeter/bin/jmeter.sh -n -t plans/fedora.jmx -Jfedora_server=futures5 -Jfedora_port=9999 -Jfedora_context=rest -Jnumthreads=5 -Jloopcount=1000 -Jfilesize_mean=1024 -Jfilesize_stddev=1 -Jtest_label=modeshape
<ajs6f>cbeer: which JMeter script is it? Aggregatiion, Read1, or Read2?11:16
<anusha>Thanks a lot. I'll start those tests now.11:18
<cbeer>ajs6f: you're on the call now too, right?
Just about to start running the high-volme test locally (on my owrkstatnio)
<cbeer>i'd argue that ticket isn't a chore.11:22
(that = https://www.pivotaltracker.com/story/show/43498953)
and i've assumed we'll restock toward the end of the week.. i can keep filing it out (especially after looking up the per diem rate for chicago.)11:32
<JasonDGI>w00t! registration!
<eddies>so weird having this call in the morning with the sun out11:35
<ajs6f>It'll be a real problem for all the vampire devs at code4lib.11:36
<eddies>mbklein totally came through and scored 2 projectors for us
don't forget to bring you video adapter thingamabobs11:37
<ajs6f>Did he get the polarized glasses for 3D?
<ajs6f>cbeer: running the stress test now. Looking good at 600 objects. Memory usage holding steady at just under 700MB. CPU not even breathing hard.11:39
cbeer; I'm going to start going back through the prototype and changing strings like "nt:file" to JcrConstants.NT_FILE and so forth.11:40
<eddies>anyone else want access to the fedora house registration spreadsheet, give me an email for google apps you want me to use11:53
<eddies>got you already
<ajs6f>Yeah, but I want to look at it from several different points of view.
<eddies>ajs6f or cbeer: either of you available right now to give me a quick walkthrough of ff-modeshape-prototype?11:56
<ajs6f>I am.
You want to get on the phone?
<eddies>i'll call you
<cbeer>ajs6f: thanks. i'm getting around to go to the office soon
<eddies>oh, you're not online, says skype
<ajs6f>ive me a sec.11:57
Chris— stress test doing okay at 1200 objects. A few exceptions have been thrown— not sure how serious they.
<cbeer>ajs6f: something about not being able to convert some kind of object to something else?11:58
<eddies>btw, it's *really* cold in chicago11:59
<ajs6f>Yeah, that's coming up from the binary GC, looks like. I was thinking more about:
This session tried to save changes to node with key 'Cannot locate child node: 87a0a8c6ae8993/ within parent: null', but it was removed by another session.
That looks ugly
Eddie— I'm on Skype.12:00
* anusha leaves
<ajs6f>Lost ya, Eddie.12:01
<eddies>skype claims you are offline12:02
<cbeer>hm. that's new to me.12:05
<JasonDGI>is the phone #'s wiki page posted yet?12:21
* jonathangee leaves12:42
<ajs6f>cbeer— made it to 2834 objects and still looking okay. Might be slowing down a little… hard to say.13:10
Guess the logs will tell.
AHHHH! I accidentlliy stopped the stress test. Well, it got well into the thousands with no noticeable slowdown.13:49
I guess I'll try running it for.
* jonathangee joins13:54
<cbeer>ajs6f, barmintor: what JAVA_OPTs are the cool kids using to tune java garbage collection these days?14:02
<barmintor>cbeer: unless you know the operating parameters really well, I think you're better off using the adaptive strategy14:03
Althought I do use class unloading for the prototype.
<eddies>barmintor: i'm going to try and pick up the stuff from northwestern this afternoon, to save us some time tomorrow14:09
are you still thinking of hitting 3floyds first thing?
<barmintor>eddies: yeah, I think it would be good to try and make that trip while people are at work. I'm open to feedbac, though.14:10
<eddies>well, i have to give the house owners a time for checkin
and plus jason is arriving into o'hare at 10:30am
so i don't want to leave him hanging for too long either14:11
<JasonDGI>w00t! Jason!
<eddies>ok. maybe we leave him hanging for a little while ;-)
<JasonDGI>oh, :(
<eddies>it's a bit of a trek to get to 3floyds
<barmintor>eddies: earlier checkin is fine w/ me14:12
<eddies>if we do that first thing, i think i can make my call w/ sg
but it would be really tight
<barmintor>as long as I have tim to pick you up
<eddies>that we can do
<barmintor>cbeer: -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+UseAdaptiveSizePolicy
<eddies>ok, how about i see how early we can check in
<barmintor>cbeer: those are all the "do what you think is best, JVM" switches
<cbeer>ajs6f: ok, i'm seeing these errors locally (too?): 'Cannot locate child node: 87a0a8c317f1e7jcr:versionStorage within parent: 87a0a8c317f1e7jcr:system'14:13
<eddies>and as long as i get the northwestern stuff today, we only have 3floyds and groceries to deal with after checkin
<ajs6f>Do they correspond to a faile request?
<eddies>barmintor: you arrive at 8am?14:14
<cbeer>ajs6f: yes.
<ajs6f>Uh oh.
<barmintor>eddies: 8:09, I think
<cbeer>ajs6f: of course, i wasn't seeing that on futures5, so... i don't know if it's a real issue14:15
<ajs6f>But then futures5 just couldn't run the test, right?
<cbeer>i don't know. and it looks like this was isolated to just one node, now that i look at it
so. hm.14:16
* github-ff joins
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/wzwVzQ
ff-modeshape-prototype/master dfc5ed9 ajs6f: Code cleanup in FedoaOXML
* github-ff leaves
<eddies>amazon just delivered our coffee grinder, scale and chemex14:20
and kettle14:21
<cbeer>ajs6f: ah, i just found where temp files are getting deleted (supposedly)14:23
oh, but that was in the filebinarystore
<ajs6f>Isn't that what we are now using?14:25
<cbeer>no, you set it to use the cache
(which i believe means infinispan)
<ajs6f>Oh, right.
<cbeer>ok, so maybe that is a bug in InfinispanBinaryStore14:26
i've almost posted in the forum
<ajs6f>They may just let them accumulate.14:29
<cbeer>that's what i'm thinking. but maybe there's something buried in the callstack that should be cleaning up the file
(not that I believe that, given the evidence)14:30
ajs6f: hm. it does look like something is accumulating in the PS Old Gen with these objects14:35
i wonder if it'll get cleaned up.. we'll see
<ajs6f>What kinds of things? (What types do they have?)
afk brb14:38
<cbeer>I think it's ChildReferences? I think other things are getting GCed14:40
* ajs6f1 joins14:42
<cbeer>i bet there's was a more controlled way to find out. like adding fewer nodes and seeing what accumulates. oh well14:43
* ajs6f leaves
<cbeer>i realized the way i was trying to gather that data was stupid. going to be a little more controlled this time14:46
<ajs6f1>The old gen objects?14:48
<cbeer>yeah. i was trying to collect it in the middle of running a lot of objects through there
* github-ff joins14:49
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/NgT9BQ
ff-modeshape-prototype/master 29714bf ajs6f: Added comment explaining blocking/nonblocking behavior of JCR binary values
* github-ff leaves
* eddies leaves14:50
* eddies joins
* eddies leaves
* eddies joins
* github-ff joins14:55
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/LB2sdw
ff-modeshape-prototype/master 4922f6c ajs6f: Changing logging fully to slf4j
* github-ff leaves
<ajs6f1>Chris— I noticed that some of the JAXRS methods weren't closing their sessions when finished.
I've pushed changes to fix that. That might have been what was causing objects to live too long. Sessions that should have gone away but didnt.
You might want to pull before continuing to test.14:56
<cbeer>ok, will do
or i'll run out of PermGen space.14:59
* github-ff joins15:01
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/103jtw
ff-modeshape-prototype/master f1f0453 ajs6f: Added class-level comment explaining rationale for Namespaces
* github-ff leaves
* JasonDGI leaves15:02
* ajs6f1 leaves
* ajs6f joins15:04
<cbeer>yeah, that might have done it. i'll try again after lunch15:17
<ajs6f>Woo-hoo! I fixed my own stupid bug!15:18
<cbeer>ajs6f: is enunciate (http://enunciate.codehaus.org/) something we should look at/use?15:21
* github-ff joins15:22
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/ZLvy7g
ff-modeshape-prototype/master 44e64b0 ajs6f: Added comments to Spring files to help make clear what beans are for
* github-ff leaves
<ajs6f>Looks very cool— Eddie and I were talking earlier today about clients and API documentation and how we can use them to engage the dev community.15:23
You want me to try adding it to see what happens?
<cbeer>ajs6f: nah, i'll give it a go.
I'm still convinced that the best single thing we can do is to have really good, really useful HTML responses in our API.
That could provide documentation right next to the code, and be a simple admin client as well.15:24
<ajs6f>Be of the Web, not in it.15:30
* github-ff joins15:31
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/2lRJmQ
ff-modeshape-prototype/master 20fbc5e ajs6f: Using JcrConstants instead of literal strings
* github-ff leaves
<ajs6f>Looks like at least 27 people are coming on Monday. I hope you guys are very successful with your booze procurement activities.15:34
Chris— we're overriding the managed version of JGrous. IS there a specific behavior we're trying to get by doing that?15:35
Or a bug that got fixed?
<cbeer>ajs6f: yes, i think so. or i pulled it along for the ride when i updated infinispan15:36
<ajs6f>Okay. No prob. I'm always trying to kill these Eclipse warnings. I hate the color yellow.15:37
<cbeer>yeah, you certainly can't downgrade now anyway. the jgroups.xml is set up for that newer version
<ajs6f>We're married to XML!
<cbeer>so, enunciate has a pretty crazy mode where it injects itself into your web.xml15:44
i'm not going to do that.
* github-ff joins15:45
[ff-modeshape-prototype] cbeer pushed 1 new commit to master: http://git.io/K9H1VA
ff-modeshape-prototype/master f33ba64 Chris Beer: add mvn plugin to generate web API docs
* github-ff leaves
* github-ff joins15:59
[ff-modeshape-prototype] cbeer pushed 1 new commit to master: http://git.io/Ud0kag
ff-modeshape-prototype/master 7cae6cf Chris Beer: datastream docs
* github-ff leaves
<cbeer>i wonder if we should submit/are submitting a general track OR session.16:26
* ajs6f leaves16:28
* avessey leaves16:35
* ajs6f joins16:36
Good question.
* avessey joins
<ajs6f>I wonder if the steering group is aware of the deadline.
Actually, a FUG presentation would be a good thing.
But maybe we can just do it all with a pub night.
<cbeer>i'm sure we have some sway with the FUG people16:37
But out of courtesy, we should make plans to give them time to schedule and so forth.
They _might_ not all be people we know.16:38
<cbeer>i think Donald at UPEI is the FUG chair.
<ajs6f>Is he planning on a FF session?16:39
anyway, sounds like a good job for the SG.16:40
<ajs6f>+1 to that.
<jonathangee>whats up?
<cbeer>do you know if Mark or Donald have already pencilled in a FF presentation at OR?16:41
(idle curiousity while modeshape is doing its thing.)
<jonathangee>i would be surprised if they haven't, but no one has told me they have either
so what i'm really saying is i don't know :)16:42
if we want to give a presentation though I'm sure there is space
is the deadline creeping up?
<ajs6f>As Chris said, "Sounds like a job for the STEERING COMMITTEE!" {superhero music}
The dealine is this month.
* github-ff joins16:44
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/igN4qw
ff-modeshape-prototype/master ecfe133 ajs6f: Added more Javadocs
* github-ff leaves
<cbeer>hm. i think modeshape might be keeping NodeKeys around.
<ajs6f>Eeeeentersting. Shouldn't need to, I wouldn't think.
Are we seeing them pile up in old gen?16:45
<cbeer>oh, wait, a bunch just got GCed.
<ajs6f>Maybe we should go back to C. At least then if we waste memory it's _our_ fault.16:46
<cbeer>must be some kind of cache
<ajs6f>Could be that we're discovering that {gulp} MODE may be one of those codebases that is inordinately sensative to GC config at scale.
<cbeer>JCR is sure liberal in its object creation16:47
* avessey leaves
<cbeer>as long as it doesn't fall over, it's ok with me.
i wonder what version of lucene they're using.16:51
<ajs6f>mvn dependecies:tree
Or Eclipse maven plugin has a lovely gui that lets you browse the hierarchy and examine poms and things.16:52
<cbeer>oh, yeah. intellij has that too
Is that current?
<cbeer>ah, good. so we can pretend there's a performance boost going to lucene 4.x
Actuall, I bet one of the first questions will be: does it do Solr?16:53
<cbeer>answer: our index is not your index.16:54
<ajs6f>We better accompnay that with some really good beer.
* jonathangee leaves16:57
<barmintor>I think that answer will be accpetable as long as the framework for wiring in an indexing service is available16:59
& also we'll have some good beer
<ajs6f>Hey, we're publisling events, right? Just use GSearch!17:00
you're fired.17:01
<cbeer>we're going to need better beer.
<ajs6f>And we'll start with good beer. I have yet to be convinced that any will be let when I get there.
Oh, hell.
<barmintor>Actually, I look forward to your breakout session on using gsearch with the fedora-4 prototype17:02
<ajs6f>Where I break out in hives?
* github-ff joins17:21
[ff-modeshape-prototype] ajs6f pushed 2 new commits to master: http://git.io/7BRSzw
ff-modeshape-prototype/master 92bfb1e ajs6f: Even more Javadocs
ff-modeshape-prototype/master aef033b ajs6f: Rearranged imports and use of Guava Collections for cleaner code
* github-ff leaves
<ajs6f>That's about as much damage as I can do in a day. I'm heading out. Good luck, Ben and Eddie. And don't drink all the beer in a desperate attempt to stay warm!17:23
o and o
* ajs6f leaves
* barmintor leaves17:38
<cbeer>it's possible we are collecting JcrXaSessions.17:49
and ISPN is still holding onto a bunch of file references17:53
* eddies leaves21:26
* eddies joins21:29
* eddies leaves
* eddies joins
* github-ff joins21:35
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/vZ_qVw
ff-modeshape-prototype/master 2d01ebf ajs6f: Added a little more tests, because that's how we do what we do...
* github-ff leaves
* eddies leaves22:33