Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* jonathangee joins08:00
* JasonDGI joins08:06
* barmintor joins09:36
* eddies joins09:37
* eddies leaves
* eddies joins
* eddies_ joins09:38
* eddies leaves09:39
* eddies_ leaves09:40
* eddies joins09:41
<cbeer>ok, i ran a slew of fedora tests last night.. starting modeshape-using-fedora-api tests now. i'll try to get something up before standup09:58
* eddies leaves10:22
* eddies joins
* eddies leaves
* eddies joins
* anusha joins10:35
<cbeer>sorry, maven--, apparently.10:36
it doesn't respect Xmx, etc.. you have to use MAVEN_OPTS instead.10:37
* anusha leaves10:40
* anusha joins10:53
<eddies>Fedora 4: We can rebuild it. We have the technology. Better…stronger…faster.10:58
i wonder if the six million dollar man is too old a reference :-P10:59
jon and i will be a couple of minutes late, hes still in a meeting
<barmintor>Just got out of a meeting, let me know if I can answer any questions11:08
<cbeer>i forgot to mention the jmx stuff i did yesterday, but i sent an email to the list11:21
eddies: should we finish standup (if we're not already) before continuing this discussion?11:25
ok, got databank running on futures2.11:26
<JasonDGI>looks like i accidentally clicked finish on a ticket, i reverted it11:28
<cbeer>here's my fedora tests from the oxford vm: https://gist.github.com/465699511:33
<anusha>cbeer: Have you installed maven in futures3?11:34
<cbeer>anusha: yes.
<anusha>where have you installed maven? Can't find it in /usr/local11:35
<cbeer>anusha: /usr/bin/mvn11:40
it should be in your $PATH
<anusha>It isn't there11:41
<cbeer>on futures3?
cbeer@futures3:~/lily-1.3$ which mvn
- /usr/bin/mvn
<anusha>In my path I mean. I did find mvn in futures311:42
<cbeer>JasonDGI: i grabbed the lily.tar.gz and ran it successfully on futures3, so you should have all your dependencies in place.11:43
and ajs6f is looking into the ff-modeshape-prototype heap problem.11:44
<JasonDGI>did you want to do everything else too chris? :P11:47
<cbeer>nope. lily scares me.11:48
<barmintor>cbeer: don't be afraid of your freedom.11:49
<anusha>cbeer, eddies: I feel like I am just repeating what chris has done. He's already posted the fedora tests done in the Oxford VM11:55
<eddies>anusha, cbeer: where are those results? i seem to have missed them11:56
<anusha>Posted in here just a while ago by Chris. Copying it https://gist.github.com/465699511:57
<cbeer>sorry. serves me right for missing last week's standups.12:13
i thought fcrepo was safe, i must have missed the tracker issue
and ajs6f reports the modeshape issue is likely a configuration issue in infinispan, keeping stuff around in the heap too long12:14
<eddies>ok, i did a little wiki gardening: https://wiki.duraspace.org/display/FF/FF+Candidate+Test+Harness and https://wiki.duraspace.org/display/FF/Test+Results12:23
we should post results on the wiki
cbeer: in the gist anusha posted, what are the x & y-axes?12:28
<cbeer>y is request time, x is just an index value.
it's not an especially meaningful 2d graph.. i need to do better analysis in R.. like with errors bars and percentiles12:29
* anusha leaves12:47
<cbeer>huh. the modeshape/infinispan caching mechanism is pretty weird. makes sense, but weird.12:50
to be fair, i think we probably need to run sequential only-write and only-read cycles here12:51
and take that as a worst-case baseline.12:52
(although, iguess the multithread tests will be fair enough. i've kept the in-RAM cache pretty small)12:56
<eddies>cbeer: where are the irc logs for this channel?13:13
i see gluck.cul.columbia.edu referenced in https://www.pivotaltracker.com/story/show/42650461
but that just looks like a blacklight install13:14
<cbeer>eddies: that's the CNAME to use13:25
i'll add it as a vhost on gluck when you set it up13:26
<eddies>ah. k13:30
i'll have to request it from the duraspace folks
hey, what version of ubuntu are the oxford vms running again
(anusha or cbeer)
oh, no anusha
Ubuntu-Server 12.04.1 LTS _Precise Pangolin_
* jonathangee leaves13:32
<cbeer>ok, something's still wrong with our modeshape config. i wonder if we lost something between the modeshape rest api testing and now.
* jonathangee joins13:34
<cbeer>i'm going to make ajs6f fix it again, i think.
<eddies>cbeer: you said you put lily on future3, right?13:50
so does anything else differ from my email:
futures1: jmeter
futures2 & 3: modeshape
futures4 & 5: lily
futures6: fedora
<cbeer>eddies: hm. i didn't see that email
i agree with putting jmeter on an isolated instance that's used to only run jmeter13:51
but.. why split modeshape and lily like that? wouldn't it be better to have both of them on the other nodes, and just run a single benchmark at a time?
i can't imagine the scale-out from 1 to 2 nodes to be dramatic (and, might be harmful, with whatever coordination the nodes need to do)13:52
<eddies>so have 4 or 5 nodes for the multinode config?
we can just shut down lily/modeshape/fedora while the other tests are running13:54
<JasonDGI>cbeer, where did you put the lily zip?14:01
<cbeer>JasonDGI: just in my home dir, but you should probably just download it again. i just did it to see if it'd run14:02
<eddies>ok, i added: https://wiki.duraspace.org/display/FF/Test+Platform
cbeer: can you just update the vm allocations to reflect reality (or the aspirational reality)
and we should indicate, at least in lily's case, which is "master" vs "slave"
ok. i'd vote vm614:03
modeshape doesn't care (except that it needs a db host somewhere in JDBC configuration.. so i guess that could go in the same place)
but first, to the office.14:04
* jonathangee leaves
<JasonDGI>so eddie, we have sudo access if you need your password fixed14:10
<eddies>no i'm all set14:11
<cbeer>ok, there's a shared group on futures1-6 called "futures"14:56
which is now our primary group, and i've set the umask to 002. who needs privacy anyawy.14:57
JasonDGI: do you know where the islandora/hierarchical model stuff Tom mentioned in his email is?15:10
i don't remember reading it15:11
<JasonDGI>i dont know, ill ask jon
jon is afk, ill have to get back to you
<cbeer>np. i figure i should know what was said before telling Tom it isn't a big deal15:12
i don't think we'd force our internal hierarchy on implementors.. but maybe we said we could
<barmintor>also a difference in organizational hierarchy for storage and exclusive hierarchy of type metadata15:17
<cbeer>ajs6f++ much more elegantly said that I would have.15:18
* ajs6f joins15:25
Chris—on this ordering-in-the-JMeter-script question, have you checked the responses? Are you getting the right ordering for operations?15:26
All— Ross Wayland says hi!15:27
<cbeer>ajs6f: yes, i had successes (except after modeshape blew up.)
<ajs6f>ModeShape blowing up meaning the problems with heap size?
Solved by eviction...
<cbeer>i was seeing some bad multithreading behavior (that i thought we had also fixed before), but didn't dig into it yet15:28
<ajs6f>That's really weird. I wonder if we're using different versions of JMeter or something...
Different semantics for ordering...
<cbeer>i'm using the one you bundled into the project15:29
<ajs6f>I suspect that the bad multithreading behavior is coming from JCR Session scope.
IOW, that different threads are hitting the same node and trying to save changes to it in a way that JCR doesn't like.
<cbeer>i don't think that should be the case. each node is actually changed in its own thread15:30
* JasonDGI leaves
<cbeer>but, like i said, i need to dig into it again
Are you seeing things like (follows)? Because that's the kind of thing I'm seeing.15:31
javax.jcr.InvalidItemStateException: This session tried to save changes to node with key 'Cannot locate child node: 87a0a8c6ae89939457b532-bd18-41c5-83be-bb2a074be56a within parent: 87a0a8c6ae89933640f3c2-bfe0-44dc-95bf-e2a2310cdc07', but it was removed by another session.
<ajs6f>Maybe threads are killing nodes faster than other threads can finish saving and reading datastreams.
<cbeer>but we fixed that with a locking change before
<ajs6f>Yeah, turning on PESSIMISTIC locking.
<cbeer>anyway, lunch. and then i'll look
<cbeer> or, if you want.. the jmeter stuff is on futures1:/data/ff-jmeter-madness15:32
<ajs6f>yeah, I should check to see if I can log into those machines, anyway. I'll poke around a bit.
<cbeer>and i'll move the modeshape prototype out of my home folder
<ajs6f>{grin} Okay.
How about /opt/ffutures/[different prototypes]15:33
Gosh, I wonder if we have to think about user-managed transactions…? That would stink.15:34
I had really hoped that JCR Session's lightweight transaction semantics would be enough.
<cbeer> - /opt is fine with me.
<ajs6f>Okay, I'll look for it there.
* ajs6f leaves16:15
* ajs6f joins16:17
<cbeer>ajs6f: that's the springification branch?16:24
<ajs6f>Yeah, yeah. But you know hat?
I think I've got a lead.16:25
I as just looking at the two-thread exmple.
It blows out, but with a lot of:
javax.jcr.nodetype.ConstraintViolationException: The mandatory property named 'jcr:data' defined in type 'nt:resource' is missing from the node at '/test:7/DS_5/jcr:content'
at org.modeshape.jcr.JcrSession$JcrPreSave.process(JcrSession.java:1706)
at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:501)
at org.modeshape.jcr.JcrSession.save(JcrSession.java:935)
at org.fcrepo.ffmodeshapeprototype.FedoraDatastreams.addDatastreamNode(FedoraDatastreams.java:147)
at org.fcrepo.ffmodeshapeprototype.FedoraDatastreams.addDatastream(FedoraDatastreams.java:81)
See where it dies in our coe?
final Node ds = jcrTools.uploadFile(session, dspath, requestBodyStream)
and I'm pretty sure that16:26
uploadFile _doesn't_ bock on the completeion of the upload.
Not a tasty German beer.
I think we've got threads smashing themselves.16:27
I'm goignt o try with a blocking implementation.
<cbeer>barmintor: can i get you to harass some columbia people for me (for Declan, too)? it'd be nice if this 2 month old thread got some attention: https://groups.google.com/forum/?fromgroups=#!topic/blacklight-development/KLvGnvUixz016:31
* github-ff joins
[ff-modeshape-prototype] cbeer pushed 1 new commit to master: http://git.io/k2BEaQ
ff-modeshape-prototype/master bf776e3 Chris Beer: Merge pull request #22 from futures/Springification...
* github-ff leaves
<barmintor>cbeer: let me walk down the hall and see who's there16:32
ok, I just laid a big guilt trip on pur junior dev. Hope you're happy!16:37
<ajs6f>Chris— one note. I just realized the Springificaton branch lost the insertion of the "test" namespace, so to run the JMEter script, you've got to add it manuall. I16:38
'll put that back in
with a quick commit
* github-ff joins16:51
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/OEE5tw
ff-modeshape-prototype/master 9b03a68 ajs6f: Added 'test' namespace back in
* github-ff leaves
<cbeer>ajs6f: ok. so, should i wait for a blocking impl of uploadfile?16:53
<ajs6f>I'm looking around right now to understand what the behavior of streaming values vs. JCR Session is.
<cbeer>ok, i'll wait. i'll update the wiki page about the test stuff in the meantime16:54
<ajs6f>It seems like setting a property to a stream begins the sttream and returns.
<cbeer>and get back to running some performance tests on the vms
(performance of the hardware tests)
<ajs6f>Well, I committed the addition of the test namespace back, so that's okay.
You can run the test against it immedaitely.
Yeah, definitely look at the ardware— I've got to reason through this thing so we're sure about what we've got in hand.16:55
<cbeer>running tests against futures2.17:10
<ajs6f>I'm heading out. I made a little progres ith this. I'm more convinced now that we have a problem with reentrant threads bashing each others Sessions.17:17
I'll take a look at it again tomorrow when I'm fresh.
Hae good night, all.
* barmintor leaves17:21
* ajs6f leaves17:24
<cbeer>thanks adam.17:25