Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* eddies leaves03:55
* eddies joins03:57
* eddies leaves
* eddies joins
* anusha joins05:19
* eddies1 joins07:09
* eddies leaves
* eddies1 leaves
* eddies1 joins
* JasonDGI joins07:46
* anusha leaves08:01
* anusha joins08:14
<eddies1>morning/evening all10:00
<eddies1>jasondgi & anusha, are you dialing in?10:02
<anusha>eddies1: can you give send me the skype contact again. Sorry!
<JasonDGI>are we doing this early today?
<cbeer>that's what we said
* ajs6f joins
Can't get in on Skype. Be there in a minute.10:03
<eddies1>you should have received a calendar invite
<ajs6f>I did— what's that got to do with it?
<cbeer>ajs6f: earlier conversation.
<cbeer>ajs6f: also, thanks for trying to clarify my ramblings.
<ajs6f>Hardly needed.
* barmintor joins10:05
* anusha leaves10:07
* jonathangee joins10:12
can someone paste that test result link again?
* anusha joins10:20
<cbeer>anyone else getting awful echo?10:27
<barmintor>I am getting that awful echo
<cbeer>it makes my head hurt this early in the morning.10:28
<ajs6f>I'm only getting it from Jonathan. Maybe he's saying everything twise?10:30
<jonathangee>i muted. did that help?10:31
<cbeer>jmeter-- # yeah, i guess i'm seeing the same behavior you were seeing, ajs6f. fire-and-forget rest calls, maybe?10:32
<cbeer>ajs6f: or maybe jmeter is fine, but modeshape returns before everything gets settled internally (more modeshape config?)10:34
<barmintor>That seals it: Cassandra and node.js10:36
<barmintor>Apparently, you can configure Infinispan to run over Cassandra.10:42
<cbeer>barmintor: yeah. infinispan is pretty cool10:43
we could also create a bunch of objects, (SLEEP) then a bunch of datastreams, (SLEEP) and then read them..
so we can get some data some how.10:44
<cbeer>anusha: sorry, you said you liked/understood 26085 over 4656995?10:48
<anusha>Cbeer: I liked 2608510:49
<cbeer>ok. the reason that's misleading is it combines all the types of requests. really, we should summarize the data rather than plotting it all10:50
<anusha>I agree
<cbeer>but i've been too lazy to figure out the right way to do that
(in R)
<anusha>the only issue I have with the 2nd graph is the x axis
<cbeer>ah. don't like the log scale?
hm. and they're uneven.10:51
<anusha>ah, what does 1.35 e +12 represent
<cbeer>oh, sorry. i read 'y'.10:52
that's just the labels I chopped off from the other graph :P
they're irrelevant.
<anusha>oh, in that case, the 2nd graph is more meaningful10:53
<barmintor>eddies: that's awesome.10:55
<cbeer>ajs6f: should we throw away ff-modeshape-prototype for the next sprint (and do your OSGi/Spring stuff right?)
or is it easier to just adapt what we have?
<barmintor>eddies: also, I'm bringing some pretty serious door prizes.
<ajs6f>Mm. Good qquestion.
Maybe we should let everyone take a look at what's there, in order to get comfortable with ModeShape's API.
Then we can have a discussion about modularization.10:56
<cbeer>whew. figured out boxplots.11:00
<JasonDGI>is futures1 configured and ready to run jmeter?11:02
<cbeer>JasonDGI: yes. let me check the wiki page before I tell you to RTFM.11:03
i think i updated it.11:04
JasonDGI: yup; at the bottom of https://wiki.duraspace.org/display/FF/Test+Platform
you'll have to manually tweak the jmx test so all the right things are active.. or refactor it like i did the fedora stuff
or reply to my email and say "sounds great; want to take a pass at the others?"11:05
ok, added fedora 1 - 20 thread boxplots to https://gist.github.com/465699511:16
it'd be nice to pivot those so all the common operations were graphed together. maybe later.
<anusha>cbeer: Great looking graphs. Is it possible to mark the key events on the y axis (min, lower quartile, median, upper quartile, max), so it makes it easier to compare. You are happy to ignore me.11:24
* anusha leaves11:29
<eddies1>barmintor: is barrier near you?11:35
<barmintor>eddies1: they're off the Long Island Rail11:36
it's about an hour from Columbia
<eddies1>they can't sell outside NY11:38
so we'd have to get it from them in NY
<barmintor>what is "it"?11:39
<eddies1>maybe you knew this, i didn't
sorry. their beer
<barmintor>yes, but in what vessel is this beer? I can easily buy it in NY, but getting it to Chicago is the tricky bit11:40
* eddies1 leaves11:41
* eddies joins
* eddies leaves
* eddies joins
* ajs6f leaves11:42
* ajs6f joins11:47
<barmintor>JasonDGI: I think I could scrape up a catapult, but it'd be suited to somewhat shorter distances :P11:53
<JasonDGI>i could try to dig up an old rocket engine if that would help11:54
<ajs6f>I just came back into the room. What on earth are you people planning!?
<barmintor>a smuggling operation
<ajs6f>Now _that's_ creative funding!11:55
* avessey joins11:59
<eddies>barmintor: so is it worth pursuing barrier (from a sponsorship/give-us-some-beer-for-free)?12:13
<JasonDGI>will they accept a "get us drunk" plea?12:14
<eddies>it's more a logistics question for how ben would transport to chicago
i won't pursue the sponsorship if we don't have a sensible means of getting the product to chicago12:15
<barmintor>eddies: exactly. as far as I know, they're a keg-only operation. I talked to one of the brewers out there about getting it out there, and he described shipping pony kegs to his family in some hazy way that I didn't understand. It's probably best to drop it unless thy can advise us on how to transport.12:16
(I should at least observe that his family actually lives in Chicago- it wasn't *totally* crazy)12:17
<ajs6f>Okay, Chris/Eddie— I think I've got a really good clue for this MS concurrency weirdness.12:19
I went ahead and "synchonized" all of the mutating methods.12:20
And I was still getting (many fewer, but still some) weird exceptions.
But here's the clue. The exceptions show up like this:The values for the '{http://www.jcp.org/jcr/1.0}content' property on node '/test:15[2]' no longer satisfy the type and/or constraints on the 'jcr:content' property definition on the 'nt:file' node type definition12:21
See that node path? See the '[2]' part?
<cbeer>i suspect it's time for another post to their forum. '
<ajs6f>It's like XPath. That means that there are _two_ nodes named test:15 and this is second one.
Now I don't know that the JMeter script is doing that.12:22
I'm thinking we are double-creating nodes in the app.
Meaning ingest() doesn't check to see that there isn't a pid of that name already.12:23
Whicih is doesn't.
I'm going to fix that now, which we would have wanted to fix anyway.
<JasonDGI>so i ran the jmeter test for lily on oxford and it finished in about 10 seconds an reported 4000 lines of successes, is that what i should expect?12:26
<cbeer>that seems.. questionably.. fast to me12:27
<eddies>cbeer: sorry, i'm tied up on a call
im gonna run it locally and see if i get the same results
<cbeer>JasonDGI: if you get me the steps you ran, i'll take a look
eddies: np. i'm going to head to the office now
i'll keep playing with the fedora graphs
<JasonDGI>cbeer: i used the link you sent, copied the jmx file and removed all non-lily components12:29
<cbeer>JasonDGI: how do i start lily, where is it running, what port, etc?12:31
all i've done with lily before is run the.. launch-test-lily or whatever it is
<JasonDGI>futures3, i have it running in a screen called lily
default ports and such12:32
<cbeer>k. i'll take a look in the office.
<JasonDGI>yeah just sudo ./lauch-test-lily.sh
<ajs6f>Chris— Steve Villereal, who is coming to code4lib and looking forward to the Hydra sessions, is wondering if your demo (Hydra demo?) will be fixed before then, or whether he should look for some other learning resource?13:05
* github-ff joins13:06
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/RqYaCQ
ff-modeshape-prototype/master a1b16fd ajs6f: Massively improved logging
* github-ff leaves
<cbeer>ajs6f: no idea. hindman and others took it and broke it at our last meeting.13:18
sorry, not hindman.13:19
asking in projecthydra13:20
<cbeer>needless to say, i've lost interest in the whole thing.
<ajs6f>That's cool. I was just doing Steve a favor by exercising my ability to contact the most powerful people in the world with a mere IRC ping.
I'll tell him not to wait for it.
<cbeer>yeah. i'm giving up harassing people in #projecthydra.. lost interest in that too. I'd have Steve send hydra-tech an email about it13:25
<ajs6f>This is insteresting. I put a check in the addDatastream() method to see that the parent object exists before agreeing to create the datastream. A large number of JMeter requests are returning 406 becaues of that. In other words, a significant number of threads are asking to create datastreams for objects that haven't yet finished creating.
Now I don't know why Fedora Classic didn't show the same behavior.
I think you' were right, Chris. We've got REST methods returning before the mutations they induce are completely settled.13:26
I'll tell him that.
<cbeer>ajs6f: yeah. i don't remember seeing that behavior before. but maybe the addition of clustering and whatnot did something bad13:27
ok. i think i'm done with my calls for now
cbeer, quick check-in on parsing/data viz?
<eddies>and then i can switch over to modeshape troubleshooting w/ ajs6f
<cbeer>eddies: on IRC? i'd like to be too lazy to move somewhere quiet
<cbeer>or just work with ajs6f
and i'll keep chugging along
(or, start chugging along. got distracted in office)13:29
<ajs6f>Eddie— scroll back a page and you'll see where MS is. We have two choices: figure out how to prevent REST methods from returning before the action against the repo is absolutely complete (which we probably want to do anyway), or13:30
<eddies>ok, i don't know what results reporting options you're weighing…as soon as you have some options to share, let me know
<ajs6f>turn off the nodetype-based validation, which will let ModeShape fly as fast as it can, with no net.
To mix metaphors.
<eddies>ajs6f: is that behavior supposed to be a feature of modeshape?13:31
<cbeer>eddies: latest stab is these box + whisker graphs: https://gist.github.com/465699513:32
(well, not the latest. but the latest i've uploaded)
<eddies>(i was referring to returning before state is updated, not nodetype val)
<cbeer>eddies: i really want to figure out how to pivot our data so we can compare e.g. fedora 1,5,10,20 threads for the different operations13:34
<eddies>ack. i just realized i didn't clarify for the testing…what fixture set are folks using?13:35
<cbeer>eddies: those fedora tests are over the jmeter-generated fixtures (1Mish files)
which i think is better than our other fixture sets for this performance analysis than the real-world files13:36
<eddies>we also need to run against at least the reduced binary data set
<cbeer>and faster than fasseg's generator
eddies: what's that?
(i should include a histogram of file sizes, i guess)13:37
<eddies>it's frank's smaller version (7GB) of his dataset
or at least under 8GB
<cbeer>eddies: i think the jmeter-generated dataset as good or better than that dataset. at least, it puts all the tuning levers into the test run.
<ajs6f>Eddie— I'd like to turn in something ModeShapey that can run the multithreaded tests today. Are you okay with me temporarily turning off the validation (the JCR schema)?
<eddies>how does turning off the validation help the concurrency problem?13:38
reads will still fail
<ajs6f>No, that part is okay. Now the problem is different. You got to stop going to meetings.13:39
The problem is as Chris previously suggested. REST methods return faster than the changes for which they are reponsible propagate.
So threads are trying to create datastreams for objects that don't fully exist yet.13:40
* jonathangee leaves
<ajs6f>But that only matters if MS validates the nodes
That's why we didn't see this against the MS HTTP API.
No validation there.
<cbeer>oh, so the changes aren't propagated until the object is fully validated?
<ajs6f>If the object doesn't validate, the changes don't propagate at all! So what happens is this:13:41
Thread1: Create object begin
Thread2: Put datastream into object begin
Thread1:try to finish object create
* jonathangee joins
<ajs6f>But object is now invalid. When it had no datastreams, it was valid. But now it has half a datastream.13:42
<eddies>isn't that just a borked test then?
<ajs6f>Well, I agree that the tests should isolate actions, but they don't seem to.
<ajs6f>If the same thread was the only thread that acted against a particular object, the problem should disappear.
I admit I'm not much of a JMeter meister, so it's not obvious to me how to do that.
I think you chave to somehow link threadgroups and for-each loops.13:44
<cbeer>within jmeter? i thought each threadgroup would run in-order and run ${numthreads} ${numloops} times.
but i don't get jmeter.
<ajs6f>Join the club. {grin}13:45
<cbeer>and.. wouldn't we see that same behavior against fedora?
<ajs6f>My logs clearly show different threads trying to add datastreams to the same object.
<cbeer>(if it was the jmeter tests)
<ajs6f>I don't know.
<eddies>there's a setting to run thread groups consecutively
<ajs6f>I don't understand it perfectly myself.
(i.e. in serial rather than parallel). but that's not what's going on here, right? this is all happening within a single thread group13:46
<ajs6f>I believe so.
thread group
- add the object
- for 1:10
- add a datastream
<JasonDGI>so i think i have the csv results for lily, is there a post-process script or something?
<cbeer>JasonDGI: upload it somewhere and i'll do something with it13:48
<JasonDGI>its on futures1, did you still want me to send it to you?
<cbeer>futures1 is fine. where is it?
in log?
<JasonDGI>no, it dumped stright into /opt/ff-jmeter-madness for some reason
<cbeer>oh, right. ./log is something i did for fedora.jmx
<ajs6f>I've got to break for a few minutes to get to my office and plug the laptop in. Be back in five.13:49
<eddies>adam: unless someone has a better idea, turning off validation seems like our best option at the moment
<JasonDGI>ill probably update that in my lily.jmx
* ajs6f leaves
<eddies>cbeer: the jmeter fixture data we were talking about earlier, you're referring to the govdocs1 data?
<cbeer>ok. i'm also happy to loo kat it
eddies: no.
eddies: read my email from 2 days ago13:50
Jmeter tests for performing reads
* anusha joins
<cbeer>from.. 28/Jan your time, I think.
or, I guess the relevant part isn't long:
<eddies>yes. i have the old test plan opened13:51
<cbeer> "- generate fixture data within the jmeter test itself, using the seed, filesize_mean and filesize_stddev, it'll generate a Gaussian distribution of filesizes around the filesize_mean."
yeah, this is only in fedora.jmx, because i also said
"If this makes sense to people, I'm happy to start converting the other tests. "
* ajs6f joins
<cbeer>JasonDGI: the lily results don't make sense to me.. i see 4 results:13:52
"Add blob and record"
"BSF Sampler" (???)_
"NOOP Sampler"
and "NOOP Sampler"
<JasonDGI>what should it be?13:53
<cbeer>well, with Fedora, we have 5 things.. something like create object, create datastream, delete object, read datastream, and ...
maybe i just don't understand the lily stuff (this was something about multipart uploads, and maybe that screws things up?)
yeah, let me look at lily.jmx, i guess.13:54
i think i just don't know what i don't know.
<JasonDGI>i quickly hacked it together based on what i understood in fedora.jmx13:55
<cbeer>k, and haven't committed it anywhere?
<JasonDGI>not yet
i can
<JasonDGI>should i be able to commit from futures1? im getting errors13:57
<cbeer>probably not. i think i just did a read-only checkout there13:58
<JasonDGI>k thanks
<cbeer>we could change that.. but then you'd have to forward keys and all that
<JasonDGI>ill do it locally
k, pushed14:01
i'm going to make a mess of it and add all the fancy stuff i put in fedora.jmx14:02
(what's the distinction between distinct records and versioned, same record?)
<JasonDGI>someone else wrote the tests, im not sure14:07
<eddies>elschlomo wrote them14:09
<cbeer>jmeter--.. all i want to do is copy steps from one window to the other...
oh. maybe merge?14:10
elschlomo: ping?14:12
i really don't get these tests. i guess it's because lily is weird, but gah.14:13
<eddies>i think versioned just creates adds each blob as a new "datastream" version14:14
<cbeer>ok, of the same datastream name though.
<eddies>whereas the other creates a new "datastream" per blob
i think
<cbeer>ok. i guess that matches what i've pieced together
so this is different than the other testing
where we're creating N datastreams on M objects
is that just how lily is? or ...?14:15
<eddies>pinging frank on gtalk14:16
i'm not really qualified to answer
says he'll be on in a min
here's my understanding of what's going on (and fasseg should correct me):14:17
- we set up a data structure in lily (a record?) which is just a label and a binary blob
- we post a binary blob (resource?) to lily and get a response back
- we take that response and create (the record?) by initializing it with a label and that response.14:18
and lily is taking care of generating internal identifiers for the resource and the recor
<fasseg>hola guys...
<cbeer>hey fasseg. lily makes my head hurt.14:19
so, what we're really testing in the lily tests is datastream upload. we don't have a concept for a fedora-object-analogue
<fasseg>quite right what you just posted before i came on...14:20
well theres the record-thingie...which is kind of an object
with a real simple model of one field and a blob14:21
<cbeer>but it's just a label + data tuple in this case?
we were talking about creating something more realistic, but chose not to in one of the calls
or just pushed it back?14:22
<cbeer>ok. i was expecting something more realistic, i guess. and i was looking through the lily docs to try to figure out how you'd even do that
it doesn't look like fields can be multivalued?
but you can have a LIST of <LINKS>14:23
<fasseg>dont think so, although you could but a xml blob in there and treat it as properties or sth
<cbeer>which can link t orecords?
<fasseg>I think so yes
<cbeer>ok. i'm going to try to construct something more realistic based on that.
at least we'll have an apples-to-pears comparison that way.
<fasseg>but you can easilz copzy the "create field handler
and create some more fields to put more strain on hbase...
<cbeer>oh, and the other thing.. we have to use beanshell to do the resource uploads because the default jmeter one doesn't do something right, correct?
<fasseg>and solr i guess
<cbeer>ha ha! i actually understand something
i guess that's a point in lily's favor.14:25
<fasseg>no way to post the file contents as a raw body from a path
only multipart
so beanshell uses httpclient to post the filecontents..
<cbeer>cool. that's fine with me.14:27
so i can create a bunch of Records that way, capture their IDs somewhere, and add them to the "object" analogue
i'm less opposed to the lily api now that I understand what it's actually for.14:28
thanks fasseg. i'll give this a go, and send an email to the list if i either get anywhere or fail miserably.
<fasseg>well the records are the "objects" but the create record sampler initializes a record model which is used later14:29
so you can have N models
* anusha leaves
<cbeer>and, i'm going to just spend my time until lunch on it.
<fasseg>okay have fun then... ping me on gtalk if you need me..
<cbeer>ah, cool use of post-processors to allow the 409 conflicts to be ok.14:30
i'll have to remember that one
<ajs6f>Eddie (and anyone else interested). I got it. There's a subtle deal going on with JCR Sessions that makes them different if you get them from the Workspace or from the Repository. We were getting ours form the Workspace. We were WRRRROOONNNNG! Our long natiional nightmare with ModeShape concurrency is over.
I still need to alter the Freemarker templates to take account of the fact that they now need copies of data and not references, but that's like ten minutes.14:31
<ajs6f>It now runs clean multithreaded.
I'll fix this Freemarker bs, push, and we can start testing at the Oxford Proving Grounds.14:32
<cbeer>+1. if you get that pushed, i'll run the whole suite of tests against fedora, modeshape and lily.. and databank too14:33
and then eddies can have data to point at and pretend means something.
<ajs6f>One other thing. The copy of the JMeter script I have (which may be way out of date) seems to be asked for datastreams via /objects/{pid}/datastreasm/{dsid}, not /objects/{pid}/datastreasm/{dsid}/content. That's why I noticed the Freemarker thing. Shouldn't it be asked for the content?
Anyway, I'll let you Jmeter honchos figure that out. I'm going back and fix this thing. Permanently.14:34
<cbeer>ajs6f: yes. you're right.
explains why read datastreams were so fast :)
<ajs6f>Ah, Fedora REST API. You magnificent bastard!
<cbeer>ajs6f: do you know where the fedora namespaces are documented?14:35
<ajs6f>You mean like the stuff that sprinkles the XML responses?
<cbeer>i'm going to adapt this jmeter/lily stuff to use names that i can keep track of
No. I'm afriad I don't
Eddie might.
<cbeer>ok. i'll just make something up
<ajs6f>Isn't that the Fedora way?14:36
<cbeer>man. if only lily was XML based
<ajs6f>Written in XForms, XQuery, XSLT, and X (the LA punk band).14:37
<JasonDGI>featuring X1114:39
<ajs6f>I take back what I said about fixing Freemarker first. There's the push.
* github-ff joins
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/cxAPIQ
ff-modeshape-prototype/master 239f9ef ajs6f: Brute force concurrency fix
* github-ff leaves
<ajs6f>The returns from the XML-based methods may not look quite right.
Let me know if we needthem for testing.
The reason I pushed is because I don't know that we even want to use Freemarker. If we are committed to the ModeShape prototype,14:40
<cbeer>doubt it. the tests are just CRUD.
<ajs6f>we should have a real discussion about templating, views, and so forth, and pick something everyone likes, not what I happened to pick off the ground.
Well, let m eknow if they're needed.
I'll fix any that are.
But for data CRUD, give it a go.14:41
I'm going to begin trying to desynchronize the mutating methods
for speed.
<cbeer> "namespace1": "prefix1",14:44
who does that?
* github-ff joins
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/zJS5mA
ff-modeshape-prototype/master 6dae019 ajs6f: Desynchronizing mutating methods
* github-ff leaves
<ajs6f>Sorry, Programmer1
That was the joke.
I desynchronized the mutating methods and everything is fine. Shold be a little faster.14:45
Also, when you run for time, Chris, you'll want to turn the logging down. It's very high, for obvios reasons.14:46
i don't understand it.
<ajs6f>I find that honest conversation (may over a soothing cup of tea) is the most important way to come to understand someone.15:01
<cbeer>so, when i try to interpolate a variable in the raw post body, all of a sudden the url it goes to changes
<ajs6f>You mean the head of the URL?
Oh, wait, I see.
Is it just a straight variable, or are you using any of he logging functions or that sort of thing?15:02
<cbeer>ajs6f: it wasn't. it was a counter.. i missed a closing }15:03
which really confuses jmeter apparently
<ajs6f>It's a gentle, simple soul.
<cbeer>ajs6f: hm. are variables in jmeter thread-local?15:09
(answer: yes)
<ajs6f>Sorry, was afk. I didn't know that anyway.15:12
* JasonDGI leaves15:26
* ajs6f leaves15:50
* ajs6f joins15:53
* jonathangee leaves16:10
* github-ff joins16:30
[ff-modeshape-prototype] ajs6f pushed 2 new commits to master: http://git.io/y4zBjQ
ff-modeshape-prototype/master 0e56efb ajs6f: Brought tests up to date with code changes for concurrency
ff-modeshape-prototype/master 6642b68 ajs6f: Added test for getDatastreamContent
* github-ff leaves
<cbeer>i'm almost at feature-parity in this jmeter test16:37
<ajs6f>Cool. Apples to apples, eh?
<cbeer>apples to pears.
<ajs6f>Let me know if the ModeShape prototype is lacking anything.
<cbeer>and that's as far as i'm going to take it
<ajs6f>Mm, I could go for a cool glass of perry.
<cbeer>i'm not going to even try to do reads
<ajs6f>The Lily prototype doesn't do reads?16:38
I can do that, if we're still working on the base I started...
<cbeer>ajs6f: this is against the lily api itself
<cbeer>i don't know. that's what i was given.16:39
<ajs6f>Okay. Well, I'm not going to go crazy on it. Haven't we technically chosen the platform anyway? {grin}
<cbeer>i think it'll be enough to suggest it isn't a good contender anyway... although I hear the SG was ok with our decision
You know what I think Lily would be very good for?16:40
A MARC store.
You know the qualities of your data there.
You can really take advantage of Lily's strengths.
I might even prototype that, if I get around to it.
<cbeer>huh. git submodules encode the original filesystem path in them16:41
s/root-relative/absolute path/
<ajs6f>Well, absolute _is_ realtive to root, right? {grin}
I didn't even known they existed. I'll have to read up on these guys.16:42
Cool. That's one of those places that centralized version control tends to fall down entirely.16:43
<cbeer>yeah. i give up. symlinks it is.
<ajs6f>What are you trying to "submodule in" to what?16:44
<cbeer>ajs6f: i had a project (ff-jmeter-madness) with submodules (hydra-jetty and fixtures). i checked it out to /data/ff-jmeter-madness originally.. and then moved it to /opt
and now git complains it can't find stuff in /data
i found two config references to /data16:45
but there must be some more hinding
<ajs6f>Ah, I see.
SO moving the repo broke it.
Not cool.
<ajs6f>I wonder why they'd use absolute paths. That doesn't seem to be very advantageous.
<cbeer>where does lily keep its data anyway?16:46
HBase, HDFS.
<cbeer>sorry, what directory on my filesystem should I blow away to get a completely clean slate?
or is that a stupid thing to want?
<ajs6f>I think part of the Lily "style" is that you shouldn't care, because you could be running over a huge grid.
Well, are you using the test script runner thing they offer?
<cbeer>uh. yeah. launch-test-lily16:47
<ajs6f>We could trace the HBase and Hadoop configs to find out where they put stuff. My guess would be in Java's temp directories, which vary OS to OS>
I think what you are secretly asking is, "Where do HBase and Hadoop put stuff when run with the Lily script?"16:48
<cbeer>ok. i'm not going to care. maybe it'll have stale data, maybe it won't.16:49
it's good enough for this benchmark anyway.
<ajs6f>Lily, you are such a jerk.
<cbeer>hm. it might be a /tmp directory16:50
<ajs6f>Yeah, that's where Java puts its temp directories on a POSIX platform.
You can find out with a quick test app.16:51
<cbeer>ok, i found it. it is in /tmp16:52
and presumably changes every startup anyway
<ajs6f>Yeah, I believe that is in the contract.
<ajs6f>Lily is heavy metal.16:53
It's clearly meant for large systems, distributed systems.
I think with it we're running into aconflict between our desire/need to move fast and Lily's desire to be very distributable.16:54
<cbeer>good thing we didn't choose it, then.16:55
ff-tech, continuing to not make stupid decisions.
<ajs6f>Well, it's the nature of our immediate project.
If this was a giant Mellon-funded deal like the 2.x effort...
it might make a lot of sense.
I'm heading out. If you run into anything where the ModeShape prototype needs help, send me an email. I'll get it later tonight. Otherwise, I feel like I've done enough damage for one day.16:57
<cbeer>ajs6f++ thanks for saving us again
<ajs6f>Hey, I lead us down the rathole. At least I can try to dig us out. {grin}16:58
Talk to ya tomorrow morning.
o and o16:59
* ajs6f leaves
<cbeer>no wonder i thought lily was pretty slow. uploading binary blobs /is/ pretty slow17:01
sometimes faster than Fedora, but sometimes pretty bad.
i wonder if multithreaded tests will look similar.
barmintor: thanks for bugging Eric for me17:15
barmintor: he somehow reports to Rob, right?17:16
yeah. over 1s to upload a 1Mish file.17:25
i wonder how that changes with clustering, if anyway.17:26
<barmintor>cbeer: yeah, he does17:38
1.5 hour conference calls about DPN are going to drive me to drink
<cbeer>good. then my offer to have Tom talk to Rob about how awesome it'd be to get that into blacklight actually makes sense17:39
you're not driven to drink already?
<barmintor>cbeer: it's going to piledrive me into drink
and tomorrow I have to meet with a "strategic review subcommittee" about our collaborative efforts17:40
or FF?
<barmintor>columbia library's in general. FF, DPN, Hydra, Duraspace, a bunch of non-tech stuff
<cbeer>oh. going to take your ball and go home?17:42
<barmintor>No, I'm just supposed to tell them what objectives from the 2010-2013 Strategic Plan for the Libraries were met successfully
<cbeer>hm. i guess i'm blissfully ignorant if we have any committees or strategic plans.17:43
<barmintor>stay in that Eden as long as you can.17:44
<cbeer>hm. if FF is an eden...
grr. modeshape ran out of file handles17:47
<barmintor>meh, so does fedora :D
<cbeer>yeah, but:17:48
open files (-n) 40960
i think if it's keeping 40k files open, something's not right.
<barmintor>that's a bunch of files
<cbeer>i mean, sending 10000 files at it in <10m is probably unnatural, but..17:49
<barmintor>there should be some cleanup17:50
I wonder if there's some delete_on_exit temp files that are being otherwise cleared out on a timer17:51
<cbeer>"I can't increase the linux ulimit. So I have resolved the problem by using PerWriteFileSync:
via https://community.jboss.org/message/771412
ok. i guess i'll wait for ajs6f17:57
as good a reason as any to head to the pub, i guess.17:58
<barmintor>did you see this: https://issues.jboss.org/browse/ISPN-130017:59
and yes, it is
it looks like there's a number of infinispan issues around file handles and the underlying lucene18:00
<cbeer>ok, so, only 4 million files18:02
<cbeer>i'm going to make sure we're not leaking file handles in our own code first
<barmintor>ok, I'm off. Until tomorrow!18:05
* barmintor leaves18:06
<cbeer>and then ran out of disk space. there's no winning, i guess.18:18
* eddies leaves19:05
* eddies joins19:07
* eddies leaves
* eddies joins
<cbeer>hm. looks like modeshape just keeps file handles open? wow.20:45