Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* eddies joins00:05
* eddies leaves
* eddies joins
* eddies leaves07:03
* eddies joins07:27
* eddies leaves
* eddies joins
* fasseg joins08:23
added this page when checking for some test data: https://wiki.duraspace.org/display/FF/Test+corpora09:18
...and btw do you guys know how to get a MIME-Type associated with an item in RDFDatabank? Hast this to be done via a RDF relation?09:20
There's no parameter while POSTing for the mime type it seems09:21
* JasonDGI joins09:46
<cbeer>fasseg++ # both of those look like good thing sto pull in09:52
i think there was also some work on file characterization the other month, i'll have to see where they put all their stuff
<fasseg>you mean the test data?
because yeah, i wouldn't want to push it to github09:53
hmm ingesting those 367 files with 38Mb takes quie the time09:57
<cbeer>into databank?
<fasseg>it's alternating between 260 and 85 ms per item..
I'll create a graph from my first few tests09:58
Oh I messed up there10:01
i created 2093 versions of the same item it seems ;)
<cbeer>i haven't looked at your jmeter yet, but it was driving me crazy yesterday10:05
my understanding of order-of-execution of the steps is clearly flawed
<fasseg>hehe, heh it's so simple it almost feels wrong10:06
<cbeer>i'm pretty close to just writing a script to generate jmeter steps :/
<fasseg>just add those controller and samplers
I'll give you a heads-up when I do a push10:07
<cbeer>thanks. it'd be nice to see another example
barmintor: for akubra-glacier, does this make sense (or is there a better way..)?:10:19
<barmintor>cbeer: I'm missing what "this" is
<cbeer>sorry, still typing :P
every time i request a copy of the glacier inventory, i want to store a copy (to load on a reboot, or whatever)
i also want to fetch a new inventory every 24h (i think that's what AWS claims is the refresh time on inventories)10:20
<barmintor>with you so far
<cbeer>i have some kind of local hashmap (in memory or otherwise) from the blob IDs to glacier archive ids10:21
every time i fetch a new inventory, i should throw away all the mappings that are older than when i requested the inventory (because they should be in the glacier inventory)
and figure out some way to merge the rest
so far, so good.
<cbeer>for fetching the inventory every 24h, should i just store a timestamp with the last glacier fetch, compare it every so often, and spin off a thread to fetch the new one as needed?
or just always fetch a new inventory every 24h (because it's a free request, i think), scheduled ... somehow.10:23
<barmintor>Hmm. I don't think there's a job scheduler in FCRepo
<cbeer>not that i found.
<barmintor>A few years ago I used Atlassian's quartz to do something like that
but I feel like there might be something bundled in already, now that I'm typing this10:24
let me poke around a little today
<barmintor>JasonDGI: ping10:33
<fasseg>@cbeer: I pushed the repo and added a small readme: https://github.com/futures/ff-databank-jmeter10:43
you dont have to use the test data set as written in the readme though, you can use any data as long as you put the path in the files.txt relative to the directory where the JMX file is10:44
i think i might just end up writing a script to generate the tests. clearly nested looping isn't something jmeter thinks they need to support
<fasseg>arg that sucks..10:46
<cbeer>yeah, so, here's what i've learned (or, think i've learned):
<fasseg>hmm here�s an example it seems: http://stackoverflow.com/questions/12993754/how-to-implement-nested-loop-in-jmeter
<cbeer>all config blocks are evaluated at the very beginning of the run
which means a config block can't reference a variable from a loop10:47
(e.g., given a list of directories, we want to loop through the files in each directory)
that's right out.
<fasseg>hmm can you add the config element as a child of the loop?
<cbeer>doesn't matter, all configs are executed at once, in the beginning of the test run10:48
<cbeer>so, i tried to stick my code into a while controller condition
but the while controller condition only has variable substitution happen once.
<fasseg>are you sure i thought i read sth diff. just a minute ago10:49
<cbeer>so, in a nested loop, it only takes the first execution and then runs that n times.
<fasseg>you can do some __javascript() magic i think
<cbeer>hm. got a reference for that?
i might fall under "putting non idempotent functions in Condition (like __counter) can introduce issues."10:51
fasseg: so, here's my broken test care (FedoraMadness.jmx): https://github.com/cbeer/ff-modeshape-jmeter/tree/this_doesnt_work10:53
and you have to symlink in the latest ff-fixtures directory as 'fixtures' into the directory you run jmeter from
i'll bang my head on it more tody10:54
<fasseg>performance on my local box for the EDRM datset: https://docs.google.com/file/d/0B5nd_qlYdcqyS2haMlZLajRMc2M/edit10:55
@cbeer: Oh are there already fixtures i though that was a work in progress..
I don't thikn I have the time to run that, I'll be off in the Weekend in about 20 mins ;)10:56
<cbeer>that's fine. i'll update your thing once i get the fixtures into jmeter11:00
barmintor: ping?11:01
trying to dial
just making sure you weren't muted
barmintor: ew. i hate overloading launchd.11:07
especially some way to make it less unpleasant to work in a development mode, huh?11:08
<barmintor>cbeer: yes, totally
<cbeer>jonathangee: (via JasonDGI): we also re-calibrated our point assignment system. sorta... i think we said 3 points was about a person-day of work
<barmintor>cbeer: The number of different components (solr on app server, a python daemon, a mysql db, a messaging server) required to get going is eroding my confidence in databank as-it-stands as a candidate. Will be very hard to build a community like this.
<barmintor>At least fcrepo's dependencies can be loaded up in one app server, you know?
<cbeer>+1, i'll stick around for a modeshape conversation
barmintor: and a (belated) +1 to that.11:23
is this a good conversation to have off-line or on-list, rather than the standup?11:25
<barmintor>cbeer +111:30
<cbeer>cbeer, scrum enforcer.
<barmintor>fasseg: https://wiki.duraspace.org/display/FF/Story+Management11:34
* jonathangee joins
<cbeer>barmintor: i'm interested in the modeshape conversation, but i'm going to get around to go to the office. can you give me a call/text if i don't respond on irc? i'll PM my #
<barmintor>cbeer: sure thing
grabbing lunch, back in a few12:05
<cbeer>ok, heading into the office. probably back online in 45m12:10
<jonathangee>back in the office from lunch12:30
So modeshape meeting is now?
or in half an hour?
<jonathangee>i think its supposed to be soon12:34
<jonathangee>whenever eddie is ready
<fasseg>chris, did you reset the JMeter task? Did i finish too early, again? ;)12:36
<cbeer>fasseg: i don't think i did anything.. not intentionally, at least12:46
ok, i'm going to spend 30 minutes angry at jmeter12:47
<fasseg>Does this task encompass the fedora tests and a wrapped databank instance so it can all be run from the command line without further effort?
<cbeer>and then try akubra-glacier stuff again
because then we might wanna add some points
<cbeer>fasseg: i don't think it matters. i think what you've done is sufficient, in my mind
once i crack the fixtures loading stuff
i'll update all 3 jmeter tests at once12:48
and then we can compare across platforms.
<fasseg>thats gonna be confusing
<cbeer>(or, someone can.. who can install databank.)
<fasseg>oh just remebered there's still a problem with spaces in filenames12:52
I'll fix that..
okay it should ingest all files now, not just the files w/o spaces in their names12:58
I see exactly the same perf degradation with mysql :/12:59
but this is getting too late for me now, I'm getting angry looks from the gf....13:00
so I'll be off for the weekend, I'll talk you on monday!
* fasseg leaves13:01
my short calls were not short
anyone need me?13:06
i think i'm going to pull myself a nice, stiff drink because it's depressing that i've spent my entire friday night on skype :-P13:07
<barmintor>modeshape call?13:12
<barmintor>Oh come on13:14
* eddies leaves
* eddies joins
* eddies leaves
* eddies joins
<barmintor>RECAP! RECAP!
<barmintor>jonathan has a very attractive ringtone13:20
<barmintor>eddies: cut out
<jonathangee>i was missing every second word before i dropped
guess not
or yes.
<barmintor>call quality kind of crap :(
JasonDGI: Did you get my email about that init.d script?13:46
<JasonDGI>let me check
yeah i have it here, do you just want the script as-is?13:48
<barmintor>JasonDGI: yes please
<JasonDGI>email sent with attachment, if that doesnt work, i will try again13:50
<barmintor>JasonDGI: got it. Did you install that seperately, or did easy_install include it?13:53
<JasonDGI>im not sure, that wasn't done by me13:55
thanks for forwarding!
<JasonDGI>i can get you an answer on monday13:56
<barmintor>supervisor hints http://bash-shell.net/blog/2012/apr/28/django-nginx-uwsgi-supervisor-on-debian/14:59
<cbeer>yay -- https://github.com/cbeer/ff-modeshape-jmeter15:02
supposedly has all 3 test cases in it
* JasonDGI leaves15:41
<cbeer>barmintor: ping?16:18
<cbeer>i just had lunch with tcramer and talked about FF possibilities16:19
one of the things we talked about was rewriting databank in (language/framework)-TBD
from my understanding of databank, it wouldn't take too much to rewrite it in ruby (given there are microservices libraries for ruby already).. does that make sense?16:20
<barmintor>My feeling is that you could write it in Java or ruby fairly quickly
<cbeer><<1 iteration?
(or, say, a hackfest?)
hackfest has more potential16:21
<cbeer>jonathangee: i assume JasonDGI is gone for the week?
<barmintor>though some of it might still be clunky; ie: If indexing is supposed to happen via an external messaging queue as a feature, it will suck to set up in all platforms
<jonathangee>yah. jason headed home a few minutes ago16:22
<cbeer>FF isn't committed to indexing via a MQ, though, right?
<barmintor>cbeer: No, but if we are re-creating databank, the question is whether Oxford is16:23
<cbeer>that is true.16:24
<barmintor>it's also possible that rewriting in Ruby will slide us the wrong way on the performance scale
unless there are some real inefficiencies in the exsitng code
<cbeer>guess we need this test harness going.
<cbeer>i think fasseg's testing wasn't quite right16:25
i think he was adding many files to a single dataset
but maybe my databank-fu is too weak to understand it
anyway, i'll send an email to ff-tech to get some more eyes on the jmeter tests.. i guess i should fork them into futures.
barmintor: once i do some more fixtures work, i'll work on the email for eddies. anything unexpected i should mention?16:46
<barmintor>I don't think so, but I'll respond to your message if you copy me
* jonathangee leaves17:22
* jonathangee joins17:40
* jonathangee leaves
<cbeer>boom: https://github.com/futures/ff-jmeter-madness17:44
fixtures running against modeshape and fedora (and maybe databank??)
now we just need a good way to analyze that data17:46
<barmintor>the data...
ok, I think I get how I can simplify this daemon config somehwat17:50
<cbeer>barmintor: it looks like columbia contributed a bunch of scanned books to OL18:11
any idea if you have easy access to those files? or should i just figure out a way to crawl archive.org to grab them?
ok, and got eddies and I an introduction to the berkeley people who know something about jcr.19:06
hm. i don't know if this is something wrong in my test setup, but 500 errors from modeshape are not promising.19:08
<httpSample t="22" ts="1357949294334" s="false" lb="ModeShape Create Node Request" rc="500" rm="org.modeshape.jcr.cache.NodeNotFoundInParentException: Cannot locate child node: 32a6fcb7505d64ab357c4c-b4ca-4887-b909-d51db731765e within parent: 32a6fcb7505d64c8a30dbd-4238-4a2e-8fbf-779483985d4e" tn="Modeshape group 1-1" dt="text">
but still way faster than fedora.19:11
even more so than before.. i guess before the upload lag might have made the comparison more favorable19:12
* barmintor leaves19:54