Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* eddies leaves01:54
* eddies joins02:12
* eddies leaves
* eddies joins
* fasseg joins04:32
morning!
* eddies leaves05:11
* eddies joins05:12
* eddies leaves
* eddies joins
<fasseg>grmblbl nuxeo API is straight from a horror movie05:32
I can't get the CMIS interface to run and when trying to use the RESTlet API i gt NUllpointers.....
the examples from the documentation do *not* work
they changed e.g. endpoints and it's not the same as in the docuementation05:33
error messages are not helping at all
rather nightmarish experience
I sinked 5h into this, do you want me to continue, or switch to soemthing diff and let some one else try it?05:34
the only thing i get to run was creating and deleting a folder at the start of the JMEter run05:35
But i Can't get a File ingested via CMIS or RESTLet
and automation API wants multipart/related ...
but i stumbled upon this test which uses a Junit Sampler and some Apache Chemistry libraries to reate the request:05:36
https://github.com/nuxeo/tools-nuxeo-cmis-jmeter
maybe we can continue from there...
Lol now im reading this: The root of the problem is that restlets in Nuxeo that are seam enabled (UploadRestlet is one of them) are meant to be used with a seam context (ie: a user does a drag/drop), not for outside call. You might want to contribute your own restlet that does not use seam context.05:41
WAARGHHHHHH! *bangs head*
I'll have a break....
* JasonDGI joins07:09
<fasseg>wow lily seems fast....07:43
ingest times don't go over 30ms it seems
and you can set your own schema using rest calls..so e.g. add a field for dc:title of type String07:44
of a field data for a blob or sth..
so easy t o use content-models
thx for the repo access ;)09:16
here are some results form Lily: https://docs.google.com/file/d/0B5nd_qlYdcqyaXhHX0VIV3I1OHc/edit09:18
always forget my mousepointer when taking screenshots ;)
My Problem at he moment is that I cannot init the fixtures:09:20
Clone of 'git@github.com:futures/ff-fixtures.git' into submodule path 'fixtures' failed
(publickey denied)
<cbeer>fasseg: oops. i can probably fix that the lazy way for now09:22
shouldn't have used a read/write url for the fixtures though, sorry09:23
hm. says you should have access. maybe you don't use git+ssh?09:24
<fasseg>nuxeo is quite nightmarish btw, I can't get the jmeter tests to run...i sinked 5h there, and then switched to Lily, maybe someone else wants to check out the nuxeo tests ? ;)
hmm lemme check..
<cbeer>fasseg: ok, fixed in the repo instead. you should be able to pull, submodule init and submodule update now?09:26
<fasseg>hmm nah still permission denied..09:27
Did you actually use my pubkey?
or should it be a password auth?
<cbeer>fasseg: i originally put in a git+ssh url for the submodule (bad of me)
so github is trying to use your pubkey auth09:28
but i just updated it to use a git:// url
<fasseg>so ill just use my key and it should work...
<cbeer>which should be non-authed.
<fasseg>ok ill just pull agina then
<cbeer>oh, maybe you need to run git submodule sync after pulling?09:29
i hate submodules.
<fasseg>and they have their problems, too i read sth about why not to use them but forgot ;)
http://somethingsinistral.net/blog/git-submodules-are-probably-not-the-answer/ ;)09:30
need some sugar....brb09:31
<cbeer>that's a little over the top.. which isn't to say they aren't confusing09:33
<fasseg>heh gotooa be loud in order to be heard ;)09:37
*gotta
btw: why tf hasn't JMeter's HttpSampler the possibility to send a file as the post entity??09:38
That's just inconvienient
okay now it seems to work for me to get the fistures09:40
*fixtures
<cbeer>good
there should only be a handful in the git repo itself
hm. and maybe i never pushed the govdocs harvester?09:41
<fasseg>files.txt is already in there or do i have to create one?
<cbeer>if you're using ff-jmeter-madness, you don't need files.txt at all
<fasseg>oh so just the "${__log(${__property(catalog,,files.txt)},OUT,,File catalog = )}" thingie?09:42
<cbeer>what's that?
<fasseg>for the CSV filename..
<cbeer>don't need one.
<fasseg><- confused now
I thought i saw that in your tests
<cbeer>in futures/ff-jmeter-madness, i rewrote all those jmeter things to just go look in the fixtures directory
and do the right thing
<fasseg>Ah in FedoraMAdness.jmx?09:44
<cbeer>ok, looks like i never finished writing the govdocs harvester. got derailed suggesting we just use it instead.
yes.
<fasseg>i was looking at ModeShapeMadness.jmx all the time
<cbeer>yeah. poorly named files would do that, wouldn't they.
<fasseg>Oh ok now i see...I'll add my tests as a group there then, since i created new *.jmx files09:45
<cbeer>sorry about that. probably should just delete modeshapemadness while you're at it09:46
<fasseg>heh kk
<cbeer>and rename that JMX file
<fasseg>and LilyMadness.jmx and NuxeoMadness.jmx ;)09:47
arg i can't copy and paste between two JMeter instances :/09:48
but you can "Save selection as.." and then "Merge"09:50
phew
<cbeer>oh, had i known that... :P09:52
<fasseg>ehhehhhheeehheehheee09:53
Do I have to run "convert-openplanet-corpus-to-fixtures.rb" as suggested in the docs?09:56
<cbeer>fasseg: you can, but what you really want is the script i haven't delivered yet, that grabs the large govdocs corpus09:57
<fasseg>okay, but for now ill take the open planets stuff09:58
<cbeer>yeah. it's a good mix of content
fasseg: ok, pushed the new script10:06
<fasseg>k thx
<cbeer>except i need to add a way to specify a path to bagit.py, maybe
at least, when i install it, it isn't in my $PATH10:07
<fasseg>i just used my local path
<cbeer>yeah, i think i tried to be too tricky though
<fasseg>which bagit version are you using?10:09
im getting these:SyntaxError: invalid syntax
File "/usr/lib/python2.7/site-packages/bagit-0.9.9-py2.7.egg/bagit.py", line 124
except Exception, e:
^
<cbeer>the same.
i don't get that error though10:10
<fasseg>hmm maybe im runnign python3
yeah that was the prob10:11
what about "/manifest-md5.txt" because i get a Problem in BSF script org.apache.bsf.BSFException: JavaScript Error: Internal Error: org.mozilla.javascript.EcmaError: TypeError: Cannot read property "length" from null10:15
and i dont have that file in fixtures/objects
<cbeer>every directory under objects should have a manifest-md5.txt in it10:16
<fasseg>yes just saw it via "find"
<cbeer>and i've pushed the script to get the govdocs
<fasseg>yeah ok ill switch scripts ;)
<cbeer>oh, i bet the openplanets script failed for you10:17
because i hardcoded a path to my bagit.py
<fasseg>nah i found that and just used my path hardcoded
<cbeer>k
<fasseg>the corpus and the manifests are there
i just get a nullpointer on the file in the Javascript forEach postprocessor10:18
<cbeer>so, here's what should be happening (and you can add debug processors to check, i guess):
the first BSF PostProcessor should find all the fixtures/objects subdirectories10:19
<fasseg>was just about to complain
<cbeer>and put them into jmeter variables FIXTURE_OBJECTS_0 to FIXTURE_OBJECTS_N
<fasseg>oops wrong chan
<cbeer>the second one ( in the ForEach ) get the objectdir variable
and goes through the manifest-md5.txt, parses it, and sticks it into FILE_0 to FILE_N10:20
<fasseg>well i dont get to the second one i think since the first one thros this:
2013/01/16 16:12:06 WARN - jmeter.extractor.BSFPostProcessor: Problem in BSF script org.apache.bsf.BSFException: JavaScript Error: Internal Error: org.mozilla.javascript.EcmaError: TypeError: Cannot read property "length" from null
<cbeer>what version of jmeter?
(I'm using 2.8)
and, do you see this log message?10:21
log.info("Loading fixture objects from " + fixtures_dir);
* barmintor joins10:22
<fasseg>just a sec on the phne atm, sry10:25
re10:33
2.8 herre too
no, can't find that in the logs: https://gist.github.com/454800410:34
shall I just push my changes, so you can take a look?10:35
hmm okay it seems to have lost some part of the directory path10:38
it's only got ${user.dir}/fixtures but my projects are in a subfolder...
oh i guess i have to set a user property for my fixture directory10:39
cbeer: right?10:40
okay now I get found object log msgs10:44
<cbeer>fasseg: what if you set that HARNESS_FIXTURES_DIRECTORY (or whatever it is) to the absolute path to your fixtures?10:51
<fasseg>yeah did that, but now, it seems it wont go into the feoreachcontroller10:52
although i can see the found objects log messages
Hmm should the Found Object log messages point to a file? since it points to a directory for me...10:54
as in: " Found Object: ...fixtures/objects/op_NovaMind"10:55
Can I push and let you take a look at it?10:56
<cbeer>found object should point at a directory
<fasseg>hmm ok then
hmm I can see the FIXTURE_OBJECTS_N in a debug postporcessor...10:59
but they wont get picked up by the for each loop
<cbeer>https://www.pivotaltracker.com/projects/68482511:01
barmintor?
JasonDGI: is there a ticket for that work?11:02
<barmintor>Jonathan has been REALLY bad about Pivotal11:03
<cbeer>https://github.com/futures/icemelt11:09
* futures-git joins11:12
[icemelt] none pushed 3 new commits to master: http://git.io/6M4-Kw
icemelt/master 77d5f0d Chris Beer: Update README.md
icemelt/master b259a6c Chris Beer: more docs
icemelt/master c53efdd Chris Beer: fix check to see if archive exists.
* futures-git leaves
<fasseg>I pushed the jmeter project with the working lily tests to https://github.com/futures/ff-jmeter-madness11:26
<barmintor>+1 lily feedback11:27
<JasonDGI>+1
<barmintor>cbeer: you're the requestor and the owner of the glacier inventory cache.11:28
<cbeer>barmintor: want to request it instead?
<barmintor>sure
<cbeer>POST /repository/blob11:34
=> get back a JSON blob
POST /repository/record
with body:11:35
"{\"action\": \"create\", \"record\": {\"type\": \"jm$record\", \"fields\": {\"jm$name\" : \"" + file.getName() + "\", \"jm$data\": " + { STUFF WE GOT BACK BEFORE } + "},\"namespaces\": { \"jmeter\": \"jm\"}}}
<fasseg>http://docs.ngdata.com/lily-docs-1_0/g3/g2/427-lily.html
cbeer: right
<cbeer>https://docs.google.com/file/d/0B5nd_qlYdcqyaXhHX0VIV3I1OHc/edit
<JasonDGI>or just declare a winner11:38
<cbeer>https://docs.google.com/file/d/0B5nd_qlYdcqyaXhHX0VIV3I1OHc/edit
oops
https://gist.github.com/26085aeb31db2a8ddd93 <- databank, fedora, and modeshape
<fasseg>https://docs.google.com/file/d/0B5nd_qlYdcqyM3pZcF96RzZLVUk/edit
<barmintor>https://github.com/NGDATA/lilyproject FYI11:42
one possible warning sign about lily is the number of developers: https://github.com/NGDATA/lilyproject/graphs/contributors11:51
<cbeer>here's an example of an actual JSON response from lily:11:53
{"id":"UUID.5eeec22a-cf45-448e-aa7e-ab5afd3f7029","version":1,"type":{"name":"ns1$record","version":1},"versionedType":{"name":"ns1$record","version":1},"fields":{"ns1$data":{"value":"qfBtZ2NhQnyrr5Y-oYbeVUhERlMAAAAE","mediaType":"application/octet-stream","size":975360},"ns1$name":"190298.ppt"},"namespaces":{"jmeter":"ns1"}}
<jonathangee>agreed. having essentially one developer is a little worrying.11:54
<barmintor>I stand by my previously asserted desire to have a candidate identified before I show up in Chicago12:01
<fasseg>that'd be very advantegous
<eddies>https://www.ohloh.net/p/lilyproject12:07
<barmintor>We also need to figure out whether there's preservation concerns re: hdfs12:08
since we'd definitely be losing transparency12:09
<fasseg>yes, there needs to be some tools or sth
<cbeer>added lily to https://gist.github.com/26085aeb31db2a8ddd9312:10
almost.
i'm surprised how bad the 5 thread version is.
not particularly worse than fedora though12:11
<fasseg>idd...12:12
<JasonDGI>lily has 3 sets of contact information http://www.lilyproject.org/lily/index.html12:17
at the bottom
<barmintor>Canadia!12:23
<JasonDGI>+1 for Canuckistan
<cbeer>fasseg: i pushed some changes to your lily tests12:27
<fasseg>kk
saw the github commits
<jonathangee>https://groups.google.com/forum/?fromgroups#!forum/lily-discuss
<cbeer>https://wiki.duraspace.org/display/REPONEXT/Prospective+platforms+for+long+term+effort12:28
that seems nice and active
<jonathangee>this is the company behind lily12:31
http://www.ngdata.com/site/about/ngdata.html
they may have a UI they are selling on top of lily12:32
<cbeer>i've pinged adam about my embedded modeshape problems12:33
eddies: for lily, just grab their distribution and run.. bin/launch-test-lily12:34
<eddies>cbeer: thanks. actually, can you just update the readme on ff-jmeter-madness to reflect that?12:35
<cbeer>sure. wasn't sure how soon you'd get started
<barmintor>gotta grab lunch before my 2pm meeting; bacl later12:40
<cbeer>updated jmeter readme12:42
<fasseg>so im gonna be off for today guys..13:01
here's my experiences with nuxeo this morning if you wanna feel my pain:13:02
grmblbl nuxeo API is straight from a horror movie
<fasseg> I can't get the CMIS interface to run and when trying to use the RESTlet API i gt NUllpointers.....
<fasseg> the examples from the documentation do *not* work
<fasseg> they changed e.g. endpoints and it's not the same as in the docuementation
<fasseg> error messages are not helping at all
<fasseg> rather nightmarish experience
<fasseg> I sinked 5h into this, do you want me to continue, or switch to soemthing diff and let some one else try it?
* fasseg leaves
<cbeer>log4j--13:13
i guess i've roped ajs6f into helping with modeshape stuff this iteration.14:01
barmintor: this just opened a couple miles away from here: www.freewheelbrewing.com14:07
i'm not thrilled with the lily api, but could probably tolerate it14:21
or, it really seems like the lily api is NOT geared towards working with large binary blobs14:23
<eddies>argh. i've been on the phone for the last 2 hrs :-P my ears are tired14:24
ok, i don't think i'm going to get a chance to look at the jmeter stuff tonight as i had hoped14:25
<cbeer>eddies: if you want, we can try to do modeshape in your morning.
<eddies>but will do so in the am
well, my "morning" starts at 12pm (UTC+8) now
so 8 hrs from now
you still going to be up?14:26
or rather, working?
<cbeer>trying to figure that out :P
yeah, i can probably be on at least an hour then14:27
that's my 8pm, i think
<eddies>cbeer: are you using brew to install jmeter? (i have an older version currently installed manually, but might as well switch to brew if it works fine)14:28
<cbeer>ok, ajs6f fixed up my ff-modeshape-prototype problems, he thinks. so i might get a chance to play with their api
eddies: i think i did, yeah
but ajs6f also embedded a version into the ff-jmeter-madness repo
<eddies>ok, i'll set my alarm for 11:45am :-P14:29
<cbeer>guess i tricked ajs6f into fixing up my poor code15:05
* JasonDGI leaves15:27
<barmintor>so we want fcrepo to have a RELS-INT annotation for datastreams so that an akubra module can know to use glacier for a datastream's content15:41
and we want the datastreams to be managed
(type M)
<cbeer>sounds good enough for a first pass
<barmintor>this means that the first version of the datastream is going to be… somewhere else
<cbeer>or you have to declare the RELS-INT first before adding the datastream15:42
<barmintor>I guess we also need a multiplexing akubra module that dispatches to the akubra mmodule
Let me make sure fcrepo doesn't audit RELS-INT15:43
this introduces a helluva a race condition :(15:46
when you ask for historical versions of a datastream, fcrepo has to know to use historical versions of RELS-INT to look for the annotation15:47
the RDF approach is nonsense15:50
this won't work
<cbeer>hm15:53
are we stuck with adding a hint to the datastream info serialization then?15:55
<barmintor>I think you need to get the information from the dsLocation, or else you're always going to have to figure out what the relevant version of RELS-INT was for a datastream content request
<cbeer>yeah, that doesn't sound fun15:56
<barmintor>and resign yourself to breaking access to that data in between the time that you update RELS-INT and the time you update the DS
it's a broken approach. A hint has to be in the dsVersion serialization.15:57
<cbeer>i guess all storage providers have an initial URI configuration compontent15:59
so using that in the dsLocation isn't so bad, right?
<barmintor>I don't think so16:02
it might be kind of weird to have a managed DS with a dsLocation that's a URL-like string, but I can't think of a way around it right now16:03
<cbeer>that's not so bad. and aligns nicely with E + R datastreams anyway16:04
needing to update all those after moving datastores sucks, but we don't support that well anyway16:05
<barmintor>cbeer: tangent- looking at the Java client, the only place glacier supports CSV serializ. is in Job output, and it only makes sense for inventory retrieval (archive is binary)16:12
<cbeer>yes, truth.
* barmintor is adding this coment to the icemelt issue
so there may not *be* a CSV format- it may just be a list
<cbeer>i think i saw a list of field names in the docs16:13
The CSV format has five columns "ArchiveId", "ArchiveDescription", "CreationDate", "Size", and "SHA256TreeHash" with the same definitions as the corresponding JSON fields.16:14
ah, cool, looks like they've updated their glacier docs since last month16:15
<barmintor>I just pasted the last few lines of IRC into the issue comments for later16:16
<cbeer>+116:18
* futures-git joins16:42
[icemelt] cbeer pushed 2 new commits to master: http://git.io/YL6Ahw
icemelt/master b9019ec Chris Beer: start adding error responses
icemelt/master 81edc39 Chris Beer: fix #2, add csv serialization for vault inventory job
* futures-git leaves
* futures-git joins16:46
[icemelt] cbeer pushed 1 new commit to master: http://git.io/ObTjgA
icemelt/master 1221b7b Chris Beer: add 404 response to jobs (hopefully it matches the AWS API
* futures-git leaves
<cbeer>barmintor: those were the error conditions i caught. i'lll grab your java tests some time and try them16:47
<barmintor>cbeer: you'd need to introduce an error- like not passing a vault name or something16:48
<cbeer>and apparently i should have run my tests before pushing. go figure.16:50
cbeer--16:51
<barmintor>bleargh16:55
<cbeer>oh, bah. i was getting test failures locally because something wasn't ruby 2.0 compatible16:56
maybe not just bad coding
<barmintor>your hint for glacier can't come from dsLocation unless the blob is already in glacier when you create the ds16:57
<cbeer>why?16:58
<barmintor>because the dsLocation will be an address to the blob you want to be in glacier16:59
<cbeer>so.. what if we just go back to the RELS-INT hinted one, with the caveat that things break if you change data sources?17:00
is that good enough for a prototype?
<barmintor>*sigh* I suppose it must be17:01
<cbeer>i don't think it's too terrible. it's no worse than when you swap out akubra stores things break.
<barmintor>yeah
<cbeer>well, ok. it's worse. but not so much worse
PATCHES WELCOME, and all that.
<barmintor>that's sounds a little like "faster than glacier!"
<cbeer>i love when tests catch stupid code.17:03
<barmintor>yeah, it makes you feel good about the process
<cbeer>all the hard work pays off
(and makes you feel guilty for not writing tests for the code that broke the other tests.. oh well)17:05
* futures-git joins17:06
[icemelt] cbeer pushed 3 new commits to master: http://git.io/FYnVnw
icemelt/master 4bb85ea Chris Beer: fixup describe job
icemelt/master c7e5ce8 Chris Beer: fix Job#new? condition
icemelt/master a8c6208 Chris Beer: fog is now parsing json bodies (because we forgot a content-type before?)
* futures-git leaves
<barmintor>this has been an afternoon of aborted code17:07
I'm trying again from Queens.
* barmintor leaves
<cbeer>jonathangee: what's the Islandora fedora API layer called again?17:41
(and, if i were looking for the code, where would I find it?)
ah, must be https://github.com/Islandora/tuque17:42
<jonathangee>yup18:41
you got it
<cbeer>eddies: i'm heading out for a bit. i should be back in time, but feel free to give me call. I'll commit to being around an hour from when I get back if it's later than that18:51
back.22:39
<eddies>cbeer: ok, i'm awake…sort of22:59
making coffee
<cbeer>k. i'm going to have dinner soon23:00
2m left23:19
unzipping23:21

Generated by Sualtam