Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* kaarefc joins01:34
* kaarefc leaves01:48
* kaarefc joins02:36
* kaarefc leaves03:16
* kaarefc joins03:38
* kaarefc leaves03:39
* kaarefc joins03:41
* eddies leaves04:17
* eddies joins04:42
* eddies leaves
* eddies joins
* eddies leaves05:43
* eddies joins05:48
* eddies leaves
* eddies joins
* eddies leaves06:16
* eddies joins06:59
* eddies leaves
* eddies joins
* ppound joins07:43
* avessey joins07:45
* JasonDGI joins07:53
* jcoyne joins08:26
<JasonDGI>whats the endpoint for the rss service? ive tried /rss and /rest/rss with no luck09:12
* kaarefc leaves09:13
<cbeer>JasonDGI: it isn't wired into the fcrepo-webapp09:22
<JasonDGI>can be it activated yet? is it ready?
<cbeer>that's why one of the tickets is to create that kitchen sink app
<ff-pivotal>Vincent Nguyen finished "fcrepo4 kitchen sink project" https://www.pivotaltracker.com/story/show/4473343309:23
* VincentNG joins09:26
<ff-pivotal>Paul Pound added comment: "have created a site http://islandora.fedora4lib.org/fcrepo/ which points at fcrepo4. Having ..." https://www.pivotaltracker.com/story/show/4449071909:27
<cbeer>ppound: can you point me at the tuque code that addes datastreams?09:28
<ppound>cbeer: there are a couple of different layers, The post function lives here https://github.com/futures/tuque/blob/1.x-fcrepo4/HttpConnection.php#L48709:31
* kaarefc joins09:32
<ppound>cbeer: the add data stream function lives here https://github.com/futures/tuque/blob/1.x-fcrepo4/FedoraApi.php#L646
cbeer: what would be the expected mime type at the data stream level? I'm not sure what I see there is correct should it be the same as at the data stream listing level?09:34
<cbeer>it should be whatever mime-type you pushed into fcrepo4.09:35
which is partly why i got suspicious about the tuque code
(and, you say this works fine in fcrepo3, right?)
<ppound>yes it always has
i'll check the mime types on the fcrepo3.6 on islandero.fedora4lib though just in case09:36
<cbeer>(i'm surpised, but..) it looks like you're forming a multipart post and adding the file as one of the parts.. fcrepo4 pulls the mime type out of the http request headers and the body out of the post body.09:37
check out http://www.php.net/manual/en/function.curl-setopt.php
see CURLOPT_INFILE
<ppound>k09:38
<cbeer>that's what I think you should be using. but let me quickly look at the fcrepo3 code and see if there's any oddities there we should have carried over.
<ppound>yeah we don't specify the INFILE option I don't think we do specify file and it might be setting the multipart stuff09:40
<cbeer>ah, so there IS code in fcrepo3 that negotiates multipart vs post body09:41
https://github.com/fcrepo/fcrepo/blob/master/fcrepo-server/src/main/java/org/fcrepo/server/rest/RestUtil.java
<ppound>cbeer: i'll trying setting the in file option
<cbeer>i guess that's for an html admin client that never existed, maybe?
<ff-pivotal>Jason MacWilliams added comment: "watching the fcrepo4 log, i can see negative sizes being reported, its probably not islandora" https://www.pivotaltracker.com/story/show/4481334909:44
Chris Beer added "Multipart POST vs POST body content stream negotiation for add/modify datastream" https://www.pivotaltracker.com/story/show/45062931
<cbeer>JasonDGI: i haven't looked at the logs yet, but it'd be helpful if you noted when the object size goes negative on this ticket: https://www.pivotaltracker.com/story/show/4496264309:45
<ff-pivotal>Chris Beer edited "Introduce ourselves on the MODE forum" https://www.pivotaltracker.com/story/show/4489920109:48
<ppound>cbeer: we maybe hitting this Passing an array to CURLOPT_POSTFIELDS will encode the data as multipart/form-data, while passing a URL-encoded string will encode the data as application/x-www-form-urlencoded.09:51
<cbeer>ppound: yes, that sounds right.
and, at least so far in fcrepo4, you don't want to do either of those
(see CURLOPT_FILE for the passing-a-string case, i think)09:52
<ff-pivotal>Vincent Nguyen edited "fcrepo4 kitchen sink project" https://www.pivotaltracker.com/story/show/4473343310:03
* eddies loves maven so much he could strangle it, set it on fire, and roll it down the stairs10:04
<JasonDGI>cbeer is it possible fcrepo4 is performing size calculations before datastreams/properties are constructed/populated?10:05
<cbeer>possible, but i think there's a double-save in th code to deal with just that.10:06
<JasonDGI>my log shows things like: create islandora:root size=327, delete islandora:root size=8648
* kaarefc leaves10:14
* barmintor joins10:20
<cbeer>barmintor: do you have any opinion about https://www.pivotaltracker.com/story/show/45062931 ?10:33
(eddies: feature request; the pivotal bot should respond to pivotal ticket ids and look up the title.. you know.. if your spare time)
<eddies>cbeer: are you able to set things up so others can re-deploy the pivotal bot?10:35
or should we just move to heroku?
<barmintor>cbeer: I'm not swayed one way or the other.10:36
<ff-pivotal>Jason MacWilliams added comment: "It looks to me like the size calculations on ingest are performed before the object is finished constr..." https://www.pivotaltracker.com/story/show/4496264310:37
<JasonDGI>how do i get a list of all objects in the repostitory?10:53
<cbeer>https://github.com/futures/fcrepo4/pull/2411:02
https://www.pivotaltracker.com/story/show/4421398111:06
<barmintor>JasonDGI: excellent question.
<ff-pivotal>Jason MacWilliams edited "Islandora running on top of fcrepo4" https://www.pivotaltracker.com/story/show/4449071911:09
<cbeer>https://www.pivotaltracker.com/story/show/4506293111:10
<ff-pivotal>Jason MacWilliams edited "Expose repo stats in Islandora" https://www.pivotaltracker.com/story/show/44813349
<JasonDGI>barmintor i found localhost:8080/rest/objects which in the background falls to ObjectService::getObjectNames, i think that does what i want11:11
* ajs6f joins11:14
<ff-pivotal>Edwin Shin edited "Repository size should not go negative" https://www.pivotaltracker.com/story/show/4496264311:15
Edwin Shin edited "Repository size should not go negative" https://www.pivotaltracker.com/story/show/44962643
<ajs6f>JasonDGI: "how do i get a list of all objects in the repostitory?"11:17
rest/objects
<eddies>http://fcrepo.org/fcrepo4/
<JasonDGI>ajs6f or ObjectService::getObjectNames in the code
<ajs6f>Right. I added that the other week.
<JasonDGI>and its awesome11:18
<ajs6f>As for the size calculations— the negativity thing sounds like a bad blunder on my part. But the odd (large) sizes after a few mutations are okay.
Remember, there are an enormous number of system nodes created/deleted altered when any versioned node is mutated.
And because they stick around (unless we change our config) they're going to keep contributing to repo footprint.11:19
<JasonDGI>that will make counting very difficult
<ajs6f>Why?
Counting what, I guess is a better question?
<JasonDGI>counting sizes of *all& nodes
<ajs6f>Those systme nodes _are_ part of the size of the node, as we discussed it before I added that code.11:20
If we want different semnatics for size,
<JasonDGI>but we only update the size when *we* add/remove a node
<ajs6f>Right.
<JasonDGI>and we only count the immediate node we added/deleted
and dont account for side effects
<ajs6f>No, we don't.
If we want to do that (and I can understand why we would)11:21
<JasonDGI>wouldnt those system nodes be classified as side effects
<ajs6f>we have to engage with the internal semantics of MS's in-JCR storage model.
Only some of which is spec'd by JCR.
Perhaps we can start by determining just how much of an impact those sysnodes have.
<cbeer>or, as I suggested, don11:22
<ajs6f>Don?
New dev?
<cbeer>'t bother accounting for anything but the jcr:data size
<ajs6f>Yes, and I agree with that.
Give Don the ticket.
<cbeer>ajs6f: i'm trying to get to the bottom of our clustering.. problem. it has to be something funky we're doing on top of MODE/ISPN, i think12:01
but i'm not sure what it is or where to start.12:02
* VincentNG leaves12:03
<cbeer>i do think we're hitting an ISPN bug that was supposedly fixed earlier
but i don't think that's the whole story
<JasonDGI>so if i ask fcrepo4 to calculate its current size (on demand) i get the correct number - the updateRespositorySize call on ingest is reporting incorrectly
* ajs6f1 joins12:04
* ajs6f leaves12:05
<cbeer>good. ship it.12:06
<ajs6f1>What's going wrong with our clustering?12:07
<cbeer>ajs6f1: the 60s+ delay before things actually appear on other nodes12:08
<ajs6f1>Right.
Hm.
is it possible that our JGroups action is funky?
doing some kind of beffering?
s/beffering/buffering
<cbeer>ajs6f1: unlikely. i've been trying to assemble a simple test case and it doesn't suffer from the same problem
<ajs6f1>What's the case?12:09
<cbeer>https://github.com/cbeer/modeshape-examples/tree/clustering-tests/modeshape-clustering-example
although i should push first
<ajs6f1>Say when you've pushed...12:10
<cbeer>pushed.12:12
<jcoyne>beffer uverfleu
<cbeer>only the readme needs a tweak
<ajs6f1>Can I run it the way the README says?12:14
<cbeer>ajs6f1: you can now.
<ajs6f1>Cool.
<cbeer>anyway, that piece works
and you should be able to launch fcrepo4 and start one of these guys12:15
<ajs6f1>Okay, let me try the case locally.
<JasonDGI>cbeer the 60s+ delay before things actually appear on other nodes ?
<cbeer>(let me double-check the behavior i was seeing)
JasonDGI: yes.
<JasonDGI>that sounds bad12:16
<cbeer>ajs6f1: ok, except i screwed up the output of the test case
<ajs6f1>Should I wait and pull?12:17
<cbeer>yeah, one sec
<ajs6f1>afk brb12:18
<cbeer>pushed.
anyway, when you do a clustering-test <-> clustering test, it works no problem12:19
fcrepo4 <-> clustering test, you get the same behavior has fcrepo4 <-> fcrepo4
* VincentNG joins12:28
<JasonDGI>exposing repo stats - do we just want object count + total bytes used?12:50
do we care about bytes in datastreams vs bytes in objects?
<ajs6f1>cbeer:not sure I understand the test case. When I run the second process, I just get:12:56
d-137-155-216:modeshape-clustering-example ajs6f$ java -DindexesPath=target/index2 -DinfinispanCacheDirPath=target/infinispanCacheDir2 -cp target/classes:target/classes/org/modeshape/example/clustering:target/dependency/* org.modeshape.example.clustering.ModeShapeExample Node1
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at org.modeshape.example.clustering.ModeShapeExample.main(ModeShapeExample.java:30)
Looks like I need an integer argument there?
I put in 1000, as in the first invocation, and now it appears to be doing something… was that right?12:57
<cbeer>yes, sorry, must have failed at copy-paste
<ajs6f1>No prob. So, how do I know the test passed? Is there something for which I am looking in the logs of the two processes?12:58
<cbeer>each log should have nodes from the other.. the first arg is the prefix to use
so you should see nodes with e.g. Node0 and Node1 prefixes
<ajs6f1>I am. So now I try running it against a fcrepo4 process, right?12:59
<cbeer>i have a version that actually runs these as a junit test, but it wasn't breaking as expected, so for now it's manual.
ajs6f1: yes. although you'll have to get your fcrepo4 invokation right
you need to tell jgroups to bind to 127.0.0.113:00
<ajs6f1>Is that paramterized or do I change config?
<cbeer>parameterized.
<ajs6f1>in repository.json or jgroups.xml?13:01
<cbeer>jgroups-*.xml
<ajs6f1>Thanks.
<cbeer>"Large file support improvement. Currently, performance can be disappointing when ingesting thousands of files."13:02
that doesn't mean what someone thinks it means.
<ajs6f1>Did we say that?13:03
<cbeer>http://duraspace.org/duraspace-sponsor-updates-fedora-futures
apparently we did
<ajs6f1>I wonder what we meant by it.
<cbeer>that's a mediocre press release.
<ff-pivotal>Vincent Nguyen finished "fcrepo4 kitchen sink project" https://www.pivotaltracker.com/story/show/44733433
<cbeer>"• It should be assumed that APIs for Fedora will be stable for new releases. "13:05
"• API compatibility is a concern"
<ajs6f1>Well, athat's why we broke out legacy-api as a module.
And we can build El Dorado API on a different track.13:06
"• Batch and data model compatibility between legacy Fedora and new versions"
Wait, what does "batch compatibility" mean?
<cbeer>i have no idea13:07
eddies!!!?!
<ajs6f1>Eddie, are you going to compatible our batches?
<cbeer>have we been committed to things we don't know about? or is the press release.. just.. bad?
<barmintor>WTF is that whole document?
<cbeer>"Formal testing and QA will validate all releases."
will we get tuxes?
<ajs6f1>Or taxes (to maintain an aging despotism)?13:08
<barmintor>that's not a press release. Those are carol's notes from the conference calls.13:10
<eddies>what?!13:11
i was on the phone w/ usgs
<ajs6f1>cbeer: running the test case against fcrepo4, I got some interesting early messages (one in each log):
<eddies>vincentng: i pushed the overlay branch, if you hadn't already seen it for the kitchen-sink app
<ajs6f1>13:09:39,091 WARN [JGRP00012] discarded message from different cluster ISPN (our cluster is modeshape). Sender was d-137-155-216-31996
WARNING: [JGRP00012] discarded message from different cluster modeshape (our cluster is ISPN). Sender was c01baaf8-d5d8-1465-f23c-66dfd9e5c37d
<cbeer>ajs6f1: normal.
ajs6f1: and seemingly not a problem13:12
<ajs6f1>See that? Are they even talking to each other?
You're sure? Because I'm not getting clustering.
<cbeer>ajs6f1: no "DEBUG Received change" messages?13:13
<ajs6f1>I mean they're talking to each other at the level of JGroups...
<eddies>oh. i see, "Batch and data model compatibility between legacy Fedora and new versions"
i have no idea what that means
<VincentNG>eddies: i saw it and applied it to my master branch. Thanks for the help!
<ajs6f1>Yeah, getting that, but does that represent clustering? Or just the repo managing its own events?
<cbeer>that's the keep-alive messages, it's a sign of at least something being on jgroups13:14
<eddies>vincentng: so you should just be able to add in other fcrepo4 modules or futures projects as dependencies
<cbeer>the first sign of clutsering is..
09:20:06,020 DEBUG New view accepted: [localhost-34097|3] [localhost-34097, localhost-30304]
09:20:06,021 INFO ISPN000094: Received new cluster view: [localhost-34097|3] [localhost-34097, localhost-30304]
<ajs6f1>Okay, wait, I just added an object to Fedora and saw it appear in the other node instantly.
<VincentNG>eddies: ya, i added foxml, dc, and glacier as a dependency
<eddies>keep in mind, however, that when the kitchen-sink gets deployed, it's going to rely on the artifacts that are published to the maven snapshot repo13:15
<ajs6f1>Eddie: Can we not add the maven snapshot repo to the pom of kitchensink?
<eddies>so we need to actually deploy snapshots regularly for the kitchen-sink app to take advantage of it
<cbeer>ajs6f1: appear in the ISPN messages, or appear in the list of nodes? and if you do it again, is it the same?
<VincentNG>eddies: so those artifacts need to be published to maven snapshot right?
<ajs6f1>List of nodes definitely. I'll have to scroll back and check the ISPN messages.13:16
<cbeer>(I see instant "received change" messages, delayed appearing in the list of nodes, but seemingly timing is a factor, so you might have made your change right before a timeout occurred or whatever the problem is)
<eddies>ajs6f: yes, the sonatype snapshot repo is in the pom. but we still need to deploy the snapshots for the artifacts it depends on
<ajs6f1>eddies: right, right. I misunderstood your point.13:17
<eddies>vincentng: yes, basically, "mvn deploy"
<cbeer>(heading to the office.)
<ajs6f1>cbeer: okay, let me try it couple of times without some time in between. I'll tell you how it goes whaen you get back.
<eddies>vincentng: should we discuss what you can work on next?13:18
<VincentNG>eddies: sure
<ajs6f1>Ah, now I'm seeing the delay.
Not quite sixty seconds, but some lieel while.
* github-ff joins13:25
[fcrepo4] jmacwilliams opened pull request #26: function to recalculate the repository size (on-demand) (master...master) http://git.io/Ygr9XA
* github-ff leaves
<JasonDGI>so i can calculate the repository size manually and if I do it several times while i'm ingesting an object (just 1) i see the repo size increase incrementally13:26
* ajs6f joins13:34
<ff-pivotal>Jason MacWilliams added comment: "Interesting note, if I ingest 1 object and calculate the repo size several times while the object is b..." https://www.pivotaltracker.com/story/show/44962643
* ajs6f1 leaves
<ff-pivotal>Jason MacWilliams edited "Expose repo stats in Islandora" https://www.pivotaltracker.com/story/show/4481334913:36
<ajs6f>cbeer, you back?13:43
<cbeer>barely. heading to a meeting in ~10m
<ajs6f>Real quick q: did you copy the configs for the test case from fcrepo4?13:44
<cbeer>the relevant bits, yes.13:45
<ajs6f>Hm. Okay. I'm trying to diff the test case and fcrepo4.
The test case doesn't use the ClusterCacheLoader. Hm, I wonder….13:46
<cbeer>there's some annoying whitespace differences, i think
oh, i wonder if i didn't commit something
you should definitely kill CCL
<ajs6f>Oh, I meant diff in a spiritual sense.
Yeah, I'm going to try that.
<cbeer>the CCL is definitely wrong, so go ahead and commit that13:47
<ajs6f>Well, hold on. I _did_ get a change in behavior (sans CCL). Now the object _never_ shows up in the test case process… !13:50
Oh, wait, wrong was I.
It just takes even longer.
<cbeer>hm. (Now i can't remember what problem removing CCL was supposed to solve)13:51
<ajs6f>I think we wanted to kill it just because rhauch said it wasn't neccessary.13:52
<cbeer>jboss++
they do know how to log
<ajs6f>So we're using TCP ping for this, right cbeer?13:53
<cbeer>yep
<ajs6f>Just trying to get my head aroun this from the bottom up.
<cbeer>it seems to work more reliably, at least.
<ajs6f>Oh, I have no qualm with it. I just want to read all the configs correctly with my human eye.
I'm thinking I may start pulling things out of jgroups-*.xml one by one as much as possible.13:54
<cbeer>i don't think jgroups is the problem, though.
jgroups works fine, infinispan is getting its changes immediately
<ajs6f>No, not on my box. It takes quite a few seconds.13:55
And then even longer for the node to appear.
<JasonDGI>where is this batch-modify-object endpoint?14:26
<ajs6f>Okay, that's my biggest data point to add to this clustering thing so far, and it isn't much of one: on my box, I do _not_ see the events occurring and then a long delay before the nodes actually appear. In fact, for nodes that appear from outside the test case repo (transferred by INSPN/JGroups) I don't see the logging event handler fire at _all_, and I don't see any logging info from MODE or INSPN. The nodes just suddenly appear. Which is pretty weird.
afk brb
* ajs6f leaves
* ajs6f joins14:29
<cbeer>JasonDGI: https://www.pivotaltracker.com/story/show/4411197114:31
<eddies>cbeer: do you have a link for the merritt spec (that DRAFT pdf doc)?14:39
i can't seem to find it again
<cbeer>eddies: search merritt storage rest api
<eddies>danke: https://wiki.ucop.edu/download/attachments/16744547/Merritt-storage-service-latest.pdf14:40
vincentng: https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide14:46
<JasonDGI>does this addDatastreams/batch/cxf stuff use soap?14:51
<cbeer>no.14:53
<eddies>ok, i'm about at my limit for the night. gonna turn in. cbeer, i'll look for you on irc when i'm up14:57
<ajs6f>o and o15:27
\quit
Urg.
* ajs6f leaves
<JasonDGI>cbeer for your rubydora batch ticket, did you make *everything* use the batch modify?15:32
<cbeer>JasonDGI: yes.
i couldn't see any reason not to.
but we also have some level of ORM happening at the rubydora layer15:33
not sure if tuque does something similar
so it was easy for me
<JasonDGI>orm?
<cbeer>VincentNG: why are we pulling in code into fcrepo-kitchen-sink? or is the work you're doing now to remove it?15:41
JasonDGI: http://en.wikipedia.org/wiki/Object-relational_mapping, or whatever it is in a fedora-centric world.
<ff-pivotal>Paul Pound added comment: "as discussed in the standup today we can probably close this as there is another ticket regarding the negativ..." https://www.pivotaltracker.com/story/show/4481334915:50
* ppound leaves15:51
<VincentNG>cbeer: do i not need to pull the code in?15:53
<cbeer>maybe i missed the reasoning, but now that we're set up to publish artifacts, shouldn't we just use them?15:54
* JasonDGI leaves
<VincentNG>cbeer, i did that before i knew we were going to make sure to publish the artifacts
<cbeer>ok, but we are and should, right?
<VincentNG>once their in the repo, then i'll remove them15:55
<cbeer>+1
<VincentNG>yes, we are
cbeer: how do i access jenkins?15:56
* github-ff joins16:12
[fcrepo4] cbeer pushed 1 new commit to master: http://git.io/Pp4Pdw
fcrepo4/master af35080 Chris Beer: MODE/ISPN config tweaks
* github-ff leaves
* travis-ci joins16:27
[travis-ci] futures/fcrepo4#135 (master - af35080 : Chris Beer): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/c8910d3d83ed...af35080eaf78
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/5052553
* travis-ci leaves
* VincentNG leaves16:38
* barmintor leaves17:40
* github-ff joins17:45
[fcrepo4] cbeer pushed 1 new commit to master: http://git.io/pWSunQ
fcrepo4/master 4026728 Chris Beer: bump infinispan version
* github-ff leaves
* jcoyne leaves17:48
* travis-ci joins18:00
[travis-ci] futures/fcrepo4#136 (master - 4026728 : Chris Beer): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/af35080eaf78...402672801b7e
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/5054934
* travis-ci leaves
* github-ff joins23:06
[fcrepo4] eddies pushed 3 new commits to master: http://git.io/jLEtEg
fcrepo4/master 77e07af Edwin Shin: Updates versions of various maven plugins
fcrepo4/master 315a1a4 Edwin Shin: Uses maven classic <reporting> instead of <reportPlugins> configuration....
fcrepo4/master 3e13308 Edwin Shin: Merge branch 'master' of github.com:futures/fcrepo4
* github-ff leaves
* travis-ci joins23:19
[travis-ci] futures/fcrepo4#137 (master - 3e13308 : Edwin Shin): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/402672801b7e...3e1330809885
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/5061214
* travis-ci leaves

Generated by Sualtam