Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* barmintor joins02:48
* barmintor leaves03:02
* anusha joins06:13
* kaarefc leaves09:01
<cbeer>anusha: do you have time to look at https://wiki.ucop.edu/download/attachments/16744573/Merritt-ingest-service-latest.pdf?version=23&modificationDate=1357327883000 and try to wrap your head around what they're doing, and if it's useful to us?09:02
* barmintor joins09:29
* barmintor_ joins09:33
* barmintor leaves
* eddies leaves09:34
* barmintor leaves09:39
* kaarefc joins09:40
<anusha>cbeer: looking at it now.09:48
* eddies joins09:58
* eddies leaves
* eddies joins
* JasonDGI joins10:09
* eddies leaves10:15
* JasonDGI leaves
<kaarefc>So, um, when would developer challenge submissions be due?10:45
I have some ugly proof of concept code that does nothing and is held together with duct tape, but I might be able to make it into some ugly proof of concept code that does /something/ and is held together with duct tape during the morning10:46
But if I've already missed my deadline I could pay attention to the conference instead, I suppose :-)
<anusha>cbeer: The document from Merritt you linked to describes the asynchronous ingest API10:47
<cbeer>kaarefc: go for it. i think we just said "morning"10:48
anusha: i'm not sure I know what an asynchronous ingest API means.10:49
<anusha>There's just one POST to /submit, where you post either a file, an object (containing many files) or a container objects (linking to many objects) to a queue
<cbeer>a merritt-managed queue?
(and, what's it doing that it needs to queue these things anyway?)10:50
<anusha>1. Send an email
<cbeer>damn. i have clustering working on the oxford VMs. and I don't know what I've done differently today than before.10:51
<anusha>page 36 & 37 describe the ig=ngest process
those CDL specs are such a disorganized mess10:52
(while appearing structurely sound..)
<anusha>Basically, uou post everything to a queue, it then builds the package (creating a series of files like ownership, object xml) and doing fixity checks and once it's done all that it updates the status of the object10:53
<cbeer>so, it's holding this object in some staging area? or it goes into the filesystem with a status flag?
<anusha>you can submit jobs in batches (by providing details in the manifest file)10:54
<cbeer>ah, so multiple-object ingest?
<anusha>and query the state of the queue for the service itself, for a batch, for n object or for a file
<cbeer>(and, do they all-pass or all-fail? does that solve our transactions issue?)
and i'm spinning up a 4-node cluster right now.10:55
<anusha>cbeer++. Can I start doing tests:)
<cbeer>i'm going to check that things get distributed appropriately, or if we need a load-balancer in front of the cluster
<anusha>I'll wait till you are happy for me to play with it
I think each object / file can fail individually. I need to read up in more detail to understand that bit10:56
I think it is a merritt managed queue. This has no technical specs, so not sure what they are using as a queue10:57
* eddies joins
* eddies leaves
* eddies joins
* eddies leaves
* eddies joins
* ajs6f joins10:58
* barmintor joins
<ajs6f>Doing Ben's work.
<barmintor>excuse me?10:59
<ajs6f>Wait for the push. :)
<anusha>From what I understand, it's holding it in a staging area before writing to disk
<eddies>hi anusha!11:00
<anusha>They also support what we had read earlier, which is the synchronous api11:01
hi eddie!
* github-ff joins
[fcrepo4] ajs6f created fcrepo-foxml (+1 new commit): http://git.io/C0P9DQ
fcrepo4/fcrepo-foxml e770c0f ajs6f: New fcrepo-foxml in support of https://www.pivotaltracker.com/story/show/44206401
* github-ff leaves
<anusha>eddies: See https://github.com/futures/ff-jmeter-testResults/tree/master/2013-02-10-FedoraTests/results11:02
<ajs6f>Ben, that was https://www.pivotaltracker.com/story/show/44206401
<anusha>These are the results from the test I did last Friday
* JasonDGI joins
* github-ff joins11:03
[fcrepo4] ajs6f opened pull request #16: New fcrepo-foxml in support of https://www.pivotaltracker.com/story/show... (master...fcrepo-foxml) http://git.io/Vm1Njg
* github-ff leaves
* anusha leaves
* anusha joins11:04
<barmintor>ajs6f: is it finished sans testing?
<ajs6f>Well, there's a sanity test to see that objects with the right PIDS appear when FOXML is thrown into the repo.11:05
No, there's not much testing of the parsing, because there's not much parsing.
Remember, I wrote it so that we could reassure people that "we're doing FOXML".11:06
And I'm more than happy to see it replaced with some other useful code.
Are we okay with the FOXML stuff staying in Scala?11:07
Because we can make an issue for fuller parsiing.
<kaarefc>Oh noes! Exactly what information should I be able to get from the JMS ATOM packages?
<ajs6f>Hey, Fedora House people… you guys didn't happen to find my scarf, did you?
kaarefc: not much right now.
A PID and the name of the operation that ccurred.
The whole point of that service is to support legacy code.
<kaarefc>Well, that's it for my little proof of concept test - I should have looked more before implementing
I was going to update a SOLR index on datastream updates11:10
<ajs6f>If you want to right anything new, let us give you a feed that makes sense for you
<kaarefc>But since I don't actually get the PID of the object updated, that's sort of impossible :-)
<barmintor>ajs6f: I've got it
<ajs6f>B— THANKS!
kaarefc: You should get the PID. Are you not getting the PID?
Who's go tthe PID?
<kaarefc>This is the ATOM I get on datastream updates:11:11
<entry xmlns="http://www.w3.org/2005/Atom"><title type="text" xml:base="http://localhost:8080/rest">addDatastream</title><category term="DC" scheme="xsd:string" label="fedora-types:pid" /></entry>
<ajs6f>Ah, you're getting the datastream name instead of the PID.
Okay, let me look into that.11:12
<kaarefc>I could be wrong, but I don't think I get *anything* over JMS on datastream updates either
<ajs6f>That would blow up legacy stuff too.
Oh, you should.
<kaarefc>I'll double check that
<ajs6f>Can you do me a huge favor and file Github issues on this stuff. I'll take care of our internal tracking, but I need to be reminded.
I'll probbly start fixing that right away to stop blocking you.
Again, tho', kaarefc, if you want to write new stuff, don't feel constrained by the legacy format (which we don't like that much ourselves). Feel free to ask for something new, or use the RSS feed if that's what suits you. We've also talked about offering webhooks or other notifications, so feel free to suggest.11:14
* eddies leaves
<kaarefc>Just for now the path of least resistance seemed to be listening for JMS messages. But any notification scheme would work for me.11:16
* eddies joins
* eddies leaves
* eddies joins
* eddies leaves
* eddies joins
<cbeer>anusha: yeah, i'm still missing something about the modeshape cluster config. i'm going to put it aside now11:18
<anusha>Cbeer: No worries. let me know when you are done. I got the ram increased in futures1 to 4GB and checked that the CPU has 4 cores.11:20
I'll increase the heap size in the jmeter.sh file to 1G
* barmintor leaves11:24
<kaarefc>I added a github issue11:26
ajs6f: As for notification stuff, I'd like some sort of push notification - it could be a hook of some sort, but RSS isn't really usable for my purposes11:30
<BHSPiMonkey>how's the hack week thing going to wrap up? I'm interested in seeing a recap of what went down this week
<kaarefc>ajs6f: JMS seems to be fune for me
<ajs6f>Okay, kaarefc.11:35
Did you find that you _do_ get messages for dtastream mutation?11:36
Does anyone have an example of what fcrepo3 produced for datastram-relevant ATOM messages?11:37
I know we must put the DSID somewhere, but I can't remember where.
<kaarefc>ajs6f: http://fedora-commons.org/documentation/3.0/userdocs/server/messaging/#messages11:38
<ajs6f>Yeah, that doesn't actually describe the possible terms. :)11:39
At least as far as I could find.
<kaarefc>No :-)
<ajs6f>Well, I guess I'll spin up an fcrepo3 and look at what it does.
<cbeer>ajs6f: i can probably grab one
<kaarefc>ajs6f: And, no, it does not seem to be the case that datastream updates trigger JMS messages11:41
Should I add a separate github issue?
<cbeer>kaarefc: https://gist.github.com/cbeer/91ceb2f680c519e7442511:42
<ajs6f>Yes, please. That's particularly weird.
Thanks, cbeer.
I'll make ours look more like that. {grin}
<cbeer>i guess i should have grabbed a delete too
but we don't have any in the error queue11:43
<ajs6f>The more examples the fewer issues kaarefc will have to file. :)
For those hacking on the code.
* jcoyne joins11:49
* github-ff joins11:56
[fcrepo4] ajs6f pushed 1 new commit to master: http://git.io/RGCKVQ
fcrepo4/master 82d9f34 ajs6f: Addded convenience predicates for detecting types of Fedora resources
* github-ff leaves
* ajs6f leaves12:22
* ajs6f joins12:23
* ajs6f leaves
* ajs6f joins
* github-ff joins12:25
[fcrepo4] eddies pushed 2 new commits to master: http://git.io/igwqJQ
fcrepo4/master 5fcf91c Edwin Shin: Removing the (temporary) fedora4lib Nexus repo references
fcrepo4/master 35cda0b Edwin Shin: Merge branch 'master' of github.com:futures/fcrepo4
* github-ff leaves
* github-ff joins12:29
[fcrepo4] ajs6f created ImproveJMSAtom (+1 new commit): http://git.io/YgT0mw
fcrepo4/ImproveJMSAtom b89a146 ajs6f: Fix for https://github.com/futures/fcrepo4/issues/17
* github-ff leaves
* github-ff joins
[fcrepo4] ajs6f opened pull request #19: Fix for https://github.com/futures/fcrepo4/issues/17 (master...ImproveJMSAtom) http://git.io/0GC1LA
* github-ff leaves
<ajs6f>That's the fix for kaarefc's first issue. It would be nice if someone else could glance at it and merge it just to make sure I'm not completely insance.12:30
* github-ff joins12:32
[fcrepo4] cbeer pushed 1 new commit to master: http://git.io/92eIIw
fcrepo4/master 68e1add Chris Beer: Merge pull request #19 from futures/ImproveJMSAtom...
* github-ff leaves
* github-ff joins
[fcrepo4] cbeer deleted ImproveJMSAtom at b89a146: http://git.io/tswj6Q
* github-ff leaves
<ajs6f>Thanks, cbeer. kaarefc: Let us know if that don't cut the mustard. Now onto the next.
<JasonDGI>ajs6f do you know what version(s) of gsearch can work with solr 4.1?12:35
<ajs6f>I do not.12:36
There may nor yet be one.
<JasonDGI>you forgot the g at the end of that expr
ajs6f ive heard that 3.5 may be the highest solr that still works with gsearch, have you heard anything about that?12:37
<ajs6f>I have not, but I haven't been following.12:38
I'd step back to 3.5.
<ajs6f>We just want to see something working. We don't need pivot facets. {grin}
<JasonDGI>but my gsearch is having trouble talking with solr12:39
<ajs6f>What version of solr?
<JasonDGI>4.1 (futures6)12:40
<ajs6f>Back it down, is my suggestion.
<JasonDGI>eddies: monitor12:42
<cbeer>JasonDGI: isn't that just a permissions error?12:49
<JasonDGI>the gsearch thing?
<JasonDGI>like file permissions, or do you think i have a user/pass in a config file wrong?12:50
<cbeer>yes. (and it was)
you were the owner of the directories
(restarting tomcat)
<jcoyne>2. (intr) to advocate a cause with the object of making converts12:52
wrong chan12:53
<ajs6f>Oakey-dokey, I got to the heart of kaarefc's second issue.12:56
The truth is simple. We never modfify datastreams. We only delete and recreate them.
So I'm going to make some changes to the legacy-api module to try to actually _modify_ things.
<cbeer>apparently you can not install gsearch before you start your solr core for the first time?12:57
<JasonDGI>is that why gsearch just disappeared?12:58
are you changing that?
<cbeer>JasonDGI: working on it.
JasonDGI: also looks like the islandora core was never loaded?12:59
and the schema/solrconfig need to be updated for solr4?
<JasonDGI>yeah, but probably not required yet13:00
<eddies>fedora4lib house folks: house meeting/standup at 2:00pm today13:01
<cbeer>JasonDGI: how/where is gsearch configured? I see this error:13:07
dk.defxws.fedoragsearch.server.errors.GenericSearchException: Connection error (is Solr running at http://localhost:8983/solr # the Solr server base url/update ?): java.net.ConnectException: Connection refused
<ajs6f>I'm going to hold on on trying to fix kaarefc's second issue until we get some of this model vs. service stragithened out, and since jcoyne is working on that now, I'll let him finish and then this'll be much easier.
<JasonDGI>i probably typed an incorrect path somewhere, there are a couple of different config files that iwill examine13:08
* jcoyne leaves
* eddies leaves
* JasonDGI leaves
* ajs6f leaves13:11
<kaarefc>Okay, so this never got to work13:12
I'm submitting the code *as is*, feel free to disqualify me :-)
You can find a branch of fcrepo at https://github.com/kaarefc/fcrepo413:13
It was supposed to make the simplest possible attempt at maintaining a SOLR index
What it actually does is parse JMS messages and print exceptions to the log :-)13:14
Also, it has the most horribly forcefully added SOLR instance hardcoded into fcrepo-webapp where it should NEVER have been
There's no documentation, and the code is basically hack upon hack13:16
But I submit it to you anyway for comments. I'll probably get it in some working state when I get back to Denmark, mostly because I wanted to see how real-time updating SOLR from Fedora messages would work out in practice13:17
It's rather doubtful that the code as is will ever have any practical use :-)
Thanks for this year's #code4lib, and for the amazing fedora futures discussions13:18
Enjoy the last time in the house!
I'm most likely offline until I reach Denmark, Friday past noon US time
* kaarefc leaves13:19
* anusha leaves13:27
* JasonDGI joins15:20
* eddies joins
* eddies leaves
* eddies joins
* barmintor joins15:24
<cbeer>dk.defxws.fedoragsearch.server.errors.GenericSearchException: Connection error (is Solr running at http://localhost:8983/solr # the Solr server base url/update ?): java.net.ConnectException: Connection refused15:28
JasonDGI: ^
eddies: https://github.com/futures/fcrepo4/pull/1615:37
eddies: https://www.pivotaltracker.com/story/show/4125817315:38
<JasonDGI>kaare's core data model?15:52
* ajs6f joins16:14
<cbeer>eddies: https://gist.github.com/BHSPitMonkey/36aa5674a24b19393387/16:35
* github-ff joins16:37
[fcrepo4] cbeer pushed 1 new commit to master: http://git.io/C8qnYw
fcrepo4/master 865cb5e Chris Beer: use the webapp test.port path
* github-ff leaves
* ajs6f leaves17:58
* barmintor leaves18:17
<cbeer>eddies: https://gist.github.com/cbeer/495744418:52
* JasonDGI leaves19:38
* eddies leaves20:15
* github-ff joins22:12
[fcrepo4] jcoyne opened pull request #20: Extract FedoraService from FedoraObjects and FedoraDatastreams (master...service) http://git.io/WMdAvA
* github-ff leaves