Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* eddies leaves02:55
* eddies joins04:52
* eddies leaves
* eddies joins
* eddies leaves07:55
* eddies joins08:05
* eddies leaves
* eddies joins
* anusha joins08:43
* JasonDGI joins09:01
* anusha leaves09:05
* eddies leaves09:17
* eddies joins
* eddies leaves
* eddies joins
* anusha joins09:18
* anusha leaves09:47
* anusha joins10:00
* ajs6f joins10:02
* anusha leaves10:30
* anusha joins10:51
<eddies>Hi ajs6f! long time no see on irc =)10:58
<anusha>my current meeting is running over. I am not sure I am going to make it to todays standup.10:59
<ajs6f>Scott and I are talking about gh-pages and JavaDocs
<eddies>anusha: can you send oxford vm info to ff-internal then?11:02
* tecoripa joins11:03
<ajs6f>Scott— Pivotal Tracker is being used for Fedora Futures issue tracking.11:04
Not the Duraspace Jira you might expect.
<tecoripa>ok -- do you have a link
<ajs6f>I just lost the call11:05
<JasonDGI>lost connection
<barmintor>https://www.pivotaltracker.com/projects/684825
<ajs6f>Scott— I thikn Eddie would have to add you to a list for you to see it.
<tecoripa>thanks, Ben
ok
* barmintor waves
<JasonDGI>calling in now
<tecoripa>yep, I'm prompted to login
<ajs6f>I'll try to call in. Will take a minute.11:06
* jgreen joins11:07
<ajs6f>back in the call.
* jgreen leaves
* jonathangee joins11:08
<tecoripa>is there a wiki page documenting tyhe engines you're testing?11:09
<ajs6f>Yes— I'll find those liks.
<tecoripa>thx
<ajs6f>https://wiki.duraspace.org/display/FF/Prospective+platforms+for+long+term+effort11:10
<anusha>eddies: Yes. In another hour or two, you should get the details.11:11
<ajs6f>Scott— Lily is at:
http://www.lilyproject.org/lily/index.html
<tecoripa>got it
<eddies>JasonDGI: https://wiki.duraspace.org/display/FF/Lily11:13
* anusha leaves11:18
* tecoripa leaves11:26
* anusha joins11:31
<JasonDGI>@eddies going for some lunch, be back in a bit
<eddies>ok
* JasonDGI leaves11:32
* github-ff joins11:35
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/WFyyTg
ff-modeshape-prototype/master c6b8b2c ajs6f: Added note about multi-instance operation to README
* github-ff leaves
* github-ff joins11:36
[ff-modeshape-prototype] ajs6f pushed 1 new commit to master: http://git.io/zkHn9A
ff-modeshape-prototype/master 46abf14 ajs6f: Added note about multi-instance operation to README
* github-ff leaves
<cbeer>ajs6f: thanks for the modeshape clustering stuff. i got the two instances working on EC2 yesterday11:50
<ajs6f>Rock an/or roll!
<cbeer>i had to use JDBC_PING in the jgroups config (got a gist somewhere i'll dig up)
<ajs6f>Cool. We wanted to do that anyway, right?
<cbeer>and i turned out clustering in modeshape itself too
<ajs6f>Did you push that back to master?11:51
Let me know if you need any help getting them onto Oxford's gear.
<cbeer>not yet, but hopefully soon.
<ajs6f>Cool. Come to think of it, I would have seen a notice from Github, so I don't know why I just asked that.11:52
<cbeer>once it's up on oxford's stuff, i'll stop draining our beer budget :P
<ajs6f>ALL RESOURCES TO BEER!
<barmintor>I'm kind of intrigued by Lily's content modeling approach. I'm not sure it's an option (cf. Oxford & archival storage), but it's interesting to think about how FCR3 would migrate to it11:56
* barmintor gets back to his dayjob
* barmintor which is apparently wondering why rubygems claims it doesn't know hydra-head11:57
<ajs6f>Ditto. It seems about as powerful as JCr's.
I really appreciate the ability to validate.
And the versioning model looks pretty elegant as well.
* anusha leaves11:58
<ajs6f>I would think that Jonathan's example of a Fedora CMA instance in JCR would translate pretty easily.
To Lily, that is.
<barmintor>ajs6f: it would also play really nicely with CModels as mixins, assuming a Lily object can have more than one recordType11:59
*and* it avoids the namespace collisions that FCR3 hsa
s/hsa/has/12:00
<ajs6f>How about just this:
http://docs.ngdata.com/lily-docs-current/g1/407-lily.html#Mixins
?
<barmintor>OH, hurr… yes, that's about it
<ajs6f>JCR has mixins, too. I'm really not seeing one as having the superior model right now.
If Lily is even one-tenth as fast as it seems… that's pretty hard to beat.12:01
<barmintor>ajs6f: the numbers seems pretty close to modeshape's, weren't they? Maybe I was looking at the wrong graph.
<ajs6f>I thought they were way, way faster?
<cbeer>iirc, lily was pretty fast and consistent in 1-thread operations
<ajs6f>Do you have the link to that Gist that had the results?
* barmintor looks for those graphs
https://gist.github.com/26085aeb31db2a8ddd9312:02
<cbeer>but under load it was on-par, i think
(fwiw, modeshape as also pretty fast in 1-thread)
<ajs6f>Yeah, they do look comparable.12:03
<cbeer>it'll be nice to run these under more controlled conditions
<ajs6f>I thikn I may have accidentally confused "elapsed" and "latency". {blush}
<cbeer>do we have the oxford VMs yet? or is that still in progress?
<ajs6f>Lily has basically no latency.
Yeah, but Anusha has yet to send out the info.
<barmintor>ajs6f: yeah, the latency numbers would be impressive return times :D
<ajs6f>Shoot for the moon!12:04
Actually, Modeshape looks a jot faster in both runs (5-thread and single-thread). But not by a huge amount. We need the real-er tests.12:05
Those awesome latency figures are interesting in their own right. I wonder how they're doing that?12:06
<barmintor>lily only runs on localhost12:07
<ajs6f>Oh. We haven't been able to test it remotely?!
<eddies>fyi, i forked lily to futures: https://github.com/futures/ff-lily12:08
<barmintor>ajs6f: I was just joking. IDK why latency is so low on that test.
<eddies>and just to be perverse, i've now submitted the first pull request to lilyproject…to correct a link in the README ;-)
<ajs6f>Har. Har. Har.
<barmintor>eddies: I wonder if they're happy or confused right now.12:09
<eddies>heh
well, now we can't say lilyproject doesn't have any pull requests, eh?12:10
holy cow, lily has a *lot* of maven modules
<ajs6f>Yeah, the Eclipse import is taking forever for me.12:14
And they're using a lot of Lily-custom plugins...
<barmintor>I vote no on any project that has learned how to do stuff with Maven.12:15
<ajs6f>That would be us, no?
<barmintor>Ehh12:16
<ajs6f>If we vote against ourselves, do we just disappear in a puff?
* barmintor makes a wobbly gesture
<eddies>yeah the eclipse import took a few mins
<ajs6f>Schroedinger's codebase.
Looks like Lily is using this:12:21
http://docs.ngdata.com/kauri/about.html
as REST services framework.
That's a new one for me.
<eddies>yeah, i just read this:12:22
Tip: when you make changes to the Lily source code, after building with Maven you can directly restart Lily. There is no packaging or deploying to do. This is because the Kauri Runtime platform on which Lily runs directly loads the project dependencies (= constructs the classpath) using your local Maven repository (~/.m2/repository).
(from http://docs.ngdata.com/lily-docs-trunk/422-lily/g1/434-lily.html)
<ajs6f>Mm.12:23
I'm not a fan of obscure frameworks.
<barmintor>surely that's optional, though
<ajs6f>_That_ is a good question.12:24
<barmintor>I'm gathering that it's just a way to more quickly test changes
but...
<ajs6f>One way to find out— by examining the code.12:25
And trying to use Lily minus Kauri.
<barmintor>oh, I get it12:27
it's not a webapp
Well, the singular developer appears to be attentive12:33
* JasonDGI joins12:35
<ajs6f>Well, it can be not a webapp and use a popular module system (e.g. OSGi).12:39
s/and/but/
Hm. It's looking to me like Kauri is just brought into play for a standalone server platform. There don't seem to be any weird dependencies. REST/HTTP is built up with ordinary JAXRS.12:45
That's not too bad.
I think we should be able to embed in our own app without difficulty.
It's not totally clear to me that we will need to fork Lily. It seems like we could just write a code base against the repo API and impl modules. I'm not sure we even need to worry about the indexer, mapreduce, pluginregistry, etc. modules.12:51
jason— have you set up a Fedora-Lily prototype project?12:58
<JasonDGI>as in project in github?12:59
<ajs6f>Yeah, akin to ff-modeshape-prototype.
<JasonDGI>i havent put anything in github yet13:00
<ajs6f>Okay, then if you like, once I get Lily built (building now) I'll make such a project— a simple webapp with Lily and JAXRS packed in.
<JasonDGI>sounds good13:01
<ajs6f>Cool. I'll ping everyone with it later today.
<JasonDGI>when does hackfest start? i need to know for travel arrangements?13:16
<ajs6f>I think it's the 5th.13:32
Man, I wish Lily would publish to Maven Central. Or anywhere, for that matter. This is a long build to watch.
<eddies>JasonDGI: you should plan to arrive on the 5th, we start work first thing on the 6th13:36
<JasonDGI>kk thanks
<eddies>ajs6f: are you using eclipse?13:37
<JasonDGI>@ajs6f yeah, it took me 20 minutes
<eddies>and if so, did you resolve all these eclipse errors for lily?
<JasonDGI>i did it in netbeans, the only errors i got were missing dependencies
<eddies>i think i just need to add stuff for the eclipse plugins to the poms13:38
sigh
<ajs6f>No, see above— I decided to try writing against a built set of Lily modules.
<JasonDGI>or get netbeans :)
<ajs6f>Eddie— these days I add it directly to the Eclipse settings.
<eddies>i don't think i have the mental bandwidth for another ide ;-)
ajs6f: what settings?
<ajs6f>Hang on, let me get th name.13:39
Settings/Maven/Lifecycle Settings
Sorry, Settings/Maven/Lifecycle Mappings
It's a piece of POM that just deals with the lifecycle mappings.
<eddies>my maven settings don't have a "lifecycle mappings" section13:40
<ajs6f>Are you on Juno?
<eddies>yah
<ajs6f>Hm.
What version of the Maven plugin?
<eddies>m2e version 1.1.0.20120530...
mebbe a bit old?13:41
checking for updates
what version do you have?13:42
<ajs6f>Hang on...13:43
<eddies>ok, i see latest m2e is 1.2.0.20120903...
<ajs6f>Well, Settings/Maven/Lifecycle Mappings is recently added.13:44
So that might the deal.
<eddies>yeah, well about time. that was a damn annoyance with m2e
<ajs6f>Agreed. It doesn't eliminate the problem totally, though.
You still have to do extra setup.13:45
But you are getting something out of it.
Integration of the Maven lifecycle with the Eclipse facilities.
<eddies>jasondgi or ajs6f, did you do the mini HBase (as noted in lily's readme) or another way?
<ajs6f>And publishing "null" Eclipse info into a Maven plugin doesn't look too hard. More people ought to do it, or even valid info.13:46
<eddies>mini aka dummy hbase
<JasonDGI>nope
<ajs6f>I'm finishing build and I want to try the tests. Haven't gotten that far.13:47
Oh, wait, I see what you mean.
No, I just took the default. That's probably part of why it took so long.
<JasonDGI>one of the zookeeper related tests failed for me, i had to disable it to continue13:48
<ajs6f>I got through first time fine...?
Just finished
<JasonDGI>i guess it just doesnt like me
<ajs6f>;)
<eddies>jasondgi: i take it this means your passport is sorted? =)13:49
<JasonDGI>its supposed to be on its way13:50
ill get an update early next week
<eddies>fyi, lily maven artifacts are published here: http://lilyproject.org/maven/maven2/deploy/14:05
<ajs6f>Oh, gosh. I wish I'd known. Oh, well, keep moving.14:06
<eddies>yeah, i just found it14:07
<ajs6f>WTF? I just finished that long build and I see no lily-repository modules in my repo. Grr. I'm going to use the URL Eddie just passed.14:08
<eddies>ajs6f: ok, that took way too long to update eclipse14:10
now i see the lifecycle mapping menu item
<ajs6f>Just finished? Yow.
<eddies>well, i was making tea
and on the phone ;-)
<ajs6f>Looks like we're both spending a lot of our day waiting.
<eddies>so what am i supposed to do now?14:11
even better, can you post your mapping in a gist? =)
pretty please
<ajs6f>I didn't make a mapping, remember/
?
Tell you what.
Either I can post my mappings from another project, to give you the idea.14:12
Or.
<eddies>oh, i see an option to "open workspace lifecycle mappings metadata" which is just an empty xml file
<ajs6f>You can just use the Eclipse "Quick Fix" context menu item for all those annoying errors.
Hit Quick Fix -> Ignore in Preferences.
<eddies>i thought that was what you had tweaked
<ajs6f>You can go back and change that to Execute if you want.
No, I never fooled with Lily in Eclipse except just bringing it in to examine it.14:13
<eddies>hrm. i didn't even try quick fix. that didn't use to work at least in previous versions of m2e
<ajs6f>I don't want to fork it, I want to use it.
It does now.
That's the big diff.
<eddies>i want my fork and eat it too14:14
<ajs6f>That's indigestion waiting to happen.
This is annoying:14:22
1) Lily's documenation is for 1.3. Their release is 2.0
2) It only covers connecting to a server (presumably running on their platform). I want to instantiate a server.14:23
I keep looking for some kind of RepositoryFactory type deal.
<eddies>that was my pull request to point to the docs for trunk14:27
<ajs6f>Ah— do we actually have the URL?
<eddies>http://docs.ngdata.com/lily-docs-trunk/index.html
<ajs6f>Thanks!
Urg. This is still not so good. I can't find anything that tells me how to produce a Repository, other than tracing the code and finding a constructor for the one kind of concrete repo that seems to exist: HBaseRepository. That constructor is not so useful for quick work:14:35
public HBaseRepository(TypeManager typeManager, IdGenerator idGenerator, RowLog wal,
HBaseTableFactory hbaseTableFactory, BlobManager blobManager, RowLocker rowLocker)
You got to build up the whole machinery.
It seems to me that Lily wants you to use their server, which isn't a fate worse than death, but would engage us with this Kauri thing.14:36
Well, let me see how hard it is to actually use that contructor,14:37
.
<JasonDGI>im definitely not looking forward to making a one line change and rebuilding lily for 20 minutes :(14:38
<ajs6f>Amen.
But if we can use Lily as a set of libraries, we're okay.14:39
That's why I really want to find out how to embed it.
Well, that constructor was a bust. I went right down the rabbit-hole of uncommented Hadoop code. There's a command-line launcher here:14:55
https://github.com/futures/ff-lily/blob/master/cr/server-test-fw/src/main/java/org/lilyproject/lilyservertestfw/launcher/LilyLauncher.java
and I think I'll see if I can hotwire it.
<eddies>barmintor: i'm making my ask to 3floyds15:09
do i ask for barrels, kegs, bottles, dealer's choice?
<ajs6f>Godspeed and good luck, Eddie. Fedora's survival as an entity may depend on this beer.
<eddies>heh15:10
cbeer: have any opinion on the matter?
<JasonDGI>it worked for the pyraminds, it will work for fedora
<eddies>fingers-crossed15:20
i'm turning in
did anything come in from anusha?
think positive thoughts about our beer ^H^H^H^H lily and i'll talk to y'all tomorrow15:22
<JasonDGI>have you guys been deploying lily manually (just moving jars) or what?15:23
<ajs6f>" Hey, let's be careful out there." http://www.imdb.com/name/nm0175700/:
I haven't yet got it running, because I so much don't want to run it on their weird platform.
I've been running around trying to buld a repo in code.
But I'm almost rady to give up.
The only way that seems available is to reuse their test amchinery.15:24
<JasonDGI>go home, its getting late ….. somewhere
<ajs6f>I am home. It snowed enough here that I stayed home.
I'm on my deck smoking my pipe watching my dogs behave like idiots in the yard.
<eddies>hey, i just got confirmation that northwestern will be providing us some of the equipment we need (e.g. projector)15:25
<ajs6f>Which makes this much less frustrating.
<eddies>nulib++
<ajs6f>Good folks.
If y'all have a chance, do visit their older library building.
Built by the family that owns International Harvester in teh 30s.
It's beautiful, fully restored.
Gorgeous wood craft.15:26
<eddies>jasondgi: does pei have any good beer we should be getting you to bring down?
<ajs6f>Bill parod took me on a tour.
Their new building is a brutalist nightmare.
<eddies>i'm guessing anything from pei isn't going to have been widely distributed outside of the island
<JasonDGI>probably, i dont drink it so i wouldnt know
<eddies>ajs6f, are you talking about the library where the stacks are like spokes out from the center?15:27
<ajs6f>That's the new one.
<eddies>that's still pretty old then :-P
<ajs6f>The librarians hate it because the stacks are structural, so it can't be adapted for new purposes.
Yeah, it's from the 60s.
When architeture died in the West.
But I don't have strong opinions or anything.15:28
<eddies>jasondgi: ping jonathan when you see him then =) unfortunately, singapore is basically tiger beer, which is basically bud :-P
<ajs6f>I'm going to back up on Lily. I think we need to just run their platform and put our app alongside it, to get anywhere with any speed.15:29
I can try some Maven trickery to make that easier.
It'll start to feel like the hated Fedora test machniery, where we're setting up and tearing down whole containers.15:30
But it shoud get us going.
Now I'm going to try launching Lily naked. Hope this works.
Oh, it starts with installing HBase, Hadoop, ZooKeeper and Solr.15:32
Okay, this will tak a bit.
<JasonDGI>X(
* JasonDGI leaves15:38
<ajs6f>What's awesome is that the Hadoop "Getting Starting" instructions (which take many paragrpahs, covering config of various kinds that is necessary) end with:
Good luck.
Thanks, Hadoop-guys.
* jonathangee leaves15:40
<ajs6f>For reference, I'm working from this:15:43
http://docs.ngdata.com/lily-docs-current/414-lily/432-lily.html
Okay, I was able to get a test Lily instance running using their "standalone" script.15:58
I think.
It shows logs, but I haven't tried to connect to it with a client and do anything yet.
I guess that's the next step.15:59
<barmintor>eddies: sorry, was in Stephen's office16:05
I'd take whatever they've got
<ajs6f>I I think Eddie went to bed ("i'm turning in").
<barmintor>oh, err… yeah16:06
<cbeer>and then kept talking
<barmintor>well, surely he put together that I was unresponsive :P
<ajs6f>He's pretty quick-witted.16:07
<cbeer>but maybe not at 4am
<ajs6f>You'd think he'd be used to it by now.
Wheeewwww.16:18
Okay, I got a JAXRS app up and talking to Lily.16:19
But the app doesn't embed Lily.
I had to run Lily separately using their gear.
I don't like that at all.
Lily has a lot of heavy, heavy dependencies. Hadoop, HBase, Zookeeper, and Solr.16:20
Packaging all that together in an easy-to-run way would be… interesting.
<cbeer>that's kinda how i feel too, just from the outside16:27
although those are all good buzzwords
<ajs6f>{grin}
I'm writing a mail to ff-internal to describe what I saw.
<cbeer>actually_seeing_things++
<barmintor>@karma doing_work16:28
hmm...
<ajs6f>Well, mostly what I didn't see— an embeddable anything.
We're talking about a multi-process system.
Okay for eScieDoc, but Rock and Roll Hall of Fame?
<barmintor>current installs are probably familiar with running a RDBMS, Tomcat and possibly an external Solr16:29
<ajs6f>Right.16:30
<barmintor>but those are easier processes to setup than Lily's dependencies
<ajs6f>But without too much diffculty (just Maven darkness) we could build Fedora 3 into a distributable Jetty.
I don't think we could that with Lily.
<barmintor>Well, there is a binary distro16:31
<ajs6f>Yeah. I know plenty of sysops who are comfortable with running Java container.
But not too many who feel the same way about Hadoop.
Of Lily?
That's what I was using.
There's also an "enterprise" version. I wonder what kind of packaging/deployment options are particular to it...16:32
<barmintor>some of this is also consequence of what we suggested was interesting about Lily, eg HDFS
<ajs6f>True.
If we want to power, we have to pay.
the power
But our budget is actually defined by our community, not ourselves.
How much can we ask them to take on?
<barmintor>also we'd want the upstream part to be embeddable in F416:33
<ajs6f>Maybe a multi-process server _can_ be effectively pacakged.
Upstream?
<barmintor>the lily part. Hadoop and hbase are separate procs, no?16:34
<ajs6f>Right.
I don't know enough about H* to know how easily they could be embedded, but I imagine it's feasible.
<barmintor>Also, if I'm listening to a Swedish punk/pop band, why is Spotify advertising a playlist of Disney love songs to me?16:35
<ajs6f>BEcause Disney owns copyright to everything.
Including you.
<barmintor>whoops, that band is Norwegian. Now it all makes sense.16:39
<ajs6f>I'm heading out. have a good night, all.16:48
* ajs6f leaves16:52