Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* mikeAtUVa leaves00:11
* mikeAtUVa joins
* ksclarke leaves00:50
* kaarefc joins01:16
* kaarefc leaves01:23
* kaarefc joins01:39
* kaarefc leaves01:58
* kaarefc joins02:10
* kaarefc leaves02:15
* kaarefc joins02:51
* kaarefc leaves04:53
* fcrepo-bot joins07:50
* nbanks joins08:13
<pivotal-bot_____>Nigel Banks started "Add Mulgara support to JMS Indexer" https://www.pivotaltracker.com/story/show/5819379008:16
* jonathangee joins08:19
<pivotal-bot_____>Nigel Banks added comment: "https://wiki.duraspace.org/display/FF/Triplestore+Setup" https://www.pivotaltracker.com/story/show/5819379008:35
Nigel Banks added comment: "https://github.com/nigelgbanks/fcrepo-jms-indexer-pluggable/compare/futures:master...patch-1?quick_pull=1" https://www.pivotaltracker.com/story/show/5819379008:40
Nigel Banks added comment: "Corrected README " https://www.pivotaltracker.com/story/show/58193790
Nigel Banks added comment: "Corrected README
https://github.com/nigelgbanks/fcrepo-jms-indexer-pluggable/compare/futures:master...patch..." https://www.pivotaltracker.com/story/show/58193790
* fcrepo-bot leaves08:52
<pivotal-bot_____>Frank Asseg added "Publish ingest benchmarking tool used for the SCC cluster comparison on github" https://www.pivotaltracker.com/story/show/5896372809:13
Frank Asseg started "Publish ingest benchmarking tool used for the SCC cluster comparison on github" https://www.pivotaltracker.com/story/show/58963728
Frank Asseg added comment: "Project is available at https://github.com/futures/benchtool" https://www.pivotaltracker.com/story/show/5896372809:14
Frank Asseg added comment: "Trying again with the new version of fedora I ran into the following exception: https://gist.github.com/fass..." https://www.pivotaltracker.com/story/show/5228954509:16
Frank Asseg finished "Publish ingest benchmarking tool used for the SCC cluster comparison on github" https://www.pivotaltracker.com/story/show/5896372809:17
* ksclarke joins09:18
<fasseg>cbeer: I have an exception happening after a couple of ingests. Can I ask you to take a look at it and maybe give me a hint what might cause inifinspan to be unable to store binary data there? And of course there was still plenty of disk space available at the time the exception happenened...09:24
https://gist.github.com/fasseg/7007565
* tecoripa joins09:40
<nbanks>Do I need to set JMS config to receive messages? See here https://wiki.duraspace.org/display/FF/Deploying+Fedora+4. Regardless how do I get fcrepo to log all the JMS messages it is producing?09:41
<fasseg>hmm I dont think it's that easy to log JMS messages09:55
I tried this before in another project and we had to handwrite something :/
so we basically had to listen for incoming msgs and log them by hand :/
<tecoripa>nbanks: not sure how fcrepo4 is wired, but in general, you need to set up a consumer (listener) to receive messages, then do something with them
that's usually something you do yourself, not something that comes bundled with a producer09:56
(though now that I think of it, a simple message logging facility for testing my be useful to distribute with fedora4)
<nbanks>fasseg: I do have the fcrepo-jms-indexer-pluggable running but it doesn't seem to recieve any JMS messages. Good to know that it's difficult to have fcrepo log the JMS messages it generates, I'll drop that for now.10:00
<fasseg>nbanks: you can maybe try and monitor the activemq queue, there are some tools for that, in order to see if anything is in there...http://activemq.apache.org/how-can-i-monitor-activemq.html10:01
<nbanks>fasseg: thanks!
<tecoripa>questions about building fcrepo4...10:02
I'm doing a naive build: I checked it out of my git repo, made sure it was up-to-date with upstream master, and then ran a simple "mvn test verify"10:03
I'm seeing some problems, one of them a show stopper.
first (not a show stopper):
when building fcrepo-metrics, I get a number of java.lang.IllegalStateException: Class blah already instrumented exceptions10:04
doesn't prevent the tests from passing, but it's pretty ugly
are these exceptions to be expected?
<fasseg>lemme try and reproduce...10:05
I have those too: Caused by: java.lang.IllegalStateException: Class org/junit/runner/notification/RunNotifier$7 is already instrumented.10:06
<tecoripa>fasseg: yep. I see they occur in othere tests, too10:07
<fasseg>it's the jacoco plugin trying to instrument classes for getting code coverage, which i think have already been instrumented because it's defined in the maven paretn pom
but I dont know enough about jacoco to give a qualified judgement, to why this happens10:08
I think Adam wrote this bit, so you might wanna ask him
<tecoripa>ok, thanks. I see a lot of those exceptions in fcrepo-storage-policy tests, too10:10
the second error, which requires that I kill the JVM with a -9 signal:10:12
java.lang.OutOfMemoryError: PermGen space
it occurs when running org.fcrepo.integration.SanityCheckIT, presumably at the end of the test ctcle10:13
<fasseg>tecoripa: can you try running it with MAVEN_OPTS="-XX:MaxPermSize=256m -Xmx512m" ?
<tecoripa>okay. I thought it might have been something like that
is that documented anywhere?10:14
I'll give it a try.
<fasseg>yeah it's on the bottom of the readme in fcrepo4
https://github.com/futures/fcrepo4
* ajs6f joins10:15
<tecoripa>fasseg: yep, I see it now. I was just going off the contibutor wiki page:10:16
https://wiki.duraspace.org/pages/viewpage.action?pageId=34654611#Contributing/GuideforNewDevelopers-Runningthetests
(told you this was a naive install)
* ajs6f1 joins10:19
* ajs6f leaves10:20
<tecoripa>fasseg: that worked, thanks. Some fcrepo-webapp tests failed, but I'll look more into that now.10:22
<fasseg>cool
<tecoripa>I added a note to the Testing page on the wiki to consult the fcrepo4 README10:28
https://wiki.duraspace.org/display/FF/Testing
<nbanks>fasseg: Have you ever set up the webconsole for activemq http://activemq.apache.org/web-console.html, or have you used jconsole to connect with JMS?10:31
<fasseg>nbanks: nah never the web console, but connecting with JConsole is pretty straight forward as long as you're on the same host10:32
but I'm not sure that you will be able to see queue stats in the JCOnsole JMX view
I never did this
<nbanks>For JMX I followed the instructions here http://activemq.apache.org/jmx.html by chaning the broker in the webapp to use "useJmx="true"" but I was unable to attach to the process or to activemq by using the url service:jmx:rmi:///jndi/rmi://localhost:61616/jmxrmi10:34
<fasseg>Can't you just connect to the JVM that ActiveMQ is running in and you will have the jmx beans exposed there? Do you have to connect via jndi?10:35
<ajs6f1>nbanks: Did you start the JVM containing the broker with flags to make JMX available?10:36
<nbanks>ajs6f1: Probably not, I just modified this file https://github.com/futures/fcrepo4/blob/master/fcrepo-webapp/src/main/resources/config/activemq.xml to useJmx="true" then recompiled everything10:37
fasseg: I can't attach to the webapp process directly, although it's listed in the console.10:38
<ajs6f1>nbanks: http://stackoverflow.com/questions/856881/how-to-activate-jmx-on-my-jvm-for-access-with-jconsole
<nbanks>ajs6f1: thanks!
<ajs6f1>JMX being available from outside the JVM is a security hole, so you have to take positive action to indicate that you want it open.10:39
<nbanks>ajs6f1: In that case we should probably do something about fcrepo-jms-indexer-pluggable as I was able to connect to that without any special configuration.10:40
<ajs6f1>In the same web container?
<nbanks>No seperate containers10:41
fcrepo-jms-indexer-pluggable is just running by itself at the moment
<ajs6f1>That's a separate container.
<nbanks>Yup
<ajs6f1>If you are looking at two separate system processes, you are certainly looking at two separate JVMs and two separate containers.
Check the Maven for fcrepo-jms-indexer-pluggable. Configuration settings for a "jetty" launch should be in there and should make clear how it is starting.10:42
Keep in mind that Maven may fork a new JVM to launch a webapp in Jetty, so the flags you pass to Maven may not be passed to the JVM containing your running webapp.10:43
<nbanks>ajs6f1: thanks, I didn't think that the flags might not get passed along.10:45
<ajs6f1>Maven (and Maven plugins) can and sometimes will fork a process for various reasons. This kind of behavior is usually adjustable, but by various means depending on who and why the fork is occuring.
<nbanks>ajs6f1: ya it seems as though -Dcom.sun.management.jmxremote* doesn't get passed along to jetty.10:46
ajs6f1: I'll take a look at the mvn jetty docs and look for where I can set this stuff up.
<ajs6f1>Right.
You may find it quicker for now to deploy into your own containers, to get the settings right, then to incorporate them in the Maven config.10:47
<nbanks>You mean like running tomcat locally and depolying fcrepo4 war to it?10:48
<ajs6f1>Right.
That will take the extra complication of Maven out of the way, for the moment.
You can concentrate on getting the JMX flags right.
<nbanks>I'll give it a shot Osman got the vanilla war working with tomcat right?
<ajs6f1>Several people have, with a couple of cmdline flags to set up persistence and the like. See awoods' exchanges on the list to this question.10:49
afk bb for standup10:50
* ajs6f1 leaves
<nbanks>ajs6f1: thanks for the help I'll check that out.10:51
* ajs6f joins10:54
* kaarefc joins
<ajs6f>Standup in 5 minutes.10:55
* barmintor joins
jonathangee: is the FCR3 GsearchDOManager still a thing? Or is GSearch driven by the messaging API now?10:56
<ajs6f>barmintor: Normally messagin.
<jonathangee>I don't get deep into the gsearch code. i actually actively try to avoid that. So I don't know about the GsearchDomManager, but I know we always set it up using messaging10:57
<barmintor>ok
<ajs6f>I know of no one doing otherwise, but then… maybe?
<barmintor>this stupid class should never have been in the fcrepo-server package10:58
ajs6f, jonathangee: thanks, just trying to work a couple of things out10:59
<ajs6f>Emotionally?11:00
* osmandin joins11:06
<fasseg>https://github.com/futures/benchtool11:10
* jongibson joins11:11
<ajs6f>Standup notes: https://wiki.duraspace.org/display/FF/2013-10-07+to+10-17+FF+Stand-up+Mtgs#id-2013-10-07to10-17FFStand-upMtgs-Wednesday2013-10-1611:14
<osmandin>fasseg: are the ingest objects random / come with f3?11:15
<ajs6f>barmintor: GsearchDomManager ay predate JMS as we know it...11:16
* osmandin leaves11:28
<fasseg>osmandin: the ingest objects are created on the fly from java.util.Random11:34
* kaarefc leaves11:40
* github-ff joins11:51
[fcrepo4] fasseg pushed 1 new commit to master: http://git.io/LN9q_A
fcrepo4/master 9bb96b9 fasseg: added Dummy AuthZ provider to minimal config
* github-ff leaves
<fasseg>oh wow!!11:53
single node performance of fcrepo4 with minmal config throughput is 17.77mb/s compared to 12.85 from fcrepo3 !!
* github-ff joins12:26
[fcrepo4] ajs6f pushed 1 new commit to RDFIteration: http://git.io/16Z5_g
fcrepo4/RDFIteration be6ebee ajs6f: RDF based in JCR hierarchy now being constructed in iterator mode
* github-ff leaves
<ajs6f> fasseg: Hooray!!!
We're gonna make it after all!
<nbanks>:D fantastic12:56
<barmintor>fasseg: I'm going to be interested to see how it runs against the 3.7.1 snapshot14:00
<fasseg>barmintor: I only ran it vs the 3.7 release until now14:01
<ajs6f>fasseg: 3.7.1 has some speedups, so don't run against that. In fact, I suggest hobbling the 3-series repo to make us look even better.14:02
<cbeer>ajs6f++14:03
<barmintor>let me spin up a new fork of 3.3 with the version number changed to 3.7.114:04
<fasseg>and some strategically placed sleeps :)14:05
<pivotal-bot_____>Nigel Banks added comment: "Had the same problem as identified here: http://permalink.gmane.org/gmane.comp.cms.fedora-commons.user/8753" https://www.pivotaltracker.com/story/show/5819379014:17
* jongibson leaves14:23
<nbanks>I can't for the life of me get jconsole to connect with a running tomcat if fcrepo4 is deployed to it.14:32
Also I can only get the alpha-1 of fcrepo4 running in tomcat, the latest doesn't seem to work.
<ajs6f>nbanks: Do you have some log output or the like you can gist?14:41
<nbanks>for the jconsole? nothing as such for alpha-2 running in tomcat I get the same error's as this fellow http://permalink.gmane.org/gmane.comp.cms.fedora-commons.user/875314:42
ahh I'm an idiot14:43
I can connect with jconsole, I had two terminals open one with JAVA_OPTS defined locally and one exported, hence I was unable to connect where JAVA_OPTS was only defined locally.14:44
<ajs6f>This is why I try to avoid using computers.
<nbanks>good thinking
Still not seeing http://activemq.apache.org/jmx.html in the MBeans panel though... maybe I need to enable useJmx="true" with JCR as well as the webapp14:45
ah it's set in jms project as well (duh)14:46
Worth noting though when running alpha-1 in tomcat, it seems to leave processes running when tomcat get's shutdown.14:48
<ajs6f>What kind of procs?14:49
<nbanks>sorry I mean threads, not sure completely though: https://gist.github.com/nigelgbanks/85a14896c39d8c14199914:52
after shutdown the java process continues to run.
<ajs6f>Give it a little while, especially if you have been exercising it with load tests. Those threads are coming out of ISPN and Mode's HIbernate layer, so there isn't too much we can do with them immediately except note them.14:54
<nbanks>ok14:55
* ajs6f leaves15:04
* ajs6f joins15:07
* jongibson joins15:27
* github-ff joins15:43
[fcrepo4] ajs6f pushed 1 new commit to RDFIteration: http://git.io/uSWd1g
fcrepo4/RDFIteration a3b51fa ajs6f: Adjusted tests in fcrepo-http-api to meet small alterations in behavior
* github-ff leaves
<ajs6f>This: http://boingboing.net/2013/10/16/skeletor-insulting-his-minions.html15:46
makes me glad awoods is not that kind of tech lead.
* nbanks leaves15:56
<pivotal-bot_____>A. "Falubrious" Soroka added comment: "https://github.com/futures/fcrepo4/pull/117" https://www.pivotaltracker.com/story/show/5730442616:00
* github-ff joins
[fcrepo4] ajs6f opened pull request #117: Rdf iteration (master...RDFIteration) http://git.io/rGkvoQ
* github-ff leaves
<ajs6f>all (especially barmintor/cbeer): Because I know you have nothing to do and sit around drinking beer and playing darts all day, I'd appreciate any feedback for that PR. It's the first move of two to stop rolling up the generation of RDF from the kernel into giant in-memory Jena Models (which blow out heap under fairly normal circumstances). The second step will be to expose the new Iterator-based RDF generation from the kernel outwards, but I'd like to g16:08
<cbeer>ajs6f: you got cut off.. you'd "like to g..."16:14
<ajs6f>cbeer: I'd like to go drink beer and ignore my responsibilities.16:15
cbeer; It was just a request for feedback on that PR.
especially with awoods out for a few days.16:16
afk bbl
* ajs6f leaves
* ajs6f joins16:20
<fasseg>hm I can use the leveldb infinispan config for running fcrepo4 in a cluster, right? Or is this for single node configurations only?16:33
<ajs6f>I think you can, but cbeer knows most about that.
<cbeer>don't see why not16:40
<ajs6f>We'll know if there's suddenly huge bang and flash that fasseg shouldn't have done that.
* jongibson leaves16:44
* jongibson joins16:45
* ajs6f leaves16:51
<fasseg>infinband 4x QDR my *** network I/O is at ~500KB/sec after a couple of seconds when the buffers are filled...16:56
I really have to catch one of those admins...
Using the leveldb config the nodes dont join a cluster :/17:20
I'm guessing the current leveldb config is not usable, and I have to update the clustered configuration by hand to use leveldb?17:21
Im off for today see you guys tomorrow17:30
* fasseg leaves
* pivotal-bot_____ leaves18:10
* pivotal-bot_____ joins18:11
* ksclarke leaves18:17
* tecoripa leaves19:26
* tecoripa joins19:33
* tecoripa leaves19:34
* jongibson leaves20:07
* ksclarke joins20:48