Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* ksclarke leaves01:21
* fasseg joins04:25
<pivotal-bot>Frank Asseg added comment: "Added this issue to modeshape: ""04:58
https://issues.jboss.org/browse/MODE-2103" https://www.pivotaltracker.com/story/show/61298284
Frank Asseg finished "Create a Testcase for large file handling in Infinispan and submit a bug report if applicable" https://www.pivotaltracker.com/story/show/6129828405:00
<fasseg>I'm now quite sure that the OOMEs when ingesting large files are caused by a modeshape component, so please vote for this bug: https://issues.jboss.org/browse/MODE-2103 :)05:37
* aawoods__ joins06:17
* awoods leaves
* aawoods__ leaves06:22
* ermadmix__ leaves07:53
* nbanks joins08:55
<pivotal-bot>Esme Cowles added comment: "fcrepo-jms-indexer-pluggable" https://www.pivotaltracker.com/story/show/61351914
Esme Cowles started "Build failing: fcrepo-jms-indexer-pluggable" https://www.pivotaltracker.com/story/show/61351914
* kaarefc leaves
<pivotal-bot>Esme Cowles added comment: "I can build fcrepo-jms-indexer-pluggable on MacOSX with no other services running. But with fcrepo4 running..." https://www.pivotaltracker.com/story/show/6135191408:57
* escowles joins08:58
* mikeAtUVa joins09:13
* ermadmix joins09:16
* ermadmix_ joins09:19
* ermadmix leaves09:21
* awoods joins09:23
<pivotal-bot>A. "Outrageac" Soroka added "Break up JcrRdfTools" https://www.pivotaltracker.com/story/show/6139121609:27
* kaarefc joins09:31
<pivotal-bot>Andrew Woods added comment: "On Linux, I get the IndexerGroupIT failures with no other services running, using the standard command: ""09:34
mvn..." https://www.pivotaltracker.com/story/show/61351914
* tecoripa joins
<awoods>All: Can someone on a Mac and someone else not on a Mac try to build "https://github.com/futures/fcrepo-jms-indexer-pluggable.git"?09:35
mvn clean install
Just looking for "git clone https://github.com/futures/fcrepo-jms-indexer-pluggable.git" and "mvn clean install"09:36
<tecoripa>I'll give it a try on my Linux workstation
* ksclarke joins
<awoods>thanks, tecoripa09:37
<tecoripa>does it need to be built within the fcrepo4 project? or will it build standalone?
<awoods>It is a stand-alone project09:38
<tecoripa>build failure09:39
I'll post the output in gist
<escowles>awoods: if the problem with fcrepo-jms-indexer-pluggable is a timing issue, then particular setups might be able to build it09:40
<tecoripa>tests failed for this class:
Running org.fcrepo.indexer.SparqlIndexerIT
<awoods>escowles: I agree, it is likely a timing issue, that for some reason has been prompted by the Versioning work.09:41
<awoods>escowles: In any case it would be good to update the tests to work.
tecoripa: It looks like your errors are with IndexerGroupIT, not SparqlIndexerIT.09:43
<escowles>awoods: is there an elegant way to make the test wait for the triplestore to catch up? i could have it retry a few times, or sleep() for a few secs, etc.
<tecoripa>ah, right09:44
<awoods>escowles: Yes, there is currently wait logic in-place... but inadequate apparently. I believe it waits for as long as 15 seconds.
I get the same errors, tecoripa.
* osmandin joins09:47
<escowles>awoods: 15 seconds seems like a long time to wait... but i'll see if a longer or indefinite wait will succeed
<awoods>escowles: thanks. But I agree, it seems like 15 should be plenty. Did you say that you are getting the same errors that tecoripa gisted only when you have F4 running? Or do you get different errors (like port conflicts)?09:48
<escowles>on my (mac) desktop, it builds fine if i have nothing running, if i have fcrepo4 running, i get a different error: IndexerGroupIT.doIndexerGroupUpdateTest(IndexerGroupIT.java:122)09:50
maybe that test is failing because the update is going to the fcrepo4 running on 8080 instead of the one run by the test?
here's the gist of that build failure: https://gist.github.com/escowles/764237909:52
<tecoripa>I'm seeing two different port bindings in the debug output:09:53
08:38:05 INFO ARJUNA012337: TransactionStatusManagerItem host: port: 34274
and DEBUG 08:38:25.807 (HttpGraphSubjects) Resolving graph subjects to a base URI of "http://localhost:36134/"
<awoods>tecoripa: There should be a third port as well for the jms.09:54
<tecoripa>[INFO] --- build-helper-maven-plugin:1.8:reserve-network-port (reserve-port) @ fcrepo-jms-indexer-core ---09:58
[INFO] Reserved port 36134 for test.port
[INFO] Reserved port 50555 for test.mgt.port
[INFO] Reserved port 43972 for test.fuseki.port
<awoods>tecoripa: right, fuseki, not jma.
* ajs6f joins10:09
* ajs6f leaves10:10
* mikeAtUVa leaves10:12
* mikeAtUVa joins10:13
* kaarefc leaves10:20
<awoods>All: to be clear, "fcr:search" only searches over node paths. correct?10:22
* ajs6f joins10:23
<awoods>All: I have not seen "full text" search capability here, only the ability to find terms within the path of nodes.
<escowles>awoods: i think the issue with fcrepo-jms-indexer-pluggable is that the queries used by waitForTriples weren't updated, so they were using a different query than the rest of the test10:27
<awoods>escowles: nice catch
escowles: can the query be factored out so that the same mistake does not happen again?10:28
<ajs6f>All: Did you know that MODE has a JDBC dirver…?
<escowles>awoods: yes, i'll do that
<awoods>escowles: Also, nice work on condensing the single-node test results.10:29
<escowles>thanks -- i should have update/delete numbers shortly
<awoods>escowles: It is helpful to see the similarities and differences across the various institutions.10:30
* gregjansen joins10:31
<pivotal-bot>Gregory Jansen added comment: "awoods: please try ITs again for this project. I changed a cargo plugin setting and I am hoping that it a..." https://www.pivotaltracker.com/story/show/61225722
<awoods>ajs6f: What did you have in mind with the JDBC connector?10:32
<ajs6f>Possible integrations. (Besides just advertising it as our feature. {grin}). E.g.:10:33
Sorry, better page:10:34
If it's as fast as their academic papers claim, it could be a real contender to supply "live" virtualized SPARQL.
And there's some cute stuff they're doing with OWL I find intriguing. cbeer and I have both noticed how fun it would be to offer infererencing queries (e.g. being able to define subproperty relationships and have queries use them).10:36
<gregjansen>Heck yeah, ajs6f. That is neat. My UNC use case: I'd love to offer SPARQL over legacy DB and data formats.10:39
i.e. linked data disseminator
<ajs6f>gregjansen: RIght. Either federating in MODE/Fedora, or using JBoss' Teiid _over_ MODE/Fedora and whatever else is in play.
And ontop over that.
<gregjansen>Teiid? Is that a Harkonnen word?10:40
<ajs6f>I used Pubby to do that over Fedora3, but our own API is actually better and supports LDP.
(fcrepo4 API, i.e.)
or see:
http://www.jboss.org/products/datavirt.html (which rhauch mentioned in that thread)
which integrates MODE and Teiid.
The more integration points we can offer without having to do any work, the better.10:42
awoods: I'm going to file a ticket for myself to set up a test integration of fcrepo4, the MODE JDBC connector, and ontop. I want to see if this thing has real legs.10:43
<pivotal-bot>Andrew Woods added comment: "With updated code, getting the following error when running "mvn clean install": ""10:44
https://gist.github.com/aw..." https://www.pivotaltracker.com/story/show/61225722
* barmintor joins
<awoods>ajs6f: It may be best to stay focused on the holiday-release... although creating the ticket as a future placeholder is fine.
<pivotal-bot>A. "Outrageac" Soroka added "Rename GraphSubjects" https://www.pivotaltracker.com/story/show/6139819210:45
<ajs6f>awoods: I'll do that. If I have time, I will try the JDBC thing, because if it's really trivial, it would be pretty cool to announce that as an unexpected feature: "Connect to Fedora with anything that speaks SQL!"10:46
No repo framework I know of does that.
* github-ff joins
[fcrepo-jms-indexer-pluggable] escowles created test-query-bugfix (+1 new commit): http://git.io/aUpNiA
fcrepo-jms-indexer-pluggable/test-query-bugfix 2886ab3 Esmé Cowles: Fixing queries used in waitForTriples calls to be the same as other checks (fixes http://www.pivotaltracker.com/story/show/61351914)
* github-ff leaves
<pivotal-bot>bug: Build failing: fcrepo-jms-indexer-pluggable (started) / owner: Esme Cowles
<ajs6f>And there should be no code involved.
* github-ff joins10:47
[fcrepo-jms-indexer-pluggable] escowles opened pull request #13: Fixing queries used in waitForTriples calls to be the same as other checks (master...test-query-bugfix) http://git.io/QQwvvg
* github-ff leaves
<gregjansen>awoods: I wonder if your current error is a JMS port conflict, since it involves the broker bean... I guess I can randomize that port too.. I will see how that's done.
<pivotal-bot>Esme Cowles added comment: "The problem was that calls to waitForTriples were using the pid, but the other method calls were using serve..." https://www.pivotaltracker.com/story/show/6135191410:48
Esme Cowles finished "Build failing: fcrepo-jms-indexer-pluggable" https://www.pivotaltracker.com/story/show/61351914
A. "Outrageac" Soroka added "Test the MODE JDBC connector" https://www.pivotaltracker.com/story/show/6139853410:49
<gregjansen>awoods: shall we make a mvn archetype for Fedora REST developers based on this cargo stuff? (write your code and have working ITs right away)10:50
<awoods>gregjansen: yes, it is the jms.port conflict.
<gregjansen>awoods: kthx
<awoods>gregjansen: an archetype would be nice... slated for later.
<ajs6f>gregjansen: Like an archetype that starts with a "Hello World" endpoint and tests?10:51
<cbeer>awoods: fcrsearch /is/ full text search, but again, only for indexed properties (but there are text extraction frameworks baked into modeshape that we could use...)
<gregjansen>ajs6f: ideally something like that.. would be nice to have equivalent setups for other languages obviously, but at least maven users would love that.10:52
<ajs6f>gregjansen: Maven covers most of the languages you'd want to use on the JVM. JRuby would be the exception.
<awoods>cbeer: I added a text document, and was unable to find it via the "full text" search.
<ajs6f>awoods: did you add it as a bitstream?
<cbeer>awoods: right. like i said, full text over indexed properties. binary content isn't.
<ajs6f>or as a text property?10:53
what cbeer says.
<awoods>cbeer: So you are saying it is "full properties text" search.
<ajs6f>if a given prop is config'd to be indexed.
<cbeer>awoods: sure, but there's also e.g. tika integration (https://docs.jboss.org/author/display/MODE/Built-in+text+extractors) for extracting content from binary properties.10:54
<awoods>cbeer: This is maybe a bit misleading: https://wiki.duraspace.org/display/FF/Design+-+Administrative+Search
<cbeer>" It will search all indexed properties."
<ajs6f>awoods: I don't _want_ us to be offering anything much in the way of full-text there. Haven't we decided to keep the core searchability very very lightweight?
If you want full-text search (and who doesn't) you should use the searching facilities we are building _outside_ the core.10:55
<awoods>cbeer: Sure. It would be great to document which properties are indexed, and how a user can define "indexed properties"
<cbeer>ajs6f: full-text is just one of modeshape's windows into their lucene index.
awoods: the /fcr:nodetypes endpoint documents that for a given repository.
<awoods>ajs6f: I agree, but was slightly misled in walking through the wiki docs linked above.10:56
<ajs6f>cbeer: Sure, but if one relies on that to provide access-type searching, one gets into the whole "synchrnous" indexing config that slow down the repo horribly.
<cbeer>awoods: and we haven't defined how we're letting people define indexed properties beyond a CND update
<ajs6f>awood; We should certainly be clear. Very clear.
* tecoripa leaves
<ajs6f>omw to my office for standup. bbs.
* ajs6f leaves10:57
<awoods>cbeer: I think defining properties in the CND is fine for now. We just need a little more clear user-documentation to that effect.
<cbeer>awoods: won't the content modeling discussions feed into that?10:58
<awoods>cbeer: It would be nice if the conversations intersected, yes.
* ajs6f joins11:00
<osmandin>is the stand up at ready talk?
<escowles>osmandin: we're on hangout: https://plus.google.com/hangouts/_/calendar/eW91cm1lZGlhc2hlbGYuY29tXzVlYzdpNXQ2Z282dTdidHI4aTVrbGJxOTUwQGdyb3VwLmNhbGVuZGFyLmdvb2dsZS5jb20.opn3ai06k1ge0no877ei3cku4o
<awoods>cbeer: And presumably for "Full Text Search" to work, "indexing" will need to be configured?11:01
<ajs6f>escowles: At code4lib?11:02
* tecoripa1 joins
<osmandin>escowles: thx
<escowles>ajs6f: yep
<ajs6f>escowles: We had some folks try for those last time, but they didn't get any.
standup? scrum-dog?11:03
<ajs6f>I guess there's considerable demand.
<gregjansen>sorry wrong hangout
<cbeer>awoods: yes. (or, I assume so. Modeshape might force some structure indexing on us)
<escowles>ajs6f: might have better luck this time with 9 scholarships instead of 2/3
<ajs6f>escowles: I'll make sure those people know about that.
<pivotal-bot>Benjamin Armintor started "Record profiling data for Benchtool 1k/2m/15" https://www.pivotaltracker.com/story/show/6124729011:05
Benjamin Armintor accepted "Record profiling data for Benchtool 1k/2m/15" https://www.pivotaltracker.com/story/show/61247290
<tecoripa1>all : we on ready talk, or google hangout?11:06
<pivotal-bot>Osman Din added comment: "Using ROME library for ATOM, instead of Abdera. Branch: ""
T..." https://www.pivotaltracker.com/story/show/61152842
<pivotal-bot>Osman Din finished "Fix integration test AtomJMSIT exception related to truncated messages" https://www.pivotaltracker.com/story/show/6115284211:07
Osman Din added comment: "Clicking finished for REVIEW. It's not the end product yet (see comments on F3 above, e.g.) -- although this d..." https://www.pivotaltracker.com/story/show/6115284211:08
<tecoripa1>usual hangout? or a new one?
<tecoripa1>my audio is going in and out... is that happening to others?11:13
moving to a different computer... back in a minute11:15
<cbeer>JERSEY 2! JERSEY 2!11:17
<ajs6f>The people have spoken.
awoods: I am in the office through Wednesday, btw11:18
* tecoripa1 leaves11:19
* tecoripa joins11:27
<awoods>cbeer/barmintor: was your drop intentional?
<barmintor>… yes?11:29
but I can come back
I thought you were talking about something w/ Greg
<awoods>barmintor/cbeer/ajs6f: do you have a moment to chat about next steps?11:30
<barmintor>awoods: yes
<ajs6f>next steps for tgiving cooking?
<cbeer>awoods: sorry. next steps for what?
<barmintor>Maybe being bombarded with complaints we can minimize11:41
<gregjansen>cbeer, fasseg: I'd like to put together a PR later today which would incorporate changes to clustered repository.json (tuning following our initial test results) Though perhaps we could branch on futures and work on that.11:55
<fasseg>gregjansen: sounds good, although I dont think I will have time to look at it until tomorrow, since it's already in the evening here12:01
<gregjansen>fasseg: great. I just want us to coordinate the tuning of these settings.12:02
<fasseg>cbeer: Can you tell me why we switched to ISPN 5.3.0.Final instead of 5.2.7.Final which comes with Modeshape? Was this just because of the leveldb addition in ISPN 5.3.0?
<ajs6f>awoods: It doesn't even build, let alone work.12:06
* awoods leaves
* awoods joins
<cbeer>fasseg: sounds likely. i don't know what other bug fixes were in 5.2.7 => 5.3.0 off the top of my head.12:07
<fasseg>kk thx
<pivotal-bot>Andrew Woods edited "HTML description of Solr setup" https://www.pivotaltracker.com/story/show/5745332612:10
Andrew Woods edited "HTML description of Solr setup" https://www.pivotaltracker.com/story/show/57453326
Andrew Woods edited "Solr query recipes" https://www.pivotaltracker.com/story/show/5745338812:11
<ajs6f>awoods: See above. I can't even get this to build. Is it building in CI?12:12
<pivotal-bot>A. "Outrageac" Soroka started "HTML description of Solr setup" https://www.pivotaltracker.com/story/show/57453326
<awoods>ajs6f: see above?
<ajs6f>ajs6f 12:06
awoods: It doesn't even build, let alone work.
<awoods>ajs6f: The build broke recently, and escowles put in a PR just before the standup which presumably fixes it...12:13
<ajs6f>I-test failures in the core modules.
Let's get that merged.
Shall I?
<awoods>ajs6f: see chat from this morning's IRC
ajs6f: I will review escowles PR and push after creating another ticket or two re:search12:14
time for lunch. I'm not as dedicated as barmintor.
* nbanks leaves12:16
* nbanks joins12:18
<pivotal-bot>Andrew Woods added "Simple Search Flow" https://www.pivotaltracker.com/story/show/61406616
Andrew Woods edited "Simple Search Flow" https://www.pivotaltracker.com/story/show/61406616
<awoods>ermadmix: ping12:19
<ermadmix>awoods: andrew?
<awoods>ermadmix: ajs6f will be helping out to get the simple flow of an external search index working: https://www.pivotaltracker.com/story/show/6140661612:20
<pivotal-bot>feature: Simple Search Flow (unstarted) / owner: A. "Outrageac" Soroka
A. "Outrageac" Soroka started "Simple Search Flow" https://www.pivotaltracker.com/story/show/61406616
<awoods>ermadmix: it may be helpful if you have initial comments related to search. Have you started working with the fcrepo-jms-indexer-pluggable project, or have you mostly been working on a mapping strategy between F4 properties and a Solr document?12:22
<fasseg>barmintor: can I ask you to clone this test and check if you can run the tests? no need to let it finish I just want to know if the pom is right...https://github.com/futures/large-files-test12:26
<ermadmix>awoods: looking the the pivotal just posted now, last week just testdrove creating objects in fcrepo and verifying the fileserialization client side, next step is creating a solr doc w/ basic properties12:27
<awoods>ermadmix: It would be ideal if you can 1) follow/install/review the work ajs6f will be doing in terms of setting the Solr integration, and 2) help establish a plan for allowing the user to map F4 properties into a Solr document.
<ajs6f>awoods: are you going to merge escowles PR, or do you want me to?12:35
<awoods>ajs6f: I am running the local build right now12:36
<ermadmix>awoods: ok, on it
<awoods>ajs6f/escowles: IndexerGroupIT.indexerGroupDeleteTest:163 expected:<2> but was:<3>
ajs6f: Can you try pulling down escowles' PR to see if it works on your side? I get the failure noted above.12:37
<ajs6f>from the pr?
<ajs6f>will do
awoods: what branch?
<awoods>ajs6f: it is from escowles' repo12:38
ajs6f: you can add escowles' repo as a remote, then cherry-pick that one commit.
ajs6f: or you can update your ".git/config" file to also pull down PRs.12:39
that sound usefull
<awoods>ajs6f: it is12:40
ajs6f: open the file ".git/config"
ajs6f: in the section labelled "[remote "origin"]"...
add the single line: "fetch = +refs/pull/*/head:refs/remotes/origin/pr/*"
ajs6f: so the [remote "origin"] section will now look like: [remote "origin"]12:41
fetch = +refs/heads/*:refs/remotes/origin/*
url = https://github.com/futures/fcrepo4.git
fetch = +refs/pull/*/head:refs/remotes/origin/pr/*
ajs6f: then do a "git pull", followed by...
ajs6f: git checkout -b pr-013 origin/pr/1312:42
<barmintor>bbl/diet coke12:46
<awoods>escowles/ajs6f: The exact fail moves around, now getting:
IndexerGroupIT.indexerGroupDeleteTest:143->doIndexerGroupUpdateTest:123 There should be 1 file
escowles: I take it that on your machine the build works over and over again?
<ajs6f>awoods: worked for me12:47
trying again12:48
<awoods>ajs6f: great, Mac?
<awoods>ajs6f: escowles is Mac as well.
<pivotal-bot>Scott Prater added "Create repository.json with ServletContainerAuthenticationProvider" https://www.pivotaltracker.com/story/show/61409536
Scott Prater started "Create repository.json with ServletContainerAuthenticationProvider" https://www.pivotaltracker.com/story/show/61409536
<awoods>ajs6f: there is a separate issue that if F4 is running, the webapp test conflicts with the JMS port.
<ajs6f>it wasn't running for me. that's a separate issue.12:49
I will push the PR, just so you can get going... although there still may be an IT timing issue.
* github-ff joins12:50
[fcrepo-jms-indexer-pluggable] awoods pushed 1 new commit to master: http://git.io/UvNxiQ
fcrepo-jms-indexer-pluggable/master b7660d4 Andrew Woods: Merge pull request #13 from futures/test-query-bugfix...
* github-ff leaves
<ermadmix>awoods: heading for New Haven for a 3PM meeting. Hoping gunman lockdown will be ended and I will be back online at 4PM.
<awoods>ermadmix: be safe12:51
* ermadmix leaves
<awoods>tecoripa: I pushed escowles updates to the fcrepo-jms-indexer-pluggable project, can you pull down the latest master and try "mvn clean install" again?
<tecoripa>awoods: yes12:52
<awoods>tecoripa: thanks
<pivotal-bot>Andrew Woods delivered "Build failing: fcrepo-jms-indexer-pluggable" https://www.pivotaltracker.com/story/show/61351914
<ajs6f>anyone know where the Solr schema in this indexer project came from? Is it the out-of-box example?12:54
<escowles>awoods: it was working for me on linux (using on UCSD VMs)
i'll try it a few times now and see if I see intermittent failures
<awoods>ajs6f: the schema looks out of the box... but only Ye knows for sure.12:55
<awoods>thanks, escowles.
<tecoripa>awoods: build failed, 1 test error
<awoods>tecoripa: thanks12:56
I guess ;)
<tecoripa>shall I gist it?
<awoods>gregjansen: the fcrepo-content-model-examples builds now... what was the change?12:57
before I continue: I have another instance of fedora4 running in the background, for the performance test, on port 8080... will that conflict with the jms test fedora?12:58
<awoods>tecoripa: yes, likely.
<tecoripa>awoods: ah, okay. let me shut that down, try again13:00
<awoods>tecoripa: I think the f4:8080 will not conflict (as the build-helper maven plugin is configured to get around that) but the jms port will conflict.
<tecoripa>awoods: shutting down the other one, running the jms test again...13:01
<awoods>tecoripa: thanks
I am interested to hear results from tecoripa and escowles.13:02
<tecoripa>awoods: BUILD SUCCESS :) now it's happy.
back to my performance test.
<barmintor>awoods, escowles: when I do the yourkit profiling, I just run benchtool against mvn jett:run
<awoods>tecoripa, that is great.
<barmintor>is that accurate?
<osmandin>afk 15
<barmintor>some of the CPU hotspots are in legacy JMS stuff, which may not be relevant?13:03
<escowles>barmintor: yes, that's what i've been doing (and against hydra-jetty for f3)
<awoods>escowles/barmintor: hopefully the other single node testers are doing the same.
<escowles>awoods: i just pulled fcrepo-jms-indexer-pluggable and now i'm getting build failures on both linux and macosx
<awoods>escowles: shucks13:04
escowles: me too, but it seems to be working for tecoripa... a shifty issue.
<pivotal-bot>Esme Cowles started "Build failing: fcrepo-jms-indexer-pluggable" https://www.pivotaltracker.com/story/show/61351914
<tecoripa>afk, bbl13:05
<awoods>All: There is an implicit expectation that all projects should be able to build even if F4 is running in the background. The build-helper-maven-plugin helps accomplish this.13:06
gregjansen: fcrepo-content-model-examples builds if F4 is not running. If F4 is running, we get a jms port conflict.13:08
<pivotal-bot>Andrew Woods added comment: "@gregoryjansen, This build succeeds if F4 is not running. If F4 is running, there is a jms port conflict." https://www.pivotaltracker.com/story/show/6122572213:09
<ajs6f>Now I'm sometimes seeing indexerGroupDeleteTest(org.fcrepo.indexer.IndexerGroupIT) Time elapsed: 15.184 sec <<< FAILURE!13:10
java.lang.AssertionError: There should be 1 file
<awoods>ajs6f: exactly
<ajs6f>I blame you, awoods. You jinxed us.
<awoods>ajs6f: I am skeptical of voodoo and magic, but maybe.13:11
<pivotal-bot>Benjamin Armintor added "Reduce calls to #getMixinNodeTypes in FedoraTypesUtils" https://www.pivotaltracker.com/story/show/6141139613:14
* travis-ci joins13:21
[travis-ci] futures/fcrepo-jms-indexer-pluggable#32 (master - b7660d4 : Andrew Woods): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo-jms-indexer-pluggable/compare/136b46bbd955...b7660d43b772
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo-jms-indexer-pluggable/builds/14503431
* travis-ci leaves
<pivotal-bot>A. "Outrageac" Soroka added comment: "Should we cache the types in FedoraResource (which doesn't outlive a request)?" https://www.pivotaltracker.com/story/show/6141139613:33
* github-ff joins13:34
[fcrepo-jms-indexer-pluggable] escowles created test-query-bugfix2 (+1 new commit): http://git.io/YlNgrA
fcrepo-jms-indexer-pluggable/test-query-bugfix2 478284b Esmé Cowles: Making tests that check for presence of triples less brittle
* github-ff leaves
<ajs6f>awoods; will you get that? (review and merge)
<awoods>you like this latest attempt, escowles?
ajs6f: yes
<escowles>awoods: yes, the new issue was whether there were 4 triples or 6 -- so i just changed the test to check for any triples at all -- should be more robust13:35
* github-ff joins13:36
[fcrepo-jms-indexer-pluggable] escowles opened pull request #14: Making tests that check for presence of triples less brittle (master...test-query-bugfix2) http://git.io/UEbKcA
* github-ff leaves
<ajs6f>is the indexer supposed to run via mvn jetty:run in the webapp module? (like fcrepo4 itself)13:37
<awoods>ajs6f: yes, it can13:38
<ajs6f>not for me. it hangs with
INFO 13:36:32.672 (DefaultLifecycleProcessor) Starting beans in phase 2147483647
<awoods>ajs6f: with a different -Djetty.port, of course.
<ajs6f>yeah, it's not port conflict.
<awoods>ajs6f: it waits for both F4 and a triplestore to be up.
<ajs6f>wait, what?!13:39
you can't run it without a triplestore? and what does it need a repo up for at all? It should start and listen...
if the repo goes done and up it should affect a message-driven component...
<awoods>ajs6f: see: https://wiki.duraspace.org/display/FF/Design+-+Indexing13:40
<ajs6f>This is no good. No one is going to install a vestigial triplestore in order to do full-text indexing.13:42
And it should certainly not need a Fedora API to be available to start up.
<awoods>ajs6f: It is in the configuration. You can configure the jms-pluggable to not try to connect to a triplestore.
<ajs6f>Yes, that ought to be the default.
<awoods>ajs6f: That is any easy config update.13:43
<escowles>we could create a couple sample configs (solr only, triplestore only, solr+triplestore, etc.)
<ajs6f>And it ought to start a forwarding broker internally and wait for messages. It has no reason to detect a Fedora API until it has already received a message.
<osmandin>+1 for sensible defaults13:44
<ajs6f>escowles: Yes, at leas tthat.
What we really want is runtime wiring, but Spring doesn't tend that way.
<awoods>ajs6f: good points. These are vary rational next steps following the initial work that was done.
<ajs6f>At least let's not require people to set up a triplestore in order to run a full-text index.13:45
(Out of the box)
<awoods>ajs6f: Out of the box it needs to do something...
<ajs6f>awoods: Out of the box it needs to start as easily and cleanly as possible.
<escowles>is there an out-of-the-box way to specify different spring configs?13:46
<awoods>ajs6f: we can choose if it is search or triplestore, or both.
<ajs6f>escowles: We could fake it with an import element in Spring,
and fill that with a system prop.
<escowles>the third indexer we have just writes files to disk -- so there's no dependency on anything else being setup
<ajs6f><import resource="${dgdfg}"/>
escowles: So it is or is not waiting on a repo?13:47
<escowles>ajs6f: it probably still waits on the repo
Actually, how?
Does it send test messages to the API?13:48
<awoods>ajs6f: What is your question?13:49
<escowles>i'm looking at that now -- i'm not seeing anything that connects to the repo
<ajs6f>escowles: Right. Neither do I.
<escowles>awoods: does the indexer connect to the repo at startup?
<ajs6f>And if so, how?
<ajs6f>That shows JMS config. That should not be connected to the repo at all.
That's just bad fabric.13:50
Which I am now going to fix.
<awoods>connects to a jms broker... which may or may not be waiting. ajs6f: does it block if F4 is not running?
<ajs6f>It certainly blocked for me. Whether because of that or not I don't know.
<awoods>escowles: still getting build failures in jms-indexer
<ajs6f>It should start its own broker, which should develop a network connection to the Fedora broker asynchronously.
<awoods>escowles: IndexerGroupIT.indexerGroupUpdateTestingFullPath:195 There should be 1 file13:51
ajs6f: make it so
<escowles>awoods: let me try with fcrepo4 running again
<pivotal-bot>A. "Outrageac" Soroka added "fcrepo-jms-indexer should maintain its own broker for robustness" https://www.pivotaltracker.com/story/show/61414712
A. "Outrageac" Soroka started "fcrepo-jms-indexer should maintain its own broker for robustness" https://www.pivotaltracker.com/story/show/6141471213:52
<awoods>ajs6f: you have been rdf-iterating... and look at all of the fun you could have been having.
<ajs6f>I don't see anyone else in here laughing.
<awoods>It is all virtual, ajs6f.13:53
<escowles>ajs6f: when you say it should have its own broker -- do you mean instead of the one fcrepo4 is running? can events be double-brokered?
<ajs6f>So is Fedora's indexing ability, right now.
escowles: Brokers are just containers through which messages pass. A message can come from repo core, go through the fcrepo-side broker, to the indexer-side broker, thence to the indexer client (or clients, when parallelizing that makes sense, which it will) and then to code that takes action.13:55
It's not going through multiple paths. There are configs that provide overloaded connectivity. We aren't that clever.
We just want to isolate the working code in the indexer
from the vagaries of whatever is happening repo-side.13:56
<awoods>escowles/ajs6f: currently, the indexer-events.xml defines a connectionFactory that defaults to the same connection-info provided by a local F4. One way or another, we need a broker running, no?
<ajs6f>And a plain client isn't the right tool.
awoods: Read what I just wrote. We have a broker running. We need another.
This is exactly the same road tecoripa and I went down with fcrepo3.
Many moons ago.
<awoods>sounds romantic13:57
<escowles>man, the repo and the indexer are such a bunch of prima donnas -- it's not enough to have one broker, they both need brokers to keep them both happy
<ajs6f>Brokers are cheap. First impressions are priceless.13:58
Brokers are just containers of messages. (With filtering and routing and so forth.) Spin 'em up and tear 'em down. They don't cost much.13:59
<awoods>gregjansen: Did you get an autoreponse on your email to the legacy fedora mailing-list?14:07
<gregjansen>awoods: no autoresponse received. I sent it again as gregory.jansen@gmail
<gregjansen>awoods: I sent it again for the first time, as gregory.jansen@gmial
<tecoripa>I'm back14:11
looks like you've all been having fun
<awoods>welcome back, tecoripa.
<tecoripa>shall I post the gist of my happy build?
<awoods>tecoripa, no need for a success-build gist.14:12
<tecoripa>ok -- is the problem that the jms plugin cannot have another Fedora running in the background? is that what causes your failures?
<escowles>tecoripa: i'm seeing intermittent failures on macosx with fcrepo4 running in the background14:14
<tecoripa>ajs6f++. Brokares ARE cheap, the more, the better.14:15
that's what messaging is all about.
* nbanks leaves14:23
* nbanks joins14:24
<gregjansen>am i getting the same JMS errors as everyone else? "class path resource [spring/jms.xml] cannot be opened because it does not exist"14:33
<escowles>gregjansen: i saw that this morning, but it went away after i pulled the changes in fcrepo414:34
escowles/gregjansen: To summarize these two threads: jms-pluggable and content-model-examples both are unable to properly build?
<gregjansen>content-model-examples -- I probably am not doing the JMS ports right yet. It runs by itself but not in tandem with other JMS servers14:35
<pivotal-bot>Frank Asseg added "Work out the cause of the large file ingest test failure with rhauch" https://www.pivotaltracker.com/story/show/61418398
<gregjansen>escowles: I just checked out futures/fcrepo4[master] and did a clean build..
<pivotal-bot>Frank Asseg started "Work out the cause of the large file ingest test failure with rhauch" https://www.pivotaltracker.com/story/show/61418398
<escowles>awoods: i think the new problem may be similar to the 4 v. 6 triples thing: multiple files making a test that looks for exactly 1 file to fail
gregjansen: hmm, probably not the same thing then14:36
<tecoripa>question about CNDs and nodetypes: CNDs are mix-ins, right? so someone could constrain the plain vanilla fcr:object or fcr:datastream nodetype by adding other node types onto the object, correct?
<pivotal-bot>Frank Asseg added comment: "This test fails for large files https://github.com/futures/large-files-test" https://www.pivotaltracker.com/story/show/61418398
<gregjansen>escowles: hang on, I am behind by a few commits still
<tecoripa>ajs6f: cool, thanks
<pivotal-bot>Frank Asseg added comment: "This issue has been reported to the modeshape folks: https://issues.jboss.org/browse/MODE-2103" https://www.pivotaltracker.com/story/show/61418398
Frank Asseg added comment: "Might be realted to usage of infinispan 5.3.0 instead of 5.2.7" https://www.pivotaltracker.com/story/show/6141839814:37
<ajs6f>Trying to do https://www.pivotaltracker.com/story/show/61414712 , having the same build probs as everyone else. awoods: did you merge escowles fix?
<pivotal-bot>feature: fcrepo-jms-indexer should maintain its own broker for robustness (started) / owner: A. "Outrageac" Soroka
Osman Din started "Change camel-case of fcr:accessRoles endpoint" https://www.pivotaltracker.com/story/show/61179680
Osman Din added comment: "Changed: ""14:38
https://github.com/osmandin/fcrepo-authz/compare/61179680" https://www.pivotaltracker.com/story/show/61179680
Osman Din finished "Change camel-case of fcr:accessRoles endpoint" https://www.pivotaltracker.com/story/show/61179680
<escowles>ajs6f: there's a second PR that's not merged yet: https://github.com/futures/fcrepo-jms-indexer-pluggable/pull/1414:39
* nbanks leaves
<awoods>ajs6f: I merged the first fix (pr-13), and was going to merge the second fix (pr-14), but it does not work for me locally... and it sounded like escowles was working on an update... so it has not been merged.
<ajs6f>k, got it.
<gregjansen>ajs6f, escowles: my jetty startup is fixed after pulling latest
<ajs6f>latest fcrepo4
<ajs6f>escowles: DId you look at barmintor's itest for the main JMS module for the repo? It shows a good technique for this, although it, too, is sensitive to counting problems.
<gregjansen>somehow I had failed to fast-forward before, probably by ignoring prompts
* nbanks joins14:41
<tecoripa>awoods: is fcrepo-auth-common part of the fedora 4 core distribution now? That is, would it be okay to create a sample repository.json sample file in the fcrepo-http-commons project that references a class in fcrepo-auth-common?14:46
<awoods>yes, and probably no
does fcrepo-http-common depend on auth-common?14:47
<tecoripa>no, I think not
<gregjansen>escowles: better late than never: https://gist.github.com/gregjan/7647595
<awoods>tecoripa: we probably do not want to introduce that dependency.14:48
<escowles>awoods: i updated the file tests in a similar manner to the triple-counting tests (check for any files, not limiting to exactly 1) and now it works for me consistently with fcrepo4 running -- i think it wasn't a port conflict, but the test being slower so the second file was getting created before the test executed
<tecoripa>awoods: I'd like to provide a sample config, like minimal, single, single-basic, etc., that sets up the wiring for basic access roles
awoods: but I can keep that sample config in fcrepo-auth-common, then.
<gregjansen>tecoripa: folks should be able to reference them via classpath: if they are using auth-common14:49
<awoods>gregjansen: right
tecoripa: I think barmintor was moving towards keeping the configs close to their dependencies. So having a config in auth-common seems like a good approach.14:50
<tecoripa>awoods: sounds good. I'll do that, then.14:51
<escowles>gregjansen: i have seen a problem using the minimal repository.json that if i restart fcrepo4, the objects aren't in the repo (but are still on disk) -- have you seen anything like that?
<awoods>escowles: jetty:run?
<gregjansen>escowles: Nope, but I purge the stores regularly due to disk constraints and have only been doing ingest tests so far14:52
* github-ff joins
[fcrepo-jms-indexer-pluggable] escowles pushed 1 new commit to test-query-bugfix2: http://git.io/Byn4WA
fcrepo-jms-indexer-pluggable/test-query-bugfix2 8355d05 Esmé Cowles: Making file tests less brittle (checking for 1 or more files, not exactly 1 file)
* github-ff leaves
<escowles>awoods: yes, using mvn jetty:run
<gregjansen>escowles: perhaps using explicit store locations would help, as per gist?
<escowles>awoods: i've been using mvn jetty:run -Dfcrepo.modeshape.configuration=classpath:/config/minimal/repository.json
* nbanks leaves
<escowles>but i'll retest with gregjansen's store locations14:53
<gregjansen>escowles: how to I specify the XOR random function.. (movin' slow)
<awoods>escowles: yes, if you do not specify all of the SystemProperty details, then files get written to tmp.
<escowles>gregjansen: XOR is the default, you can specify -Drandom.impl=java.util.Random to use that instead
<awoods>escowles: something like /tmp/fcrepo4-data
<escowles>awoods: i'm seeing files in fcrepo-webapp/target/binaries (using the minimal config)14:54
<gregjansen>escowles: I will see if that helps. My random data is taking forever to generate while f4 is running, but went faster (much) when f3 was running.
escowles: I don't think that those are all of the fiels
<awoods>escowles: there is a ticket to change tmp to cwd: https://www.pivotaltracker.com/story/show/6100433614:55
<pivotal-bot>feature: Default Storage Parameter Directory: tmp to cwd (unstarted) / owner:
<gregjansen>escowles: well, well, well. java.util.Random is now the faster one here too.14:57
<escowles>gregjansen: very strange
java.util.Random was about 35x faster than XORShiftRNG
i wonder if java.util.Random interacts with hardware RNG differently on different machines?14:58
<ajs6f>afk bbs15:05
* github-ff joins15:08
[fcrepo4] mikedurbin opened pull request #167: Updated fcr:versions to return more useful triples, fcr:content to work ... (master...versioning-2) http://git.io/BaO8Mg
* github-ff leaves
<pivotal-bot>Mike Durbin added comment: "https://github.com/futures/fcrepo4/pull/167" https://www.pivotaltracker.com/story/show/60769514
Mike Durbin finished "Review and possibly revise or extend the fcrepo-http-api endpoints relating to versions to ensure they are useful ..." https://www.pivotaltracker.com/story/show/60769514
* ajs6f leaves15:09
* ajs6f joins15:10
<pivotal-bot>Mike Durbin added "Add alternate versioning policy." https://www.pivotaltracker.com/story/show/6142169015:15
Mike Durbin edited "Add alternate versioning policy." https://www.pivotaltracker.com/story/show/61421690
Mike Durbin edited "Better expose nodes of type nt:frozenNode." https://www.pivotaltracker.com/story/show/6113303015:17
Mike Durbin started "Better expose nodes of type nt:frozenNode." https://www.pivotaltracker.com/story/show/61133030
Osman Din started "Fix Checksum Mismatch Message" https://www.pivotaltracker.com/story/show/6100798815:28
<gregjansen>ingest exceptions during 5 thread single node test: https://gist.github.com/gregjan/764831515:30
<cbeer>ok. i think i just wasn't running with a large enough heap size in my tests last week. boosted it to 2g and things seem better15:32
<pivotal-bot>A. "Killmation" Soroka added comment: "Perhaps it would be appropriate for a new package: ""15:45
to include these two clas..." https://www.pivotaltracker.com/story/show/61391216
<fasseg>awoods: I seem to be unable to add single users as collaborators to https://github.com/futures/large-files-test/settings/collaboration Can you add rhauch as a collaborator for that repo?
<awoods>fasseg: on a call
* github-ff joins15:46
[fcrepo-jms-indexer-pluggable] ajs6f created BetterBrokerConfig (+1 new commit): http://git.io/wdeYag
fcrepo-jms-indexer-pluggable/BetterBrokerConfig 03c29ec ajs6f: Added config to use an internal broker for isolation from repo broker
* github-ff leaves
<ajs6f>awoods: if you are on a call, do you want me to review/merge escowles latest fix?15:48
<awoods>ajs6f: sure
<ajs6f>escowles: did you make a pr for that yet?15:49
<escowles>i added it to: https://github.com/futures/fcrepo-jms-indexer-pluggable/pull/14
<ajs6f>k thnx
escowles: getting:15:54
java.lang.AssertionError: expected:<2> but was:<3>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at org.fcrepo.indexer.IndexerGroupIT.indexerGroupDeleteTest(IndexerGroupIT.java:161)
Another counting prob?
<escowles>ajs6f: yes, looks like another place where it's looking for a specific number of files...15:56
<ajs6f>escowles: well, at least that is familiar...
escowles: Can we develop a better pattern than counting? Like looking for specific markers instead?
(Or ensuring that we don't find specific markers, or whatever is appropriate to a given test.)15:57
Especially if we go to multithreaded tests, counting is going to get very brittle.
<escowles>ajs6f: i agree about counting being brittle -- i'm mostly updating it to handle "x or more" instead of "exactly x" to make it less brittle15:58
This is all about tests of the "framework", right, not tests of the individual indexers?
<escowles>yes, it's the IT that's failing
<ajs6f>What if we had a "testing indexer" that just persisted and deleted the stuff as it comes in?15:59
Just an in-memory list of messages.
<escowles>so you could just ask it whether it got a delete message or not...?
<ajs6f>Then we could just look in that list to see if IndexerGroup properly distributed them.
escowles: Right. No counting at all.
And we're checking the behavior of the one component about which we actually care (in this test): IndexerGroup.16:00
<escowles>yes, that's probably a better way to test it
<ajs6f>Okay, so a "TestIndexer" that just saves the messages in a list (or other searchable structure).
And can tell you what it has and hasn't received.
It'd be faster, too. No touching the disk.
No need to spin up any backend for that test.16:01
* fasseg leaves
<escowles>so it would have the normal update() and remove() methods, plus receivedUpdate() and receivedRemove() calls to test whether they had been invoked or not
<ajs6f>yeah, that sort of thing. Maybe even receivedUpdateForPid(String pid) or the like.16:02
To check on a given message.
<escowles>right, so it would just store the pids it had received messages for in a set or something
<ajs6f>It wouldn't even have to save the messages, just a notation that one of a certain kind for a certain pid had been recevied.
yeah, something like that. Maybe a MultiMap<String,Messages> for all the messages on a given pid. Or maybe even that is too much.16:03
<escowles>that might lead to counting again
<ajs6f>Not if we a) use an off-the-shelf multimap, and b) use messages with unique IDs.16:04
But i don't care: whatever is simplest.
Maybe just List<String> with the pids of received massges.
Whatever is enough to verify IndexerGroup.16:05
afk bbs16:06
* fasseg joins
* ajs6f leaves
<pivotal-bot>Osman Din added comment: "> Datastream.java:173 should respond with an error message that includes the ContentDigest.asURI("SHA-1", dsCh..." https://www.pivotaltracker.com/story/show/6100798816:09
Osman Din finished "Fix Checksum Mismatch Message" https://www.pivotaltracker.com/story/show/61007988
Osman Din started "Parameterize JMS port in fcrepo-authz ITs" https://www.pivotaltracker.com/story/show/61155312
* ajs6f joins
* github-ff joins16:10
[fcrepo4] sprater opened pull request #168: Sample repository.json for secured repository (master...fcrepo-61409536) http://git.io/TH-fbQ
* github-ff leaves
<pivotal-bot>Scott Prater added comment: "PR: https://github.com/futures/fcrepo4/pull/168" https://www.pivotaltracker.com/story/show/6140953616:11
<awoods>fasseg: rhauch has been added16:12
* ermadmix joins16:14
<tecoripa>all: is there a way to specify the path to the repo.xml file as flag on the java command line? -Dfcrepo.repo.configuration, or something like that?
<awoods>tecoripa: That is currently specified in the web.xml as:16:17
tecoripa: so it could be tricky changing that out.16:18
afk -- lunch16:19
<tecoripa>awoods: okay. I'll update my test documentation to reflect that, then.
<ajs6f>tecoripa: What about repo.xml are you trying to alter?16:22
<tecoripa>I'm trying to specify a repo.xml that contains the activation for Basic Role Based Access Control
<osmandin>tecorpia: were you able to test basic auth?16:26
<tecoripa>osmandin: working on it. still getting my test environment properly configured, creating correct repo profiles.16:27
<ajs6f>tecoripa: For our own convenience, would we want to breakout a sys prop for that, or install some other config mechanism in repo.xml?16:28
<osmandin>tecorpia: i have a branch that might be helpful in set up: https://github.com/osmandin/fcrepo4/tree/auth-exp
<tecoripa>ajs6f: since repo.xml seems to be a top-level configuration file, I'd be inclined to break out a sys prop for that. The changes I need to make are to include some authz pep beans in the file itself.16:29
osmandin: I'll check that out, thanks.16:30
osmandin: your branch has the files in the usual places, just changed to use the basic access roles module, correct?
<osmandin>tecorpia: yes, but it's based on an older master.. so it might complain about some config files that were moved around in the past few days16:31
<tecoripa>heres' waht I'm working on: a test repository profile others can use to do performance tests with authz enabled.16:32
<osmandin>tecorpia: but otherwise it should work with the basic authz
<tecoripa>osmandin: thanks. I'm thinking of something a little more generic -- more like instructions for others to set up their environment.16:33
here's what I have now:
<gregjansen>tecoripa: the only link between the repo config and the authz provider is that the authz provider must be configured first (hence the dependency link between repo and authzprovider in the repo.xml file). As long as we can ensure a start order between bean files, we can break it out into a separate config file..16:36
tecoripa: by repo config, I mean repo.xml
<tecoripa>gregjansen: ah, of course. yes -- are spring bean files processed in standard unix lexical order?16:37
<ajs6f>gregjansen/tecoripa: There's a dependency? Doesn't that represent a weakness in the repo construction? If a configured authZ component isn't live when it's wanted, shouldn't the repo just wait until it is, instead of falling over?16:38
(As a voice on the wind mutters… "OSGi dynamic services…")
<tecoripa>tecoripa covers head, cowers under desk16:39
<ajs6f>tecoripa: It is a really bad idea to rely on beans coming up in any particular order. That is guaranteed to be a problem for somebody, somewhere, sometime. Probably one of us.
<tecoripa>ajs6f: I'm not sure what happens now, if the authenticationProvider can't be found.16:40
<ajs6f>You can force an order with imports… but that's weak cheese, for my money.
<gregjansen>ajs6f: it certain does represent a weakness. the authz provider depends on the PEP, which is fair enough, but the repo factory depends on a singleton pattern (!!) for getting the configured provider.
<ajs6f>tecoripa: Well, I know what I would _like_ to happen. The repo just waits. (having logged an appropriate message)
<tecoripa>ajs6f: so for the short term, is it preferable then to hardcode the dependency in repo.xml, where the fact that you explicitly stated it there implies that it will be present, and it's okay to fall over if it isn't?16:41
<ajs6f>gregjansen: Urg. We do that a good bit. I know I've done some of that, and I'm not proud.
<gregjansen>the repofactory will try to configure the repo with a authz provider, using the classname in json.. this results in reflection, calling getInstance(), and failing that a default constructor
<ajs6f>tecoripa: I guess that's the best road forward. But maybe file a ticket to remind us to flexible-ize the repo?16:42
<gregjansen>this is mode config stuff.
<ajs6f>gregjansen: Oh, crap. This is _MODE_ stuff?
<gregjansen>_OH YEAH_
trust me, I would not invent this (this decade)
<ajs6f>See here:16:43
<gregjansen>I guess we could create another layer, urgh
<ajs6f>We're not getting anything better anyime soon.
rhauch and I have been around this block.
"All problems in computer science can be solved by another level of indirection."
<gregjansen>escowles: I got less than credible read performance on F3, 1005 mb/s16:45
<ajs6f>a gig per second?16:46
or that bits?
<awoods>ajs6f: I thought you were going to review/push escowles pr-14 in jms-pluggable?
<ajs6f>or millibits.
<gregjansen>ajs6f: bytes, but those are just pointers to bytes
<ajs6f>awoods: See above. It doesn't fix the problem quite, and we now have a better approach planned. You were on a call.
escowles: Unless you want that PR in anyway?16:47
I didn't see anything else in it, but maybe I missed something?
<awoods>escowles: you are working the new approach, I take it?
<escowles>awoods: yes, a test indexer that just keeps track of what messages it has received so we can just ask it instead of checking for files, etc.16:48
<ajs6f>awoods: We should probaby do something like that for fcrepo-jms.
<gregjansen>ah, er, um, where did the configs go again?
<awoods>ajs6f: fcrepo-jms? you mean the module within fcrepo4?16:49
<ajs6f>Although it was good that osmindin got down and dirty with the Atom, since we were making bad stuff.
awoods: Yeah. There's an itest there that does counting, too. It's less brittle, but it makes assumptions about how many messages are emitted for a given action, which is wrong.
That's controlled in kernel, and fcrepo-jms shouldn't be testing that.
<awoods>ajs6f: got it. Yes, a cleaner, more reliable approach would be nice for the ATOM tests.16:50
<tecoripa>gregjansen: you mean this? https://wiki.duraspace.org/display/FF/Configuration+Options+Inventory
<ajs6f>awoods: Why don't you file a forty point ticket for "Make everything cleaner and more reliable."
awoods: Just wrap the whole thing up.16:51
Maybe just a thousand point ticket for "Make all the stakeholders happy."
<gregjansen>tecoripa: perfect
<tecoripa>gregjansen: yeah, that page is a lifesaver. I've been reading it all day long.
<gregjansen>tecoripa++ indeed
<tecoripa>okay, NOW I think I'm ready to start running the uathz performance tests.16:54
it took a while...
<ajs6f>Out for the day. See y'all tomorrow. http://www.youtube.com/watch?v=Jmg86CRBBtw16:56
* ajs6f leaves
<barmintor>As far as I can tell, ~80% of the CPU used on ingest is actually used in looking the changed node up again in the legacy event processing16:59
IOW: Going from a string path to a node, and filtering the mixins, is relatively expensive17:00
<awoods>barmintor: and the legacy-event-processing is our JMS messaging?17:01
<barmintor>awoods: yes
<awoods>barmintor: So that is good and bad. Since it is our code, we can optimize it... but we certainly need the JMS capability.17:02
<barmintor>awoods: Soooort of- we are limited in our ability to optimize it, outside of minimizing how often we do it
<awoods>barmintor: We have circled on how to most effectively/efficiently filter JMS messages... but apparently have not hit the mark yet.17:03
barmintor: There was talk of JCR-2.1 helping
<cbeer>barmintor: but isn't the event stuff going to be "solved" with the JCR 2.1 changes? (there's a ticket about that somewhere...)
<barmintor>cbeer/awoods: I have no idea
I'm just looking at reports
Memory wise, the version checkpoints are apparently the big hotspots17:05
<awoods>barmintor: as a upper-limit on performance, does it make sense to disable JMS and Versioning and run tests again?17:07
<barmintor>awoods: that appears to be the lion's share of both CPU and memory, yes
<awoods>barmintor/cbeer: https://www.pivotaltracker.com/story/show/5453060617:08
<pivotal-bot>feature: Leverage Event.getMixinNodeTypes() in DefaultFilter (unscheduled) / owner:
<barmintor>it would also help if we could pass a caching node ref off to the listeners directly, and have them ask the node for its path (instead of going from path->node)
but there would be some moderate gains in caching DS nodes in the Type util functions, since (b/c of the conditional structure) they get loaded twice17:09
<awoods>barmintor: Does such an object currently exist: caching-node-ref?17:10
* barmintor starts looking
<awoods>side question: how do you achieve the "meta" IRC comment?
<barmintor> "/me metacomment"17:11
I think /em works too
* awoods wondering if it works
<barmintor>awoods achievement unlocked: IRC Spam
<awoods>barmintor: How do you want to move forward with you JMS and Versioning discoveries?17:12
<barmintor>awoods: I'm going to start looking at slimming down the FedoraTypesUtils uses
and re-run the Yourkit profiles, which were not too divergent between 1k and 2m payloads
* tecoripa looks on /barmintor as a role model. with his own CND, no less
* fasseg leaves17:16
* fasseg joins
<barmintor>awoods: Just more reading here: the peak memory in create (FedoraContent.create) is nearly all in MODE. FCRepo doesn't even register 1% objects created or allocation size17:30
<pivotal-bot>Andrew Woods added comment: "Thanks, @osmandin. At first glance, this looks good. In order to provide comments on your proposed updates ..." https://www.pivotaltracker.com/story/show/6115284217:31
Andrew Woods rejected "Fix integration test AtomJMSIT exception related to truncated messages" https://www.pivotaltracker.com/story/show/61152842
<awoods>barmintor: good to know17:32
* fasseg leaves17:42
<pivotal-bot>Andrew Woods delivered "Create a Testcase for large file handling in Infinispan and submit a bug report if applicable" https://www.pivotaltracker.com/story/show/6129828417:44
* fasseg joins17:46
<barmintor>I know this is an aesthetic issue, but: All of these anonymous classes in FedoraTypesUtils don't make it *easier* to decipher the profiling results.18:03
FedoraTypesUtils$3.apply is opaque
<awoods>barmintor: personally, I find the code hard to read (and it appears to be hard to decipher profiling results). It seems, however, to be a required concession to keep ajs6f's interest.18:06
barmintor: I am disappointed to hear that it transcends aesthetics into practical analysis.18:07
<barmintor>awoods: it's just hard to read18:08
s/read/parse which function it is/
* ermadmix leaves18:11
<pivotal-bot>Andrew Woods added comment: "One very minor comment, then ready to squash in ship.18:19
https://github.com/futures/fcrepo4/pull/166/files#r79..." https://www.pivotaltracker.com/story/show/61166142
Andrew Woods rejected "Alter FedoraRepositoryNodeTypes, FedoraRepositoryWorkspaces, FedoraIdentifiers and FedoraRepositoryNamespaces to ..." https://www.pivotaltracker.com/story/show/6116614218:20
<tecoripa>awoods: do you know where Fedora writes its data when using the infinispan basic config?18:22
 "storage" : {        "cacheName" : "FedoraRepository",        "cacheConfiguration" : "${fcrepo.infinispan.cache_configuration:config/infinispan/basic/infinispan.xml}",        "binaryStorage" : {            "type" : "cache",            "dataCacheName" : "FedoraRepositoryBinaryData",            "metadataCacheName" : "FedoraRepositoryMetaData"        }
I'm looking all over for it, mostly in tmp, but I can't find it
and my root filesystem is almost full :(
* awoods looking18:23
tecoripa: Have you specified the -Dfcrepo.ispn.repo.CacheDirPath option?18:24
<tecoripa>nope, but I will now :)
<awoods>tecoripa: otherwise, there are defaults in the file I linked above ^^
tecoripa: namely, "target/..."
<tecoripa>hmm, but I wasn't seeing anything in target/FedoraRepository/storage
<awoods>tecoripa: I assume you will looking in the cwd of the process from which the repository was started?18:26
<tecoripa>I looked at the inifinspan.xml file, too.
I clearly have something whacked with my environment. I'll keep looking.18:27
I was hoping that it *would* write below the cwd, or in the target/ subdirectory. I've got all the Fedora stuff running on a huges local filesystem.18:28
but when I launch the webapp, then run benchtool, my root partition starts filling.
and nothing gets written to the target/ directory.
<awoods>tecoripa: have you looked in "/tmp"?18:29
tecoripa: Are you running as jetty:run or in tomcat?
<tecoripa>yes, and I did find some stuff there, from earlier, but nothing current.
<awoods> tecoripa: Are you running as jetty:run or in tomcat?18:31
I see a bunch of directories in /tmp, with what looks like timestamps names18:32
<awoods>tecoripa: then it should definitely be in "target"
<tecoripa>ls -l 1385410835452-0
total 68
drwxrwxr-x. 7 sprater sprater 4096 Nov 25 14:20 binaries
drwxrwxr-x. 2 sprater sprater 4096 Nov 25 14:20 changes
-rw-rw-r--. 1 sprater sprater 60473 Nov 25 14:20 documents_000001.bin.gz
that doesn't look like an infinspan storage artifact, does it?
<awoods>tecoripa: those are likely from the backup/restore IT
<tecoripa>yeah, that's what I thought.18:33
ok, I'm going to launch again, and see if I can determine what is getting written where, using lsof or some such utility.
* github-ff joins18:37
[fcrepo-authz] awoods pushed 1 new commit to master: http://git.io/KhVGrA
fcrepo-authz/master dc224da osmandin: Converted fcr:accessRoles endpoint to lowercase
* github-ff leaves
<pivotal-bot>Andrew Woods added comment: "Resolved with: https://github.com/futures/fcrepo-authz/commit/dc224da564e58157d28ff5ec1c674056addaa365" https://www.pivotaltracker.com/story/show/6117968018:38
Andrew Woods delivered "Change camel-case of fcr:accessRoles endpoint" https://www.pivotaltracker.com/story/show/61179680
* github-ff joins18:41
[fcrepo-authz] awoods pushed 1 new commit to master: http://git.io/PO_9Qg
fcrepo-authz/master fd50d1c Andrew Woods: Add .travis.yml
* github-ff leaves
<bljenkins>Project fcrepo-authz build #2: SUCCESS in 4 min 3 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-authz/2/
<tecoripa>okay, I tracked down the culprit, thanks to lsof:18:44
npow why it is putting it there is odd.18:45
<bljenkins>Project fcrepo-authz build #3: SUCCESS in 3 min 45 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-authz/3/
<tecoripa>it's almost as if it's interpreting fcrepo.ispn.repo.CacheDirPath as string literal.
<awoods>tecoripa: yes, https://github.com/futures/fcrepo4/blob/master/fcrepo-kernel/src/main/java/org/fcrepo/kernel/spring/ModeShapeRepositoryFactoryBean.java#L7218:46
tecoripa: it uses the literal to make it clear which SystemProperty can be set to change a give directory.
<tecoripa>right, that makes sense18:47
<awoods>tecoripa: apparently not
<tecoripa>but it shouldn't use the literal to actaully name the directory.
I'm going to look at the syntax of my repository.json file18:48
<pivotal-bot>Andrew Woods added comment: "Pending comment: ""18:49
https://github.com/osmandin/fcrepo4/commit/a971fed3a681bcc644f3e5666679be43631d761d#commit..." https://www.pivotaltracker.com/story/show/61007988
Andrew Woods rejected "Fix Checksum Mismatch Message" https://www.pivotaltracker.com/story/show/61007988
<tecoripa>so here's what I have in repository.json:18:50
"storage" : {
"cacheName" : "FedoraRepository",
"cacheConfiguration" : "${fcrepo.infinispan.cache_configuration:config/infinispan/basic/infinispan.xml}",
"binaryStorage" : {
"type" : "cache",
"dataCacheName" : "FedoraRepositoryBinaryData",
"metadataCacheName" : "FedoraRepositoryMetaData"
* fasseg leaves
* fasseg joins
<tecoripa>which all looks okay, and does not match fcrepo.ispn.CacheDirPath or fcrepo.ispn.repo.CacheDirPath, which are system properties
<awoods>is that a question?18:51
<tecoripa>in fact, I have several directories which look like system property names:
drwxrwxr-x. 5 sprater sprater 4096 Nov 25 16:08 fcrepo.ispn.CacheDirPath
drwxrwxr-x. 5 sprater sprater 4096 Nov 25 16:08 fcrepo.ispn.repo.CacheDirPath
drwxrwxr-x. 7 sprater sprater 4096 Oct 25 15:12 .
drwxrwxr-x. 3 sprater sprater 4096 Oct 25 15:12 com.arjuna.ats.arjuna.common.ObjectStoreEnvironmentBean.default.objectStoreDir
drwxrwxr-x. 3 sprater sprater 4096 Oct 25 15:12 com.arjuna.ats.arjuna.objectstore.objectStoreDir
drwxrwxr-x. 3 sprater sprater 4096 Oct 25 15:12 fcrepo.ispn.binary.CacheDirPath
no, not a question...
<awoods>tecoripa: did you trace the logic I sent a moment ago?
<tecoripa>but wondering if there's a bug? or if I'm simply misconfigured somewhere18:52
<tecoripa>although I'm going with the plain vanilla configurations, available out of the box
<awoods>tecoripa: did you trace the logic I sent a moment ago?
<tecoripa>I see that it loads the system properties...18:53
I'll trace the logic of that
* nbanks joins18:55
<cbeer>barmintor: ping?18:56
<barmintor>cbeer: yo
<cbeer>profiling question... i'm trying to track down my exploding heap problem
after ~100k object
<cbeer>so i'm running yourkit against a http-api IT that just creates a bunch of objects
<cbeer>and took a snapshot after ingesting a bunch of objects.. i'm poking around the memory panel trying to make sense of what i'm looking at18:58
i've found a big nasty ConcurrentHashMap and want to know who created it18:59
is that a GC Root?
<barmintor>I would look at the backtrace under the "Allocations" tab to see if I can find it there
<tecoripa>awoods: got it. I'll set the system properties, then.19:02
<awoods>tecoripa: That is the surest way to know where things are going.19:03
<cbeer>barmintor: ok, thanks. i'll poke around some more and maybe bug you tomorrow.
* tecoripa leaves
<fasseg>set theme=pyhy19:04
* fasseg leaves19:05
* fasseg joins
im off for today see you tomorrow...19:09
* fasseg leaves
* nbanks leaves19:15
* travis-ci joins19:19
[travis-ci] futures/fcrepo-authz#2 (master - fd50d1c : Andrew Woods): The build was fixed.
[travis-ci] Change view : https://github.com/futures/fcrepo-authz/compare/dc224da564e5...fd50d1ce6271
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo-authz/builds/14519471
* travis-ci leaves
<barmintor>cbeer: I think i get your question. On the prev version of YK, I could get the backtraces for classes. I think this is a metric to configure somehwhere.19:22
Blog ?s for tomorrow: Where is that big ConcurrentHashMap? Do SimpleObserver and the Event queue impls all need to look the node up from the path?19:23
<cbeer>ok. after turning off indexing and the simple observer session, it looks like i'm not going to kill my heap19:31
so.. there's something funky in that long-lived observer session
and a RAMDisk-backed lucene index is a bad idea.19:32
(although our production query configuration is configured not use a ramdisk, i guess)19:33
so maybe it's just that observer that's killing me19:34
* github-ff joins
[fcrepo4] awoods pushed 2 new commits to master: http://git.io/K65WjA
fcrepo4/master b65d0e0 Michael Durbin: Updated fcr:versions to return more useful triples, fcr:content to work on old versions.
fcrepo4/master 49613d5 Andrew Woods: Merge pull request #167 from mikedurbin/versioning-2...
* github-ff leaves
<pivotal-bot>Andrew Woods delivered "Review and possibly revise or extend the fcrepo-http-api endpoints relating to versions to ensure they are usefu..." https://www.pivotaltracker.com/story/show/6076951419:35
Andrew Woods accepted "Review and possibly revise or extend the fcrepo-http-api endpoints relating to versions to ensure they are useful..." https://www.pivotaltracker.com/story/show/6076951419:37
Andrew Woods accepted "Create a Testcase for large file handling in Infinispan and submit a bug report if applicable" https://www.pivotaltracker.com/story/show/61298284
Andrew Woods accepted "Change camel-case of fcr:accessRoles endpoint" https://www.pivotaltracker.com/story/show/61179680
* nbanks joins19:43
* travis-ci joins19:48
[travis-ci] futures/fcrepo4#1269 (master - 49613d5 : Andrew Woods): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/ee3c9ff6eaa7...49613d5a18d4
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14521372
* travis-ci leaves
<bljenkins>Project fcrepo-fixity-corrupter build #490: SUCCESS in 1 min 14 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/490/19:51
Project fcrepo-kitchen-sink build #671: STILL UNSTABLE in 2 min 22 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/671/19:53
Yippie, build fixed!19:54
Project fcrepo-jms-indexer-pluggable build #278: FIXED in 4 min 43 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-jms-indexer-pluggable/278/
* nbanks leaves20:26
<awoods>ping: cbeer
* osmandin leaves20:28
<cbeer>awoods: hi20:31
cbeer: I am testing the fcr:sparql endpoint
I have created a property on a node
I can use fcr:search to find "Bob"
and the property shows up in the UI.
however, searching via fcr:sparql is not working for me20:34
<cbeer>what query are you using?
<awoods>curl -XPOST --upload-file query-0.txt http://localhost:8888/rest/fcr:sparq
<cbeer>sure. what's query-0.txt?
<awoods>PREFIX dc: <http://purl.org/dc/terms/>
SELECT ?subject WHERE {
?subject dc:creator "Bob"
<cbeer>is dc: really /terms/? i thought we made it /elements/1.1/?
<awoods>yes, it is elements/1.1
let me try...
PREFIX dc: <http://purl.org/dc/elements/1.1/>20:37
SELECT ?subject WHERE {
?subject dc:creator "Bob"
<cbeer>good. we used to have dc: as /terms/, but that was bad of us.20:38
and maybe this is an argument for trying to implement variable predicates.. although i hate to think about what we'd have to do to get there20:39
<awoods>I used /elements/1.1 when creating the property...
but /terms when querying: https://wiki.duraspace.org/pages/diffpages.action?pageId=39027499&originalId=4007551820:40
thanks for the quick help.
cbeer: fyi, it appears that the properties do not need to be specified in the CND to be indexed.20:43
I created some properties with: PREFIX aw: <http://awoods.com/>
and those are returned by queries to both fcr:search and fcr:sparql20:44
(note: the aw prefix was not defined in the CND)
<cbeer>huh. that's not the behavior i was seeing. can you write some ITs to confirm that?20:46
<pivotal-bot>Andrew Woods accepted "Create design document for simple admin search" https://www.pivotaltracker.com/story/show/6071383820:47
<awoods>cbeer: sure
* awoods leaves21:44
* awoods joins
<pivotal-bot>Andrew Woods added "Custom Property IT" https://www.pivotaltracker.com/story/show/6144662822:22
Andrew Woods started "Custom Property IT" https://www.pivotaltracker.com/story/show/61446628
Andrew Woods added comment: "You do not feel that these new classes would be kernel-ish?" https://www.pivotaltracker.com/story/show/6139121622:34
* github-ff joins22:36
[fcrepo4] awoods created custom-properties (+1 new commit): http://git.io/DXIvUg
fcrepo4/custom-properties 3b30e4c Andrew Woods: Add integration test for SPARQL query over previously undefined properties...
* github-ff leaves
* github-ff joins22:37
[fcrepo4] awoods opened pull request #169: Add integration test for SPARQL query over previously undefined properti... (master...custom-properties) http://git.io/o53onQ
* github-ff leaves
<pivotal-bot>Andrew Woods added comment: "Resolved with: https://github.com/futures/fcrepo4/pull/169" https://www.pivotaltracker.com/story/show/61446628
Andrew Woods finished "Custom Property IT" https://www.pivotaltracker.com/story/show/61446628
* travis-ci joins22:48
[travis-ci] futures/fcrepo4#1270 (custom-properties - 3b30e4c : Andrew Woods): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/commit/3b30e4c299b1
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14526269
* travis-ci leaves
<bljenkins>Project fcrepo-fixity-corrupter build #491: SUCCESS in 3 min 50 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/491/23:45
Project fcrepo-kitchen-sink build #672: STILL UNSTABLE in 8 min 40 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/672/23:54

Generated by Sualtam