Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* ksclarke leaves02:25
* kaarefc joins03:31
* kaarefc leaves03:36
* kaarefc joins04:03
* kaarefc leaves04:15
* kaarefc joins04:24
* kaarefc leaves04:31
* kaarefc joins04:40
* kaarefc leaves07:07
* kaarefc joins08:18
* gregjansen joins08:55
* gregjansen leaves
* gregjansen joins
<mikeAtUVa>For the performance testing done so far, when creating an object with several datastreams do you wrap all the updates for a node in a transaction or perform them separately? I suspect performance would be better with the transaction since we'd only create one version and only have on session that gets committed. This would also likely be the preferred way of doing such operations anyway.08:57
* gregjansen leaves09:00
<escowles>mikeAtUVa: i think all of my tests so far have involved only a single datastream per object -- but i agree that we should be testing multiple datastreams, and wrapping the whole object in a transaction would be the way most people would want to do that09:06
<mikeAtUVa>escowles: if the overhead is in session.save or versionManager.checkin then even wrapping those two calls in a transaction could conceivably improve performance (unless the finalization of the transaction takes just as long).09:08
escowles: in any event, I think we'll have options to discuss at today's meeting.
* ajs6f joins09:14
* kaarefc leaves09:22
<pivotal-bot>Andrew Woods accepted "Allow versioning to be turned off." https://www.pivotaltracker.com/story/show/6115175009:45
* ksclarke joins09:47
* tecoripa joins09:55
* ajs6f1 joins09:57
* ajs6f leaves09:58
* ermadmix joins09:59
<pivotal-bot>Gregory Jansen added "Enhance AuthZ documentation" https://www.pivotaltracker.com/story/show/6121262010:00
* tecoripa leaves10:02
<pivotal-bot>A. "Mutchkin" Soroka added comment: "This may or may not be worth doing pre-modularization: much of the work to establish clean module b..." https://www.pivotaltracker.com/story/show/61084786
A. "Mutchkin" Soroka added comment: "See: ""10:03
https://community.jboss.org/message/846393
for a warning from @rhauch of MODE about the diff..." https://www.pivotaltracker.com/story/show/60257148
<ajs6f1>awoods: ping?10:04
<awoods>ajs6f1
<ajs6f1>For https://www.pivotaltracker.com/story/show/57310572
<pivotal-bot>feature: Improve/update Maven descriptive metadata (unscheduled) / owner: A. "Mutchkin" Soroka
<ajs6f1>do you mind if I just push to master?
It's a totally non-functional thing.
<awoods>ajs6f1: go for it
<ajs6f1>Time to achieve description!10:05
<awoods>make it happen
<ajs6f1>awoods: I'm assuming that everyone listed under "Technical Team" here: https://wiki.duraspace.org/display/FF/Project+Team goes into Maven metadata as a "developer". Yes?10:11
<cbeer>awoods: i'm at an all-day conference today. probably around on IRC, but will miss our calls.
mikeAtUVa: i've been doing single-datastream adds, but batching them (and improving the batching API) is on my todo list10:12
<awoods>ajs6f1, yes
<ajs6f1>k
<cbeer>mikeAtUVa: but initial fcrepo4 performance looked dreadful (success, i guess)
<awoods>cbeer: thanks, we will be missing you. I assume you return on Friday?
<cbeer>awoods: yes.10:13
<awoods>cbeer: what is the topic of the conference?
<cbeer>awoods: stanford IT.10:14
<pivotal-bot>Gregory Jansen edited "Enhance AuthZ documentation" https://www.pivotaltracker.com/story/show/6121262010:15
Esme Cowles added comment: "It makes sense that, in a clustered scenario with self-hosted config, that the common repo would have to con..." https://www.pivotaltracker.com/story/show/6025714810:17
A. "Mutchkin" Soroka started "Improve/update Maven descriptive metadata" https://www.pivotaltracker.com/story/show/5731057210:19
Gregory Jansen added comment: "I think the first step will be to establish what information ModeShapes passes to our PEP when versions a..." https://www.pivotaltracker.com/story/show/60769426
A. "Mutchkin" Soroka added comment: "The crucial question (to which we don't yet know the answer) is: to what extent may repo configs di..." https://www.pivotaltracker.com/story/show/6025714810:20
Esme Cowles added comment: "Does self-hosting changes that fact? If divergent configs can cause problems for clustering, wouldn't havin..." https://www.pivotaltracker.com/story/show/6025714810:23
Gregory Jansen edited "Add a configuration point for list of valid access roles" https://www.pivotaltracker.com/story/show/60341238
A. "Mutchkin" Soroka added comment: "It doesn't change the fact that we have to deal with it, but I think we may want to deal with it _b..." https://www.pivotaltracker.com/story/show/6025714810:24
* tecoripa joins10:25
* osmandin joins10:28
<ajs6f1>tecoripa: what is you time zone?10:31
<awoods>tecoripa = central
<ajs6f1>Is that gmt -6
cause eastern is -510:32
<awoods>sounds right, then.
I was thinking I was -8
<ajs6f1>k
<awoods>but maybe that depends on daylight savings or not
I must be -7 now
<tecoripa>ajs6f1: Central/Chicago. It's 9:32 AM right now here.10:33
<ajs6f1>k— then −6 is correct. thnx
awoods: Maven metadata goes against gmt to avoid DST questions
mikeAtUva: what's yr github id?10:34
<mikeAtUVa>ajs6f:mikedurbin
<awoods>tricky
<ajs6f1>thnx
* github-ff joins10:35
[fcrepo4] ajs6f pushed 1 new commit to master: http://git.io/SzLGKA
fcrepo4/master 34b5ecc ajs6f: Updating developer metadata
* github-ff leaves
<pivotal-bot>A. "Mutchkin" Soroka added comment: "https://github.com/futures/fcrepo4/commit/34b5ecc6f6e5ea85e27a864bdad1736ca78cefb1" https://www.pivotaltracker.com/story/show/5731057210:36
Scott Prater added "Write draft About page for Fedora 4 documentation" https://www.pivotaltracker.com/story/show/6121636210:39
<cbeer>escowles: looks like your fcrepo4 presentation is doing pretty well in the voting, huh'10:43
?
<escowles>cbeer: pretty good -- it's been as high as #4, now it's at #8 -- so i think it's pretty likely to get in
cbeer: tho i somehow neglected to mention it to declan, so that was awkward...10:44
* github-ff joins10:48
[fcrepo4] ajs6f pushed 1 new commit to master: http://git.io/hD5y6g
fcrepo4/master 29f4faf ajs6f: Improved Maven metadata for modules
* github-ff leaves
<pivotal-bot>Gregory Jansen edited "Add a configuration point for list of valid access roles" https://www.pivotaltracker.com/story/show/60341238
Gregory Jansen edited "Basic roles: acceptance testing" https://www.pivotaltracker.com/story/show/55769268
A. "Outrageac" Soroka added comment: "https://github.com/futures/fcrepo4/commit/29f4faf3faee11a8cb2503a6e933b15fdf16b2aa" https://www.pivotaltracker.com/story/show/5731057210:49
A. "Outrageac" Soroka finished "Improve/update Maven descriptive metadata" https://www.pivotaltracker.com/story/show/57310572
* edInCo joins10:51
* travis-ci joins
[travis-ci] futures/fcrepo4#1245 (master - 34b5ecc : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/ed4fa23450c1...34b5ecc6f6e5
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14316742
* travis-ci leaves
* ajs6f1 leaves10:53
<pivotal-bot>Andrew Woods accepted "Improve/update Maven descriptive metadata" https://www.pivotaltracker.com/story/show/5731057210:54
Andrew Woods added comment: "Pending review comments. Please put back in "Finished" state when done." https://www.pivotaltracker.com/story/show/6105760810:55
Andrew Woods rejected "Run fcrepo3 benchmarks on Stanford server" https://www.pivotaltracker.com/story/show/61057608
* gregjansen joins
* ajs6f joins
<osmandin>readytalk?10:59
<awoods>yes
866-740-1260, participant code: 2257295
<pivotal-bot>Scott Prater started "Refactor Test Clusters wiki page into Test Platform Profiles" https://www.pivotaltracker.com/story/show/6107756011:03
<awoods>gregjansen: https://wiki.duraspace.org/display/FF/Design+-+Content+Modeling
gregjansen: https://wiki.duraspace.org/display/FF/Fedora+3+to+Fedora+4+Concept+Mapping11:04
gregjansen: https://wiki.duraspace.org/display/FF/Fedora+3+Object+representation+in+Fedora+411:05
* travis-ci joins
[travis-ci] futures/fcrepo4#1246 (master - 29f4faf : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/34b5ecc6f6e5...29f4faf3faee
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14317374
* travis-ci leaves
<ajs6f>I'm here
<tecoripa>I'm here11:06
<mikeAtUVa>I'm here.
<ajs6f>all: Is there some reason why we moved our checkstyles and formating configs into a separate git repo / maven artifiact?
Shouldn't they just be in the main fcrepo4 parent project?
* barmintor joins11:07
<awoods>barmintor: 866-740-1260, participant code: 2257295
<mikeAtUVa>Who is taking notes?
<awoods>I will
<pivotal-bot>A. "Outrageac" Soroka finished "Create sample Performance Testing Profile (PTP) for common use case" https://www.pivotaltracker.com/story/show/6097389211:13
* kaarefc joins
<awoods>tecoripa: https://github.com/futures/benchtool11:16
<tecoripa>awoods: thanks, got it
<barmintor>awoods: we can totally handle it offline, just wanted to mention it11:21
<bljenkins>Yippie, build fixed!11:24
Project fcrepo4 build #1425: FIXED in 23 min: http://ci.fcrepo.org/jenkins/job/fcrepo4/1425/
A. Soroka: Improved Maven metadata for modules
Project fcrepo-fixity-corrupter build #474: SUCCESS in 1 min 40 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/474/11:26
Project fcrepo-kitchen-sink build #660: STILL FAILING in 6 min 33 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/660/11:32
<awoods>tecoripa: https://wiki.duraspace.org/display/FF/Design+-+Administrative+Search11:46
<pivotal-bot>A. "Outrageac" Soroka started "Remove statics from NodePropertiesTools" https://www.pivotaltracker.com/story/show/6086561811:55
<osmandin>afk12:00
* github-ff joins12:10
[fcrepo4] ajs6f created UnstaticNodePropertiesTools (+1 new commit): http://git.io/2TPCig
fcrepo4/UnstaticNodePropertiesTools 0fdfc80 ajs6f: Unstatic-ifying NodePropertiesTools
* github-ff leaves
* github-ff joins
[fcrepo4] ajs6f opened pull request #162: Unstatic-ifying NodePropertiesTools (master...UnstaticNodePropertiesTools) http://git.io/pRvqJQ
* github-ff leaves
<pivotal-bot>A. "Outrageac" Soroka added comment: "https://github.com/futures/fcrepo4/pull/162" https://www.pivotaltracker.com/story/show/60865618
A. "Outrageac" Soroka finished "Remove statics from NodePropertiesTools" https://www.pivotaltracker.com/story/show/60865618
<ajs6f> awoods: hold off on that PR ^^^ for a sec, might be a problem12:12
* github-ff joins12:17
[fcrepo4] ajs6f pushed 1 new commit to UnstaticNodePropertiesTools: http://git.io/Lo3Xnw
fcrepo4/UnstaticNodePropertiesTools 7e045a2 ajs6f: Correction for fcrepo-transform
* github-ff leaves
<ajs6f>awoods: ^^^ there we go. Good to go now.
* ajs6f leaves12:21
<pivotal-bot>Gregory Jansen added "Create project with example content models" https://www.pivotaltracker.com/story/show/61225722
<gregjansen>tecoripa: here is the content models ticket: https://www.pivotaltracker.com/story/show/61225722
<pivotal-bot>feature: Create project with example content models (unscheduled) / owner: Gregory Jansen
Gregory Jansen started "Create project with example content models" https://www.pivotaltracker.com/story/show/6122572212:22
* travis-ci joins12:24
[travis-ci] futures/fcrepo4#1247 (UnstaticNodePropertiesTools - 0fdfc80 : ajs6f): The build has errored.
[travis-ci] Change view : https://github.com/futures/fcrepo4/commit/0fdfc80991ab
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14322072
* travis-ci leaves
* ajs6f joins12:25
barmintor: Peter Kriens (big OSGi honcho) just wrote a nice blog post: http://blog.osgi.org/2013/11/osgi-services-versus-extenders.html that has some good insight into the whole OSGi μservices thing and why it connects to dynamicity. I think it's a little relevant if we move to OSGi because we'll have to think about CDI and Declarative Services and how they do and don't make sense for us.12:29
<pivotal-bot>A. "Outrageac" Soroka added "File ticket for MODE to investigate a better configuration API" https://www.pivotaltracker.com/story/show/61226386
<barmintor>ajs6f: All I want is someone to say "OSGi is better" or "CDI is better", so I know what bridge to use.12:30
<ajs6f>barmintor: http://en.wikipedia.org/wiki/Childhood's_End
<barmintor>I get it- you're Arthur C Clarke, and I'm Stanley Kubrick12:31
<ajs6f>I have nothing like Clarke's optimism.12:32
* travis-ci joins12:33
[travis-ci] futures/fcrepo4#1249 (UnstaticNodePropertiesTools - 7e045a2 : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/0fdfc80991ab...7e045a204482
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14322496
* travis-ci leaves
<bljenkins>Project fcrepo-fixity-corrupter build #475: SUCCESS in 1 min 49 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/475/12:35
* fasseg joins12:40
sry about missing the call, I was in Traffic Hell, stood for almost 2h after a garbage truck tipped over :/12:41
<ajs6f>I think you mean a mobile garbage _repository_.12:43
<fasseg>yes and it was leaking information like hell
<ajs6f>Nothing smells better than bit rot.
<cbeer>ah ha. https://github.com/futures/fcrepo4/blob/master/fcrepo-http-commons/src/main/resources/config/rest-sessions/repository.json isn't set up to do asynchronous indexing12:49
<pivotal-bot>Scott Prater finished "Refactor Test Clusters wiki page into Test Platform Profiles" https://www.pivotaltracker.com/story/show/61077560
<ajs6f>So when we said that indexing is slow, we meant _synchronous_ indexing is slow?
<cbeer>yes. which we knew before.
<ajs6f>"We shall not cease from exploration, and the end of all our exploring will be to arrive where we started and know the place for the first time."12:50
<cbeer>so, after versioning and indexing is off, the initial numbers don't look awful
time to gather real numbers..
<ajs6f>Fedora 4: It Doesn't Look Awful.
* kaarefc leaves12:51
<ajs6f>https://issues.jboss.org/browse/MODE-2101
<escowles>Fedora 4: Now You Can Turn Off The Slow Bits™
<ajs6f>(Pursuing improvements to MODE's config API)12:52
We really ought to register "The Slow Bits" as a trademark.
<osmandin>Fedora 4 Calculus (latest ajs6f discovery)
<cbeer>but they're very stable bits.
<ajs6f>TYPE-SAFE calculus!12:53
<tecoripa>echo "stable" | sed 's/b//'
<ajs6f>Eventual consistency, my impatient friend.12:54
<tecoripa>... is another way to say entropy.
<escowles>Fedora 4: Durable When We Get Around To It
<ajs6f>Persistence is....
barmintor: Don't like annotation-based config? Tired of writing reams of XML wiring? Enjoy: https://github.com/spring-projects/spring-scala/wiki/Functional-Bean-Configuration12:56
Beans are the results of things that produce beans. That is all ye know, and all ye need know.
<tecoripa>spring + scala? oh my.
beans, beans, the beautiful fruit.
<ajs6f>If the bean soup tastes bad, add red pepper. It can't hurt.12:57
afk bbl
* ajs6f leaves
<osmandin>afk 30
<bljenkins>Project fcrepo-fixity-corrupter build #476: SUCCESS in 1 min 23 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/476/12:59
* kaarefc joins
* ajs6f joins13:02
<bljenkins>Project fcrepo-kitchen-sink build #661: STILL FAILING in 4 min 14 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/661/13:03
* ermadmix leaves13:05
<cbeer>ok, 250 objects in, and fcrepo4 is twice as fast as fcrepo3 (when versioning and indexing are disabled, at least.)13:06
<escowles>cbeer: can you ingest with indexing disabled, and then turn it on later?13:07
<cbeer>escowles: yep. there's some build index if missing option13:08
<gregjansen>bbi30
<cbeer>i don't know if you can enable it without a restart though
<escowles>having to restart isn't a huge problem -- though of course it would be nice to put the repo into "ingest mode", do a big batch ingest, and then put it back in normal mode without having to restart13:09
<cbeer>and, indexing performance doesn't really worry me.. the indexing code isn't that great and is running lucene 3 (and updating to 4 will cut memory usage and boost performance)
<ajs6f>Are we still trying to push people to provide query functionality _outside_ the core?13:10
<cbeer>i sure hope so.
<ajs6f>then it matters even less
<tecoripa>ajs6f, cbeer: then it doesn't merit its own test profile, at least right now, I take it13:12
* gregjansen leaves13:13
* tecoripa leaves13:24
<pivotal-bot>Scott Prater started "Fedora performance tests with/without active basic roles and PEP enforcement" https://www.pivotaltracker.com/story/show/5941731413:28
<awoods>cbeer: good news about your performance results: take-away: turn off versioning and indexing13:29
ajs6f: review PR-162 now
<ajs6f>tecoripa: I don't htink so. I think we just want to avoid crippling ourselves.
<cbeer>s/turn them of/\1 turn them off it you care about raw throughput/
<ajs6f>awoods: what's that?
<awoods>s/review/reviewing/
* gregjansen joins13:30
<awoods>ajs6f: i.e. I am reviewing PR-162 now
<ajs6f>awoods: What is PR-162?
<awoods>https://github.com/futures/fcrepo4/pull/162
<ajs6f>Oh, right. Enjoy.
It's subminor.
<awoods>ajs6f: I was hoping it would be a step towards reducing our PowerMock overhead.13:31
<ajs6f>It is, but working just one class doesn't do that much.13:32
Working on JcrRdfTools and FedoraTypesUtils would do a lot more.
<awoods>ajs6f: add them to the list.
<ajs6f>afk bbs
What list?
* ajs6f leaves
* ajs6f joins13:35
<awoods>ajs6f: ...the list of classes to be "de-static'd"13:38
<ajs6f>Where is that list/
?
<awoods>ajs6f: In my heart
<ajs6f>"We're getting into a whole weird area…" — Joel Hodgson as Joel Robinson13:39
* ermadmix joins13:40
* fasseg leaves13:42
* github-ff joins
[fcrepo4] awoods pushed 1 new commit to master: http://git.io/ATwW6g
fcrepo4/master 3937171 ajs6f: Unstatic-ifying NodePropertiesTools...
* github-ff leaves
<pivotal-bot>Andrew Woods added comment: "Resolved with: https://github.com/futures/fcrepo4/commit/39371715f9449c906c74b384c00a7f3d0daf400f" https://www.pivotaltracker.com/story/show/6086561813:43
Andrew Woods delivered "Remove statics from NodePropertiesTools" https://www.pivotaltracker.com/story/show/60865618
* github-ff joins
[fcrepo4] awoods closed pull request #162: Unstatic-ifying NodePropertiesTools (master...UnstaticNodePropertiesTools) http://git.io/pRvqJQ
* github-ff leaves
* fasseg joins13:46
<awoods>All: We are testing the 15-person limit on g-hangouts... please join NOW: https://plus.google.com/hangouts/_/event/co1c4sgi4sbhudj4o6t9u09tc0813:54
<gregjansen>join us for the Fedora flash mob
<awoods>All: this is not a meeting "per se". you do not have to talk... just join.
<ajs6f>I just tried and got "This video call is full"14:01
I feel so alone.
* travis-ci joins14:02
[travis-ci] futures/fcrepo4#1251 (master - 3937171 : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/29f4faf3faee...39371715f944
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14326584
* travis-ci leaves
<awoods>All: test over, limit is still 10 apparently14:03
<ajs6f>Google just handle to amount of love we bring it.
can't handle, that is.
<bljenkins>Project fcrepo-fixity-corrupter build #477: SUCCESS in 1 min 26 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/477/14:05
<ajs6f>barmintor/gregjansen/tecoripa: In light of the new authZ infrastructure, does this: 4789590114:15
still make sense?
Sorry, https://www.pivotaltracker.com/story/show/47895901
<pivotal-bot>feature: Move SessionFactory out of fcrepo-http-commons into a new fcrepo-authn module with default impl for HTTP. (unscheduled) / owner: A. "Outrageac" Soroka
<barmintor>ajs6f: goos question14:16
err… "good". No, wait: I'll stick with "goos".
<ajs6f>A goose question is one that honks loudly.14:17
<cbeer>A geuze question is tasty.14:18
<ajs6f>But funky.14:19
<awoods>All: one more try: https://plus.google.com/hangouts/_/76cpjrnjr0b02kam9q1stf0s4k?hl=en
this is your chance: ajs6f14:20
<barmintor>Grr. Is there anything less intuitive than line numbers not being the default option in durned IDE?14:21
<ajs6f>Eclipse?14:22
<barmintor>yeah
I can't really have MODE and ISPN open in the same workspace
<ajs6f>http://stackoverflow.com/questions/1311720/where-to-tweak-an-eclipse-to-change-the-default-settings-used-when-creating-a-ne14:23
No great answer there, but a few workaraounds.14:24
"These go to 11."
<bljenkins>Project fcrepo-fixity-corrupter build #478: SUCCESS in 1 min 1 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/478/14:25
Project fcrepo-kitchen-sink build #662: STILL FAILING in 3 min 52 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/662/14:29
<osmandin>afk14:34
<barmintor>at org.modeshape.jcr.value.binary.infinispan.InfinispanBinaryStore$LockFactory$ISPNLock.unlock(InfinispanBinaryStore.java:666)14:40
\m/14:41
escowles: the stack you gisted yesterday was for the File config, right?14:42
it's actually a different issue than the one fasseg was having?
<escowles>i think it was for leveldb
because the file config succeeded
<barmintor>Hmm...14:43
<escowles>i posted two gists: https://gist.github.com/escowles/7572678 (leveldb running under jetty)14:44
https://gist.github.com/escowles/7573019 (war file running under tomcat with default config)
<barmintor>Ah, right- that one I disreagrded (no offense)
you didn't have one that leveldb under tomcat, or single file under jetty?14:45
<escowles>i don't thinks so -- i did the ispn configs basic, bdb and chained under jetty earlier, if you want me to gist the stacktraces from those14:46
<awoods>afk14:53
<barmintor>oh, I think I get it: This looks like the file store b/c it's running OOM when it stores the metadata, and leveldb is the blob store?14:56
<ajs6f>ask bbs14:57
afk
* awoods leaves15:00
* awoods joins
* ajs6f leaves15:08
* ajs6f joins15:09
<osmandin>afk (meeting)15:21
<pivotal-bot>A. "Outrageac" Soroka added "Spring cleaning" https://www.pivotaltracker.com/story/show/6124006415:25
<osmandin>is it possible to use a different parser for abdera.. noticed there's a dependency on axiom15:26
<ajs6f>I don't know, but if it's just using javax.xml, it should be pluggable.
It would decrease our footprint to resolve to one parser.
<osmandin>as an aside, relying on parsing xml in unit tests seems to be an interesting idea15:27
<ajs6f>Meaning you find it too heavy?
<barmintor>awoods: I'm plowing through lots of ISPN and MODE trying to find out what leveldb's problem is, but worth noting that the leveldb library has been broken out and largely rewritten15:28
<osmandin>No, I mean it's a bit flaky (or adds to flakiness)
<ajs6f>Well, the alternative seems to be to use regexps?
which is awfully brittle.
<osmandin>Hypothetically speaking, in a unit test if a parser randomly fails, is that a failure of the test?15:30
<ajs6f>It would make the test fail, so I guess so.
It's not a failure in the sense that neither the test nor code is wrong.
<osmandin>Yes, that's what I mean
<ajs6f>Are you seeing that? I haven't ever seen our XML-using unit tests fail.15:31
<osmandin>Not a failure of commission..
<ajs6f>A failure of trust.
<osmandin>Yes, I'm seeing that in fcrepo-jms..
<ajs6f>Is _that_ the intermittent death thing?
<osmandin>yes
<ajs6f>I only remember seeing the integration tests in jms fail...
the unit tests are failing?15:32
<osmandin>The integration test fail sometime due to the parser complaining.. the message seems to be OK for the most part.. although I did find some truncated messages (which is a different story)
<ajs6f>Right. So it's not the unit tests?15:33
<osmandin>If I ask it to parse again, the parser works fine
<ajs6f>Parsing the exact same text?
<osmandin>Yes
<ajs6f>Maybe it's using a _really_ non-deterministic tree automaton! (Little joke for the XML fans out there.)15:34
{crickets}
<escowles>ajs6f: i get the feeling there aren't many XML fans out there...
<ajs6f>I haven't met any in this room yet.
<barmintor>ajs6f++ / just kidding, I don't get it.15:35
<osmandin>Maybe specify that don't proceed with the unit test until you have parsed it right (or made sense of it)!!
an infinite loop
<ajs6f>http://en.wikipedia.org/wiki/Tree_automaton
<barmintor>ajs6f: That spring-scala config thing was nonsense
<ajs6f>Eh?
It seemed to parse for me.
http://tata.gforge.inria.fr/
^^^ awesome online text about tree automata. Good airplane reading.15:36
<barmintor>tree automaton: http://en.wikipedia.org/wiki/Ent
<ajs6f>osmandin: To be clear: are we talking about a unit test, or an integration test?
<osmandin>IT15:37
<ajs6f>Okay.
In which case, it is entirely appropriate to use a parser, because that is in fact what will happen in the real world. We can choose a better parser...
<osmandin>But do you think there should be some logic on the sender end...15:38
<ajs6f>To validate?15:39
<osmandin>to coordinate if the parser cannot make sense of the text
<ajs6f>For a resend?
<osmandin>yeah
<ajs6f>No. That's not a good pattern for messaging. That's not the server's job.
We should be _able_ to resend.
But we should not have special logic that considers _when_ to resend.
<osmandin>is it the job of JMS then
<ajs6f>It's the job of the client. If the client can't understand a message, it's the client's job to do whatever may be appropriate, including asking for a resend.15:40
We can introduce ack/nak or other patterns, but that's a totally different game than what we are currently playing, which is simple announcement.
Our events are not (currently) transactional.15:41
The client may not care. The client may take other action than asking for a resend.
If this problem is rooted
(XML joke!)
in the fact that we are producing bad messages, that's what we need to fix.15:42
But it sounds like we just have a crappy parser in use for tests?
<osmandin>bad messages, meaning?
<ajs6f>Improperly formatted.
Can we start by trying a different/better parser?15:43
<osmandin>I think I'll give another library a quick try
<ajs6f>I'm a big fan of transactional messaging fabrics, but that's a long road to start going down at the end of the year.
+1 to trying another library.
<osmandin>Do you know if there's a reason atom was chosen15:44
<ajs6f>Incidentally, it's the job of the JMS broker to resend _iff_ the message was garbled in xmission. (checksum error)
The format as it stands is a legacy from Fedora 3.
We should definitely do something better.
But we expect to have to supply legacy support there.15:45
Lots of people have workflow built over that crappy format.
<barmintor>Atom/rss--
<ajs6f>They're good— in the right places.15:46
<osmandin>Ok, I thought there was someone who liked atom etc
<ajs6f>It's our painful heritage, like a racist uncle to whom you have to be nice at family gatherings.
<osmandin>Ok, thx15:47
<barmintor>and yet, we don't discuss supporting FCFG for Fedora 4. Then again, no one was sadistic enough to build systems around it.
well, except the obvious one.
<ajs6f>barmintor: Right. It's the integration points that will cause migration pain.
<pivotal-bot>Mike Durbin deleted "Optional Versioning" https://www.pivotaltracker.com/story/show/6088232815:51
<ajs6f>afk blbl15:58
* ajs6f leaves
<barmintor>well, there are definitely some points at which FileCacheStore also appears to want to hold a object-to-store in memory. I could see reducing copying a fair amount by using either the streaming or NIO libraries16:23
especially since they appear to use NIO to write (FileChannels), but byte arrays to pass around the buffers.16:24
<awoods>barmintor: What are you suggesting with your comment about the leveldb library having been broken out?16:37
<barmintor>awoods: that there's a question of how much time to invest in library that might not even be what we release on16:38
<awoods>barmintor: are tracing the root-cause of some of the performance issues down to leveldb, itself?16:39
are >you< ...
<barmintor>awoods: No, the error where files > 1G crash
<awoods>barmintor: ok, so that is a leveldb issue.16:40
<barmintor>awoods: yeah, these are all, AFAIK, issues at the MODE and ISPN level
though it might be worth demonstrating that in some tests against their codebase16:41
<awoods>barmintor: so maybe I am not understanding you correctly...
barmintor: are you seeing an issue with leveldb?
<barmintor>awoods: gtalk incoming
<fasseg>barmintor: that's the exception im seeing most often when ingesting > 1gb, but the thread is not very helpful :( https://community.jboss.org/thread/19872916:42
<awoods>https://plus.google.com/hangouts/_/76cpjrnjr0b02kam9q1stf0s4k?hl=en
<pivotal-bot>Frank Asseg added comment: "@awoods Yes, I could download the file using curl, and a bitwise comparison with the original was successful" https://www.pivotaltracker.com/story/show/6111902016:50
<osmandin>afk
* tecoripa joins16:53
<pivotal-bot>Andrew Woods added comment: "@frankasseg, It would be good to document the fact that the REST-API has demonstrated the ability to downlo..." https://www.pivotaltracker.com/story/show/6111902016:54
Benjamin Armintor added "System test project for large files" https://www.pivotaltracker.com/story/show/6124711016:55
<awoods>All: fasseg in the ticket above ^^^ just noted that he was able to download a 300-GB file via the REST-API, which good to know.
<pivotal-bot>Benjamin Armintor accepted "Record Profiling data for Benchtool 1k/1k/15" https://www.pivotaltracker.com/story/show/61135088
* ajs6f joins16:56
* ajs6f leaves
<pivotal-bot>Benjamin Armintor added "Recrod profiling data for Benchtool 1k/2m/15" https://www.pivotaltracker.com/story/show/6124729016:57
Benjamin Armintor edited "Record profiling data for Benchtool 1k/2m/15" https://www.pivotaltracker.com/story/show/61247290
* github-ff joins16:59
[fcrepo4] ajs6f created SpringShortcuts (+1 new commit): http://git.io/UL3XSw
fcrepo4/SpringShortcuts a8f930f ajs6f: Intermediate commit
* github-ff leaves
* tecoripa leaves17:02
* tecoripa1 joins17:05
<pivotal-bot>Frank Asseg added comment: "Will do..." https://www.pivotaltracker.com/story/show/6111902017:06
<fasseg>awoods: where do you want me to document the succesful 300gb ingest/retrieval on a single node fedora? https://wiki.duraspace.org/display/FF/Fedora+4.0+Key+Capabilities or https://wiki.duraspace.org/display/FF/How+to+handle+large+files?17:08
<awoods>fasseg: on a call
<fasseg>kk
<gregjansen>fasseg: I just found a test results page, where I put my results, let me get you the link17:20
<pivotal-bot>Frank Asseg added comment: "Added an info paragraph about the successful ingest/retrieve to https://wiki.duraspace.org/display/FF/How+to..." https://www.pivotaltracker.com/story/show/6111902017:21
<fasseg>ah ok :)
* travis-ci joins17:22
[travis-ci] futures/fcrepo4#1252 (SpringShortcuts - a8f930f : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/commit/a8f930f6e815
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14336611
* travis-ci leaves
<gregjansen>this was the one I found.. not sure that it is quite right: https://wiki.duraspace.org/display/FF/Single-Node+Performance+Testing
mainly b/c of the title, though I notice that it does contain cluster results17:23
<fasseg>hmm yeah this seem old ;)17:24
*these
<gregjansen>yes
If you want, I think we could use a new template for a test result details page... (or the big table where we can all post results, w/links to test profiles)17:25
Esme may already be working in this direction..
I think FF Candidate test harness was really about testing modeshape and similar candidate platforms17:26
<fasseg>gregjansen: I guess it's not really about test results, it's about that specifically large size files have been successfully ingested/retrieved using the REST API. This was a pure functional test and not a performance test...17:27
<gregjansen>fasseg: aha, sorry17:28
* gregjansen leaves
* gregjansen joins17:29
* gregjansen leaves
<bljenkins>Project fcrepo-fixity-corrupter build #479: SUCCESS in 2 min 20 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/479/17:30
* tecoripa1 leaves17:35
<bljenkins>Project fcrepo-kitchen-sink build #663: STILL FAILING in 11 min: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/663/17:42
* kaarefc leaves17:43
<awoods>fasseg: I see you found a location for the largefile download, good.17:54
<fasseg>yep...
<awoods>what time is it, fasseg?17:55
<fasseg>but ill be off now for the night it's almost 0am ;)
hehe ^^
* edInCo leaves18:01
* fasseg leaves
* ermadmix leaves18:03
* ksclarke leaves18:23
* tecoripa joins18:33
* tecoripa leaves
* osmandin leaves20:05
* tecoripa joins20:43
* tecoripa leaves
* ksclarke joins21:17
<cbeer>hm. there is something really wrong with infinispan + leveldb. probably worth sending them an issue21:52
<awoods>cbeer: barmintor has been talking about pulling together one or more integration tests against Modeshape and/or Infinispan that demonstrate the bugginess to include in bug reports to the appropriate project.21:59