Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* kaarefc joins00:55
* kaarefc leaves01:35
* ksclarke leaves02:04
* kaarefc joins02:07
* fasseg joins08:01
Im getting excpetion in the AtomJmsIT when building fcrepo now: org.apache.abdera.parser.ParseException: org.apache.axiom.om.OMException: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </conttestDatasramr>; expected </content>.08:12
Are you guys getting that too?08:13
<pivotal-bot>Frank Asseg added "Test ingestion of large files on a single node" https://www.pivotaltracker.com/story/show/6111902008:20
<escowles>fasseg: i'm not seeing that -- i just updated and built fcrepo4 with no errors
<fasseg>escowles: thanks...ill check my env then08:21
<pivotal-bot>Frank Asseg added "Test a federated Fedora instance with a number of files > 1gb" https://www.pivotaltracker.com/story/show/6111930408:24
Frank Asseg added "Test a federated Fedora instance serving a file larger than 100gb" https://www.pivotaltracker.com/story/show/6111937008:26
Frank Asseg added "Document the limitations found when using File System Federation" https://www.pivotaltracker.com/story/show/6111945608:28
Frank Asseg edited "Test a federated Fedora instance with a number of files > 1gb" https://www.pivotaltracker.com/story/show/61119304
Frank Asseg edited "Test a federated Fedora instance serving a file larger than 100gb" https://www.pivotaltracker.com/story/show/61119370
* gregjansen joins08:30
<pivotal-bot>Frank Asseg added "Create a wiki page describing how to handle large files efficiently" https://www.pivotaltracker.com/story/show/6111977608:33
A. "Mutchkin" Soroka added comment: "Yep. I didn't realize it at the time because I thought we would be able to eliminate all Jena types..." https://www.pivotaltracker.com/story/show/5924323608:55
Frank Asseg added comment: "The infinispan leveldb configuration is producing OutOfMemory errors, consistently during serialization." https://www.pivotaltracker.com/story/show/6111902008:57
Frank Asseg added comment: "The basic configuration also produces OutOfMemory errors during serialization when ingesting large files" https://www.pivotaltracker.com/story/show/6111902008:58
Frank Asseg added comment: "Using the file configuration ingestion of a single file > 1gb is possible but when doing multiple ingests of..." https://www.pivotaltracker.com/story/show/6111902008:59
* gregjansen leaves09:01
* kaarefc leaves09:06
* osmandin joins09:13
<pivotal-bot>Frank Asseg added comment: "Using the file/repository.json configuration seems to work for 10 files of 1gb each: ""09:15
With
"-Dfcrepo.modesha..." https://www.pivotaltracker.com/story/show/61119020
Andrew Woods started "Test ingestion of large files on a single node" https://www.pivotaltracker.com/story/show/6111902009:19
* ajs6f joins09:20
<pivotal-bot>Andrew Woods accepted "Alter outward-facing contracts in fcrepo-kernel to expose new iterator-based machinery for RDF." https://www.pivotaltracker.com/story/show/5924323609:25
Andrew Woods started "Alter fcrepo-http-api to take advantage of new iterator-driven RDF machinery." https://www.pivotaltracker.com/story/show/5924401409:27
* ksclarke joins
* ermadmix joins09:31
* kaarefc joins09:32
* kaarefc leaves09:36
* kaarefc joins09:37
* tecoripa joins09:46
<osmandin>afk09:49
<pivotal-bot>Frank Asseg added comment: "Using the file configuration an ingest of 100 files of 1gb was also succesful" https://www.pivotaltracker.com/story/show/6111902009:58
* ajs6f1 joins10:00
* ajs6f leaves10:01
<ajs6f1>barmintor: Should we be using Spring's @Controller instead of @component for the JAX-RS resource classes?10:03
barmintor: Then again, should we be using @Service for the o.f.k.service classes, instead of @Component?10:05
Actually, those are questions for anyone who knows more about Spring than do I, which would be anyone.10:09
* gregjansen joins10:15
* kaarefc leaves
<barmintor>ajs6f1: IDK if Spring will scan @Service, but in principal we should be using those annotations rather than the Spring stereotypes10:22
<awoods>fasseg: The intermittent ATOM test failure is known, and being worked by osmandin... although there is no ticket: http://irclogs.fcrepo.org/2013-11-18.html (search for "ATOM-something")10:23
<fasseg>awoods: kk, but it worked for me everytime now so I only had this once
<ajs6f1>barmintor: I understand @Service to have the same semantics as @Component, plus. See: http://docs.spring.io/spring/docs/3.0.x/spring-framework-reference/html/beans.html#beans-stereotype-annotations
barmintor: Ditto @Controller...
<awoods>fasseg: it is intermittent
<ajs6f1>Maybe we should just try it?10:24
<fasseg>awoods: dont know the word have to look up ;) (meaning it's a race condition of sorts?)
<barmintor>ajs6f1: I'm sorry, I was talking about
<fasseg>"occurring at irregular intervals; not continuous or steady" I see10:25
<osmandin>fasseg regarding that issue I plan to create / work on a ticket
<awoods>ajs6f1/barmintor: sounds like there is no harm in using the more specific annotations... and in regards to future Spring versions (which we likely care less about) there may someday be additional benefit.10:27
<ajs6f1>Semantics are good.10:28
<gregjansen>hey folks, shall I start a fcrepo-example-content-models project? I'm generating some CNDs for UNC stuff and it seems like git is the tool for sharing these..10:29
<barmintor>ajs6f1: I was thinking of https://hk2.java.net/2.2.0-b22/apidocs/org/jvnet/hk2/annotations/Service.html, sorry
<ajs6f1>barmintor: What's the problem? Just because we're trying to discuss _every DI framework in the Java landscape all at once_ you're getting confused? Sheesh...10:30
<barmintor>ajs6f1: you have sounded me out10:31
<ajs6f1>Let's use them _all_. Then we're be sure to use th right one.
<osmandin>ajs6f1: any experience with sezpoz?
<mikeAtUVa>fasseg: there's an old ticket about eSciDoc stories for Fedora versioning needs, but I can't access the referenced results. Is there a ticket or wiki page that summarizes this information?10:33
<ajs6f1>osmandin: No, but looks cute. Like a lightweight CDI… does it have a good community around it?
<fasseg>mikeAtUVa: errr, we actually completely stopped the work on escidoc, so I dont think these apply anymore
<osmandin>ajs6f1: probably no . . . and i think only a few projects use it.. (like jenkins)10:34
<fasseg>But I know that one of the features FIZ was interested in was a "whole object versioning"
<ajs6f1>osmandin: We have a page for our choices at:
<fasseg>which they missed in fcrepo3 which escidoc used as a basis
<ajs6f1>https://wiki.duraspace.org/display/FF/Comparison+-+Wiring
osmandin: Want to add a column for sezpoz?10:35
<mikeAtUVa>fasseg: I see. Thanks! I'm starting a wiki page for design around versioning needs and I'll be sure to start/continue the conversation about whole object versioning as things develop.10:36
<osmandin>ajs6f1: sure.. and how about a column for JDK 9 (or 10 or 11) for modularization?!!
<ajs6f1>osmandin: <sarcasm> I'm holding my breath for Jigsaw. </sarcasm>
<osmandin>one of these days...10:37
<awoods>gregjansen: re: fcrepo-example-content-model, go for it.
<fasseg>mikeAtUVa: kk
<ajs6f1>osmandin: One of these days they might catch up with OSGi. {grin}
<gregjansen>awoods: great, thinking of making a minimal test config as well (exercises CND REST for all files in a folder?)10:39
<awoods>which ticket is this, gregjansen?10:40
<gregjansen>I have one to create CND models for UNC, awoods. So this is scope creep on that one. I will make a new ticket if you still think it is worthwhile. https://www.pivotaltracker.com/story/show/6098251810:41
<pivotal-bot>feature: Prepare and test UNC content models for F4 (unstarted) / owner: Gregory Jansen
Gregory Jansen started "Prepare and test UNC content models for F4" https://www.pivotaltracker.com/story/show/60982518
Gregory Jansen started "Run benchmark tests for single Fedora node on server hardware." https://www.pivotaltracker.com/story/show/60983816
<awoods>gregjansen: got it
All: what line would you like to use for today's standup? ReadyTalk or G-hangout?10:42
<gregjansen>I'd like to find out if hangout can support ~15 as rumored
<awoods>gregjansen: http://gappstips.com/news/view/137/hangout-limit-increased-to-15-participants10:43
<mikeAtUVa>gregjansen: I have a conflicting apointment and won't make the 11:00 scrum today...
<awoods>gregjansen: apparently this random person on the Internet says G-hangout now supports 15
<gregjansen>mikeAtUVa: email works if you cannot make it10:44
<cbeer>awoods: i linked to the google FAQ yssterday
awoods: it has to be a google apps linked hangout.
<gregjansen>awoods: yeah, rumoring new google features seems like a great way to SEO
<awoods>cbeer: so we should stick with ReadyTalk?
<cbeer>08:04 ] @cbeer> awoods: https://support.google.com/a/answer/2677443?hl=en10:45
<gregjansen>duraspace has a google apps account, right?
<cbeer>awoods: or you should activate the thing and save us all.
<awoods>gregjansen: we do
<gregjansen>awoods, cbeer: seems to indicate that any member of apps account can start a hangout with 15 people (no mention of participants having to be apps members)10:47
awoods: but you would have "start the trial" of Google+ Premium
<cbeer>gregjansen: the page there says you have to activate the feature, though.
<gregjansen>cbeer: yah
awoods: why don't we do ready talk today.. then activate this thing?10:48
<awoods>gregjansen: it is a plan
gregjansen/cbeer: activated... but it will apparently take some hours to come into effect.10:51
<barmintor>Grr
fcrepo-kernel cannot depend on http-commons10:52
<ajs6f1>Hell no.
<cbeer>what's in http-commons that we need in -kernel?10:53
<barmintor>org.fcrepo.http.commons.session.BypassSecurityServletAuthenticationProvider
<cbeer>why do we need that in -kernel anyway?
<ajs6f1>afk bbs10:54
* ajs6f1 leaves
<awoods>gregjansen ^^
<gregjansen>hey guys
we should move that thing yes
I will look at it. (hopefully not too many dependencies from commons)
barmintor: we can move that. it isn't tied to anything else.10:55
READY TALK (in case you missed it)10:56
<barmintor>gregjansen: looking at it now
<gregjansen>barmintor: it handles servlet authN and bypasses authZ.. should only be required within containers, but that doesn't mean it has to be in http-commons10:57
<tecoripa>on my way10:58
<pivotal-bot>Mike Durbin added "Consider how best to expose nodes of type nt:frozenNode." https://www.pivotaltracker.com/story/show/6113303010:59
* ajs6f joins
<pivotal-bot>Mike Durbin edited "Consider how best to expose nodes of type nt:frozenNode." https://www.pivotaltracker.com/story/show/61133030
<barmintor>gregjansen: Sorry, my row of cubes is popular today11:00
<ajs6f>I'm here.11:01
<cbeer>here11:02
awoods> 866-740-1260, participant code: 225729511:03
* mikeAtUVa leaves11:04
<ajs6f>here
* tecoripa1 joins
<pivotal-bot>Benjamin Armintor unstarted "Upgrade to Jersey 2.0" https://www.pivotaltracker.com/story/show/5220082311:06
<fasseg>escowles: What configuration are you using on your cluster? the vanilla config/clustered/repository.json?
<escowles>i'm only using a single node with the minimal config
<awoods>escowles/others: Do we prefer using the benchtool versus jmeter?
<escowles>awoods: the tickets said benchtool, so that's what i'm using -- but i do prefer it to jmeter11:07
awoods: e.g., https://www.pivotaltracker.com/story/show/60556948
<pivotal-bot>feature: Single-node F4 Ingest Benchmarks (started) / owner: Esme Cowles
Osman Din added comment: "Comments (on set up): ""11:08
https://wiki.duraspace.org/display/FF/Comments+on+Access+Roles+Module" https://www.pivotaltracker.com/story/show/60615670
<cbeer>readytalk--
<pivotal-bot>Benjamin Armintor added "Record Profiling data for Benchtool 1k/1k/15" https://www.pivotaltracker.com/story/show/6113508811:09
Benjamin Armintor started "Record Profiling data for Benchtool 1k/1k/15" https://www.pivotaltracker.com/story/show/61135088
Benjamin Armintor added "Take implicit http-commons dep out of jcr" https://www.pivotaltracker.com/story/show/6113571211:11
Benjamin Armintor edited "Take implicit http-commons dep out of jcr" https://www.pivotaltracker.com/story/show/6113571211:12
<cbeer>ajs6f: have you seen http://www.day.com/specs/jcr/2.0/3_Repository_Model.html#3.7.1.7 Primary Item?11:17
i wonder if we should be using that instead of our well-known fcr/jcr:content11:18
<ajs6f>jcr:content is a well-known thing, but there's no reason we should use both, right?
<cbeer>well, jcr:content/jcr:data is an nt:file thing. (i wonder if jcr:data is the primary item for jcr:content.. hm...)11:19
<ajs6f>I just mean that anyone looking for bitstreams will look in jcr:content/jcr:data
<cbeer>ajs6f: ah, yes. nt:file has primary item jcr:content11:20
beats me where jcr:content is defined.
<ajs6f>Then we're already doing it, because that's how awesome we are.11:21
<cbeer>i guess it must be a higher level concept.
<ajs6f>cbeer: I think maybe in nt:resource?
<cbeer>ajs6f: no, i mean where the definition of jcr:content is. it might be implicit
ajs6f: and, we're not really doing it. we just go directly to jcr:content
<ajs6f>It only has to be defined on a given nodeType, right?
cbeer: Oh, you mean _our_ api should look for the primary item?11:22
<cbeer>ajs6f: exactly!
instead of going fcr:content => jcr:content => magic
<ajs6f>cbeer: I'd be fine with that. It would be more robust.
<cbeer>we can go from fcr:content => primary item => magic
<ajs6f>Magic bits...11:23
<cbeer>definitely a post-december thing. but it seems like the right thing to do
<ajs6f>Yeah. It shouldn't be that hard.
<cbeer>ajs6f: and, you're right. jcr:content is a nt:resource
i missed that
and jcr:data is the primary type of nt:resource too
<ajs6f>There used to be a good diagram of this stuff somewhere in the standard, but I don't remember where.11:24
* tecoripa1 leaves
<cbeer>ajs6f: ok. java wiring question for you.
ajs6f: for MODE-2098, i'm going to be implementing some JCR 2.1 methods11:25
<ajs6f>Use 12 gauge copper wire.
<cbeer>should i just do modeshape-api overrides to provide those until MODE switches to JCR 2.1?
or is there something more clever?
<ajs6f>You mean casts?
In our client code?11:26
(client of MODE, ie)
<cbeer>ajs6f: i guess i'm all over the stack. there will be methods in the modeshape JCR impl that match the JCR 2.1 signatures
what's the right way to add those before modeshape gets dragged up to JCR 2.1?11:27
<ajs6f>You could dynamic-proxy it, but given that it's all temporary, I wouldn't bother. I would cast and add TODOs to remind us to remove the casts later.
<cbeer>ajs6f: cast to modeshape-jcr-apis?
<ajs6f>yeah, cast to whatever types actually feature the new bahviors11:28
<cbeer>ok.
<ajs6f>We do stuff like that now to get at MODE-specific functioanlity.
<cbeer>i know. it just seems weird to define these JCR 2.1 things in a MODE-specific interface
as is.
anyway, you'll probably like the new API signature:
T Node.getPropertyAs( String relPath, Class<T> type );
<ajs6f>yeah, but it's temporary. There are better ways (e.g. proxying), but is it worth the effort?11:29
Is that to get the value of the prop?
<cbeer>ajs6f: nope. this is already way more effort than i wanted to spend on the problem.. all to save a couple lines of code in our app :P
ajs6f: exactly.
<ajs6f>What happens when I ask for a String-valued prop with value "foo" as a number?11:30
Exception?
<cbeer>ajs6f: i think it goes through the existing value factory impl
<ajs6f>K
What about multi-valued props?
<cbeer>works just fine.
String[] Node.getPropertyAs( String relPath, String[].class );11:31
(so the API doc says, at least.)
and, if MODE was actually pluggable, we could presumably make it do our property => rdf translation, right?
<ajs6f>Oh, cool. So we use arrays. That's nice. We might want to consider an iterator version, for lots of values in a props
<cbeer>yeah, i think i saw an iterator version too.. not sure if that was a rhauch invention or not11:32
<ajs6f>cbeer: yeah, if MODE was pluggable, we could make it dance and sing and lots of other stuff. See my conversation with rhauch yesterday.
cbeer: They don't see themselves as developing components, they see themselves as developing _a_ component.11:33
* barmintor sings "If a MODE were pluggable" to the tune of "If I were a rich man"11:34
<cbeer>barmintor: huh. James just chimed in on blacklight-development.11:37
* mikeAtUVa joins
<barmintor>cbeer: he's neck-deep in C, with a bespoke std lib. He might miss BL a little.11:38
<cbeer>poor guy.
<barmintor>does anyone know, off the top of their head, where fcrepo.modeshape.configuration is set?11:39
<ajs6f>Isn't that a system prop?
<barmintor>ajs6f: I think so, but that doesn't answer my question
<ajs6f>meaning it is set by the deployer at runtime.
I think awoods introduced some defaulting behavior.11:40
<cbeer>oh, that reminds me. ajs6f: do you know what the rules are for interpolating java properties? e.g. for https://www.pivotaltracker.com/story/show/6078849211:41
<pivotal-bot>feature: Single system property to specify directory for all fcrepo content (unstarted) / owner:
<cbeer>can we just do something like ${fcrepo.modeshape.storage.directory:${fcrepo.home}/storage}11:42
<awoods>barmintor: https://github.com/futures/fcrepo4/blob/master/fcrepo-webapp/src/main/resources/spring/repo.xml
<ajs6f>Depends on what is doing the interpolating. I don't know what MODE would do with that syntax. Love that JSON!
Spring would handle it fine, ironically.
Arg. OpenRDF registers the wrong mimetype for n-triples. It changed from text/plain _years_ ago.11:43
<barmintor>awoods: I'm dumb- does that set the property, or interpret the property?11:44
awoods: honest question- I thought the Spring config was interpolating a val set elsewhere, but I'm confused
<awoods>barmintor: This syntax: "${fcrepo.modeshape.configuration:classpath:/config/rest-sessions/repository.json}" ...11:45
barmintor says: use the system-property "fcrepo.modeshape.configuration"...
or if it is not found, default to "classpath:/config/rest-sessions/repository.json"
<barmintor>oh, ok11:46
thank you
<awoods>np
barmintor/cbeer: There is some additionally logic to set TMPDIR defaults here: https://github.com/futures/fcrepo4/blob/master/fcrepo-kernel/src/main/java/org/fcrepo/kernel/spring/ModeShapeRepositoryFactoryBean.java#L7211:48
...but there is a ticket to change TMPDIR to CWD. https://www.pivotaltracker.com/story/show/6100433611:49
<pivotal-bot>feature: Default Storage Parameter Directory: tmp to cwd (unstarted) / owner:
<ajs6f>afk bbl11:54
* ajs6f leaves11:58
<osmandin>afk
* ajs6f joins12:03
* github-ff joins12:29
[fcrepo4] barmintor created move-configs (+1 new commit): http://git.io/a_YGAQ
fcrepo4/move-configs d896529 Benjamin Armintor: relocating JSON configs to align with dependency tree
* github-ff leaves
* github-ff joins12:31
[fcrepo4] barmintor opened pull request #159: relocating JSON configs to align with dependency tree (master...move-configs) http://git.io/j6TH4A
* github-ff leaves
<barmintor>awoods gregjansen ^^
<awoods>barmintor: is there a ticket?12:32
<gregjansen>barmintor: looking
<awoods>barmintor: If you select "finish"ed, it will automatically go into my to-review queue.
<pivotal-bot>Benjamin Armintor started "Take implicit http-commons dep out of jcr" https://www.pivotaltracker.com/story/show/61135712
Benjamin Armintor added comment: "https://github.com/futures/fcrepo4/pull/159" https://www.pivotaltracker.com/story/show/6113571212:33
Benjamin Armintor finished "Take implicit http-commons dep out of jcr" https://www.pivotaltracker.com/story/show/61135712
<awoods>barmintor: thanks
barmintor: are we getting rid of fcrepo-jcr?12:36
<barmintor>awods:No, it has code in it
<gregjansen>barmintor: so where did the configs go?
<barmintor>awoods
<gregjansen>github UI hides it..
<barmintor>gregjansen: one went to kernel, the others went to http-commons
and the ISPN configs went to kernel12:37
<awoods>gregjansen: select: "Show Diff Stats"
<gregjansen>barmintor: THX! ok cool, so modules that have no http endpoints could use the kernel config file, but otherwise they will use a servlet authN aware one.
<barmintor>yeah, that was my thinking12:38
<gregjansen>barmintor: well I wish I had thought of that
barmintor: also, I was probably too worried about breaking everything downstream.12:39
<barmintor>:P you were busy getting the real work done
<awoods>barmintor: fcrepo-jcr was only created to hold these config files. Now fcrepo-jcr is left with a single interface: FedoraJcrTypes. What is the rationale for moving the config files?
<barmintor>awoods: it actually had a commonly used set of String constants
<awoods>barmintor: you mean: FedoraJcrTypes?
<barmintor>awoods: yes
<gregjansen>awoods, barmintor: going back to my IDE unless you need me..12:40
<awoods>barmintor: Is there anything else left in fcrepo-jcr?
<barmintor>awoods: Doesn't look it12:41
<awoods>barmintor: If not, I would say nuke that project
barmintor: if we indeed want to move those configs.
<barmintor>Hmm. I think there's a good argument for moving the configs; I need to think about moving the constants interface12:42
just to make sure there's no libs that depend on -jcr but not -kernel
and to forecast some pkg export scenarios closer to release12:43
<awoods>barmintor: I am fine with moving the configs. But can you humor me with the "good arguments"?12:44
<barmintor>awoods: of course!12:45
I think those configs should have refs to classes like the BypassFilter12:46
and those classes need to know about HTTP
* travis-ci joins
[travis-ci] futures/fcrepo4#1228 (move-configs - d896529 : Benjamin Armintor): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/commit/d8965299d4ca
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14264014
* travis-ci leaves
<osmandin>afk 30
<barmintor>but kernel doesn't, and shouldn't
and still needs IT
isolating them this way gives developers a cue as to what the reasonable dependencies are12:47
this is really my argument: b/c the nature of the MODE configs prevents us from catching dependency errors, I think we should locate the example configs in contexts where they are legitimate12:49
<gregjansen>awoods: IOW, if you are writing a module and you want RESTful ITs, you can just depend on http-commons and the http-api test-jar (hopefully) then choose appropriate config from http-commons.. (or default minimal HTTP aware one from http-api test)
<bljenkins>Project fcrepo-fixity-corrupter build #464: SUCCESS in 1 min 15 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/464/12:50
<awoods>barmintor/gregjansen: got it
barmintor: are you going to do a quick analysis of the fcrepo-jcr project (i.e. can FedoraJcrTypes be moved)?12:51
<barmintor>awoods: in progress
<pivotal-bot>Scott Prater added comment: "Test - Setup Profile: https://wiki.duraspace.org/display/FF/Test+-+Setup+Profile%3A+Image+Collection" https://www.pivotaltracker.com/story/show/60974242
<barmintor>ack, of course12:52
<tecoripa>^^^^^ This might be a useful profile to do some performance testing against: it's not as large as most of the other profiles being worked on, but it does represent a real-world, and fairly typical, use case.
<barmintor>awoods: those contstants need to be available in client contexts in which the runtime is unnecessary12:53
<bljenkins>Project fcrepo-kitchen-sink build #656: STILL FAILING in 3 min 15 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/656/
Yippie, build fixed!12:56
Project fcrepo-jms-indexer-pluggable build #247: FIXED in 7 min 19 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-jms-indexer-pluggable/247/
<awoods>barmintor: so you are happy leaving the project around with containing the one interface.
<pivotal-bot>Andrew Woods delivered "Take implicit http-commons dep out of jcr" https://www.pivotaltracker.com/story/show/6113571212:59
<barmintor>awoods: that's the reason I'd offer for it. fcrepo-kernel is a big dep if you just want to use the constants
* github-ff joins13:00
[fcrepo4] awoods pushed 1 new commit to master: http://git.io/7HmFRQ
fcrepo4/master 34aab66 Andrew Woods: Merge pull request #159 from futures/move-configs...
* github-ff leaves
<awoods>barmintor: done
* github-ff joins
[fcrepo4] awoods deleted move-configs at d896529: http://git.io/W7_YRw
* github-ff leaves
<ajs6f>cbeer: Did you write FedoraHtmlResponsesIT?13:09
<cbeer>ajs6f: which one is that?
<ajs6f>the one that appears to run an automated browser to exercise the HTML stuff?13:10
<cbeer>ajs6f: ah, yes. i wrote that.
<ajs6f>It's one of the last things I have to fix for this next endpoint to work, and I have no idea what to do with it.
I'm getting npes and other nasties...
any way I could get some help with it?13:11
<cbeer>ajs6f: from the IT itself? did you break the HTML?13:12
<ajs6f>I assume so, but I can't tell what the test is actually trying to do. I mean, I get the coarse-grained behavior from method names, but I have never seen this gargoylesoftware thing before.
<cbeer>ajs6f: launch a browser13:13
click around.
<ajs6f>and then?
<cbeer>make sure what we think should happen happens.
<ajs6f>and when the tests still don't run?
<cbeer>i think you're trying to read too much cleverness into that test
* travis-ci joins13:14
[travis-ci] futures/fcrepo4#1230 (master - 34aab66 : Andrew Woods): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/760b84277b0c...34aab66bc26e
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14265795
* travis-ci leaves
<cbeer>ajs6f: sorry, i was answering what the test was trying to do.
<ajs6f>Oh, right. I think I'm spending a lot of time editing Velocity HTML. Yay.
<bljenkins>Project fcrepo-fixity-corrupter build #465: SUCCESS in 59 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/465/13:18
* github-ff joins13:19
[fcrepo4] ajs6f created RdfIterationAgainAndAgain (+1 new commit): http://git.io/8uRrkg
fcrepo4/RdfIterationAgainAndAgain 36c6051 ajs6f: Changed FedoraNodes.describe() to stream, everything works except HTML ITs
* github-ff leaves
<bljenkins>Project fcrepo-fedora3-federation-connector build #243: UNSTABLE in 5 min 6 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fedora3-federation-connector/243/13:22
Project fcrepo-jms-indexer-pluggable build #248: UNSTABLE in 6 min 0 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-jms-indexer-pluggable/248/13:23
<pivotal-bot>Frank Asseg added comment: "single files of 2, 4, 8, 16, 32 and 64 GB ingest fine using the single-file config." https://www.pivotaltracker.com/story/show/6111902013:24
* ajs6f leaves13:27
<pivotal-bot>Mike Durbin added "Allow versioning to be turned off." https://www.pivotaltracker.com/story/show/6115175013:31
Mike Durbin started "Allow versioning to be turned off." https://www.pivotaltracker.com/story/show/61151750
* ajs6f joins13:35
* ermadmix leaves
<pivotal-bot>Osman Din added comment: "Added: ""13:36
https://wiki.duraspace.org/display/FF/Comments+on+Access+Roles+Module
for task of '. . . to test the..." https://www.pivotaltracker.com/story/show/60615670
Osman Din finished "Test Basic AuthZ" https://www.pivotaltracker.com/story/show/6061567013:37
<bljenkins>Project fcrepo-fixity-corrupter build #466: SUCCESS in 1 min 38 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/466/13:42
<pivotal-bot>Osman Din added "Fix integration test AtomJMSIT exception related to truncated messages" https://www.pivotaltracker.com/story/show/6115284213:45
Osman Din added comment: "See also: https://www.pivotaltracker.com/story/show/45629439" https://www.pivotaltracker.com/story/show/6115284213:46
Osman Din started "Fix integration test AtomJMSIT exception related to truncated messages" https://www.pivotaltracker.com/story/show/61152842
* ermadmix joins13:54
<pivotal-bot>Andrew Woods added comment: "@osmandin, do you feel like this diff represents the intention you are trying to communicate with the "Test..." https://www.pivotaltracker.com/story/show/6061567013:55
Andrew Woods added comment: "@osmandin, or rather, maybe just add wiki "comments" to @gregoryjansen's page directly." https://www.pivotaltracker.com/story/show/6061567013:58
Osman Din added comment: "@awoods yes, comments make more sense. I'll also get rid of the copied content and include a link. There's a ..." https://www.pivotaltracker.com/story/show/6061567014:00
Andrew Woods added comment: "@osmandin, I think your notes in https://wiki.duraspace.org/display/FF/Comments+on+Access+Roles+Module woul..." https://www.pivotaltracker.com/story/show/6061567014:01
<ajs6f>Is there a runnig demo of fcrepo4 up somewhere?14:03
<awoods>only if futures6 is up
<ajs6f>what's that url?14:04
<awoods>http://futures6.fcrepo.org/
<ajs6f>Doesn't look like it's up.
<awoods>nope14:05
Anusha said they were having issues
hence the week-long kitchen sink Jenkins failure.
<bljenkins>Project fcrepo-fixity-corrupter build #467: SUCCESS in 1 min 14 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/467/14:06
Yippie, build fixed!14:12
Project fcrepo-fedora3-federation-connector build #245: FIXED in 7 min 42 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fedora3-federation-connector/245/
<pivotal-bot>Andrew Woods edited "Take implicit http-commons dep out of jcr" https://www.pivotaltracker.com/story/show/6113571214:13
Andrew Woods accepted "Take implicit http-commons dep out of jcr" https://www.pivotaltracker.com/story/show/61135712
<bljenkins>Project fcrepo-kitchen-sink build #657: STILL FAILING in 7 min 54 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/657/14:14
<tecoripa>question about cascading updates: are there any internal properties on objects whose update would trigger a cascading update to all descendant nodes?14:15
i.e., cascading updates that fedora would need to do to keep its internal state consistent?
<pivotal-bot>Andrew Woods added "Parameterize JMS port in fcrepo-authz ITs" https://www.pivotaltracker.com/story/show/6115531214:17
Osman Din added comment: "@awoods I've added all stuff as comment to Greg's page. Couldn't find the delete option for the pages I create..." https://www.pivotaltracker.com/story/show/6061567014:18
Andrew Woods added comment: "As far as you have witnessed, @frankasseg, is the "file" config the only configuration that will work with ..." https://www.pivotaltracker.com/story/show/61119020
<awoods>osmandin: In the wiki, do you have a "Tools" menu/drop-down in the top-right?14:19
<pivotal-bot>Frank Asseg added comment: "@awoods : Yes all the other ones fail for me with Java Heap OOMEs at the infinispan level during some byte a..." https://www.pivotaltracker.com/story/show/6111902014:20
<osmandin>awoods: I can see the drop down..but no delete
<awoods>osmandin: let me change that...
osmandin: maybe now, maybe in the next 20min your should see a "delete" option in that menu (it sometimes takes a moment for the permissions to propagate).14:22
<osmandin>awoods: thx14:23
<awoods>osmandin: let me know if you see the "delete"
<escowles>gregjansen: i'm getting this error when i try to use the updated benchtool with fcrepo3: Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity.14:29
<osmandin>awoods: now i'm seeing even less options.. copy/remove disappeared14:37
copy/move
<gregjansen>escowles: interesting, I wonder how that got in there14:38
escowles: there must be some recycling of an HttpEntity?
<escowles>i think the initial fcrepo3 uploads fail b/c of auth, then retry with the user/pass, but the entity isn't repeatable
i'm trying to enable preemptive auth to fix this14:39
<gregjansen>that makes the most sense.. we certainly don't want to repeat the random data gen
<barmintor>that's weird, how did I do that14:43
<osmandin>awoods: I have the delete option now14:45
awoods: thx
afk14:47
<pivotal-bot>Scott Prater added comment: "Test - Setup Profile: Trees14:51
https://wiki.duraspace.org/display/FF/Test+-+Setup+Profile%3A+Trees" https://www.pivotaltracker.com/story/show/60974242
<awoods>gregjansen/escowles/fasseg: There are a couple of PRs on the benchtool. Is it safe to assume the three of you are reviewing/committing those?14:57
<escowles>awoods: yes, i'm working on them right now, and i assume gregjansen and fasseg will review when i'm ready14:58
<fasseg>awoods: will do!
<awoods>awoods: what'a team
<gregjansen>awoods: whenever things are ready.. someone can probably merge mine, if it merges fine14:59
<fasseg>escowles: I already updated the PRNG though...so maybe greg's commit is not necessary, sry about that...
<tecoripa>are we meeting now in google hangouts?
for doc talk?
<awoods>yes
<tecoripa>ok
<escowles>tecoripa: omw
* github-ff joins15:01
[fcrepo4] mikedurbin opened pull request #160: Added support for unversioned fedora nodes. (master...versioning) http://git.io/DP4HYA
* github-ff leaves
<pivotal-bot>Mike Durbin added comment: "Added very basic section to the Versioning page on the wiki: ""
https://wiki.duraspace.org/display/FF/Versionin..." https://www.pivotaltracker.com/story/show/61151750
Mike Durbin finished "Allow versioning to be turned off." https://www.pivotaltracker.com/story/show/6115175015:02
<gregjansen>fasseg: I am rebasing my PR to resolve conflicts with current master
<fasseg>gregjansen: Sorry I didn't see your pull request, I already updated the random data geanration implementation unsing an XOrShiftRNG from uncommons-math. Sorry bout that but I closed the PR
gregjansen: Please check https://github.com/futures/benchtool/blob/master/src/main/java/org/fcrepo/bench/BenchToolInputStream.java first that's in master
<gregjansen>sounds great, sorry about duplication
<fasseg>my bad, I didn't notice the PR and did the update earlier today15:03
<gregjansen>fasseg: uncommonly good
<fasseg>but it's a good sign we chose the same implementaion for the RNG ;)15:04
<gregjansen>fasseg: yes, I think so
* tecoripa1 joins
<awoods>gregjansen: google-hangout?15:06
<barmintor>escowles: is this a thing being worked on right now - javax.jcr.nodetype.ConstraintViolationException: Unable to determine a valid node definition for the node "/objects/benchfc4-143/DS1/45" in workspace "default" of "repo"15:16
<escowles>barmintor: i've got a PR in that fixes this (adds a unique prefix to the generated object names to avoid this)15:17
<barmintor>escowles: okey doke. So I should be running from a branch of benchtool?
<escowles>barmintor: my update is in the branch multi-function -- adds reads/deletes/updates and fixes a fcrepo3 preemptive auth issue15:18
<barmintor>righy-o
<fasseg>escowles: I added 2 comments to the PR at https://github.com/futures/benchtool/pull/3/files15:21
<escowles>fasseg: i'm in the docs meeting right now -- i'll address those in a bit
<fasseg>escowles: kk15:22
<barmintor>escowles: awesome
it looks like there's something else that happens periodically (javax.jcr.InvalidItemStateException: This operation cannot be performed when the session has pending changes)15:23
but at least I'm collecting some data
Wow. "Threads: Live: 73 Peak: 75 Daemon: 32"15:25
* jcoyne joins15:26
<escowles>barmintor: is 75 a lot of threads?
<ajs6f>barmintor: Is it a lot compared to where we were?
* github-ff joins15:29
[fcrepo4] ajs6f force-pushed RdfIterationAgainAndAgain from 36c6051 to 9f5a55a: http://git.io/qHFP1A
fcrepo4/RdfIterationAgainAndAgain 3752994 ajs6f: Changed FedoraNodes.describe() to stream, everything works except HTML ITs
fcrepo4/RdfIterationAgainAndAgain 9f5a55a ajs6f: All tests passing for FedoraNodes.describe() streaming
* github-ff leaves
<pivotal-bot>A. "Mutchkin" Soroka added comment: "https://github.com/futures/fcrepo4/pull/161" https://www.pivotaltracker.com/story/show/5924401415:30
* github-ff joins
[fcrepo4] ajs6f opened pull request #161: Rdf iteration again and again (master...RdfIterationAgainAndAgain) http://git.io/XKITPA
* github-ff leaves
<ajs6f> awoods/cbeer: Got the time for a merge? ^^^
<awoods>ajs6f: on a call
<ajs6f>I CAN'T BELIEVE IT! AWOODS IS ON A CALL!15:31
<barmintor>escowles, ajs6f: I'd have to run numbers again (or check at home), but I think it's substantially more than FCR315:32
<ajs6f>barmintor: Does that mean we win?
<barmintor>ajs6f: we jad already won
<ajs6f>It feels good.
<cbeer>barmintor: iirc, lots of those threads are MODE/ISPN internals.15:33
<barmintor>cbeer: yes
cbeer: and KahaDB, and Hibernate,15:34
<cbeer>Kaha is activemq?
<ajs6f>Yep.15:35
The xactional store under it.
<cbeer>who pulls in hibernate?
<ajs6f>We should probably do something to tune AMQ a bit.
<escowles>so that's what kaha is...
<ajs6f>I think MODE uses Hb.
But I could be remembering wrong.
<barmintor>MODE -> Arjuna -> Hibernate15:36
<ajs6f>Arjuna is apretty name.
I could see naming a daughter Arjuna.
<barmintor>I'm naming my kid Reaper Thread15:37
<ajs6f>Your kid is going to grow up like "A Boy Named Sue".
<barmintor>ok, maybe "Tex Cleanup Service"
<ajs6f>Much better.15:38
<tecoripa1>so Google Drive has frozen my browser. stopping, rejoining the hangout...15:39
* travis-ci joins15:42
[travis-ci] futures/fcrepo4#1233 (RdfIterationAgainAndAgain - 9f5a55a : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/36c60510fe83...9f5a55a46cec
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14272833
* travis-ci leaves
* github-ff joins15:44
[fcrepo4] ajs6f pushed 1 new commit to RdfIterationAgainAndAgain: http://git.io/_qSNmQ
fcrepo4/RdfIterationAgainAndAgain 64cfac8 ajs6f: Improved unit test
* github-ff leaves
<bljenkins>Project fcrepo-fixity-corrupter build #468: SUCCESS in 57 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/468/15:45
Project fcrepo-fedora3-federation-connector build #246: UNSTABLE in 3 min 52 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fedora3-federation-connector/246/15:48
<barmintor>awoods: So, I have the YourKit output for the 1k/1k/15 test. It's ~690M. What to do with it?15:56
<ajs6f>Put it in Fedora!
afk bbs
<awoods>barmintor: one moment15:57
* jonathangee leaves15:58
* travis-ci joins
[travis-ci] futures/fcrepo4#1235 (RdfIterationAgainAndAgain - 64cfac8 : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/9f5a55a46cec...64cfac837e4f
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14273542
* travis-ci leaves
<awoods>barmintor: back16:00
* tecoripa1 leaves16:01
<awoods>barmintor: Do you think anyone will ever open up those yourkit output files? Would it be more useful if the appropriate screenshots were just captured?
* ajs6f leaves
<barmintor>awoods: there's far too much info for screenshots
<bljenkins>Project fcrepo-fixity-corrupter build #469: SUCCESS in 57 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/469/
* ajs6f joins
<barmintor>I'll see if there's some kind of export I can do
<awoods>barmintor: My concern is that a yourkit outpuf file will never be looked at again.16:02
barmintor: or rather, when someone wants to know the status of F4 performance, they will be looking for a graph on a wiki.16:03
<barmintor>That's jmeter's business
yourkit is profiling the internals
<awoods>barmintor: That is not to say that keeping the ~690M output is not a good idea.
* github-ff joins
[fcrepo4] ajs6f pushed 1 new commit to RdfIterationAgainAndAgain: http://git.io/CEcwAg
fcrepo4/RdfIterationAgainAndAgain e5d2c34 ajs6f: Another unit test
* github-ff leaves
<ajs6f>No one who isn't actually dev'ing is going to be interested in the kinds of thing YK tells us.16:04
<awoods>ajs6f: agreed
* jcoyne leaves
<bljenkins>Project fcrepo-kitchen-sink build #658: STILL FAILING in 3 min 11 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/658/
<awoods>And those who are dev'ing, most will want the summary of what was gleaned from the YK output. Beyond this week, I am not sure if anyone will open that YK file.16:05
<bljenkins>Yippie, build fixed!16:06
Project fcrepo-fedora3-federation-connector build #247: FIXED in 5 min 19 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fedora3-federation-connector/247/
<barmintor>The broad lesson so far: All ofthe CPU and memory is used in MODE and ISPN code
<ajs6f>This is not _too_ surprising.
<awoods>barmintor: that is good and bad
good in that it is not our fault, bad in that it will be harder to fix.
* jcoyne joins16:07
<barmintor>though it's possible we could trim some of it down if we did something to cache a node's types in the FedoraTypesUtils functions (it looks like a small savings)
but that's also for small files, I assume we also need a larg(er) file test?
<awoods>barmintor: yes, large files tests16:08
barmintor: Have you seen in significant -time- sinks?
s/in/any/
<tecoripa>awoods: you can change the order of pages on the wiki: by default it's alphabetical, but the second you manually move one of the pages to a different spot in the order, it becomes manual.
awoods: but it looks like this is something only a Space Admin can do? I'm not seeing the controls for it.16:09
awoods: https://confluence.atlassian.com/display/DOC/Changing+the+Sequential+Order+of+Pages
* github-ff joins16:10
[fcrepo4] ajs6f pushed 1 new commit to RdfIterationAgainAndAgain: http://git.io/lL6UAg
fcrepo4/RdfIterationAgainAndAgain 0272bee ajs6f: Better unit test
* github-ff leaves
<awoods>tecoripa: pages sorted16:11
<pivotal-bot>Scott Prater added comment: "I added four CRUD Workflow profiles under "Test - Workflow Profiles for Performance Testing": https://wiki..." https://www.pivotaltracker.com/story/show/60974242
Scott Prater finished "Create use case for common preservation repository profile" https://www.pivotaltracker.com/story/show/60974242
* travis-ci joins16:12
[travis-ci] futures/fcrepo4#1237 (RdfIterationAgainAndAgain - e5d2c34 : ajs6f): The build has errored.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/64cfac837e4f...e5d2c34a07a4
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14274543
* travis-ci leaves
* github-ff joins16:13
[fcrepo4] ajs6f pushed 1 new commit to RdfIterationAgainAndAgain: http://git.io/LYV6VQ
fcrepo4/RdfIterationAgainAndAgain b8f4ec7 ajs6f: Licenses...
* github-ff leaves
<ajs6f>"Space Admin". That's a cool phrase. I'd like to be a "Space Admin".
<tecoripa>I *knew* you were going to jump on that.16:14
I was just waiting.
<ajs6f>I'm already in orbit!
<barmintor>awoods: really, it's all JcrSession.sav and JcrVersionManager.checkin
<ajs6f>In my Space Admin capsule!
<tecoripa>Say hello to Major Tom (Cramer)
<ajs6f>I'll ask him what kind of shirts he wears.
barmintor: isn't JcrSession.sav the committing of a session?16:15
<barmintor>ajs6f: yes
<ajs6f>It's a question: if we're using MODE efficiently, do we want to spend time/effort to make MODE more efficient?16:16
* tecoripa leaves
<barmintor>ajs6f: at some point, our performance is dependent on those libraries. Don't know how much alternative there is.
<ajs6f>barmintor: True dat.16:17
<barmintor>or we re-write over Akubra 2.0
* barmintor ducks
<awoods>barmintor: That is certainly an argument for making Versioning optional: https://www.pivotaltracker.com/story/show/61151750
<pivotal-bot>feature: Allow versioning to be turned off. (finished) / owner: Mike Durbin
* barmintor runs
* barmintor hides
<ajs6f>I'm writing a totally Trippi-based persistence layer.16:18
* gregjansen leaves
* travis-ci joins
[travis-ci] futures/fcrepo4#1239 (RdfIterationAgainAndAgain - 0272bee : ajs6f): The build has errored.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/e5d2c34a07a4...0272beec277b
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14274937
* travis-ci leaves
<ajs6f>If we can publicly beat F3, I guess it becomes a matter of priorities?
<barmintor>"Your persistence layer is Trippi?" "Dude, it's *totally* trippy."
<ajs6f>I've seen some repos that bear that out.
<awoods>can we use JcrSession.save() more efficiently?16:19
<barmintor>awoods: I don't think so
<ajs6f>Like, by not saving things?
<awoods>are we calling it too much?
<ajs6f>No.
We don't call it more than once per xaction or request.
And we have to call it that often.16:20
<awoods>In which case, should we try to dive into Mode to make JCR.save() more efficient?
<bljenkins>Project fcrepo-fixity-corrupter build #470: SUCCESS in 57 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/470/
<ajs6f>That's why I'm calling it a question of priority.
If we need more speed, that's the kind of thing we need to do.
But that's at the cost of functionality we could be building.
So… do we need more speed?16:21
Do we just have to beat F3?
<awoods>ajs6f: maybe there is still an unanswered question of "do we need more speed"... exactly
<ajs6f>Do we have other marks to meet?
<awoods>We have to do two things:
<ajs6f>How fast is fast enough?
<mikeAtUVa>I imagine we can be more efficient in versioning. I've submitted a pull request to turn off the versioning, might be worth a comparison.
<escowles>ajs6f: everything instantaneous everywhere
<ajs6f>DONE!
<awoods>Beat F3 on single node setup
<barmintor>28% of the CPU time was spent coordinating versons
<ajs6f>Ow.
<awoods>Show that by adding more nodes to a cluster we can proportionally improve throughput16:22
<barmintor>so… versioning is much, much slower
<ajs6f>It's okay, mikeAtUva is fixing that.
<mikeAtUVa>We can improve that.... we haven't tuned any of the property version settings so we're likely making lots of redundant copies.
<ajs6f>Makes me wonder how much of Fedora3's vaunted snail-like pace was due to versioning.
Can we version props independently of their nodes?16:23
<bljenkins>Project fcrepo-fedora3-federation-connector build #248: UNSTABLE in 4 min 34 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fedora3-federation-connector/248/
<ajs6f>Maybe we need to develop some flexibility in versioning, so that some portions of the repo can useit while others don't.16:24
<mikeAtUVa>There's rather sophisticated per-property settings, though it all has to be in the node definition upfront and our use of wildcards there leave me at a loss as to how to best configure things by default.
<ajs6f>Hm.
Our type system is utterly organic. Maybe we should revisit it and see what kind of refactoring might benefit it.
(JCR types, not Java types)16:25
<bljenkins>Yippie, build fixed!16:26
Project fcrepo-jms-indexer-pluggable build #253: FIXED in 6 min 54 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-jms-indexer-pluggable/253/
* travis-ci joins16:27
[travis-ci] futures/fcrepo4#1241 (RdfIterationAgainAndAgain - b8f4ec7 : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/0272beec277b...b8f4ec76f702
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14275132
* travis-ci leaves
<ajs6f>awoods: Is the sitemap a good target for streaming? It's not RDF, but it could be huge...16:28
We're using JAXB for it now, I think.16:29
Most of the other RDF endpoints don't present the 'threat" that describe().
<awoods>ajs6f: sounds like a good candidate for streaming.16:30
<ajs6f>They're not likely to explode and spew triples like a bar of soap in the microwave.
We always wanted to use a library for sitemaps anyway.
Should I close:
https://www.pivotaltracker.com/story/show/59244014
<pivotal-bot>feature: Alter fcrepo-http-api to take advantage of new iterator-driven RDF machinery. (started) / owner: A. "Mutchkin" Soroka
<ajs6f>and make new tix for the remaining endpoints?16:31
<awoods>ajs6f: smaller tickets for specific endpoint would be great.16:32
<ajs6f>k
<awoods>s/kk/k/
<pivotal-bot>A. "Mutchkin" Soroka finished "Alter fcrepo-http-api to take advantage of new iterator-driven RDF machinery." https://www.pivotaltracker.com/story/show/59244014
<ajs6f>No, not that much okay. Just a smidgeon.
I don't want to spoil you.
<awoods>k
<pivotal-bot>A. "Mutchkin" Soroka added "Alter sitemap generation to stream results" https://www.pivotaltracker.com/story/show/6116607416:33
A. "Mutchkin" Soroka added "Alter FedoraRepositoryNodeTypes and FedoraRepositoryNamespaces to use iterating RDF generation" https://www.pivotaltracker.com/story/show/6116614216:34
<escowles>fasseg: i've addressed your int/long file size and jena comments -- kept the RNG for choosing objects to read/update/delete in random order16:35
<pivotal-bot>A. "Mutchkin" Soroka edited "Alter FedoraRepositoryNodeTypes, FedoraRepositoryWorkspaces, FedoraIdentifiers and FedoraRepositoryNamespac..." https://www.pivotaltracker.com/story/show/61166142
A. "Mutchkin" Soroka added "Alter FedoraFixity and FedoraDatastreams to use iterating RDF generation" https://www.pivotaltracker.com/story/show/6116632216:36
<ajs6f>Is there currently any way to see what you've done in a txaction? If you "GET" the txaction itself, do you get anything interesting?
I don't see such an endpoint, and it might be useful...16:37
* jcoyne leaves
<mikeAtUVa>Here's a question that reflects my ignorance: Where is the data persisted when we run fcrepo-http-api integration tests and why might it persist between subsequent invocations of "mvn clean install -Dtest=FedoraVersionsIT -pl fcrepo-http-api"
<cbeer>ajs6f: i don't think we ever figured out how to get the changeset out of a session
<ajs6f>cbeer: Right. That's MODE-internal, I guess. JCR doesn't offer that, I don't think.
Oh, well.
<cbeer>mikeAtUVa: that's the behavior i noticed too. awoods swears up and down data isn't being persisted to $TMPDIR, but I still think he's wrong.16:38
<ajs6f>"Just commit it and see what changed!"
<escowles>i'm seeing data in my $TMPDIR still
<ajs6f>mikeAtUVa/cbeer: I use -rf :fcrepo-http-api and haven't seen that problem.
<awoods>data is created in your TMPDIR if you start F4 (jetty:run)
<ajs6f>Oh, I haven't been running from jetty.16:39
<awoods>I will try once again to see if fcrepo-http-api is stashing somewhere.
<bljenkins>Project fcrepo-fixity-corrupter build #471: SUCCESS in 1 min 12 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/471/16:40
<barmintor>sneaky: what looked like ISPN classes eating up CPU are actually MODE classes in an org.infinispan package16:41
thanks for nothing, Java access controls!16:42
<awoods>mikeAtUVa: Have you tried to clear out your /tmp of 1384* and modeshape-binary-store?16:43
<bljenkins>Yippie, build fixed!
Project fcrepo-fedora3-federation-connector build #249: FIXED in 5 min 8 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fedora3-federation-connector/249/
<awoods>mikeAtUVa: and then run your tests again, and again?
<mikeAtUVa>awoods: trying that now...16:44
<ajs6f>awoods: Do you have a sense of how far away we are from beating F3 at single-node?
<awoods>escowles? cbeer? ^^16:45
<escowles>right now, f4 is beating f3 at single-node ingest -- haven't gotten as far as read/update/delete testing yet
<cbeer>awoods: yes, that's how i clear up that kind of failure. except it's not tmp on a mac, it's $TMPDIR
(resolves somewhere in /var/private/...)
<ajs6f>Almost there… stay on target...16:46
<bljenkins>Project fcrepo-kitchen-sink build #659: STILL FAILING in 6 min 8 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/659/
<awoods>cbeer/mikeAtUVa: and you have to delete the $TMPDIR after each build?
<cbeer>awoods: just when persistent data causes a problem.
<escowles>ajs6f: though f4 is faster only with the minimal profile (no query/indexing) -- does that count? i'm not sure what the ramifications are
<cbeer>like tests that have hard-coded identifiers, or when i changed out the CND namespaces, etc16:47
<ajs6f>We may have to parse the results for our audience. {grin}
<escowles>awoods: yes, i don't have to delete $TMPDIR/fcrepo4-data after each build, just when i want to clear repo contents
<awoods>escowles: the integration tests create $TMPDIR/fcrepo4-data when you run them?16:48
escowles: I am not seeing that.
<osmandin>afk16:50
<barmintor>holy crap is ISPN a big project
* osmandin leaves
<ajs6f>out for the day… see y'all soon.16:51
* ajs6f leaves
<mikeAtUVa>awoods: the behavior I've witnessed is: 1) clear the /tmp directory and the test run fine, run one again with "mvn clean install -Dtest=FedoraVersionsIT -pl fcrepo-http-api" and it fails because of residual data.... I clear the data again, and when I run that command ALL the tests are run again and pass (presumably)16:52
<awoods>mikeAtUVa: what exactly are you deleting in the /tmp?16:53
<mikeAtUVa>awoods: a couple directories that start with 1384 a "fcrepo4-data" directory and "hsperfdata" (if i started it with connection options)16:55
<awoods>connection options?
<mikeAtUVa>awoods: if I set flags at runtime to allow a debugger to connect.16:56
awoods: perhaps that screws with the deleting of temp files...16:57
<escowles>awoods: when i do jetty:run, data that i'm loading shows up in fcrepo-webapp/target16:58
<awoods>mikeAtUVa: an interesting possibility
escowles: that actually makes sense, since whether data goes to target or /tmp is determined by the property set by maven : "integration-test=true", which is set with jetty:run17:00
mikeAtUVa: I just ran the following command twice successfully (without deleting anything in between runs): "mvn -Djacoco.skip clean install -Dtest=FedoraVersionsIT -pl fcrepo-http-api"17:01
<mikeAtUVa>awoods: unfortunately the test that fails isn't yet in master, but I'd believe my setup is the problem.17:02
awoods: does it write stuff to /tmp when it runs?
<awoods>mikeAtUVa: note: cbeer is also having some sort of similar issues, although he has not surfaced the exact details or nature of the issue.17:03
<mikeAtUVa>awoods: also, does it just run that one test, or does it run everything in that module
<awoods>mikeAtUVa: yes, the backup/restore integration tests create /tmp/1384* dirs17:04
mikeAtUVa: and /tmp/fcrepo4-data and /tmp/hsperfdata_* are created
mikeAtUVa: it runs everything, the command should be -Dit.test=17:05
<mikeAtUVa>awoods: it looks like my maven syntax was indeed the problem... running "mvn clean install -Dit.test=FedoraVersionsIT -pl fcrepo-http-api" works over and over again... when you run it as -DTest it must not set some property that causes the data to be stored in the target dir.17:07
<awoods>mikeAtUVa: that is good to hear. I wonder if anyone else has issues. escowles? cbeer?17:08
All: Regarding testing and performance, it is interesting to highlight the fact that fasseg is completely unable to ingest any files that are larger than 1-GB unless he uses the repository.json:storage:binaryStorage:type="file" configuration: https://github.com/futures/fcrepo4/blob/master/fcrepo-http-commons/src/main/resources/config/single-file/repository.json#L2217:12
escowles/cbeer/barmintor/gregjansen: have you seen this as well ^^^17:13
<escowles>awoods: i haven't tried files that large -- i'll try tomorrow am17:14
<barmintor>awoods: I'l definitely have a look
tho one expects that there is a stacktrace somewhere? fasseg?17:15
<awoods>barmintor: This is what I have seen from fasseg: https://www.pivotaltracker.com/story/show/6111902017:16
<pivotal-bot>feature: Test ingestion of large files on a single node (started) / owner: Frank Asseg
<barmintor>useful, thank you!
<escowles>awoods: i can confirm -- i can ingest a 4GB file (2x Xmx) with file storage, but get OutOfMemoryError if I switch to leveldb17:30
<barmintor>this is an un-chunked config, I assume?17:31
<awoods>escowles: thanks. barmintor: seems like it must be.17:32
barmintor: do we have an example of a chunked config in the codebase?
<barmintor>awoods: the clustered conf is chunked17:33
but leveldb doesn't appear to be17:34
<cbeer>barmintor: what do you mean by chunked and unchunked?17:39
<barmintor>cbeer: whether ISPN is breaking things up into chunks17:40
<cbeer>that's configurable now?
<barmintor>I assume the fact that the clustered config let's you specify a chunk size meant it was17:41
<cbeer>pretty sure there's just a default size17:42
<barmintor>Hmm...
<cbeer>escowles: can you try with any of the other infinispan-backed configs, and do you have a stacktrace?17:44
barmintor: after https://github.com/futures/fcrepo4/commit/d8965299d4cad48b2b5e36d9813b1e9cd876debf, what's the point of fcrepo-jcr?17:47
i thought we originally created it just to hold those configs..
<barmintor>cbeer: so that clients can access the constants without pulling in the fcrepo-kernel dependency
<cbeer>do we have such clients? i guess we're envisioning modeshape sequencers or something?17:48
<barmintor>or a java-based REST client
<cbeer>ah, good point. ok. i just thought that one class looked pretty lonely
<barmintor>me, too. I was going to axe ituntil I realized that (for example) our IT suites for the REST api would need access to fcrepo-kernel17:49
<pivotal-bot>Chris Beer finished "Run fcrepo3 benchmarks on Stanford server" https://www.pivotaltracker.com/story/show/6105760817:51
Chris Beer added "Run simple case fcrepo4 benchmarks" https://www.pivotaltracker.com/story/show/61172394
<escowles>cbeer: here's a stacktrace i see when trying to upload 4GB file using leveldb storage17:56
<barmintor>(please be a gist, please be a gist)17:57
<cbeer>barmintor: 200 lines, pasted into IRC!
<barmintor>:)
<cbeer>or, java's shortest stack trace ever.18:00
* barmintor Googles "esme stacktrace a minute ago"18:01
didn't work.
<cbeer>https://gist.github.com/escowles/757267818:02
<barmintor>But I turned this up: http://esme.apache.org/
<cbeer>escowles has his own top-level apache project?
<barmintor>I always knew he had a governance model18:03
<cbeer>too bad his employer can't sign a CLA.
<barmintor>heh
<cbeer>escowles: is it possible the maven jetty runner can't handle files that large18:04
?
<barmintor>it looks like something along those lines
<cbeer>0.2597 objects/s (objects per second)18:05
that sets a nice low bar for fcrepo4 performance to beat.18:06
<escowles>i didn't know i'd been promoted to a top-level apache project...18:09
<awoods>escowles: you are always the last to find out18:10
<escowles>sorry about forgetting to paste in the gist url -- luckily cbeer is psychic
<pivotal-bot>Eric James added "configuration documentation " https://www.pivotaltracker.com/story/show/61173578
Eric James started "configuration documentation " https://www.pivotaltracker.com/story/show/61173578
<escowles>i'll have to try running fcrepo4.war in tomcat to see if that fixes it...18:11
<barmintor>escowles: It's worth noting that you have a very different error reported than fasseg, so we need a stacktrace over there as well18:12
<escowles>barmintor: i thought i got an OutOfMemoryErrror the first time, but that's not what i saw the second and third times
* ermadmix leaves18:13
<escowles>barmintor/cbeer: ok, here's a different gist with an OutOfMemoryError when uploading under tomcat: https://gist.github.com/escowles/757301918:19
interesting fact: there was a file in $CATALINA_HOME/temp/ms-ispn-binstore5321658938415300322hashing that grew to 4GB, then vanished after the stacktrace18:20
so i think the file is getting streamed and written to disk just fine, but something else is trying to load it into mem
<pivotal-bot>Frank Asseg added comment: "Ingest of a single 300gb file using single-file configuration was successful on my local machine: ""18:24
Throughput..." https://www.pivotaltracker.com/story/show/61119020
Frank Asseg added comment: "I will create follow up tickets.." https://www.pivotaltracker.com/story/show/61119020
* ksclarke leaves18:26
<cbeer>escowles: this is using https://github.com/futures/fcrepo4/blob/master/fcrepo-kernel/src/main/resources/config/infinispan/leveldb/infinispan.xml?18:29
barmintor: did you ever figure out (or, even need to figure out) how that TMPDIR magic works?18:32
<barmintor>cbeer: didn't need to18:33
* awoods leaves18:39
* awoods joins18:40
<escowles>cbeer: i was using the WAR file in fcrepo-webapp/target/fcrepo-webapp-4.0.0-alpha-3-SNAPSHOT.war with no special properties set18:48
so i assume it was using config/single/repository.json and config/infinispan/leveldb/infinispan.xml18:49
<cbeer>ok
<barmintor>criminy, trying to look at the MODE/ISPN interaction18:58
going home for the night, adios
* barmintor leaves
<cbeer>mikeAtUVa: versioning seems like a huge performance hit, huh?19:02
<pivotal-bot>Andrew Woods added comment: "@frankasseg, when you say "the file was available" were you able to download it via the REST-API, or you ju..." https://www.pivotaltracker.com/story/show/6111902019:26
* fasseg leaves19:44
<pivotal-bot>Andrew Woods added "Change camel-case of fcr:accessRoles endpoint" https://www.pivotaltracker.com/story/show/6117968020:13
Andrew Woods edited "Change camel-case of fcr:accessRoles endpoint" https://www.pivotaltracker.com/story/show/61179680
* ksclarke joins20:14
<pivotal-bot>Andrew Woods accepted "Test Basic AuthZ" https://www.pivotaltracker.com/story/show/6061567020:28
Andrew Woods added comment: "@cbeer, could you create a small table at the top of this page that summarizes and highlights the F3 results?" https://www.pivotaltracker.com/story/show/6105760820:30
Andrew Woods edited "Run fcrepo3 benchmarks on Stanford server" https://www.pivotaltracker.com/story/show/61057608
Andrew Woods accepted "Create use case for common preservation repository profile" https://www.pivotaltracker.com/story/show/6097424220:48
* github-ff joins21:08
[fcrepo4] awoods pushed 1 new commit to master: http://git.io/rPzaXQ
fcrepo4/master 6444f24 ajs6f: Change FedoraNodes.describe() to stream...
* github-ff leaves
* github-ff joins21:09
[fcrepo4] awoods closed pull request #161: Rdf iteration again and again (master...RdfIterationAgainAndAgain) http://git.io/XKITPA
* github-ff leaves
<pivotal-bot>Andrew Woods delivered "Alter fcrepo-http-api to take advantage of new iterator-driven RDF machinery." https://www.pivotaltracker.com/story/show/59244014
Andrew Woods added comment: "Resolved with: https://github.com/futures/fcrepo4/commit/6444f2417b975ad42ee33a4d00192d1bb27be65c" https://www.pivotaltracker.com/story/show/59244014
* fcrepo-bot joins21:16
<bljenkins>Project fcrepo-fixity-corrupter build #472: SUCCESS in 5 min 40 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/472/21:53
* github-ff joins21:54
[fcrepo4] awoods pushed 2 new commits to master: http://git.io/77F1iA
fcrepo4/master f342fe4 Michael Durbin: Added support for unversioned fedora nodes.
fcrepo4/master ed4fa23 Andrew Woods: Merge pull request #160 from mikedurbin/versioning...
* github-ff leaves
<pivotal-bot>Andrew Woods delivered "Allow versioning to be turned off." https://www.pivotaltracker.com/story/show/61151750
Andrew Woods accepted "Alter fcrepo-http-api to take advantage of new iterator-driven RDF machinery." https://www.pivotaltracker.com/story/show/5924401421:55
<bljenkins>Project fcrepo-fedora3-federation-connector build #250: UNSTABLE in 18 min: http://ci.fcrepo.org/jenkins/job/fcrepo-fedora3-federation-connector/250/22:06
Project fcrepo-jms-indexer-pluggable build #255: UNSTABLE in 19 min: http://ci.fcrepo.org/jenkins/job/fcrepo-jms-indexer-pluggable/255/22:07
* travis-ci joins22:08
[travis-ci] futures/fcrepo4#1244 (master - ed4fa23 : Andrew Woods): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/6444f2417b97...ed4fa23450c1
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/14289358
* travis-ci leaves
* fcrepo-bot leaves22:19
<bljenkins>Project fcrepo-fixity-corrupter build #473: SUCCESS in 5 min 52 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/473/23:09
Project fcrepo4 build #1423: UNSTABLE in 1 hr 1 min: http://ci.fcrepo.org/jenkins/job/fcrepo4/1423/00:05
md5wz: Added support for unversioned fedora nodes.

Generated by Sualtam