Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* ksclarke leaves01:46
<pivotal-bot>Esme Cowles added comment: "I've successfully tested this manually with 1TB files (both in repo storage and a federated filesystem)." https://www.pivotaltracker.com/story/show/7123629206:14
Esme Cowles finished "Largefile byte offset read" https://www.pivotaltracker.com/story/show/7123629206:15
* dwilcox joins07:11
<pivotal-bot>Andrew Woods accepted "Hierarchy: 404 on object PUT" https://www.pivotaltracker.com/story/show/7098476808:09
Andrew Woods accepted "Hierarchy: Datastreams created at a level too high" https://www.pivotaltracker.com/story/show/70984934
Andrew Woods accepted "Refactor CRUD in FedoraNodes and FedoraContent for transparent auto-hierarchy support." https://www.pivotaltracker.com/story/show/71230608
Andrew Woods accepted "Hierarchy: Location response URL on object creation is unexpected" https://www.pivotaltracker.com/story/show/70984436
Andrew Woods added comment: "@escowles, please add a note to that effect here: ""08:12
https://wiki.duraspace.org/display/FF/Large+File+Ingest+a..." https://www.pivotaltracker.com/story/show/71236292
Andrew Woods rejected "Largefile byte offset read" https://www.pivotaltracker.com/story/show/7123629209:06
* mikeAtUv joins09:07
<pivotal-bot>Andrew Woods added "tmp" https://www.pivotaltracker.com/story/show/7157227209:17
Andrew Woods edited "tmp" https://www.pivotaltracker.com/story/show/71572272
Andrew Woods edited "Improve performance of range retrieval" https://www.pivotaltracker.com/story/show/7142977009:18
Andrew Woods edited "Support external datastreams" https://www.pivotaltracker.com/story/show/71324222
Andrew Woods edited "Turn on checkstyle rules for tests for jms indexer project" https://www.pivotaltracker.com/story/show/7132380409:19
Andrew Woods edited "Turn on checkstyle rules for tests for jms indexer project" https://www.pivotaltracker.com/story/show/71323804
Andrew Woods edited "Turn on checkstyle rules for tests for jms indexer project" https://www.pivotaltracker.com/story/show/71323804
Andrew Woods edited "Internal SPARQL query should support specifying the subject URI instead of a variable" https://www.pivotaltracker.com/story/show/71322140
Andrew Woods edited "Add references to the HTML rendering of an object" https://www.pivotaltracker.com/story/show/7128004009:20
Andrew Woods edited "Add references to the HTML rendering of an object" https://www.pivotaltracker.com/story/show/71280040
Andrew Woods edited "Create new version API should respond with Location header" https://www.pivotaltracker.com/story/show/7103945609:22
Andrew Woods edited "Create new version API should respond with Location header" https://www.pivotaltracker.com/story/show/71039456
Andrew Woods edited "Missing properties on federation MOVE" https://www.pivotaltracker.com/story/show/70990730
Andrew Woods edited "Update REST API docs with new POST/PUT behavior for creating objects" https://www.pivotaltracker.com/story/show/7096562009:23
Andrew Woods edited "Document ContainerRolesPrincipalProvider and HeaderPrincipalProvider" https://www.pivotaltracker.com/story/show/70710348
Andrew Woods edited "Document Id Translator Pattern" https://www.pivotaltracker.com/story/show/69451630
Andrew Woods edited "Version history documentation" https://www.pivotaltracker.com/story/show/6974410409:24
Andrew Woods edited "Hide/disable interface add object/ds option if FilesystemConnector is read-only" https://www.pivotaltracker.com/story/show/65899330
Andrew Woods deleted "Reduce "One-click-run" Logging" https://www.pivotaltracker.com/story/show/6349324609:27
Andrew Woods deleted "Remove fcrepo-jcr" https://www.pivotaltracker.com/story/show/6868022209:28
Andrew Woods added this label to 7 stories https://www.pivotaltracker.com/story/show/6645409669574878705103767069591670772010706717626662771209:30
Andrew Woods deleted "tmp" https://www.pivotaltracker.com/story/show/7157227209:31
* ksclarke joins09:35
<pivotal-bot>Andrew Woods added comment: "We should get verification of these config files from at least two other F4 developers who use eclipse.09:37
Sen..." https://www.pivotaltracker.com/story/show/71262298
Andrew Woods edited "Update Eclipse/IntelliJ formatting settings files" https://www.pivotaltracker.com/story/show/71262298
* tecoripa joins09:50
<pivotal-bot>Kevin Clarke added comment: "@awoods Yes, I will continue to move this forward this week" https://www.pivotaltracker.com/story/show/7126229809:54
Esme Cowles added comment: "I've updated the wiki with a note about range retrieval support and testing capabilities: ""09:56
https://wiki.duras..." https://www.pivotaltracker.com/story/show/71236292
Esme Cowles finished "Largefile byte offset read" https://www.pivotaltracker.com/story/show/71236292
* escowles joins09:57
<pivotal-bot>Andrew Woods added comment: "https://wiki.duraspace.org/pages/diffpagesbyversion.action?pageId=39028652&selectedPageVersions=31&selected..." https://www.pivotaltracker.com/story/show/71236292
Andrew Woods accepted "Largefile byte offset read" https://www.pivotaltracker.com/story/show/71236292
Kevin Clarke added comment: "Updated to use 118 line length to work around an eclipse bug: https://bugs.eclipse.org/bugs..." https://www.pivotaltracker.com/story/show/7126229809:58
<awoods>ksclarke: Are you planning on having time for this ticket: https://www.pivotaltracker.com/story/show/7064865810:15
<pivotal-bot>feature: How many objects can F4 hold? (unstarted) / owner: Kevin Clarke
<ksclarke>awoods: I am... the main thing to it is writing the expanding structure... I'll use my existing script once that's added in and it can run happily in the background on ec210:16
<awoods>ksclarke/escowles: what did we decide on for the optimal jcr structure: 128/128/128 or 256/256/256?10:19
* scossu joins
<escowles>awoods/ksclarke: i was thinking 256/256/256 (17m files) or even 512/512/512 (134m files)10:20
once the auto-hierarchy is in place, we could just create objects at the top level, and keep creating them until it slowed down10:21
<ksclarke>awoods, 128/128/128 was what was discussed, but remind me what that exactly means in terms of objects and hierarchy on each level...
<awoods>escowles: re:auto-hierarchy, yes, but that implementation is still a ways off.10:22
escowles: however, we could simply leverage the existing PidMinter to do that work for us in the meantime.
* longshou joins10:26
<awoods>ksclarke: 256/256/256 would map to a jcr hierarchy that has a pairtree like structure with two hex characters (256 possibilities) at each of three levels.
<escowles>awoods/ksclarke: then lets do 256/256/256 for both the filesystem and the repo to have a good apple-to-apple comparison10:27
<ksclarke>okay, so it would be a set scheme rather than one that continues to expand down the hierarchy
<awoods>ksclarke/escowles: yes
<ksclarke>ok, re: 256/256/256
<awoods>256 it is10:28
<escowles>also, i've been using 10KB datastreams -- is that too small?
<ksclarke>for the larger runs I've been using 1MB
there was more of a distinct effect vs. when I just used 1KB (benchtool defaults)10:29
* escowles has enough disk space for that (~160GB)
<awoods>on a call10:30
escowles/ksclarke: There is probably value in having small datastreams to focus the test on simply "how many objects can F4 support".10:34
escowles/ksclarke: As a follow-on, it would be good to increase the datastream size, of course.10:35
<ksclarke>awoods: so suggestion for size? 10KB?
<awoods>escowles/ksclarke: 10KB sounds like a reasonable initial pass.
<escowles>awoods: okay, i'll stick with 10KB, and we can try 1MB or larger if we have time for a second pass10:36
<pivotal-bot>Kevin Clarke added comment: "10KB datastreams; a jcr hierarchy that has a pairtree like structure with two hex characters (256 possibili..." https://www.pivotaltracker.com/story/show/7064865810:37
<escowles>ksclarke: reading your comment, i'm not sure we're talking about the same structure: i was thinking 256obj/256obj/256ds -- but your comment sounds like 256obj/256obj/256obj/1(?)ds10:47
<ksclarke>I was thinking one ds per obj10:49
<escowles>ok, i'll do 256obj/256obj/256obj/1ds (10KB)
<pivotal-bot>Esme Cowles added comment: "Coordinating with #70648658: 3-level object hierarchy (256 objects at each level), with the leaf objects eac..." https://www.pivotaltracker.com/story/show/7072457210:53
* scossu1 joins10:56
* scossu leaves
* longshou leaves10:57
<ksclarke>escowles are you writing this in bash or java?11:07
* dwilcox leaves11:25
<pivotal-bot>Andrew Woods edited "/jcr:content is accessible in some cases as a resource path." https://www.pivotaltracker.com/story/show/7141204611:32
Andrew Woods added comment: "@longshous, could you please create some simple graphs showing the results of your tests?11:38
A new page, simil..." https://www.pivotaltracker.com/story/show/7072462211:39
Andrew Woods rejected "Load test CRUD operations with concurrent users" https://www.pivotaltracker.com/story/show/70724622
Andrew Woods edited "Document Id Translator Pattern" https://www.pivotaltracker.com/story/show/69451630
Andrew Woods added comment: "This task can be simplified, since the filesystem connector will always be read-only (for now)." https://www.pivotaltracker.com/story/show/6589933011:41
Andrew Woods edited "Federation: How many files can be managed?" https://www.pivotaltracker.com/story/show/7072457211:43
Andrew Woods edited "Test repo fcr:export at scale" https://www.pivotaltracker.com/story/show/70648736
Andrew Woods edited "Turn on checkstyle rules for tests for jms indexer project" https://www.pivotaltracker.com/story/show/7132380411:44
Andrew Woods edited "Improve performance of range retrieval" https://www.pivotaltracker.com/story/show/7142977011:45
Andrew Woods edited "Support external datastreams" https://www.pivotaltracker.com/story/show/7132422211:46
Andrew Woods edited "Internal SPARQL query should support specifying the subject URI instead of a variable" https://www.pivotaltracker.com/story/show/7132214011:47
* longshou joins11:48
* edInCo joins11:52
* dwilcox joins11:56
<escowles>ksclarke: in bash -- mostly copying files using bash and then checking the status with curl12:00
<ksclarke>escowles: I'd be interested in seeing the part where you create the 256/256/256 structure... perhaps there will be something I can reuse from it12:17
<pivotal-bot>Kevin Clarke estimated "Turn on checkstyle rules for tests for jms indexer project" as 1 point https://www.pivotaltracker.com/story/show/7132380412:18
Esme Cowles added comment: "I agree it would be nice to be able to write the properties files in a separate directory (e.g., what if the..." https://www.pivotaltracker.com/story/show/7123457613:06
Esme Cowles started "Federation: How many files can be managed?" https://www.pivotaltracker.com/story/show/70724572
Andrew Woods started "How many objects can F4 hold?" https://www.pivotaltracker.com/story/show/70648658
Andrew Woods edited "How many objects can F4 hold?" https://www.pivotaltracker.com/story/show/70648658
Andrew Woods added comment: "This issue may fall out with @longshous' hierarchy work." https://www.pivotaltracker.com/story/show/7141204613:17
Esme Cowles added comment: "Ideally, the new version URI should be in the Location header and the body too." https://www.pivotaltracker.com/story/show/7103945613:19
Andrew Woods edited "Document ContainerRolesPrincipalProvider and HeaderPrincipalProvider" https://www.pivotaltracker.com/story/show/7071034813:21
Andrew Woods edited "Test repo fcr:export at scale" https://www.pivotaltracker.com/story/show/7064873613:25
Esme Cowles estimated "Improve performance of range retrieval" as 2 points https://www.pivotaltracker.com/story/show/7142977013:33
Esme Cowles estimated "Create new version API should respond with Location header" as 1 point https://www.pivotaltracker.com/story/show/7103945613:34
Esme Cowles estimated "Add references to the HTML rendering of an object" as 1 point https://www.pivotaltracker.com/story/show/71280040
Esme Cowles estimated "Revisit Holiday-Release tests" as 5 points https://www.pivotaltracker.com/story/show/70724604
Chris Beer added "Concurrent REST requests that mint new mixins (e.g. PATCH with rdf:types) can cause RepositoryExceptions" https://www.pivotaltracker.com/story/show/7159876013:49
Chris Beer edited "Concurrent REST requests that mint new mixins (e.g. PATCH with rdf:types) can cause RepositoryExceptions" https://www.pivotaltracker.com/story/show/71598760
Longshou Situ added comment: "@awoods It looks like we should explain the abnormal longer response time when the first thread was runnin..." https://www.pivotaltracker.com/story/show/7072462214:06
Longshou Situ started "Load test CRUD operations with concurrent users" https://www.pivotaltracker.com/story/show/70724622
Andrew Woods added comment: "I can better comment once I see the graphs." https://www.pivotaltracker.com/story/show/7072462214:11
Esme Cowles estimated "Support external datastreams" as 3 points https://www.pivotaltracker.com/story/show/7132422214:27
Esme Cowles started "Version history documentation" https://www.pivotaltracker.com/story/show/6974410414:29
* scossu1 leaves14:47
<pivotal-bot>Esme Cowles added comment: "The Viewing Old Versions section of the versioning feature page already has a description of the version his..." https://www.pivotaltracker.com/story/show/6974410414:49
Esme Cowles started "Update REST API docs with new POST/PUT behavior for creating objects" https://www.pivotaltracker.com/story/show/70965620
Andrew Woods added "Include external JavaScript dependencies" https://www.pivotaltracker.com/story/show/7160733414:59
Andrew Woods edited "Include external JavaScript dependencies" https://www.pivotaltracker.com/story/show/7160733415:00
* mikeAtUva leaves15:01
<pivotal-bot>Andrew Woods added comment: "See attached for example (duracloud-pom.xml)" https://www.pivotaltracker.com/story/show/7160733415:06
* scossu joins15:17
<pivotal-bot>Andrew Woods added comment: "Looks sufficient." https://www.pivotaltracker.com/story/show/6974410415:19
Andrew Woods accepted "Version history documentation" https://www.pivotaltracker.com/story/show/69744104
Benjamin Pennell added comment: "The Current version is here: ""15:23
https://github.com/bbpennel/fcrepo-extract-mods-properties-sequencer
The..." https://www.pivotaltracker.com/story/show/71400142
* scossu leaves15:25
* mikeAtUVa joins15:27
* scossu joins15:39
<pivotal-bot>Andrew Woods added "getChildren() on FedoraResource" https://www.pivotaltracker.com/story/show/7161129815:40
Andrew Woods edited "getChildren() on FedoraResource" https://www.pivotaltracker.com/story/show/71611298
Andrew Woods edited "Document hierarchy/feature problem areas" https://www.pivotaltracker.com/story/show/7161150815:42
Andrew Woods added "Document hierarchy/feature problem areas" https://www.pivotaltracker.com/story/show/71611508
* dwilcox leaves15:51
* benpennell leaves15:54
* benpennell joins
* longshou leaves
* mikeAtUVa leaves16:01
<awoods>afk16:02
* dwilcox joins16:16
* dwilcox leaves16:17
* dwilcox joins16:22
* dwilcox leaves16:23
* scossu leaves17:03
* dwilcox joins17:13
* dwilcox leaves17:38
* tecoripa leaves18:01
* github-ff joins18:40
[fcrepo4] escowles created move-to-existing (+1 new commit): http://git.io/A9IO1g
fcrepo4/move-to-existing b7ebc7a Esmé Cowles: Adding check for existing destination of a MOVE, and adding IT to check it
* github-ff leaves
* github-ff joins18:42
[fcrepo4] escowles opened pull request #368: Adding check for existing destination of a MOVE, and adding IT for it (master...move-to-existing) http://git.io/lQEWRw
* github-ff leaves
* travis-ci joins18:54
[travis-ci] fcrepo4/fcrepo4#1997 (move-to-existing - b7ebc7a : Esmé Cowles): The build passed.
[travis-ci] Change view : https://github.com/fcrepo4/fcrepo4/commit/b7ebc7a766ef
[travis-ci] Build details : http://travis-ci.org/fcrepo4/fcrepo4/builds/25558011
* travis-ci leaves
* github-ff joins19:05
[fcrepo4] escowles pushed 1 new commit to move-to-existing: http://git.io/1qyejg
fcrepo4/move-to-existing ca593fe Esmé Cowles: Also fixing COPY method
* github-ff leaves
<pivotal-bot>Esme Cowles added comment: "I have updated the wiki to combine the multiple POST entries, and correct the PUT/POST logic and new header/..." https://www.pivotaltracker.com/story/show/70965620
Esme Cowles finished "Update REST API docs with new POST/PUT behavior for creating objects" https://www.pivotaltracker.com/story/show/7096562019:06
* ksclarke leaves19:11
* travis-ci joins19:20
[travis-ci] fcrepo4/fcrepo4#1999 (move-to-existing - ca593fe : Esmé Cowles): The build passed.
[travis-ci] Change view : https://github.com/fcrepo4/fcrepo4/compare/b7ebc7a766ef...ca593fef1e54
[travis-ci] Build details : http://travis-ci.org/fcrepo4/fcrepo4/builds/25559438
* travis-ci leaves
* edInCo leaves19:39
* dwilcox joins20:10
<pivotal-bot>Longshou Situ added comment: "PR https://github.com/fcrepo4/fcrepo4/pull/366 updated, with graph in wiki https://wiki.duraspace.org/disp..." https://www.pivotaltracker.com/story/show/7072462220:19
Longshou Situ added comment: "Attached crudPerformance.txt, the test results that used to draw the graphs for CRUD on ob..." https://www.pivotaltracker.com/story/show/7072462220:23
Longshou Situ finished "Load test CRUD operations with concurrent users" https://www.pivotaltracker.com/story/show/70724622
* dwilcox leaves20:35
<pivotal-bot>Chris Beer added "Concurrent PUT requests that create hierarchy can result in unexpected results" https://www.pivotaltracker.com/story/show/7163222420:43
Andrew Woods added comment: "Do you see another way around this issue other than disallowing SNS?" https://www.pivotaltracker.com/story/show/7163222420:57
* ksclarke joins21:43
awoods is there a page I should put the iiif stuff?21:44
<awoods>ksclarke: let me look
ksclarke: either as content of this page, or as a new child page. Your choice.21:45
https://wiki.duraspace.org/display/FF/Design+-+Image+Serving
<ksclarke>ok, thanks
<pivotal-bot>Kevin Clarke added comment: "First pass at IIIF / Fedora 4 integration / support --> https://wiki.duraspace.org/pages/viewpage.action?pa..." https://www.pivotaltracker.com/story/show/7099170422:45
Kevin Clarke estimated "Strawman design of F4 and IIIF" as 1 point https://www.pivotaltracker.com/story/show/7099170422:46
Kevin Clarke started "Strawman design of F4 and IIIF" https://www.pivotaltracker.com/story/show/70991704
* github-ff joins23:00
[fcrepo4] awoods pushed 1 new commit to master: http://git.io/hjNBTg
fcrepo4/master b53b4dd lsitu: Add integration test for testing CRUD concurrent performance with up...
* github-ff leaves
* travis-ci joins23:12
[travis-ci] fcrepo4/fcrepo4#2002 (master - b53b4dd : lsitu): The build passed.
[travis-ci] Change view : https://github.com/fcrepo4/fcrepo4/compare/ef798d9951f8...b53b4dd80319
[travis-ci] Build details : http://travis-ci.org/fcrepo4/fcrepo4/builds/25569116
* travis-ci leaves
<pivotal-bot>Andrew Woods added comment: "@longshous, you may be interested in this: https://blogs.oracle.com/CoreJavaTechTips/entry/using_enhanced_f..." https://www.pivotaltracker.com/story/show/7072462223:19
Andrew Woods added comment: "Resolved with: https://github.com/fcrepo4/fcrepo4/commit/b53b4dd8031991c76d1949f2a3b017171b320c1f" https://www.pivotaltracker.com/story/show/70724622

Generated by Sualtam