Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* kaarefc joins04:25
* fasseg joins05:07
* kaarefc leaves05:11
* kaarefc joins05:20
* fasseg leaves07:52
* jonathangee leaves08:20
* jonathangee joins08:22
* nbanks joins08:41
* fasseg joins08:45
nigel: regarding https://www.pivotaltracker.com/story/show/5522724009:00
<pivotal-bot____>bug: Change web interface to upload Data-streams via POST instead of GET (unstarted) / owner:
<fasseg>can you tell me where in the web interface I can reproduce a GET datastream upload, IM not able to find such an endpoin tin the fcrepo-http-api either...09:01
and "Create new datastream" is a POST> request for me...09:03
nbanks: ^^09:04
<pivotal-bot____>Frank Asseg added comment: "Can you describe where" https://www.pivotaltracker.com/story/show/55227240
* jay joins
* kaarefc leaves09:06
<nbanks>fasseg: I can't seem to access that link? I'm getting "You are not allowed to view the requested page."09:18
<fasseg>it's a bug ticket which was requested by you: "Change web interface to upload Data-streams via POST instead of GET"09:19
description is: "I tried uploading a 2GB file as a datastream through the web interface. It failed, and upon closer inspection the Network tab in Chrome stated the attempted request was a GET rather than a POST."
<nbanks>The web page generated by velocity. I'll take a screen shot one sec.09:21
<fasseg>kk thx
<nbanks>ah problems running the webapp, I may be a min09:22
<pivotal-bot____>Nigel Banks added comment: "Attaching screen-shot of where I uploaded the file." https://www.pivotaltracker.com/story/show/5522724009:27
<nbanks>fasseg: Should be on the ticket now.
<fasseg>but that's a POST for me...09:28
no GET in there...
hmm maybe with a different id..? i only tried auto generated
<nbanks>Try uploading a very large file.09:29
<fasseg>with ID it's a PUT as I would expect09:30
hmm ill try...
<nbanks>In chrome I see blob:http%3A//localhost%3A8080/49356e24-d0a3-4ee2-908d-96d06b3a804509:31
Says it's a GET request
* kaarefc joins09:32
<fasseg>hmm yes chrome tells me it's a GET, firebug tells me it's a POST...Ill take a look at the javascript09:38
there seems to be javascript code in there, that sets window.location in the chrome browser after pressing the submit button...09:41
but the add datastream request itself seems to be a POST09:42
<nbanks>The POST fails in chrome, so I wonder if we could safely remove that window.location statment.09:45
<fasseg>hmm but we can not return a redirect from the rest api...09:49
so I guess we need to do some redirection in the web front end
<nbanks>Why can't we do the redirect server-side?09:50
<fasseg>because we return a 201/200 response and not a 30x09:51
<nbanks>Ya the rest API returns that value, but doesn't velocity wrap the REST api? Hmm I guess not since they share the same URL's.09:53
Couldn't the client-side redirect be done in the callback for the POST request? Rather than just after the submit?09:54
<fasseg>right that's what I thought, but it seems there is code in there that should already do that at crepo4/fcrepo-http-api/src/main/resources/views/common.js:7209:56
so im a bit confused...
err sry in line 3109:57
ill have to debug that...09:58
<nbanks>http://www.w3schools.com/ajax/ajax_xmlhttprequest_onreadystatechange.asp Seems that readyState should only be "4" when the server replies that request was successful or failed.09:59
* kaarefc leaves10:00
* kaarefc joins10:01
<fasseg>yes and when debugging this seems to work right, but I guess chrome captures a script timeout after X seconds...10:04
oh and btw: http://www.w3fools.com/ ;)
<nbanks>lol10:05
* ksclarke joins10:06
<nbanks>Have you been able to successfully upload a large file with firefox? I've just tested but it just seems to hang without any feedback.10:08
<fasseg>I thought so yes..10:09
lemme try again
<nbanks>I tried a 2GB file
<fasseg>nah 2gb fails for me also it seems...10:14
it's possible to do larger file uploads in modern browsers (not IE9 though) according to this: http://stackoverflow.com/questions/4856917/jquery-upload-progress-and-ajax-file-upload/4943774#494377410:17
* kaarefc leaves10:18
* nbanks leaves10:20
<pivotal-bot____>Frank Asseg added comment: "The GET request seems to be JavaScript redirection after an XmlHttpRequest..So I changed the titel of this t..." https://www.pivotaltracker.com/story/show/5522724010:37
Frank Asseg added comment: "Uploading large files via XMLHttpRequest() seems to be a problem.
One solution could be to use Blob.slice(..." https://www.pivotaltracker.com/story/show/55227240
Frank Asseg edited "Fix upload via web interface for files > ~1GB" https://www.pivotaltracker.com/story/show/55227240
* nbanks joins10:38
<fasseg>nbanks: I updated a ticket a bit to reflect our investigations...10:41
I guess we could use a plain HTML form for uploading large files though, right?
<nbanks>I don't see why we couldn't, although I'm not sure why we can't use ajax for the same purpose either.10:44
* jonathangee leaves
<fasseg>it seems the whole file is put into memory before POSTing in XHR which fails when the file is getting to large10:45
* jonathangee joins
<fasseg>no streaming :/
<nbanks>ah ic10:47
looks like a plain form is the only reasonable solution for now
<fasseg>or maybe theres a timeout for XHR...10:48
still investigating...
might be the safest solution..10:49
<pivotal-bot____>Andrew Woods edited "Publish javadocs" https://www.pivotaltracker.com/story/show/5491237210:53
* awoods joins10:54
<fasseg>nbanks: so upload via basic HTML form works for files larger than 1GB but then fcrepo crashed with a OutOfMemory error :)10:58
* github-ff joins10:59
[fcrepo4] robintaylor opened pull request #110: Fix FedoraRepositoryBackupTest.java (master...master) http://git.io/v91xEQ
* github-ff leaves
<nbanks>yikes
Do you run fcrepo with export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=1024m"
<fasseg>nah as described in the README i ran with "MAVEN_OPTS="-Xmx512m" mvn jetty:run" :)11:01
but the file upload should never cause an out of memory error...that's a nice DOS attack ...11:02
<nbanks>I thought I changed that on the site?
<fasseg>lemme try again...
<nbanks>You right, I only changed it for the build portion.11:03
but locally I just have it set to always be -Xmx1024m -XX:MaxPermSize=1024m so I never noticed it failing for large files.
Is there no standup or did I miss it? The difference in time zone has me a bit messed?11:09
<fasseg>can you try uploading a 2gb file by adding such a form to fcrepo4/fcrepo-http-api/src/main/resources/views/common-node-actions.vsl: https://gist.github.com/anonymous/651085811:12
isnt the sprint postponed since there are no dev available? Im not officialy on this sprint, though but I read something in awoods email...11:13
<awoods>fasseg: postponed indeed11:14
fasseg: That is not to say that interested developers are not completely welcomed to proceed with tickets.11:15
<fasseg>hehe ofc...nigel and I are taking a look at one right now ;)
<awoods>fasseg: I see that, great.
and Robin Taylor has submitted a PR11:16
<fasseg>and nbanks it still fails for me with 1gig heap and perm space...
<nbanks>k sounds good, I'll continue with the work I'm doing for islandora over fedora 4. It's not looking good for merging in the code for node events for federated files, seems like it needs to be postponed till modeshape 4. https://issues.jboss.org/browse/MODE-2040#comment-12803175
CURL works for me for large files, with the given settings, I'm in the process of testing it with the form you gist'ed11:22
Yup it fails with the form though, with out of memory errors.11:24
<fasseg>strange curl works for me too...
multipart/formdata maybe?
<nbanks>maybe? I'll launch with yourkit and see what's going on.
<pivotal-bot____>Andrew Woods added "Backup/Restore test failure on MAC" https://www.pivotaltracker.com/story/show/5671261011:25
Andrew Woods started "Backup/Restore test failure on MAC" https://www.pivotaltracker.com/story/show/56712610
Andrew Woods edited "Backup/Restore test failure on MAC" https://www.pivotaltracker.com/story/show/56712610
Nigel Banks added comment: "https://issues.jboss.org/browse/MODE-2040#comment-12803175" https://www.pivotaltracker.com/story/show/5598533411:26
* github-ff joins11:27
[fcrepo4] awoods pushed 1 new commit to master: http://git.io/iLnIhw
fcrepo4/master 15d2277 Robin Taylor: Fix FedoraRepositoryBackupTest.java - Get the canonical path of the tmp directory to better mimic what happens in the class being tested.
* github-ff leaves
<pivotal-bot____>Andrew Woods added comment: "Resolved with: http://git.io/iLnIhw" https://www.pivotaltracker.com/story/show/56712610
Andrew Woods finished "Backup/Restore test failure on MAC" https://www.pivotaltracker.com/story/show/56712610
Andrew Woods delivered "Backup/Restore test failure on MAC" https://www.pivotaltracker.com/story/show/56712610
* jongibson joins11:35
<fasseg>org.infinispan.transaction.synchronization.SyncLocalTransaction retains 861 heapspace in byte[] on my local machine...11:38
861 mb
<nbanks>Ya I see that, but I'm not sure what to make of it.11:41
It's a giant mostly empty byte array11:42
<fasseg>We might be doing something wrong using modeshape's transactions...11:43
taking a look at the datastream class atm...
hmm does org.modeshape.jcr.value.binary.infinispan.ChunkOutputStream store the complete binary data in a Cache<String,byte[]> ?11:56
in line 106 each chunk gets added to such a Cache, this is where the outofmemory exception occurs for me...11:57
<nbanks>line 106?11:59
in Datastream.java?12:00
* travis-ci joins12:01
[travis-ci] futures/fcrepo4#981 (master - 15d2277 : Robin Taylor): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo4/compare/ead4e8ba8c7f...15d227747435
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo4/builds/11199972
* travis-ci leaves
<fasseg>nah in ChunkOutputStream
modeshape
<bljenkins>Project fcrepo4 build #1115: UNSTABLE in 35 min: http://ci.fcrepo.org/jenkins/job/fcrepo4/1115/12:02
awoods: Fix FedoraRepositoryBackupTest.java - Get the canonical path of the tmp directory to better mimic what happens in the class being tested.
<pivotal-bot____>Andrew Woods accepted "Backup/Restore test failure on MAC" https://www.pivotaltracker.com/story/show/56712610
<nbanks>ah12:03
hmm kinda seems like it in InfinispanBinaryStore.java line 24112:13
every write utlimately goes to that buffer which is stored in memory?12:14
if that's the case though, why doesn't it trigger an out of memory error when doing the curl request? Shouldn't it got persisted to the cache as well in that case?12:18
<barmintor>awoods: editing the schedule, but managers approved commiting to B6,7,812:28
cbeer ^^12:29
<cbeer>barmintor: works for me, afaik.12:30
<nbanks>awoods: Could I get push permissions for https://github.com/futures/islandora_solution_pack_collection?12:32
<awoods>on a call
<nbanks>awoods: cool, whenever you get the time.12:33
<awoods>nbanks: you should be good now13:03
barmintor, cbeer: It is great to see you on the sprint schedule!13:04
<nbanks>awoods: thanks!13:28
* jongibson leaves13:58
* nbanks leaves14:06
* nbanks joins14:11
* nbanks leaves14:41
* nbanks joins15:37
* jongibson joins15:38
* nbanks leaves15:42
* nbanks joins15:47
* nbanks leaves15:53
* jay leaves16:04
* nbanks joins16:48
* nbanks leaves16:54
* barmintor leaves17:20
* nbanks joins17:48
* nbanks leaves17:55
* ksclarke leaves18:14
* nbanks joins18:49
* nbanks leaves18:54
* nbanks joins19:50
* nbanks leaves19:55
* jongibson leaves20:08
* jay joins20:13
* jay leaves20:15
* jay joins20:22
* jay leaves20:42
* nbanks joins20:50
* nbanks leaves20:57
* ksclarke joins21:31
* nbanks joins21:51
* nbanks leaves21:56
* nbanks joins22:52
* nbanks leaves22:58
* ksclarke leaves23:40
* nbanks joins23:52
* nbanks leaves23:59
* nbanks joins00:53
* kaarefc joins00:54
* kaarefc leaves
* kaarefc joins00:55
* kaarefc leaves
* kaarefc joins
* nbanks leaves00:58
* kaarefc leaves01:14
* nbanks joins01:53
* nbanks leaves01:59
* kaarefc joins02:42
* nbanks joins02:54
* nbanks leaves03:04
* nbanks joins03:17
* nbanks leaves03:22
* nbanks joins03:25
* nbanks leaves03:30
* nbanks joins04:00
* nbanks leaves04:57
* kaarefc leaves05:17
* kaarefc joins05:21
* nbanks joins05:23
* nbanks leaves05:27
* nbanks joins05:29
* fasseg leaves06:21

Generated by Sualtam