Log of the #fcrepo channel on chat.freenode.net

Using timezone: Eastern Standard Time
* f4jenkins leaves03:42
* f4jenkins joins03:45
* escowles_ leaves07:28
* escowles joins07:29
* manez joins08:00
* manez leaves08:05
* manez joins08:13
* coblej joins08:15
* manez leaves08:26
* manez joins08:27
* benpennell joins08:35
* awoods joins08:39
* escowles leaves09:00
* whikloj joins09:03
* dhlamb joins
* dwilcox joins09:06
* yamil joins09:12
* bseeger joins09:25
* peichman joins09:29
* bryjbrown joins09:35
* benpennell leaves09:49
* bryjbrown leaves
* benpennell joins09:58
* coblej leaves10:19
* bryjbrown joins10:27
* coblej joins10:36
* benpennell leaves11:10
* github-ff joins11:16
[fcrepo4] escowles closed pull request #1216: Fcrepo 2538 (master...fcrepo-2538) https://git.io/v7fne
* github-ff leaves
* bseeger leaves11:25
* travis-ci joins11:31
fcrepo4/fcrepo4#5085 (master - 8694fcf : Esmé Cowles): The build passed.
Change view : https://github.com/fcrepo4/fcrepo4/compare/7e9c62d1bca3...8694fcf72714
Build details : https://travis-ci.org/fcrepo4/fcrepo4/builds/256095755
* travis-ci leaves
* coblej leaves11:50
* bseeger joins11:52
* dbernstein joins11:58
* dbernstein leaves12:01
* bseeger leaves12:25
* github-ff joins12:32
[fcrepo-specification] escowles opened pull request #177: Incorporating language about interoperability (master...interop) https://git.io/v7TU1
* github-ff leaves
* benpennell joins12:34
* coblej joins12:51
* bryjbrown leaves
* coblej leaves12:55
* coblej joins13:00
* mcritchlow joins13:10
<peichman>looking into the possible memory leak issue discussed on the tech call yesterday13:14
examination of the heap dumps shows a problem area might be the org.fcrepo.event.serializer.JSONLDSerializer class13:15
its ObjectMapper looks like it is creating but then never releasing Jackson JSON serializer and deserializer objects; 1 each for each event13:16
each of those is 1MB in size, and by the time of the last heap dump there were 228,300 of each of them, with a total memory usage of 456MB13:17
* mcritchlow leaves13:51
* mcritchlow joins13:52
* mcritchlow leaves
* mcritchlow joins
* peichman leaves13:53
* peichman joins13:56
<whikloj>peichman: good work14:03
* coblej leaves14:04
<peichman>whikloj: now the hard(er) part: figuring out what code is causing that, and how14:05
<whikloj>peichman: yes, acoburn re-wrote most of that (I think) so he knows it best. Is there a JIRA ticket opened?
<peichman>not yet, I've been looking at 4.7.0 code (since that is what we are running)14:06
I will take a look at the diff of JSONLDSerializer between 4.7.0 and 4.7.[34]14:07
<whikloj>peichman: between 4.7.0 release and now not much has changed in the event-serialization14:08
<peichman>good to know
<whikloj>4.7.0 was released on November 1st, 2016 and this PR was merged on the 6th. https://github.com/fcrepo4/fcrepo4/pull/1136/files Nothing since then14:09
peichman: However this could be a long standing issue that everybody just throws RAM at :)
<peichman>I sorta have that feeling :-)14:10
(especially since that is a common response to Tomcat problems)
* coblej joins
<peichman>whikloj: that change just looks like using a newer, better name for one of the imports, but otherwsie no changes14:13
<whikloj>peichman: yeah14:14
peichman: and JsonLDSerializer is so small14:15
peichman: Maybe if we just instantiated one serializer here?14:16
https://github.com/fcrepo4/fcrepo4/blob/fcrepo-4.7.0/fcrepo-jms/src/main/java/org/fcrepo/jms/DefaultMessageFactory.java#L72
instead of in every getMessage call
peichman: you should open a ticket though, get all this information in there. Ping acoburn on the ticket. He understands it best and could give context to some design decisions14:18
<peichman>whikloj: will do14:20
<whikloj>peichman++
* peichman leaves14:56
* mcritchlow leaves14:58
* peichman joins
* coblej leaves15:08
* coblej joins15:25
* bseeger joins15:55
* dbernstein joins16:12
<awoods>peichman: I am trying to reproduce the jvisualvm scenario you have in your ticket.16:18
peichman: Do you have suggestions on your setup?
<peichman>awoods: I have added the JMXRemoteLifecycleListener to Tomcat16:19
awoods: the JVM parameters I was using for Tomcat are in my original fedora-tech post16:20
<awoods>peichman: Do you have documentation of a link to your github... or anything that can expedite my get the JMXRemoteLifecycleListener added?16:21
peichman: Do you have documentation or a link to your github... or anything that can expedite my getting the JMXRemoteLifecycleListener added?
<peichman>awoods: https://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html#JMX_Remote_Lifecycle_Listener_-_org.apache.catalina.mbeans.JmxRemoteLifecycleListener
awoods: for this simple testing purposes, I kept SSL off and disabled authentication16:22
<awoods>peichman: it seems like JMXRemoteLifecycleListener is to facilitate connecting with a remote tomcat.16:23
<peichman>awoods: once that is set up, you should be able to use VisualVM to connect to {server}:10001
<awoods>peichman: I am currently able to connect my jvisualvm with my tomcat...
<peichman>awoods: true; I haven't tried setting this up where I had VisualVM running on the same machine as the Tomcat JVM16:24
* coblej leaves
<peichman>awoods: okay; the screenshots I shared were from the "Monitor" tab
<awoods>peichman: then what?16:25
<peichman>awoods: and then there's a "Heap Dump" button on that tab, too
<awoods>peichman: I am not seeing "dominator_tree"
<peichman>awoods: ah, that's in a different app, the Eclipse Memory Analyzer: https://www.eclipse.org/mat/
<awoods>peichman: That would explain it16:26
<peichman>awoods: there is a way to analyze the heap dumps in VisualVM, but I ran into memory problems (!!!) when trying to load them16:27
* dwilcox leaves
<awoods>peichman: In terms of reproducing your scenario, do you think a simple POST loop will do the trick?
<peichman>awoods: it should, since the key seems to be the number of events16:28
<awoods>peichman: right
<peichman>awoods: so filling the repo wqith empty containers should do it :-)
<awoods>peichman: I am hoping to reproduce your situation, then see the difference with code changes.
peichman: There may be a simple fix by changing JsonLDSerializer.java16:29
peichman: but I need to be able to see the problem before knowing if a fix works.
peichman: your Eclipse Memory Analyzer would provide that16:30
<peichman>awoods++
awoods: and as noted in the ticket, I will have some time to work on this next week16:31
<awoods>peichman: thanks for that
* benpennell leaves16:49
* whikloj leaves17:01
* bseeger leaves
* yamil leaves17:15
* peichman leaves17:34
* dhlamb leaves17:43
* github-ff joins17:49
[fcrepo4] dbernstein opened pull request #1217: Fixes incorrect redirection of URIs ending /fcr:tx in HTML UI to /fcr… (master...fcrepo-2539) https://git.io/v7T5g
* github-ff leaves
* jonathangee leaves19:27
* benpennell joins19:45
* benpennell leaves19:47
* github-ff joins19:55
[fcrepo4] awoods closed pull request #1217: Fixes incorrect redirection of URIs ending /fcr:tx in HTML UI to /fcr… (master...fcrepo-2539) https://git.io/v7T5g
* github-ff leaves
* travis-ci joins20:08
fcrepo4/fcrepo4#5087 (master - 34795a1 : dbernstein): The build passed.
Change view : https://github.com/fcrepo4/fcrepo4/compare/8694fcf72714...34795a150749
Build details : https://travis-ci.org/fcrepo4/fcrepo4/builds/256260614
* travis-ci leaves
* peichman joins22:49
* peichman leaves23:12
* awoods leaves00:08