Log of the #duraspace-ff channel on chat.freenode.net

Using timezone: Eastern Standard Time
* ksclarke leaves00:23
* kaarefc joins01:43
* kaarefc leaves02:09
* kaarefc joins02:15
* awoods leaves02:28
* kaarefc leaves02:34
* awoods joins02:40
* kaarefc joins03:09
* kaarefc leaves04:35
* awoods leaves04:54
* awoods joins
* escowles leaves06:02
* fcrepo-bot joins08:45
* ajs6f joins08:59
awoods: ping
<pivotal-bot>A. "Horbulaco" Soroka accepted "Make indexing gear work with Elasticsearch" https://www.pivotaltracker.com/story/show/6251208009:01
A. "Horbulaco" Soroka edited "Factor execution handling into abstract superclasses" https://www.pivotaltracker.com/story/show/62554588
A. "Horbulaco" Soroka accepted "Factor execution handling into abstract superclasses" https://www.pivotaltracker.com/story/show/62554588
A. "Horbulaco" Soroka edited "More integration tests for the named-fields indexer gear" https://www.pivotaltracker.com/story/show/6226275809:02
A. "Horbulaco" Soroka edited "Unit tests for indexer framework" https://www.pivotaltracker.com/story/show/62262512
A. "Horbulaco" Soroka edited "fcrepo-jms-indexer should maintain its own broker for robustness" https://www.pivotaltracker.com/story/show/61414712
* github-ff joins09:12
[fcrepo-jms-indexer-pluggable] ajs6f pushed 1 new commit to master: http://git.io/HQQgrw
fcrepo-jms-indexer-pluggable/master cac6a03 ajs6f: Trying to reenable test metrics on Sonar
* github-ff leaves
* ermadmix joins
<ajs6f>ermadmix: Any luck with the indexer tests?09:20
I've been playing with the settings for memory and JaCoCo, trying to find a stable place that both runs well and executes our test metric gear.
* nbanks joins09:22
<ermadmix>no such luck, I tried 2 more times the process that may be the problem consistently around 950,000k (not incrementing as I said before).09:24
<ajs6f>ermadmix: Sorry, I think that got garbled. Can you say that again?09:25
<ermadmix>read a similar issue caused by 2 apps in a tomcat trying to access same jar in /lib, solution was to put jar's in each apps web xml
<ajs6f>We aren't using Tomcat in any way at all in that code base.09:26
<ermadmix>no such luck, I tried 2 more times. The process that may be the problem is consistently around 950,000k (not incrementing as I said before).
<ajs6f>The process is an AMQ process?
<ermadmix>but maybe in test having the fcrepo and indexer running in the same test container is causing something similar?09:27
<ajs6f>ermadmix: Sure. You tell me. :) Seriously, can you try running this via YourKit and see what the heap actually contains?09:28
afk bbs.09:29
* ajs6f leaves
* travis-ci joins
[travis-ci] futures/fcrepo-jms-indexer-pluggable#63 (master - cac6a03 : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo-jms-indexer-pluggable/compare/fb79c66377bd...cac6a033bcae
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo-jms-indexer-pluggable/builds/15586157
* travis-ci leaves
<ermadmix>don't know how to identify the process, other than it leaves the build/test output hanging, and when I kill the process the buld/test ends (gives the the reactor summary results and stops)09:30
* kaarefc joins09:35
* nbanks leaves09:39
* ksclarke joins09:43
<ermadmix>never used yourkit, just downloading now, the memory leak check function?09:44
* fcrepo-bot leaves09:51
<ermadmix>is there a license server for yourkit available?09:53
* md5wz__ leaves09:56
* kaarefc leaves10:22
<cbeer>ermadmix: this is a blow-the-heap failure after i re-added the jacoco stuff?10:33
<ermadmix>cbeer: not sure when it was introduced, just noticed it yesterday10:34
<bljenkins>Project fcrepo-jms-indexer-pluggable build #391: ABORTED in 1 hr 50 min: http://ci.fcrepo.org/jenkins/job/fcrepo-jms-indexer-pluggable/391/11:02
A. Soroka: Trying to reenable test metrics on Sonar
* osmandin joins11:10
<bljenkins>Yippee, build fixed!11:11
Project fcrepo-jms-indexer-pluggable build #392: FIXED in 8 min 17 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-jms-indexer-pluggable/392/
<osmandin>Speaking of YourKit, has anyone tried accessing f4 using jconsole etc?11:12
<pivotal-bot>Chris Beer accepted "Fix indexer Sonar metrics" https://www.pivotaltracker.com/story/show/62593158
<osmandin>When I tried jconsole, the connection was getting a timeout11:14
* kaarefc joins11:16
* osmandin leaves11:23
* ajs6f joins11:28
Wow.11:29
cbeer: I'm almost unhappy I got the test metrics back for the indexer stuff. They tell a sad story.
<cbeer>ajs6f:want me to break the build instead?11:30
<ajs6f>cbeer: :0
I want to have my cake and eat it, too.
Even better if it was a taco instead of cake.
* kaarefc leaves11:31
<ajs6f>Well, time to start fixing things...11:32
* kaarefc joins11:34
* kaarefc leaves11:35
* gregjansen joins12:04
* github-ff joins12:09
[fcrepo-jms-indexer-pluggable] ajs6f opened pull request #25: Sonar fixes (master...SonarFixes) http://git.io/1_Lniw
* github-ff leaves
<pivotal-bot>A. "Horbulaco" Soroka added "Sonar fixes for indexer" https://www.pivotaltracker.com/story/show/62645934
A. "Horbulaco" Soroka started "Sonar fixes for indexer" https://www.pivotaltracker.com/story/show/62645934
A. "Horbulaco" Soroka finished "Sonar fixes for indexer" https://www.pivotaltracker.com/story/show/6264593412:10
A. "Horbulaco" Soroka delivered "Sonar fixes for indexer" https://www.pivotaltracker.com/story/show/62645934
* github-ff joins
[fcrepo-jms-indexer-pluggable] ajs6f pushed 1 new commit to master: http://git.io/Km-9vQ
fcrepo-jms-indexer-pluggable/master 7f4103e ajs6f: Merge pull request #25 from futures/SonarFixes...
* github-ff leaves
* github-ff joins
[fcrepo-jms-indexer-pluggable] ajs6f closed pull request #25: Sonar fixes (master...SonarFixes) http://git.io/1_Lniw
* github-ff leaves
<pivotal-bot>A. "Horbulaco" Soroka added comment: "Merged with12:11
https://github.com/futures/fcrepo-jms-indexer-pluggable/commit/7f4103ef916fbebe1eb27b..." https://www.pivotaltracker.com/story/show/62645934
A. "Horbulaco" Soroka accepted "Sonar fixes for indexer" https://www.pivotaltracker.com/story/show/62645934
* github-ff joins
[fcrepo-jms-indexer-pluggable] ajs6f deleted SonarFixes at a65712e: http://git.io/PiBlwA
* github-ff leaves
* ajs6f leaves12:17
* ajs6f joins12:19
* travis-ci joins12:32
[travis-ci] futures/fcrepo-jms-indexer-pluggable#65 (master - 7f4103e : ajs6f): The build passed.
[travis-ci] Change view : https://github.com/futures/fcrepo-jms-indexer-pluggable/compare/cac6a033bcae...7f4103ef916f
[travis-ci] Build details : http://travis-ci.org/futures/fcrepo-jms-indexer-pluggable/builds/15595942
* travis-ci leaves
* ermadmix leaves13:07
* ermadmix joins13:15
* ksclarke leaves13:33
* kaarefc joins13:38
* kaarefc leaves
* ermadmix leaves13:47
* ksclarke joins
* kaarefc joins
* ermadmix joins13:51
* gregjansen leaves14:12
* gregjansen joins14:16
* gregjansen leaves14:23
* gregjansen joins14:25
<fasseg>It seems I can't build master atm Im getting PermGen errors even with -XX:MaxPermSize=1024m...Can you guys build with tests atm?
hmm seems it is jacoco again....14:27
<awoods>fasseg: The only way I have been able to build for the last month is with -Djacoco.skip14:30
<fasseg>grml14:31
<ajs6f>I have no problem building.14:32
<awoods>fasseg: what does grml mean?14:33
<ajs6f>It's a terrible markup language,
<awoods>hmm, I am not sure I see the connection.
<ajs6f>I'm joking. He said, "Grumble."14:34
<fasseg>just a grunting sound, to express discontent
lol
ajs6f: do you have JAVA_OPTS set?14:35
<ajs6f>MAVEN_OPTS=' -Xmx2048M -XX:MaxPermSize=1024M '
That's all.
<fasseg>to increase permsize?
<ajs6f>^^^
<fasseg>hmm doenst work for me and the argline for permgen in the jacoco config in the pom.xml is gone...14:36
<ajs6f>cbeer: Was doing somehting with jacoco to make it run in CI, but IDK what.
<fasseg>so profiles could be used...14:37
<cbeer>ajs6f: actually turn it back on, with a new version of jacoco.
<fasseg>for CI, I mean
<cbeer>i don't understand why you guys have trouble though.. gluck is also a linux box and isn't broke14:38
<ajs6f>awoods + fasseg = troublemakers!
<fasseg>hehe
cbeer: different JVM maybe?
ill give openjdk a try14:39
* gregjansen leaves14:40
* github-ff joins14:46
[fcrepo-jms-indexer-pluggable] ajs6f opened pull request #26: Improverate unit tests (master...ImproverateUnitTests) http://git.io/IwJPug
* github-ff leaves
<pivotal-bot>A. "Horbulaco" Soroka started "Update fcrepo-jms-indexer-pluggable to use Supplier memoization machinery from Guava" https://www.pivotaltracker.com/story/show/62410034
A. "Horbulaco" Soroka finished "Update fcrepo-jms-indexer-pluggable to use Supplier memoization machinery from Guava" https://www.pivotaltracker.com/story/show/62410034
A. "Horbulaco" Soroka added comment: "https://github.com/futures/fcrepo-jms-indexer-pluggable/pull/26" https://www.pivotaltracker.com/story/show/6226251214:47
* github-ff joins14:48
[fcrepo-jms-indexer-pluggable] ajs6f closed pull request #26: Improverate unit tests (master...ImproverateUnitTests) http://git.io/IwJPug
* github-ff leaves
<pivotal-bot>A. "Horbulaco" Soroka added comment: "https://github.com/futures/fcrepo-jms-indexer-pluggable/commit/b334c3c962ee8d9f311d0c90718e3f7bc12..." https://www.pivotaltracker.com/story/show/62262512
* github-ff joins14:49
[fcrepo-jms-indexer-pluggable] ajs6f deleted ImproverateUnitTests at df70a5b: http://git.io/O_Rp1A
* github-ff leaves
* gregjansen joins14:55
* ajs6f leaves14:57
* ajs6f joins14:58
* gregjansen leaves14:59
* gregjansen joins15:10
* github-ff joins15:11
[fcrepo4] fasseg created fix-jacoco-permgen (+1 new commit): http://git.io/U2r1Lg
fcrepo4/fix-jacoco-permgen c646a09 fasseg: added permsize setting to argLine of the failsafe plugin
* github-ff leaves
<fasseg>cbeer & awoods: As I understood it 'jacoco.agent.it.arg' is used to set some required properies in the CI env...So I just added a -XX:MaxPermSize=128m to the argLine. With that addition the builds passes for me:https://github.com/futures/fcrepo4/compare/fix-jacoco-permgen Is it ok to merge this change, or will I break the CI setup?
<ajs6f>fasseg: Are you aware that the previously-set properties (which might come from CLI) are available in ${argLIne}?15:12
<fasseg>nope...
<ajs6f>You don't want to override props that people are setting for themselves.
You want to add to them.
Let me find you an example...
* gregjansen leaves15:13
<fasseg>it's addition, and you can still overwrite using the jacoco.agent.it.arg property, but I fear when these builds become parallelized, we dont want every thread to grab all the Heap and permgen assigend to the main maven thread...
wont this blow up?15:14
<ajs6f>Good question— it has to do with the surefire forking settings, which we have never touched.
<fasseg>so we want jacoco to run with as minimal as possible permgen, and since we do know how much is required for our build, I'd assume it's safe to set...
and it's certainly better than a failing build.15:15
<ajs6f>I'm not saying it's not safe to set, I'm saying it must be overridable.
<fasseg>it is, using the jacoco.agent.it.arg property
* gregjansen joins
<ajs6f>No, the total argLine prop.
If I use my MAVEM_OPTS, you are now discarding them and using the ones you've baked into the code.15:16
I want you to use mine, if I set them.
<fasseg>but then well blow up the heap if we tell people to use 1024m permgen as in the README
and two jacoco processes are running or sth
<ajs6f>We will let people blow up the heap if they want.15:17
But we will give them an appropriate default.
That's the behavior we want.
<fasseg>but other tests might fail if the permgen is to small...
<ajs6f>Sure, that's where the appropriate default comes in.
If I i try to build with an explicitly
set permgen that is too small.
<fasseg>so we would give ppl a either you can run the tests or you can run the jacoco inspections but either way it will fail if you dont choose one
<ajs6f>That's my fault.
* gregjansen leaves15:18
* gregjansen joins
<ajs6f>No, we will let people set the params they want, but if they forget (or don't know) to set them, we will inject useful ones.
I have lots of room on my box. I want to build faster than you can. I shouldn't be restricted to the minimal settings that will work for you.
<fasseg>but then the build fails with our prposed README values
<ajs6f>Then our README values are wrong.15:19
Not our code.
<fasseg>and if we decrease the README values i fear tests will fail...
<ajs6f>I don't understand that at all. How will increasing the heap and PermGen that we suggest cause failure?
<fasseg>I got more than one jacoco process running at one time, they seem to overlap15:20
<ajs6f>Found the example:
<artifactId>maven-surefire-plugin</artifactId>
          <configuration>
            <argLine>-Xms512m -Xmx1024m -XX:MaxPermSize=512m ${argLine} </argLine>
          </configuration>
        </plugin>
See how I reinject the argLIne at the end?
That let's the user ovverride my choices.
<fasseg>so if I assign a heap size of 1g to be used by maven and 1g to be used by jacoco I will have 3g mem used15:21
yes, but I dont thin you do get my point
<ajs6f>Used by JaCoCo? I don't think we can set that directly. We can only control what surefire uses.
<fasseg>if we use the same commandline, we will break things.
<ajs6f>It's runs for me, it runs for cbeer, it runs for Jenkins, it runs for Sonar. I don't think it's guaranteed to break things.15:22
If it breaks things for you and awoods,
we need to fix that, but I don't
<fasseg>atm there is a property jacoco.agent.it.arg which can be set, introduced by chris, and I think that makes more sense than injecting the whole maven commandline, since it's a completely different process
<ajs6f>It is _not_ a _completely_ diffferent process. It is a subprocess.
It makes sense to inherit choices.
<fasseg>nahhh, please try to get my point: we need different settings for the Unit tests the integration tests and the jacoco processes...15:23
since the unit tests and ITs need much more permgen than jacoco
<ajs6f>I get that.15:24
I disagree.
<fasseg>and we have at least 2 jacoco processes for one maven process, so we're wasting a lot of ram, and I for one do not have 3gb available to run it all on my laptop...
<ajs6f>I think that's a surprising configuration.
If you were willing to document exactly how to override your choices (and it was a matter of a single CLI flag or so), I would buy this.15:25
<fasseg>and I think (1GB heap + 1GB permgen) * 2 for just one jacoco process if we can have (256m heap and 128m permgen) + (1gb + 1gb) does not make sense
its not my choice, the property is already in there....
<ajs6f>If I have to edit Maven config because your laptop is smaller than mine, that seems unreasonable to me.15:26
<fasseg>it think chris added it for the CI env
no you dont
<ajs6f>Okay. Tell me how to execute a build with larger allocations (since I have that luxury).
<fasseg>but i would have to edit my maven config because my laptop is smaller than your's15:27
if it would be the other way around -> ${argline} in the property
you couldn't before
<ajs6f>I don't see how that makes sense, but if you test it I'll believe you.
* gregjansen leaves15:28
<fasseg>you can't run it on a machine with less than 2 * (heapsize + permsize) free space if you're using the same opts for jacoco as for the main maven build, since it's to distinct JVMs...15:32
<ajs6f>I understand that.
I want to be able to override whatever choices you make to be able to build on small machines without editing code.15:33
<fasseg>so if we make using the argline the default and we require 1024m permsize we will also use 102m permsize for jacoco...
<ajs6f>I think my machine looks more like what most devs use than yours.
I amy very well be wrong.
<fasseg>you can: just use the property jacoco.agent.it.arg
<ajs6f>Show me.
<fasseg>it's already in there: in master
I DID NOT ADD THIS
* ksclarke leaves15:34
<ajs6f>You are asking to change the allocations. I am asking what flags with what settings I would (and will, if this change goes in) I will have to use to reset them (or enlarge them).
Because I thikn most people will want to do that.
<fasseg>with the current config, awoods and me can't build..with the small addition im suggesting at least I can build, which does raise tha chance that some radnom user can also build it, doesn't it?15:35
<ajs6f>Yes, and that most users will experience painful slowdowns.
<fasseg>from the jacoco run? no they won't
<ajs6f>In any event, we need to document this.
<fasseg>there's no difference for me
<ajs6f>I already have, for weeks in the spring, until I bumped the allocations, _because Jenkins was failing_.15:36
<fasseg>permsize is just new classes allocation space, so once you have enough, adding more does not help
<ajs6f>We are _not_ talking about a minimally successful config to build, we ware talking about what will provide the best experience for the most people (and pseudo-people like CI).15:37
<fasseg>and I did not change the possibility to override the the permsize using the property jacoco.agent.it.arg which is most likely already used on jenkins...
<ajs6f>Which you still haven't explained to me how to use.
DOCUMENTATION!
<fasseg>since I dit not add it, and im wquite sure you know how to use java properties
<ajs6f>I'm not helping you get any further with this, and we're both starting to lose patience. I'm going to let awoods et al help bring this to a successful conclusion.15:38
<fasseg>good choice...15:39
<ajs6f>http://www.youtube.com/watch?v=kn481KcjvMo
<fasseg>some kittens?15:40
<ajs6f>"You got to know when to fold 'em."
* gregjansen joins
<fasseg>christ not him...
that's worse than rickrolling
<ajs6f>He's got a lovely beard.
<fasseg>ok now im all soft and mushy in the head...15:43
<ajs6f>awoods: Currently setting up fcrepo4 on the new Jenkins instance here. Hopefully we'll move quick with this and give an Xmas present to some lucky sysadmin at Columbia.
Fedora 4: Makes you soft and mushy in the head.15:44
<barmintor>ajs6f: you linked the wrong song http://www.youtube.com/watch?v=BCNORo_tmKs
<ajs6f>barmintor: That is better than all of the Xmas music I've heard this year put together.15:45
And I want all of those costumes.
<bljenkins>Project fcrepo-fixity-corrupter build #573: SUCCESS in 1 min 21 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/573/15:47
<barmintor>ajs6f: I just wish I had been born a rock Viking like that dude at 2:46
<ajs6f>barmintor: You are a beer Viking. A cultured, unstereotypical Viking.15:48
<barmintor>Not the same. Worse beards. Less guitars and sequins.
<ajs6f>You never met Erik the Glam. He didn't get very far in Iceland.15:49
<bljenkins>Project fcrepo-kitchen-sink build #729: STILL UNSTABLE in 5 min 48 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/729/15:53
* ermadmix leaves16:07
<ajs6f>out for the day, see y'all later
* ajs6f leaves
* gregjansen leaves
* github-ff joins16:10
[fcrepo4] fasseg pushed 1 new commit to fix-jacoco-permgen: http://git.io/bUQ1TQ
fcrepo4/fix-jacoco-permgen 6c9b547 fasseg: added some documentation for the jacoco properties to the README
* github-ff leaves
* github-ff joins16:12
[fcrepo4] fasseg pushed 1 new commit to fix-jacoco-permgen: http://git.io/bOiuZw
fcrepo4/fix-jacoco-permgen cf43e16 fasseg: fixed awkward grammar
* github-ff leaves
* github-ff joins16:17
[fcrepo4] fasseg force-pushed fix-jacoco-permgen from cf43e16 to 4ce2388: http://git.io/kADuEg
fcrepo4/fix-jacoco-permgen 4ce2388 fasseg: Added default perm size setting fo jacoco integration test plugin and...
* github-ff leaves
* github-ff joins16:18
[fcrepo4] fasseg opened pull request #216: Fix for test failures due to JaCoCo permgen errors (master...fix-jacoco-permgen) http://git.io/KGjdew
* github-ff leaves
<fasseg>awoods: I added this PR for fixint the JaCoCo perm gen errors, and added a small usage documentation to the README, as Adam requested...16:19
https://github.com/futures/fcrepo4/pull/216
* ivanuschka joins16:31
* fasseg leaves16:34
<bljenkins>Project fcrepo4 build #1577: FAILURE in 25 min: http://ci.fcrepo.org/jenkins/job/fcrepo4/1577/16:35
* ivanuschka leaves16:42
* ermadmix joins16:54
* github-ff joins16:58
[fcrepo4] cbeer closed pull request #216: Fix for test failures due to JaCoCo permgen errors (master...fix-jacoco-permgen) http://git.io/KGjdew
* github-ff leaves
<bljenkins>Project fcrepo4 build #1578: NOW UNSTABLE in 23 min: http://ci.fcrepo.org/jenkins/job/fcrepo4/1578/16:59
* danaja joins17:17
* danaja leaves17:19
* gochaa joins17:20
* ksclarke joins17:24
* gochaa leaves17:27
* kaarefc leaves17:32
* kaarefc joins17:33
<bljenkins>Yippee, build fixed!17:38
Project fcrepo4 build #1579: FIXED in 39 min: http://ci.fcrepo.org/jenkins/job/fcrepo4/1579/
frank asseg: Added default perm size setting fo jacoco integration test plugin and17:39
Project fcrepo-fixity-corrupter build #574: SUCCESS in 2 min 51 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-fixity-corrupter/574/17:42
Project fcrepo-kitchen-sink build #730: STILL UNSTABLE in 6 min 47 sec: http://ci.fcrepo.org/jenkins/job/fcrepo-kitchen-sink/730/17:49
* kaarefc leaves18:01
* kaarefc joins18:08
* kaarefc leaves18:13
* kaarefc joins18:18
* kaarefc leaves18:21
<pivotal-bot>Chris Beer accepted "Run performance testing using minimal-default and tweak settings" https://www.pivotaltracker.com/story/show/6247120018:24
Chris Beer accepted "Feature Documentation: Versioning" https://www.pivotaltracker.com/story/show/62258628
Chris Beer accepted "fcrepo-transform fails to consider node mixin types for the selection of a transform" https://www.pivotaltracker.com/story/show/62512344
Chris Beer added comment: "https://github.com/futures/fcrepo-jms-indexer-pluggable/commit/da3aba77656fa93fc8e007617a5f50fd348bc630" https://www.pivotaltracker.com/story/show/6241003418:26
Chris Beer delivered "Update fcrepo-jms-indexer-pluggable to use Supplier memoization machinery from Guava" https://www.pivotaltracker.com/story/show/62410034
Chris Beer accepted "Update fcrepo-jms-indexer-pluggable to use Supplier memoization machinery from Guava" https://www.pivotaltracker.com/story/show/62410034
Chris Beer accepted "confirm adding a solr doc to standalone via fcrepo-jms-indexer-pluggable" https://www.pivotaltracker.com/story/show/61582228
* ermadmix leaves18:42
* ermadmix joins19:28
* ermadmix leaves19:59
* ermadmix joins20:28
* ksclarke leaves20:56
* ermadmix leaves21:00
<pivotal-bot>Chris Beer added comment: "Updated https://wiki.duraspace.org/display/FF/Configuration+Options+Inventory#ConfigurationOptionsInventory-M..." https://www.pivotaltracker.com/story/show/6249961621:40
* ermadmix joins21:44
<pivotal-bot>Chris Beer estimated "Document cache store configurations and trade-offs." as 1 point https://www.pivotaltracker.com/story/show/6249961621:48
Chris Beer finished "Document cache store configurations and trade-offs." https://www.pivotaltracker.com/story/show/62499616
Chris Beer delivered "Document cache store configurations and trade-offs." https://www.pivotaltracker.com/story/show/6249961621:49
Chris Beer accepted "Document cache store configurations and trade-offs." https://www.pivotaltracker.com/story/show/62499616
* ermadmix leaves22:16
* ermadmix joins22:34
* ksclarke joins22:55
* ermadmix leaves23:06
* ermadmix joins23:37
* ermadmix leaves00:11
* ermadmix joins00:27
* kaarefc joins00:48
* kaarefc leaves00:50

Generated by Sualtam