Log of the #fcrepo channel on chat.freenode.net

Using timezone: Eastern Standard Time
* arebenji leaves00:03
* dwilcox joins07:16
* bseeger joins08:44
* arebenji joins08:54
* osmandin joins09:08
* acoburn joins09:19
* ajs6f leaves09:30
* ajs6f joins09:55
awoods/dwilcox: So, that Lyrisis thing. Was that just a big waste of time?09:57
acoburn: Can you take a look at https://jira.duraspace.org/browse/FCREPO-2042?focusedCommentId=50150 and tell me if you think that's crazy?
<acoburn>ajs6f: I don't think that's crazy09:58
<ajs6f>acoburn: Might answer a lot of the "I want to batch edit" questions.09:59
acoburn: But atomicity would be tricky.
<acoburn>ajs6f: yes, very tricky
ajs6f: one could wrap all of it in a transaction
<ajs6f>acoburn: Okay, I'm looking forward to seeing your POC after OR.10:00
<awoods>ajs6f: The due diligence solidified core principles of DuraSpace both internally and with many of the DuraSpace members. In that sense it was constructive.
<ajs6f>awoods: I look forward to hearing about those core principles. I was and am perfectly unaware of them.10:01
* mikeAtUVa joins
* peichman joins
<awoods>ajs6f: Then you will get a real kick out of debra and Jonathan's OR2016 plenary talk.10:02
<acoburn>ajs6f: OK, I'll write it in haskell — that way if it compiles, we know it will be correct10:03
<ajs6f>awoods: That will be the first time I ever get a kick out of a plenary talk.
<awoods>ajs6f: double bonus!
<ajs6f>awoods: I am not looking forward to being kicked by Debra and Jonathan.10:04
acoburn: For the use case in hand, I don't know if he needs atomicity. A running averager might do fine for him.10:05
acoburn: Maybe the atomicity comes from putting API-X in front of the combined 3store - repo system.10:34
<acoburn>ajs6f: yes, quite possibly. I think that arrangement has already been described as an API-X use case10:35
<ajs6f>acoburn: If so, and you know here, can you put that as a comment on that ticket. It might help move Christopher Johnson forward.10:36
* jcoyne joins10:38
<acoburn>awoods: have you done much with the fcr:backup and fcr:restore endpoints in medium to large repos?10:39
<awoods>acoburn: no... although I expect we may hit memory limits.
<acoburn>awoods: I was chatting with someone last week who had tried this on a ~30K resource repo and it was taking a long time with no clear indication that anything was happening
awoods: I wonder whether some kind of logging statement could be added10:40
<ajs6f>acoburn: Isn't that exactlly what you'd expect?
<acoburn>ajs6f: that's exactly what I'd expect
<awoods>acoburn: certainly. That is exactly the type of feedback we are looking for from folks who can start piloting the capability.10:41
<acoburn>something like: "resources 1-1000 exported….", "resources 1001-2000 exported", etc
<ajs6f>acoburn: Wait, what would you expect? I'd expect nothing to happen for a long time, is what I was saying.
<acoburn>ajs6f: I would expect some sort of indication that Fedora hadn't just frozen up
<ajs6f>acoburn: Oh. You have much higher expectations than I do. I'd really like to get rid of those endpoints They should be external scripts, instead.10:42
<acoburn>ajs6f: I completely agree that they should be external scripts
<ajs6f>Didn't we deprecate those endpoints?
<acoburn>ajs6f: only the fcr:export, fcr:import10:43
<ajs6f>acoburn: Okay, I'll open a ticket.
<acoburn>ajs6f: which I _didn't_ remove in that PR of mine
ajs6f: I got stuck with the question of "how is exporting any different than an HTTP GET request"
<ajs6f>acoburn: Yeah, but that was because of ordering for the work and code deatils, not because there's any disagreement that they should go, right?
acoburn: Exporting is just a GET with a certain representation selected.10:44
<acoburn>ajs6f: so how is that any different than using an Accept header?
ajs6f: and why would the HTTP api present the internal serialization in the first place?
<ajs6f> acoburn: It's generally not. I think we're agreeing but sing different words.10:45
acoburn: The JCR stuff mst go right away. I don't know why we would keep the endpoints themselves around any longer, but I trust you that it's easier to do the work that way.
acoburn: That's what you said in that PR, right?10:46
<acoburn>ajs6f: the other point I got stuck on was this — if the fcr:export endpoint is removed from the core, is it going to re-emerge in -exts? or is it completely going away?
ajs6f: if it's completely going away, that's pretty easy to do
<ajs6f>acoburn: No, gone. Just gone. It was always redundant.\
* whikloj joins
<acoburn>ajs6f: ok, that implies completely removing fcrepo-serialization and _not_ adding it into -exts
<ajs6f>I think we just weren't as RESTafarian in the beginning, or we would never have introduced it.10:47
acoburn: Great!
<acoburn>ajs6f: if awoods is OK about completely removing fcrepo-serialization, I would be happy to do that
<acoburn>ajs6f: what I can't do right now is reimplement it as some pluggable extension module that goes into -exts
<ajs6f>acoburn: Whoa, why did we want to do that?10:48
<acoburn>ajs6f: maybe awoods can clarify — I certainly don't need it10:49
<ajs6f>acoburn: If we want to make anything pluggable, it's the JAX-RS message-translators that seriialize ordinary HTTP bodies.
<acoburn>ajs6f: my goal in working that ticket remains squarely focused on removing modeshape dependencies from the HTTP layers, which will open up possibilities for new impls10:50
<awoods>acoburn/ajs6f: yes, when we say /fcr:export and /fcr:import are deprecated, we mean they are to be removed... not moved to -exts.10:51
<ajs6f>acoburn:awoods: Sounds like two tickets got crossed: "Remove deprecated stuff" and "Remove JCR stuff"
<acoburn>awoods: cool, I'll update that PR then
<awoods>acoburn/ajs6f: /fcr:backup and /fcr:restore are a different issue, for now, at least... since we need them to migrate to ModeShape5.
awoods: Why?10:52
<awoods>ajs6f: why what?
<ajs6f>awoods: Why would we need them to go to MODE5?
<acoburn>awoods: shouldn't they be part of some utility and not the HTTP interface?
<ajs6f>acoburn: +1
They are _profoundly_ unRESTful.10:53
<awoods>ajs6f: because the backup/restore is how the modeshape team is supporting the migration.
<ajs6f>awoods: That means we _could_ use it for that purpose. It does not mean we must or even should.
awoods: I would much rather write the scripts and use them. They will have more longevity.10:54
<awoods>ajs6f: we currently have the backup/restore endpoints... and that is the recommended path from the modeshape team, seems like an easy win.10:55
ajs6f: once the bump to mode5 is over, we can retool or potentially remove backup/restore
<ajs6f>awoods: If that's your call… I'm not going to spend time with it anyway.
Are those endpoints just dumping JCR/XML?10:57
<awoods>acoburn/ajs6f: the alternative is camel-toolbox/fcrepo-serialization (with binaries enabled) to get your repository out, followed by the infant: https://github.com/jwestgard/fcrepo-serial-restore10:58
ajs6f: yes
<awoods>ajs6f/acoburn: if that tooling were in place, that could be an attractive alternative.10:59
<ajs6f>awoods: That's an immiedaite alternative. We need something that doesn't require Camel. Just two scripts, dump and restore, acting over the HTTP API.
awood: They could just write a big pile of folders, RDF, and bits, just like the Camel stuff, but using only a simple script to do it.
awoods: fcrepo-serial-restore is one half of it.
<awoods>ajs6f: the other half is done: https://github.com/fcrepo4-exts/fcrepo-camel-toolbox/tree/master/fcrepo-serialization11:01
<ajs6f>awoods: No. That means installing a whole bunch of stuff to get Camel running. I am talking about a single, single-file script.
<awoods>ajs6f: That is why we created fcrepo-serialization11:02
<ajs6f>awoods: fcrepo-serialization was thoroughly the wrong approach. It's server-side. That was wrong.11:03
<awoods>ajs6f: ??? have you seen https://github.com/fcrepo4-exts/fcrepo-camel-toolbox/tree/master/fcrepo-serialization ?
<ajs6f>awoods: Yes, I don't understand what is so hard about this. I have said twice now, that module depends on Camel (no good) and fcrepo-serialization is server-side (no good). I am talking about a simple client-side script without dependencies on, Karaf, Camel, or anything else but the presence of a Fedora API.11:05
<awoods>ajs6f: point of clarification, there are two fcrepo-serializations: https://github.com/fcrepo4-exts/fcrepo-camel-toolbox/tree/master/fcrepo-serialization and https://github.com/fcrepo4/fcrepo4/tree/master/fcrepo-serialization, I am talking about the former, you are talking about the latter.11:06
<ajs6f>awoods: No. I am talking about both. The former is no good for my purpose because it carries a huge load of baggage (Karaf, Camel, etc.). The latter is no good because it is a server-side product, and that is completely the wrong deisgn for this.11:07
awoods: fcrepo-serial-restore is the right design. We just need the other half of it.
<awoods>ajs6f: A new script for getting resources out of the repository in a standardized format would be great... but I would argue that the camel/fcrepo-serialization does that. I would also argue that getting the community on-board with camel is a service to everyone. I would also argue that effort towards making it simple to get the camel machinery running is better value than writing yet another tool to do the same thing without the11:18
error handling, notification, workflow, etc offered by camel. Flashback to fcrepo-message-consumer.
<ajs6f>awoods: I don't agree with pretty much any of that. camel/serialization does that, yes. So do the current endpoints. There are lots of ways to do this. I'm not arguing that w aren't doing it. I'm arguing that we could be doing it better. Getting the community on-board with Camel for those purposes for which Camel is suited is fine. Dragging the community into dependence on a large external framework for a simple task like dumping the repo is not. Making Karaf a11:22
<acoburn>I am thinking that this would be a simple script — sort of like the current migration-utils or namespace-utils, etc11:24
<ajs6f>acoburn: Yes.
<acoburn>it wouldn't be complicated, it wouldn't have runtime dependencies
<ajs6f>acoburn: Amen.
<acoburn>it's just a jar that gets executed from the command line
<ajs6f>acoburn: As opposed to Karaf + Camel + a massive pile of JARs + startup scripts + network exposure11:30
All in order to iterate over resources and do GETs on each
* bseeger leaves11:49
<acoburn>awoods: what is the difference b/t junit.Assert.assertFalse and jgroups.util.Util.assertFalse?12:01
<awoods>acoburn: I have never used the jgroups version... not sure.12:02
<acoburn>mind if I replace them as I come across them?
awoods: we should also replace @AutoWired with @Inject12:04
<awoods>acoburn: I don't see a problem if you are updating files relevant to an issue you are working. If, however, you are just updating classes as you see them, that sounds like a separate ticket... all assuming that there is no difference in the two flavors of "assertFalse"
<acoburn>awoods: that's fair12:05
<awoods>acoburn: likewise, a housekeeping ticket for AutoWired to Inject makes sense.
<acoburn>awoods: https://jira.duraspace.org/browse/FCREPO-2046 and https://jira.duraspace.org/browse/FCREPO-204712:09
* mgibney joins12:21
* bseeger joins12:42
* github-ff joins12:47
[fcrepo4] acoburn opened pull request #1046: Replace use of org.jgroups methods with standard junit or java.util equivalents (master...fcrepo-2046) https://git.io/vrNNF
* github-ff leaves
* github-ff joins13:08
[fcrepo4] acoburn opened pull request #1047: Replace @Autowired annotations with the standard @Inject annotation (master...fcrepo-2047) https://git.io/vrNhM
* github-ff leaves
* github-ff joins13:39
[fcrepo4] awoods pushed 1 new commit to master: https://git.io/vrATk
fcrepo4/master 42c89b8 Aaron Coburn: Remove jcr-based serialization component (#1043)...
* github-ff leaves
* github-ff joins13:55
[fcrepo-transform] awoods pushed 1 new commit to master: https://git.io/vrAIm
fcrepo-transform/master e8846c3 Aaron Coburn: Remove compile dependency on Modeshape (#23)...
* github-ff leaves
* travis-ci joins14:01
fcrepo4-exts/fcrepo-transform#80 (master - e8846c3 : Aaron Coburn): The build passed.
Change view : https://github.com/fcrepo4-exts/fcrepo-transform/compare/1d9d2852c7ce...e8846c364533
Build details : https://travis-ci.org/fcrepo4-exts/fcrepo-transform/builds/134527865
* travis-ci leaves
* bseeger leaves14:03
* github-ff joins14:05
[fcrepo4] awoods pushed 1 new commit to master: https://git.io/vrAtL
fcrepo4/master b3ae98e Aaron Coburn: Replace use of org.jgroups methods with standard junit or java.util versions (#1046)...
* github-ff leaves
* mgibney leaves14:23
* travis-ci joins14:28
fcrepo4/fcrepo4#4496 (master - b3ae98e : Aaron Coburn): The build passed.
Change view : https://github.com/fcrepo4/fcrepo4/compare/42c89b88bb00...b3ae98e50a63
Build details : https://travis-ci.org/fcrepo4/fcrepo4/builds/134530533
* travis-ci leaves
* acoburn leaves14:40
* github-ff joins14:41
[fcrepo-module-auth-rbacl] acoburn opened pull request #31: Replace @Autowired annotation with @Inject (master...fcrepo-2047) https://git.io/vrAsu
* github-ff leaves
* acoburn joins14:44
* github-ff joins14:47
[fcrepo-module-auth-xacml] acoburn opened pull request #50: Replace @Autowired with @Inject (master...fcrepo-2047) https://git.io/vrAGM
* github-ff leaves
* github-ff joins14:50
[fcrepo-module-auth-webac] acoburn opened pull request #66: Replace @Autowired with @Inject (master...fcrepo-2047) https://git.io/vrAZn
* github-ff leaves
<acoburn>awoods: those are the last PRs related to that ticket
<awoods>acoburn: thanks14:51
<mikeAtUVa>awood: is fcrepo.ispn.repo.cache/dataFedoraRepository part of our long term persistence or just a cache? Can I stop fedora, delete it and restart?15:06
awoods: ^^^
<awoods>mikeAtUVa: that is the location of the leveldb files15:08
mikeAtUVa: i.e. your objects
<mikeAtUVa>awoods: it seems once you've run fedora 4.5.0 it changes those files such that you can't roll back to 4.4.0... is this a known behavior?15:09
<awoods>mikeAtUVa: yes, the compatibility only goes one direction.15:10
mikeAtUVa: have you possibly compromised production data?15:11
<mikeAtUVa>awoods: no, I don't think so... but I get an error that I'm pretty sure I didn't get on previous versions and was hoping to test it without having to spin up a new repository to try to replicate the scenario on a previous version.15:12
awoods: basically I can't view resources that have lots of children due to file handle limits. I seem to recall doing to before (though of course it took a long time when there are 10s of thousands of children, but it worked.15:13
<awoods>mikeAtUVa: have you considered getting off of leveldb?
<mikeAtUVa>awoods: now if I get a resource that has lots of children I get a "Caused by: java.io.FileNotFoundException: /usr/local/projects/fedora4/fcrepo-home/data/fcrepo.ispn.repo.cache/dataFedoraRepository/569536.sst (Too many open files) "15:14
awoods: I'd be happy to... are there instructions to migrate?
<awoods>mikeAtUVa: you can use /fcr:backup followed by /fcr:restore... although ajs6f has an even better approach.15:15
<ajs6f>mikeAtUVa: See discussion above ^^^15:16
<mikeAtUVa>awoods, ajs6f: I saw it... my goal right now it to quickly get to the point where the repo works so I can continue pushing data into it...15:17
awoods: what alternative to leveldb would you recommend?
<awoods>mikeAtUVa: either MySQL or PostgreSQL... your choice.15:18
mikeAtUVa: https://wiki.duraspace.org/display/FEDORA4x/Configuring+JDBC+Object+Store
<ajs6f>The fact that we have to do migrations just to make a repo keep working is a little depressing.
<mikeAtUVa>ajs6f: we're still in early-adopter mode.
ajs6f: I know that because half the time I try to do something it's obvious no one's done it before.15:19
<ajs6f>mikeAtUVa: We (the people advertising this software) is on version 4.5 something.
<mikeAtUVa>ajs6f: I don't catch your drift... does the version number imply a level of stability?15:21
* github-ff joins15:22
[fcrepo4] awoods closed pull request #1047: Replace @Autowired annotations with the standard @Inject annotation (master...fcrepo-2047) https://git.io/vrNhM
* github-ff leaves
* mikeAtUVa wanders down the hall to request a mysql database... bbs.
* github-ff joins
[fcrepo-module-auth-rbacl] awoods closed pull request #31: Replace @Autowired annotation with @Inject (master...fcrepo-2047) https://git.io/vrAsu
* github-ff leaves
<ajs6f>mikeAtUVa: Yes. Since the 4 is locked for advertising purposes, the 5 is actually the major version number. We have 5 major version releases and we have people with corruption for which our recomendation is: migrate your repo to a different backend. The one we've been recommending for years sucks.15:23
* github-ff joins
[fcrepo-module-auth-webac] awoods pushed 2 new commits to master: https://git.io/vrA8G
fcrepo-module-auth-webac/master a29853d Aaron Coburn: Replace @Autowired with @Inject...
fcrepo-module-auth-webac/master 9b68509 Andrew Woods: Merge pull request #66 from acoburn/fcrepo-2047...
* github-ff leaves
* github-ff joins
[fcrepo-module-auth-xacml] awoods pushed 2 new commits to master: https://git.io/vrA8n
fcrepo-module-auth-xacml/master a0276e5 Aaron Coburn: Replace @Autowired with @Inject...
fcrepo-module-auth-xacml/master 02fd6c0 Andrew Woods: Merge pull request #50 from acoburn/fcrepo-2047...
* github-ff leaves
<acoburn>mikeAtUVa: have you tried increasing the open file limit on your server?15:26
<mikeAtUVa>acoburn: I did... but I don't know how crazy to go... and because I was pretty sure it worked before I was trying to identify what version of fedora broke it15:27
<acoburn>mikeAtUVa: for systems I manage, I don't hesitate to set it to 6500015:28
mikeAtUVa: I think the default tends to be around 1024
* travis-ci joins15:29
fcrepo4/fcrepo-module-auth-webac#141 (master - 9b68509 : Andrew Woods): The build passed.
Change view : https://github.com/fcrepo4/fcrepo-module-auth-webac/compare/0d3308c24967...9b685093314f
Build details : https://travis-ci.org/fcrepo4/fcrepo-module-auth-webac/builds/134552127
* travis-ci leaves
<mikeAtUVa>acoburn: I set it to 4096... I'll try 65000 if you think that's reasonable... I think there's probably a file leak if you need to go that high... but perhaps I'm old fashioned.
* travis-ci joins
fcrepo4/fcrepo-module-auth-xacml#110 (master - 02fd6c0 : Andrew Woods): The build passed.
Change view : https://github.com/fcrepo4/fcrepo-module-auth-xacml/compare/b22743ec9f0c...02fd6c03cec1
Build details : https://travis-ci.org/fcrepo4/fcrepo-module-auth-xacml/builds/134552170
* travis-ci leaves
<acoburn>mikeAtUVa: I was also using Erlang on those systems, so a crazy-high file limit wasn't that crazy15:30
mikeAtUVa: 4096 seems pretty reasonable15:31
* travis-ci joins15:32
fcrepo4/fcrepo-module-auth-rbacl#60 (master - d40761b : Andrew Woods): The build passed.
Change view : https://github.com/fcrepo4/fcrepo-module-auth-rbacl/compare/b6af388704d2...d40761b9e9f0
Build details : https://travis-ci.org/fcrepo4/fcrepo-module-auth-rbacl/builds/134552096
* travis-ci leaves
<mikeAtUVa>acoburn: if I still can't request a resource with 100k children when tomcat7 is allowed to open 65000 files, should I report it as a bug?15:33
<acoburn>mikeAtUVa: I'd think that was a bug15:34
<mikeAtUVa>acoburn: well, 65000 files is apparently enough... no bug then.15:36
<acoburn>mikeAtUVa: good enough — though I'm rather surprised that leveldb needs that number of open filehandles.15:37
<mikeAtUVa>awoods, ajs6f: should we consider dropping leveldb as the default configuration for all but the single-click run? I know it's a huge pain for new installations to have to set up a database a username and password, but I feel at some point making the setup hard in favor of preventing corruption issues down the road would be a service to adopters.15:38
<ajs6f>mikeAtUVa: Isn't that what we do recommend?
<mikeAtUVa>ajs6f: I suppose we recommend it, but if you just drop in the release war files they "just work" using the minimal-default config which uses level-db.15:44
* travis-ci joins15:45
<ajs6f>mikeAtUVa: Sounds like a ticket to be filed. We ought not speak out of both sides of our mouth at once.
<travis-ci>fcrepo4/fcrepo4#4497 (master - ded76af : Aaron Coburn): The build passed.
Change view : https://github.com/fcrepo4/fcrepo4/compare/b3ae98e50a63...ded76af2258f
Build details : https://travis-ci.org/fcrepo4/fcrepo4/builds/134552017
* travis-ci leaves
<mikeAtUVa>ajs6f: I'm sort of suggesting that shipping a war file that doesn't work until you conciously make storage choices is better than shipping one that "works" out of the box with potentially flawed choices already made.
<ajs6f>mikeAtUVa:awoods: We could try another, more portable SQL backend. Derbby or something.
<mikeAtUVa>ajs6f: awoods: it may only apply to lazy folks like me who instead of reading the whole manual, just try to get it to work and read-up on the problems that arise.15:46
<ajs6f>ajs6f:awoods: At what point does the bad press we get from people with corruption outweigh the convenience of LevelDB?15:48
<awoods>mikeAtUVa: fyi, the modeshape5 branch does exactly what you suggest: no default database in the war, but uses leveldb for the one-click.15:56
mikeAtUVa: your review would be helpful: https://jira.duraspace.org/browse/FCREPO-199415:57
<mikeAtUVa>ajs6f: see, as always, awoods is (at least) one step ahead of me:)
<ajs6f>awoods: We should start labelled the "one-click" explicitly as a demo part, not for other usage.15:58
afk bbl15:59
* ajs6f leaves
<mikeAtUVa>awoods: once I get my ingests churning again I'll take that modeshape 5 for a spin...16:01
<awoods>mikeAtUVa: thank you! that would be much appreciated.16:02
* osmandin leaves16:46
* arebenji leaves16:51
* mikeAtUVa leaves17:04
* peichman leaves17:06
* dwilcox leaves17:09
* dwilcox joins17:16
* acoburn leaves17:24
* whikloj leaves18:00
* jcoyne leaves18:38
* jcoyne joins18:42
* dwilcox leaves22:14
* arebenji joins22:17
* dwilcox joins22:24
* jcoyne leaves23:48
* f4jenkins leaves23:59
* f4jenkins joins00:00
* arebenji leaves00:04
* dwilcox leaves00:53