Log of the #fcrepo channel on chat.freenode.net

Using timezone: Eastern Standard Time
* dchandekstark joins03:26
* dchandekstark leaves03:30
* dchandekstark joins05:27
* dchandekstark leaves05:31
* dchandekstark joins06:42
* dchandekstark leaves06:46
* manez joins08:08
* dwilcox joins08:17
* dwilcox leaves08:18
* jenlindner joins08:35
* jenlindner leaves08:36
* whikloj joins08:54
* bseeger joins09:09
* ajs6f joins09:13
* apb18 joins09:14
* esm_ joins09:19
* ajs6f leaves09:44
* ajs6f joins
* peichman joins09:49
* mikeAtUVa joins09:58
* manez_ joins10:21
* manez leaves
* dhlamb joins10:25
* bseeger leaves10:27
* dwilcox joins10:48
* dwilcox leaves10:55
* dwilcox_ joins
* esm_ leaves10:57
* esm_ joins10:58
* jenlindner joins11:09
* jenlindner leaves11:14
* jenlindner joins11:39
* jenlindner leaves11:40
* dwilcox_ leaves11:42
<ajs6f>dhlamb: are you super-worried about If-Match? because my impression is that, as escowles says, If-Modified-Since will handle any but the most ultra-high-throughput concurrency arrangements.11:51
<apb18>ajs6f: Would it be possible for the repository to guarantee that modification date always increments by at least the minimal unit of precision after an update?12:11
<ajs6f>apb18: I suppose, in theory. I can't see that going down well with the audit crowd.12:12
apb18: We might leave that kind of thing open in a spec, but for the "community impl" I think we have to be as precise as is practical. I would rather just have a "number of successful mutating requests since last PUT" or something like that.12:13
apb18: About _that_, you could make a similar guarantee without getting into questions about the recordation of time.12:14
The Recordation of Time would have been a good name for a mediocre prog rock band in the 60s.12:15
<apb18>ajs6f: prog rock hit really its stride in the '70s anyway12:16
<ajs6f>apb18: https://www.youtube.com/watch?v=iwMlmzFKxvE
apb18: Borderline as to progness, but all the way in as to wonderfulness.12:17
* acoburn joins
apb18: it looks like version 1.4 of the karaf features namespace has been published: http://karaf.apache.org/xmlns/features/v1.4.012:20
<apb18>ajs6f: That actually fits my mood at the moment. Space and time collide, yet all is well.
<acoburn>apb18: I asked about it yesterday on their IRC channel
<ajs6f>acoburn: The collision of space and time? Space and time have an IRC channel?12:21
<apb18>acoburn: excellent! I didn't realize they had an IRC channel. I had a few questions in the past that went unanswered in the mailing list a long time ago
<ajs6f>And a mailing list?!?
<acoburn>apb18: apache-karaf12:22
<esm_>karaf just seems so rough around the edges12:50
<acoburn>esm_: a lot of OSS projects are rough around the edges12:52
<esm_>I don't know. The bar is pretty high these days.12:53
But I suppose it is true, if considering the entire universe of OSS projects, the vast majority are rough. But the good ones have definitely raised the bar.12:54
Maybe I have unreasonable expectations.
* manez joins13:06
* manez_ leaves
* esm_ leaves13:15
<ajs6f>esm_: They're a small team. When you look at the design, the code, the features, the accomplishment is amazing. Admittedly, the docs stink and there's a lot of polish wanted.13:25
esm_: But it's open source, right? You can take them up over that bar and help them meet your expectations. Just send a PR, or for helping with docs:13:27
just use the snazzy bookmarklet to send a patch.
* esm_ joins13:41
* manez leaves13:44
* manez joins13:46
* esm_ leaves13:51
* esm_ joins14:02
* esm_ leaves14:03
<dhlamb>ajs6f: i think it's mostly paranoia over precision for me. i planned on rolling with time before I realized that If-Modified-Since only uses second precision.14:33
ajs6f: i'd feel more comfortable for millisecond
ajs6f: which I guess we sorta get with the ETag the way it's generated?14:34
<ajs6f>dhlamb: Hm. I did not realize this: https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.3.114:36
dhlamb: that is crap.
* esm_ joins
<ajs6f>escowles: ^^^
<dhlamb>ajs6f: other options are to use ntp, which can be fine if you have everything in house14:37
ajs6f: but full ETags for RDF would make the problem go away14:38
<ajs6f>dhlamb: Full (strong?) ETags for RDF is not arriving anytime soon. We would do that today if we could.
<dhlamb>ajs6f: so is the issue that you can't control order of triples?14:39
<acoburn>dhlamb: if you have a distributed system that depends on each system having a synchronized clock is a recipe for disaster
<dhlamb>ajs6f: or just not having a canonical format?
<escowles>dhlamb: yes, and a dozen other ways RDF serializations can vary
dhlamb: you'd have to serialize to disk after every update in order to make the byte-for-byte identical guarantees that http requires for strong etags
<ajs6f>dhlamb: Strong is really quite strong.14:40
dhlamb: One blank line difference? Not good enough.
<dhlamb>acoburn: yeah, wasn't keen on the idea. just presenting options. had seen suggested a couple of times.
<ajs6f>Backing out of the RDF-specific part of the conversation, https://www.infoq.com/news/2009/06/rest-ts14:42
Arguably, the habit of concurrency management that asks the server to manage conflicts is just not a good way to do REST.
Doesn't mean I don't want to do something useful for dhlamb et al. Just that we should be trying as hard as possible to use idempotency etc.14:43
<dhlamb>ajs6f: so at this point just using the weak ETag now and pretending it's strong actually offers precision i'd live with.14:45
ajs6f: but i doubt it's gonna stay like that forever
ajs6f: if we get into violating spec territory
<ajs6f>dhlamb: URG. tjohnson will mess you up for doing that.
dhlamb: "pretending it's strong"14:46
<dhlamb>ajs6f: lol, read "dirty string operation"
<ajs6f>dhlamb: What exactly is the need? Are you trying to serialize requests against the repository?
dhlamb: Why do you have many mutating requests against the same URI? (I'm not saying you're doing anything wrong, I'm sincerely wondering what the use case here is.)14:47
<mikeAtUVa>ajs6f, dhlamb: any app that allows concurrent edits to the same item requires this.14:48
<dhlamb>ajs6f: just concurrency control
ajs6f: every asych op is going to follow the pattern of using the ETag or last modified and using conditional updates
<ajs6f>mikeAtUVa: No, any app that allows concurrent edits and _does not prevent conflicts itself_ requires this.
dhlamb: And CLAW can't serialize inside itself before issuing requests to Fedora?14:49
<acoburn>ajs6f: one issue is if you have multiple apps writing to the repo
<mikeAtUVa>ajs6f: how would you prevent conflicts without the ability to have a single atomic rest request that was an "update if not changed".
<dhlamb>ajs6f: that's the point of the headers14:50
<ajs6f>mikeAtUVa: You have an intermediate form locally against which you can check, and crucially, CLAW has that, in Drupal.
<dhlamb>ajs6f: but i have people coming at the same dataset from two angles
<ajs6f>acoburn: Yes, but I'm trying to understand the interaction between CLAW and Fedora, not the general problem.
<mikeAtUVa>ajs6f: if I udnerstand that, it means only one app can act against the repo.14:51
<ajs6f>dhlamb: But they're both coming through the same layer of CLAW, aren't they?
mikeAtUVa: No, absolutely not.
<dhlamb>ajs6f: either drupal or fedora, and i have to suss those out. which I can. but to make that process safe, i need to optimisitically lock
<ajs6f>dhlamb: Sure, I'm saying that you do _not_ have to lock against Fedora. You can lock against Drupal. You control the path between Drupal and Fedora, right.14:52
<dhlamb>ajs6f: i'll be using version vectors to manage concurret edits and know when to ignore messages
<mikeAtUVa>ajs6f: I don't want to derail dhlamb's actual use case, but I am curious if there's a model I'm missing where you can guarantee not to trample other user's changes without having either locking support in the repository (etag use in this way is optimistic locking).
<ajs6f>mikeAtUVa; Certainly not. but that's not what dhlamb is trying to do. Let's stick to that.14:53
<mikeAtUVa>ajs6f: thanks.
<dhlamb>mikeAtUVa: yeah, it's either this, or i have to do pessimistic locks, which i don't want to introduce unless i have to
<ajs6f>mikeAtUVa: Actually, I hate to say it, but I take that back. You can lock something other than the repo. For example, Zookeeper.
mikeAtUVa: But it's not really important. dhlamb has an obvious thing to deal with that isn't Fedora, and that is Drupal.14:54
<dhlamb>ajs6f: don't get me started on the no PUT
dhlamb: PATCH only in D8
there i go talking to myself again
<mikeAtUVa>dhlamb: we had an fcr:lock endpoint that we axed a while back. I can't say I'm super sad, it didn't work with transactions given our use of JCR.14:55
<ajs6f>dhlamb: If you have a clear state in Drupal (because of clever version techniques) all you have to do is replicate that clear state in the repo. Not pass every message along.
dhlamb: I don't know much about Drupal— it doesn't allow PUT? That seems strange.
<dhlamb>ajs6f: state is never trustworthy from just one source if i'm multi-master
<mikeAtUVa>ajs6f: good point... you could force all your apps to lock against something else. I'm pretty sure I know why that didn't occur to me;)
<dhlamb>ajs6f: i have to compare. and when i do, and i want to synchronize state. i need some form of locking to make sure i don't trample changes.14:56
<ajs6f>dhlamb: Synchronizing your Drupal instances? That's not my problem. {grin}
dhlamb: I'm not trying to solve your problem of deriving a clear state. I'm saying:
1) It's not Fedora's problem. Because
2) Fedora isn't a great place to solve that kind of problem.
<dhlamb>ajs6f: you mean you'd be willing to support conditional update to only a second's precision14:59
<ajs6f>dhlamb: If you just want timestamping, we should take up that problem by itself. It's not that hard to provide good timestamping. We might not be able to provide it in If-Modified-SInce, but we can do something.
<dhlamb>ajs6f: i'm fine with millisecond prcecision any way i can get it. i'd help you do that15:00
ajs6f: which is why i would be willing to "pretend". fun fact: i'll stoop there if i have to.15:01
<ajs6f>dhlamb: Okay, let's take up acoburn's problem. You have a Fedora API backed by processes distributed across three Amazon zones. You get, depending on where your request ended up, one of three different timestamps, as a single update percolates across the country. Does this work?
<mikeAtUVa>ajs6f, dhlamb: just to be clear, does fcrepo4 really lack a way to issue an update (PUT OR PATCH) that will fail if the resource has changed since our last get? (verified either through very precise timestamp or etag)15:02
<dhlamb>mikeAtUVa: at this point, i don't think so
<ajs6f>mikeAtUVa: I don't know. I don't think I've ever used Fedora.
<dhlamb>ajs6f: just use one timezone15:03
<ajs6f>dhlamb: Not a question about timezones. The question is about network latency.
<dhlamb>ajs6f: but the real problem is the percolation
<ajs6f>dhlamb: Right. Distribution is hard. Trying to make it look transaction makes it harder.
<dhlamb>ajs6f: strong ETag would solve it15:05
<ajs6f>dhlamb: Patches welcome. I may be goofy, but barmintor and escowles and acoburn are pretty smart, and they agree that strong ETags aren't going to happen.15:06
dhlamb: I worry about the kinds of guarantees you seem to want to make. That changes from one direction will always get reflected through the system, I get that. I want that myself. I'm selling that to other people. But you seem to want to guarantee something about a coherent state over the system. I'm not sure I understand this.
<mikeAtUVa>This is such a technically easy problem to solve, just working around/through/with standards makes it hard.15:07
<ajs6f>mikeAtUVa: You're right. HTTP—
mikeAtUVa: The kind of thinking that led to junk like the World Wide Web just doesn't work in the long term.15:08
mikeAtUVa: Or at scale.
<apb18>mikeAtUVa: From what I understand from this conversation, (a) Fedora does not have strong ETags, this disallowing If-Match; (b) If-Unmodified-Since can only be used to 'second' precision, because RFC2616 says so.
<dhlamb>mikeAtUVa: ajs6f: yeah seriously. just being able to use something that's unique and not byte for byte would be fine. but its the standard at that point
<mikeAtUVa>apb18: yup... so neither of those standards allow us to meet the need of making a request that fails if the resource has been updated.15:09
<ajs6f>dhlamb: We can stick a timestamp into messages. We just can't stick it in one particular header. I don't understand what the horror of that is.
<dhlamb>ajs6f: yeah, i'd totally use that15:10
<ajs6f>mikeATUVa: Neither of those standards prevents us from doing that. They don't "not allow" us to do it. They just don't give us the tools prebuilt. You know, HTTP headers are not a magic enumerated thing that cannot be augmented.
<apb18>ajs6f: Would this be a fedora API specification-level suggestion?15:11
<mikeAtUVa>ajs6f: agreed... but should I make a PR that has a custom protocol for this sort of optimistic locking?
<ajs6f>dhlamb: If you want to think about something that works particularly well for RDF, we could look at getting a larger conversation started. I'm not suggesting we slow down whatever work might be useful immediately, just that you're probably not the only person in the worl who wants to update triples on a server.15:12
<mikeAtUVa>ajs6f: the simplest implementation would be to have a revision number triple with each GET, and then if you POST/PATH with a revision number triple that doesn't match, you get a 409.
<ajs6f>apb18: It could start that way. I think we want to look around. What are some other (beyond RDF) types of media that have a very strong and clear sense of "equivalence" that isn't byte-comparison? Have communities that use those media solved this problem in a way that can teach us?15:13
<dhlamb>ajs6f: i'm sure some combo of the path to the resource + timestamp for modified would be a good place to start if you want something cheap to calculate
<mikeAtUVa>dhlamb: why don't you just embed fedora in your app like the hydra folks do, and limit it to a single thread so you don't have to worry about concurrency:P
<ajs6f>mikeAtUVa: That's what I meant earlier about a "number of successful updates since last PUT" counter.
<dhlamb>ajs6f: but then that's weak, and using it in that capacity runs afoul of standard15:14
mikeAtUVa: no
<mikeAtUVa>ajs6f: wearing my "user" hat, I don't really care how it's accomplished so long as it is, because without it, you can't really write a real application against fedora.15:15
<ajs6f>dhlamb: I don't really want to use timestamp so much. I'm not dead set against it, but I think we can do better. And we're not talking about strong ETags. We're just not.
mikeAtUVa: I think that's a gross exaggeration. But it doesn't really matter.
<dhlamb>ajs6f: right. i'm fine with weak if it's good enough. honestly, the implementation right now, as it stands, is fine if we can do it under the guise of different headers15:16
ajs6f: with the understanding that it's the hash of the timestamp with millisecond precision
<ajs6f>dhlamb: I don't think I understand you. Are you talking about sticking a weak etag in some other header with a totally different semantics? That sounds quite confusing to me.15:17
dhlamb: Why is ms precious particularly important to you? All that seems to matter is whether something has changed? What does clock time have to do with it at all?
ms precision, sorry
<apb18>I'm seeing this header as containing some kind of token that is opaque to the consumer, and we could implement it by a timestamp, the weak etag string, an integer containing number of updates, etc15:18
<dhlamb>apb18: i am down wiht any of those things15:19
<mikeAtUVa>ajs6f: I suspect there are more than one update per second sometimes.
<ajs6f>Expensive as hell, tho.
<dhlamb>apb18: the more we get into lamport timestamps or something, the better
<ajs6f>mikeAtUVa: That's irrelevant. I'm saying that what time a request was served doesn't matter. All that matters is the order of requests.15:20
<acoburn>dhlamb: X-Fedora-VClock: foo.bar.baz
<dhlamb>ajs6f: right, you can establish pre-order through means other than time
<mikeAtUVa>ajs6f: I thought timestamp meant the timestamp that the resource was last modified...
<ajs6f>dhlamb: Right, and establishing it with time tangles you up in a whole bunch of other stuff.
mikeAtUVa: Sure. But we don't have to use a timestamp for this. As apb18 said ^^^15:21
<dhlamb>ajs6f: well, i mean... so i plan on donig this anyway, but was going to embed version vectors in RDF
ajs6f: if this would be useful to the fedora api, i'd be glad to help
<ajs6f>Although I don't like reusing the weak ETag string. That's just going to get confusing.
<dhlamb>ajs6f: incrementer would be enough
ajs6f: i'll be on the hook for storage a second one, so i know what version drupal thinks it has15:22
<ajs6f>dhlamb: I think we have to suss out a couple of things: is this in the API, or just a convenient feature of the ref impl? (Because we need to think about how hard this is going to be for distributed impls).
<dhlamb>ajs6f: but that's MY problem
<ajs6f>dhlamb: Let's rip into this on a tech call, okay? I'd like to hear from other people.
<dhlamb>ajs6f: well, i know what version fedora thinks drupal has. and if i know what version drupal thinks fedora has, i can solve my problem
ajs6f: and generate the pre-order15:23
ajs6f: yeah, love to talk about. i've had conflicts the past few weeks so have been out of it
<ajs6f>dhlamb: You are already assuming that Fedora ha a single version, and that you can retrieve it. That's single-node thinking.
<dhlamb>ajs6f: i assume whatever node has that version15:24
ajs6f: and am talking about just a single fedora
because in my problem drupal is the other node
<ajs6f>dhlamb: Right. The API is not going to make that assumption. It can't. But this is all grist for the mill.
dhlamb: No, I am talking about a multinode Fedora. Distributed Fedora impls.
<dhlamb>ajs6f: i'm not asking for that
<ajs6f>dhlamb: You are not the only person asking for things. {grin}15:25
dhlamb: acoburn is nice. We don't want to ruin his day by asking him to implement things that are easy on a single node and nightmarish in distribution.
<dhlamb>ajs6f: interval tree clocks is a generalization of what i'm talking about. you would need something like that for distribution if that's what you want
<acoburn>dhlamb is correct15:26
<apb18>+1 for tech call. I think the LDP and HTTP specs make this a sufficiently hard problem for everybody that we should aim for an API-spec solution, if possible
<ajs6f>dhlamb: This is exactly why we need to understand a fuller suite of uses.
apb18: Agreed. I think the question of whether that is possible is the first one to answer. If possible.
<apb18>If possible and passible.15:27
<ajs6f>dhlamb: https://wiki.duraspace.org/display/FF/2016-08-25+-+Fedora+Tech+Meeting15:28
Slather it on.
* apb18 needs to listen to another Popol Vuh album now15:29
<ajs6f>apb18: Take it to the next level:
<dhlamb>ajs6f: struggling to describe conversation in bullet point form... moment15:32
<ajs6f>dhlamb: Part of what would be nice here would be a proper PATCH language. Like Andy Seaborne's https://afs.github.io/rdf-patch/.15:34
dhlamb: https://github.com/afs/rdf-delta15:35
dhlamb: Part of the solution in the big picture is to stop talking about mutations and instead about differences. That's why we're all using Git and not SVN.15:36
<dhlamb>ajs6f: ^^reading now15:44
ajs6f: so you wanna be able to merge on conflict?15:46
<ajs6f>dhlamb: I am interested, for the future, in the following claim. We are currently trying to come to a shared understanding of the state of a resource over time by exchanging representations and accompanying metadata. It will be more fruitful _eventually_ to try exchanging representations of the changes in state we each understand to be of interest. And yes, that move does introduce the possibility of… compromise.15:57
<dhlamb>ajs6f: yeah, that'd be pretty slick16:04
ajs6f: but to be continued...
ajs6f: I have to go be a dad now16:05
ajs6f: ttyl
<ajs6f>dhlamb: Sweet gig. Enjoy.
<dhlamb>acoburn, escowles, mikeAtUVa, apb18: nice talk everyone
cya later16:06
* dhlamb leaves16:10
* manez leaves16:13
<ajs6f>Hah. I was reading the LDP spec and I only now notice that all of their RDF examples include strong ETags. Nice, guys.16:31
* ajs6f leaves16:36
* apb18 leaves16:38
* peichman leaves16:45
* esm_ leaves16:49
* whikloj leaves16:55
* escowles leaves17:14
* acoburn leaves17:17
* github-ff joins17:41
[fcrepo-camel-toolbox] awoods closed pull request #106: add support for Fedora 4.6.x (master...fcrepo-2121) https://git.io/v6wGL
* github-ff leaves
* travis-ci joins17:59
fcrepo4-exts/fcrepo-camel-toolbox#279 (master - 0f22a56 : Aaron Coburn): The build passed.
Change view : https://github.com/fcrepo4-exts/fcrepo-camel-toolbox/compare/e86b7ad4f8ca...0f22a56f3844
Build details : https://travis-ci.org/fcrepo4-exts/fcrepo-camel-toolbox/builds/153685925
* travis-ci leaves
* dwilcox joins20:01
* dwilcox leaves20:36
* mikeAtUVa leaves21:04
* peichman joins21:29
* peichman leaves21:30
* f4jenkins joins23:31

Generated by Sualtam