diff --git a/development/issues/Filters.md b/development/issues/Filters.md new file mode 100644 index 0000000..9e075a6 --- /dev/null +++ b/development/issues/Filters.md @@ -0,0 +1,124 @@ +# Filters need more specification + +Given that all data in Elm is immutable, it is VERY important that we shape our +data model the right way. + +The Matrix spec doesn't seem sufficiently clear on how certain endpoints +cooperate with the usage of filters, however, and this may raise some +misrepresentation. + +I have no familiarity with the +[Server-Server API](https://spec.matrix.org/v1.6/server-server-api) and I'm +basing my perspective of the timeline purely on the linear timeline as presented +in the [Client-Server API](https://spec.matrix.org/v1.6/client-server-api). +Section [7.6 Syncing](https://spec.matrix.org/v1.6/client-server-api/#syncing) +suggests that batch tokens can be seen as marked waypoints between two events, +and that the timeline can be seen as one with a +[strict total ordering](https://en.wikipedia.org/wiki/Total_order#Strict_and_non-strict_total_orders). + +# Filters and endpoints + +Currently, three endpoints support filters: + +- [`/sync`](https://spec.matrix.org/v1.6/client-server-api/#get_matrixclientv3sync) +- [`/messages`](https://spec.matrix.org/v1.6/client-server-api/#get_matrixclientv3roomsroomidmessages) +- [`/context`](https://spec.matrix.org/v1.6/client-server-api/#get_matrixclientv3roomsroomidcontexteventid) + +## /sync + +The `/sync` endpoint gets you the latest events in the timeline, as long as +they match the criteria of the filter. From my understanding, the endpoint is +defined as follows: + +![Representation of the /sync endpoint.](/development/issues/img/sync.png) + +As you can see: + +1. With no filter, the endpoint is clear. +2. With a filter, the endpoint is clear if the most recent event on the +timeline meets the filter's criteria. +3. With a filter, the endpoint is **NOT** clear if the most recent event +doesn't meet the filter's criteria. + +There are points to be made that the `next_batch` token is set at the end of +the timeline, but it can also make sense to return the `next_batch` token at +the most recent event that matches the filter. + +The spec doesn't seem to suggest either. + +## /messages + +The `/messages` endpoint is a little trickier, and some of the inputs aren't +exactly clear. What should happen when the user inserts invalid input? + +![Representation of all possible inputs for the /messages endpoint.](/development/issues/img/messages.png) + +When asking people in the +[Matrix spec channel](https://matrix.to/#/#matrix-spec:matrix.org): + +1. Some have argued that the endpoint should return no events, as the +homeserver should stop iteration once it has _passed_ the `to` token. + +2. Some have implied that the endpoint should iterate until it has reached any +of the limits, as the batch tokens are opaque and homeservers shouldn't be +expected to know the relative position of two tokens. + +However, when using filters, **another** issue rises of where tokens should +start and end: + +![Representation of what the /messages endpoint returns given certain filters.](/development/issues/img/messages2.png) + +As can be seen, the spec doesn't seem to verify where the `end` token should +point to. For the **circles only** filter, there's an argument to be made to +put the `end` batch token right after the last event: that way, we wouldn't +skip the next **square** and **star** event in case we switch to a different +filter. + +## /context + +If we jump to an event on the timeline, we are able to get the context of the +event and see what events have been sent around the same time. + +![Representation of what the /messages endpoint returns given certain filters.](/development/issues/img/context.png) + +At first, the issues may seem similar to the ones presented in the `/messages` +endpoint. However, the `/context` endpoint has the major disadvantage that it +doesn't show the relative location of the endpoint on the timeline. + +### An example + +Suppose we joined a public room yesterday, then turned off our client during +the night, and turned it back on today. During the night, some people sent so +many events that the `/sync` endpoint has announced a gap to us this morning. + +However, in one of the most recent events, one of the room members replies to +some event in the past! Luckily, we can use `/context` to jump to that event - +but where in the timeline is this event located? Was this event sent last +night, or before we joined the room yesterday? + +![Representation of what the /messages endpoint returns given certain filters.](/development/issues/img/context2.png) + +Since batch tokens are opaque values, we as the client cannot use them to +determine where the messages is located relative to the timeline that we're +familiar to. Or can we? + +This behaviour heavily depends on how `/messages` works on undefined values: + +1. If the endpoint stops as soon as it's _passed_ the `to` token, then one can +take two batch tokens _(e.g. `batch_token_1` and `batch_token_5`)_ and call the +endpoint once in both directions. _(Backwards and forwards)_ One of two will +return an empty list of events, which hints at the relative position of the two +tokens. + +2. If the endpoint only stops _at_ the `to` token, then the only way to +determine the relative position of the event is to keep paginating `/messages` +in either direction until you hit familiar events. _(Unrelated note: this can +be improved by picking a filter as specific as possible that eventually hits +one of our familiar events.)_ + +# To summarize + +At first, I wrote an [issue for a spec clarification]() on this, but now it seems that it's necessary to write an MSC about it. I'd like to get feedback though, so here's an open letter to all interested people first! + +The MSC would probably be a request to clarify filtering in the spec. It won't be just a clarification though, as it would mean setting so many specifics that it's likely at least one client will not have implemented them accordingly. + diff --git a/development/issues/img/context.png b/development/issues/img/context.png new file mode 100644 index 0000000..a944f7a Binary files /dev/null and b/development/issues/img/context.png differ diff --git a/development/issues/img/context2.png b/development/issues/img/context2.png new file mode 100644 index 0000000..55be8a9 Binary files /dev/null and b/development/issues/img/context2.png differ diff --git a/development/issues/img/messages.png b/development/issues/img/messages.png new file mode 100644 index 0000000..26500a4 Binary files /dev/null and b/development/issues/img/messages.png differ diff --git a/development/issues/img/messages2.png b/development/issues/img/messages2.png new file mode 100644 index 0000000..4337d3e Binary files /dev/null and b/development/issues/img/messages2.png differ diff --git a/development/issues/img/sync.png b/development/issues/img/sync.png new file mode 100644 index 0000000..e73bf8b Binary files /dev/null and b/development/issues/img/sync.png differ