Merge branch '3-timeline' of github.com:noordstar/elm-matrix-sdk-beta into 3-timeline
commit
10c7075bef
|
|
@ -0,0 +1,138 @@
|
||||||
|
# Timeline
|
||||||
|
|
||||||
|
Given the complex nature of the Timeline design, it deserves some explanation of
|
||||||
|
the design. This document aims to describe how the Elm SDK designs the Timeline,
|
||||||
|
so that other projects may learn from it.
|
||||||
|
|
||||||
|
## API endpoint disambiguations
|
||||||
|
|
||||||
|
Generally speaking, there are a few API endpoints with similar design:
|
||||||
|
|
||||||
|
- The [`/sync` endpoint](https://spec.matrix.org/v1.9/client-server-api/#get_matrixclientv3sync),
|
||||||
|
which gets the events that the homeserver received most recently.
|
||||||
|
- The [`/messages` endpoint](https://spec.matrix.org/v1.9/client-server-api/#get_matrixclientv3roomsroomidmembers),
|
||||||
|
which gets any events in the topological order.
|
||||||
|
|
||||||
|
As noted in the Matrix spec:
|
||||||
|
|
||||||
|
> Events are ordered in this API according to the arrival time of the event on
|
||||||
|
> the homeserver. This can conflict with other APIs which order events based on
|
||||||
|
> their partial ordering in the event graph. This can result in duplicate events
|
||||||
|
> being received (once per distinct API called). Clients SHOULD de-duplicate
|
||||||
|
> events based on the event ID when this happens.
|
||||||
|
|
||||||
|
For this reason, the Elm SDK maintains **two independent timelines** that are tied
|
||||||
|
together when necessary to form a coherent timeline.
|
||||||
|
|
||||||
|
## Elm design
|
||||||
|
|
||||||
|
For those unfamiliar, the Elm Architecture breaks into three parts:
|
||||||
|
|
||||||
|
- **Model** - the state of the application
|
||||||
|
- **View** - a way to turn your state into meaningful information
|
||||||
|
- **Update** - a way to update your state based on the Matrix API
|
||||||
|
|
||||||
|
Since these concepts are compartmentalized, it is impossible to make an API call
|
||||||
|
while executing the **view** function; the Elm SDK must at all times find a way
|
||||||
|
to represent its state.
|
||||||
|
|
||||||
|
## Timeline
|
||||||
|
|
||||||
|
Concerning the Matrix timeline, it is meant to create a representation
|
||||||
|
(**Model**) of the timeline, find a way to represent (**View**) it, and find a
|
||||||
|
simple way to adjust it with every incoming Matrix API result. (**Update**)
|
||||||
|
|
||||||
|
First, we define what a timeline batch is.
|
||||||
|
|
||||||
|
### Timeline batch
|
||||||
|
|
||||||
|
A timeline batch is something that most Matrix API endpoints return. It is a
|
||||||
|
little piece of the timeline and contains the following four pieces of
|
||||||
|
information:
|
||||||
|
|
||||||
|
1. A list of events that are part of the timeline.
|
||||||
|
2. A Filter for which all provided events meet the criteria.
|
||||||
|
3. An end batch token that functions as an identifier.
|
||||||
|
4. _(Optional.)_ A start token. If not provided, it indicates the start of the
|
||||||
|
timeline.
|
||||||
|
|
||||||
|
Here's an example of such a timeline batch:
|
||||||
|
|
||||||
|
```
|
||||||
|
|-->[■]->[■]->[●]->[■]->[■]->[●]-->|
|
||||||
|
| |
|
||||||
|
|<--- filter: only ■ and ● --->|
|
||||||
|
| |
|
||||||
|
start: end:
|
||||||
|
<token_1> <token_2>
|
||||||
|
```
|
||||||
|
|
||||||
|
When the Matrix API later returns a batch token that starts with `<token_2>`,
|
||||||
|
we know that we can connect it to the batch above and make a longer list of
|
||||||
|
events!
|
||||||
|
|
||||||
|
At first, this seems quite simple to connect, but there are some difficulties
|
||||||
|
that come up along the way.
|
||||||
|
|
||||||
|
### Challenge 1: different filters, different locations
|
||||||
|
|
||||||
|
When two timeline batches have different filters, we do not know their
|
||||||
|
respective location. For example, the following two timeline batches COULD
|
||||||
|
overlap, but it is also possible they don't:
|
||||||
|
|
||||||
|
```
|
||||||
|
|-->[■]->[■]->[●]->[■]->[■]->[●]-->|
|
||||||
|
| |
|
||||||
|
|<--- filter: only ■ and ● --->|
|
||||||
|
| |
|
||||||
|
start: end:
|
||||||
|
<token_1> <token_2>
|
||||||
|
|
||||||
|
|
||||||
|
|-->[★]->[★]->[★]->[★]-->|
|
||||||
|
| |
|
||||||
|
|<-- filter: only ★ -->|
|
||||||
|
| |
|
||||||
|
start: end:
|
||||||
|
<token_3> <token_4>
|
||||||
|
```
|
||||||
|
|
||||||
|
Realistically, there is currently no way of knowing without making more API
|
||||||
|
calls. However, just making more API calls isn't a solution in Elm because of
|
||||||
|
its architecture.
|
||||||
|
|
||||||
|
> **SOLUTION:** As described in the **View** function, we may assume that
|
||||||
|
overlapping timeline batches have overlapping events. If they overlap yet have
|
||||||
|
no overlapping events, then their filters must be disjoint. If the filters are
|
||||||
|
disjoint, we do not care whether they're overlapping.
|
||||||
|
|
||||||
|
### Challenge 2: same filters, same spot
|
||||||
|
|
||||||
|
Suppose there is a known timeline batch, and we're trying to **Update** the
|
||||||
|
timeline to represent the timeline between `<token_1>` and `<token_2>` for a
|
||||||
|
different filter:
|
||||||
|
|
||||||
|
```
|
||||||
|
|-->[■]->[■]->[●]->[■]->[■]->[●]-->|
|
||||||
|
| |
|
||||||
|
|<--- filter: only ■ and ● --->|
|
||||||
|
| |
|
||||||
|
start: end:
|
||||||
|
<token_1> <token_2>
|
||||||
|
```
|
||||||
|
|
||||||
|
If we wish to know what's in there for a different filter `f`, then:
|
||||||
|
|
||||||
|
1. If `f` equals the filter from the timeline batch, we can copy the events.
|
||||||
|
2. If `f` is a subfilter of the batch filter (for example: `only ■`) then we can
|
||||||
|
copy the events from the given batch, and then locally filter the events
|
||||||
|
that do no match filter `f`.
|
||||||
|
3. If the batch filter is a subfilter of `f`, then we can use an API call
|
||||||
|
between the same batch tokens `<token_1>` and `<token_2>`. In the worst
|
||||||
|
case, we receive the exact same list of events. In another scenario, we
|
||||||
|
might discover far more events and receive some new batch value `<token_3>`
|
||||||
|
in-between `<token_1>` and `<token_2>`.
|
||||||
|
4. If neither filter is a subfilter of the other and the two are (at least
|
||||||
|
partially) disjoint, then they do not need to correlate and any other batch
|
||||||
|
values can be chosen.
|
||||||
|
|
||||||
|
|
@ -2,7 +2,7 @@ module Internal.Values.Timeline exposing
|
||||||
( Batch, Timeline
|
( Batch, Timeline
|
||||||
, empty, singleton
|
, empty, singleton
|
||||||
, mostRecentEvents
|
, mostRecentEvents
|
||||||
, addSync, insert
|
, insert
|
||||||
, encode, decoder
|
, encode, decoder
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
@ -16,6 +16,29 @@ timeline is quite a complex data type, as it is constantly only partially known
|
||||||
by the Matrix client. This module exposes a data type that helps explore, track
|
by the Matrix client. This module exposes a data type that helps explore, track
|
||||||
and maintain this room state.
|
and maintain this room state.
|
||||||
|
|
||||||
|
This design of the timeline uses the batches as waypoints to maintain an order.
|
||||||
|
The Matrix API often returns batches that have the following four pieces of
|
||||||
|
information:
|
||||||
|
|
||||||
|
1. A list of events.
|
||||||
|
2. A filter for which all of the events meet the criteria.
|
||||||
|
3. An end batch token.
|
||||||
|
4. _(Optional)_ A start batch token. If it is not provided, it is the start of
|
||||||
|
the timeline.
|
||||||
|
|
||||||
|
Here's an example of such a timeline batch:
|
||||||
|
|
||||||
|
|-->[■]->[■]->[●]->[■]->[■]->[●]-->|
|
||||||
|
| |
|
||||||
|
|<-- filter: only ■ and ●, no ★ -->|
|
||||||
|
| |
|
||||||
|
start: end:
|
||||||
|
<token_1> <token_2>
|
||||||
|
|
||||||
|
When the Matrix API later returns a batch token that starts with `<token_2>`,
|
||||||
|
we know that we can connect it to the batch above and make a longer list of
|
||||||
|
events!
|
||||||
|
|
||||||
|
|
||||||
## Batch
|
## Batch
|
||||||
|
|
||||||
|
|
@ -47,8 +70,11 @@ import FastDict as Dict exposing (Dict)
|
||||||
import Internal.Filter.Timeline as Filter exposing (Filter)
|
import Internal.Filter.Timeline as Filter exposing (Filter)
|
||||||
import Internal.Tools.Hashdict as Hashdict exposing (Hashdict)
|
import Internal.Tools.Hashdict as Hashdict exposing (Hashdict)
|
||||||
import Internal.Tools.Iddict as Iddict exposing (Iddict)
|
import Internal.Tools.Iddict as Iddict exposing (Iddict)
|
||||||
|
import Internal.Tools.Json as Json
|
||||||
import Json.Decode as D
|
import Json.Decode as D
|
||||||
import Json.Encode as E
|
import Json.Encode as E
|
||||||
|
import Recursion
|
||||||
|
import Recursion.Traverse
|
||||||
import Set exposing (Set)
|
import Set exposing (Set)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -129,7 +155,7 @@ type Timeline
|
||||||
{ batches : Iddict IBatch
|
{ batches : Iddict IBatch
|
||||||
, events : Dict String ( IBatchPTR, List IBatchPTR )
|
, events : Dict String ( IBatchPTR, List IBatchPTR )
|
||||||
, filledBatches : Int
|
, filledBatches : Int
|
||||||
, mostRecentSync : ITokenPTR
|
, mostRecentBatch : ITokenPTR
|
||||||
, tokens : Hashdict IToken
|
, tokens : Hashdict IToken
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -140,22 +166,6 @@ type alias TokenValue =
|
||||||
String
|
String
|
||||||
|
|
||||||
|
|
||||||
{-| When syncing a Matrix room to its most recent state, add the most recent
|
|
||||||
batch to the front of the Timeline.
|
|
||||||
-}
|
|
||||||
addSync : Batch -> Timeline -> Timeline
|
|
||||||
addSync batch timeline =
|
|
||||||
case insertBatch batch timeline of
|
|
||||||
( Timeline tl, { start, end } ) ->
|
|
||||||
let
|
|
||||||
oldSync : ITokenPTR
|
|
||||||
oldSync =
|
|
||||||
tl.mostRecentSync
|
|
||||||
in
|
|
||||||
Timeline { tl | mostRecentSync = end }
|
|
||||||
|> connectITokenToIToken oldSync start
|
|
||||||
|
|
||||||
|
|
||||||
{-| Append a token at the end of a batch.
|
{-| Append a token at the end of a batch.
|
||||||
-}
|
-}
|
||||||
connectIBatchToIToken : IBatchPTR -> ITokenPTR -> Timeline -> Timeline
|
connectIBatchToIToken : IBatchPTR -> ITokenPTR -> Timeline -> Timeline
|
||||||
|
|
@ -236,158 +246,11 @@ empty =
|
||||||
{ batches = Iddict.empty
|
{ batches = Iddict.empty
|
||||||
, events = Dict.empty
|
, events = Dict.empty
|
||||||
, filledBatches = 0
|
, filledBatches = 0
|
||||||
, mostRecentSync = StartOfTimeline
|
, mostRecentBatch = StartOfTimeline
|
||||||
, tokens = Hashdict.empty .name
|
, tokens = Hashdict.empty .name
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
{-| Decode a Timeline from a JSON value.
|
|
||||||
-}
|
|
||||||
decoder : D.Decoder Timeline
|
|
||||||
decoder =
|
|
||||||
D.map5
|
|
||||||
(\batches events filled sync tokens ->
|
|
||||||
Timeline
|
|
||||||
{ batches = batches
|
|
||||||
, events = events
|
|
||||||
, filledBatches = filled
|
|
||||||
, mostRecentSync = sync
|
|
||||||
, tokens = tokens
|
|
||||||
}
|
|
||||||
)
|
|
||||||
(D.field "batches" <| Iddict.decoder decoderIBatch)
|
|
||||||
(D.map2 Tuple.pair
|
|
||||||
(D.field "head" decoderIBatchPTR)
|
|
||||||
(D.field "tail" <| D.list decoderIBatchPTR)
|
|
||||||
|> D.keyValuePairs
|
|
||||||
|> D.map Dict.fromList
|
|
||||||
|> D.field "events"
|
|
||||||
)
|
|
||||||
(D.succeed 0)
|
|
||||||
(D.field "mostRecentSync" decoderITokenPTR)
|
|
||||||
(D.field "tokens" <| Hashdict.decoder .name decoderIToken)
|
|
||||||
|> D.map recountFilledBatches
|
|
||||||
|
|
||||||
|
|
||||||
decoderIBatch : D.Decoder IBatch
|
|
||||||
decoderIBatch =
|
|
||||||
D.map4 IBatch
|
|
||||||
(D.field "events" <| D.list D.string)
|
|
||||||
(D.field "filter" Filter.decoder)
|
|
||||||
(D.field "start" decoderITokenPTR)
|
|
||||||
(D.field "end" decoderITokenPTR)
|
|
||||||
|
|
||||||
|
|
||||||
decoderIBatchPTR : D.Decoder IBatchPTR
|
|
||||||
decoderIBatchPTR =
|
|
||||||
D.map IBatchPTR decoderIBatchPTRValue
|
|
||||||
|
|
||||||
|
|
||||||
decoderIBatchPTRValue : D.Decoder IBatchPTRValue
|
|
||||||
decoderIBatchPTRValue =
|
|
||||||
D.int
|
|
||||||
|
|
||||||
|
|
||||||
decoderIToken : D.Decoder IToken
|
|
||||||
decoderIToken =
|
|
||||||
D.map5 IToken
|
|
||||||
(D.field "name" decoderTokenValue)
|
|
||||||
(D.field "starts" <| D.map Set.fromList <| D.list decoderIBatchPTRValue)
|
|
||||||
(D.field "ends" <| D.map Set.fromList <| D.list decoderIBatchPTRValue)
|
|
||||||
(D.field "inFrontOf" <| D.map Set.fromList <| D.list decoderITokenPTRValue)
|
|
||||||
(D.field "behind" <| D.map Set.fromList <| D.list decoderITokenPTRValue)
|
|
||||||
|
|
||||||
|
|
||||||
decoderITokenPTR : D.Decoder ITokenPTR
|
|
||||||
decoderITokenPTR =
|
|
||||||
D.oneOf
|
|
||||||
[ D.map ITokenPTR decoderITokenPTRValue
|
|
||||||
, D.null StartOfTimeline
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
decoderITokenPTRValue : D.Decoder ITokenPTRValue
|
|
||||||
decoderITokenPTRValue =
|
|
||||||
D.string
|
|
||||||
|
|
||||||
|
|
||||||
decoderTokenValue : D.Decoder TokenValue
|
|
||||||
decoderTokenValue =
|
|
||||||
D.string
|
|
||||||
|
|
||||||
|
|
||||||
{-| Encode a Timeline to a JSON value.
|
|
||||||
-}
|
|
||||||
encode : Timeline -> E.Value
|
|
||||||
encode (Timeline tl) =
|
|
||||||
E.object
|
|
||||||
[ ( "batches", Iddict.encode encodeIBatch tl.batches )
|
|
||||||
, ( "events"
|
|
||||||
, E.dict identity
|
|
||||||
(\( head, tail ) ->
|
|
||||||
E.object
|
|
||||||
[ ( "head", encodeIBatchPTR head )
|
|
||||||
, ( "tail", E.list encodeIBatchPTR tail )
|
|
||||||
]
|
|
||||||
)
|
|
||||||
(Dict.toCoreDict tl.events)
|
|
||||||
)
|
|
||||||
, ( "mostRecentSync", encodeITokenPTR tl.mostRecentSync )
|
|
||||||
, ( "tokens", Hashdict.encode encodeIToken tl.tokens )
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
encodeIBatch : IBatch -> E.Value
|
|
||||||
encodeIBatch batch =
|
|
||||||
E.object
|
|
||||||
[ ( "events", E.list E.string batch.events )
|
|
||||||
, ( "filter", Filter.encode batch.filter )
|
|
||||||
, ( "start", encodeITokenPTR batch.start )
|
|
||||||
, ( "end", encodeITokenPTR batch.end )
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
encodeIBatchPTR : IBatchPTR -> E.Value
|
|
||||||
encodeIBatchPTR (IBatchPTR value) =
|
|
||||||
encodeIBatchPTRValue value
|
|
||||||
|
|
||||||
|
|
||||||
encodeIBatchPTRValue : IBatchPTRValue -> E.Value
|
|
||||||
encodeIBatchPTRValue =
|
|
||||||
E.int
|
|
||||||
|
|
||||||
|
|
||||||
encodeIToken : IToken -> E.Value
|
|
||||||
encodeIToken itoken =
|
|
||||||
E.object
|
|
||||||
[ ( "name", encodeTokenValue itoken.name )
|
|
||||||
, ( "starts", E.set encodeIBatchPTRValue itoken.starts )
|
|
||||||
, ( "ends", E.set encodeIBatchPTRValue itoken.ends )
|
|
||||||
, ( "inFrontOf", E.set encodeITokenPTRValue itoken.inFrontOf )
|
|
||||||
, ( "behind", E.set encodeITokenPTRValue itoken.behind )
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
encodeITokenPTR : ITokenPTR -> E.Value
|
|
||||||
encodeITokenPTR token =
|
|
||||||
case token of
|
|
||||||
ITokenPTR value ->
|
|
||||||
encodeITokenPTRValue value
|
|
||||||
|
|
||||||
StartOfTimeline ->
|
|
||||||
E.null
|
|
||||||
|
|
||||||
|
|
||||||
encodeITokenPTRValue : ITokenPTRValue -> E.Value
|
|
||||||
encodeITokenPTRValue =
|
|
||||||
E.string
|
|
||||||
|
|
||||||
|
|
||||||
encodeTokenValue : TokenValue -> E.Value
|
|
||||||
encodeTokenValue =
|
|
||||||
E.string
|
|
||||||
|
|
||||||
|
|
||||||
{-| Get an IBatch from the Timeline.
|
{-| Get an IBatch from the Timeline.
|
||||||
-}
|
-}
|
||||||
getIBatch : IBatchPTR -> Timeline -> Maybe IBatch
|
getIBatch : IBatchPTR -> Timeline -> Maybe IBatch
|
||||||
|
|
@ -516,9 +379,47 @@ invokeIToken value (Timeline tl) =
|
||||||
|
|
||||||
{-| Under a given filter, find the most recent events.
|
{-| Under a given filter, find the most recent events.
|
||||||
-}
|
-}
|
||||||
mostRecentEvents : Filter -> Timeline -> List String
|
mostRecentEvents : Filter -> Timeline -> List (List String)
|
||||||
mostRecentEvents _ _ =
|
mostRecentEvents filter (Timeline timeline) =
|
||||||
[]
|
mostRecentEventsFrom filter (Timeline timeline) timeline.mostRecentBatch
|
||||||
|
|
||||||
|
|
||||||
|
{-| Under a given filter, starting from a given ITokenPTR, find the most recent
|
||||||
|
events.
|
||||||
|
-}
|
||||||
|
mostRecentEventsFrom : Filter -> Timeline -> ITokenPTR -> List (List String)
|
||||||
|
mostRecentEventsFrom filter timeline ptr =
|
||||||
|
Recursion.runRecursion
|
||||||
|
(\p ->
|
||||||
|
case getITokenFromPTR p.ptr timeline of
|
||||||
|
Nothing ->
|
||||||
|
Recursion.base []
|
||||||
|
|
||||||
|
Just token ->
|
||||||
|
if Set.member token.name p.visited then
|
||||||
|
Recursion.base []
|
||||||
|
|
||||||
|
else
|
||||||
|
token.ends
|
||||||
|
|> Set.toList
|
||||||
|
|> List.filterMap (\bptrv -> getIBatch (IBatchPTR bptrv) timeline)
|
||||||
|
|> List.filter (\ibatch -> Filter.subsetOf ibatch.filter filter)
|
||||||
|
|> Recursion.Traverse.traverseList
|
||||||
|
(\ibatch ->
|
||||||
|
Recursion.recurseThen
|
||||||
|
{ ptr = ibatch.start, visited = Set.insert token.name p.visited }
|
||||||
|
(\optionalTimelines ->
|
||||||
|
optionalTimelines
|
||||||
|
|> List.map
|
||||||
|
(\outTimeline ->
|
||||||
|
List.append outTimeline ibatch.events
|
||||||
|
)
|
||||||
|
|> Recursion.base
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|> Recursion.map List.concat
|
||||||
|
)
|
||||||
|
{ ptr = ptr, visited = Set.empty }
|
||||||
|
|
||||||
|
|
||||||
{-| Recount the Timeline's amount of filled batches. Since the Timeline
|
{-| Recount the Timeline's amount of filled batches. Since the Timeline
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue