Merge pull request #17 from noordstar/3-timeline

Add Timeline as data type

docs/timeline.md

@@ -0,0 +1,138 @@
+# Timeline
+
+Given the complex nature of the Timeline design, it deserves some explanation of
+the design. This document aims to describe how the Elm SDK designs the Timeline,
+so that other projects may learn from it.
+
+## API endpoint disambiguations
+
+Generally speaking, there are a few API endpoints with similar design:
+
+- The [`/sync` endpoint](https://spec.matrix.org/v1.9/client-server-api/#get_matrixclientv3sync),
+which gets the events that the homeserver received most recently.
+- The [`/messages` endpoint](https://spec.matrix.org/v1.9/client-server-api/#get_matrixclientv3roomsroomidmembers),
+which gets any events in the topological order.
+
+As noted in the Matrix spec:
+
+> Events are ordered in this API according to the arrival time of the event on
+> the homeserver. This can conflict with other APIs which order events based on
+> their partial ordering in the event graph. This can result in duplicate events
+> being received (once per distinct API called). Clients SHOULD de-duplicate
+> events based on the event ID when this happens. 
+
+For this reason, the Elm SDK maintains **two independent timelines** that are tied
+together when necessary to form a coherent timeline.
+
+## Elm design
+
+For those unfamiliar, the Elm Architecture breaks into three parts:
+
+- **Model** - the state of the application
+- **View** - a way to turn your state into meaningful information
+- **Update** - a way to update your state based on the Matrix API
+
+Since these concepts are compartmentalized, it is impossible to make an API call
+while executing the **view** function; the Elm SDK must at all times find a way
+to represent its state.
+
+## Timeline
+
+Concerning the Matrix timeline, it is meant to create a representation
+(**Model**) of the timeline, find a way to represent (**View**) it, and find a
+simple way to adjust it with every incoming Matrix API result. (**Update**)
+
+First, we define what a timeline batch is.
+
+### Timeline batch
+
+A timeline batch is something that most Matrix API endpoints return. It is a
+little piece of the timeline and contains the following four pieces of
+information:
+
+1. A list of events that are part of the timeline.
+2. A Filter for which all provided events meet the criteria.
+3. An end batch token that functions as an identifier.
+4. _(Optional.)_ A start token. If not provided, it indicates the start of the
+    timeline.
+
+Here's an example of such a timeline batch:
+
+```
+       |-->[■]->[■]->[●]->[■]->[■]->[●]-->|
+       |                                  |
+       |<---   filter: only ■ and ●   --->|
+       |                                  |
+     start:                              end:
+    <token_1>                          <token_2>
+```
+
+When the Matrix API later returns a batch token that starts with `<token_2>`,
+we know that we can connect it to the batch above and make a longer list of
+events!
+
+At first, this seems quite simple to connect, but there are some difficulties
+that come up along the way.
+
+### Challenge 1: different filters, different locations
+
+When two timeline batches have different filters, we do not know their
+respective location. For example, the following two timeline batches COULD
+overlap, but it is also possible they don't:
+
+```
+       |-->[■]->[■]->[●]->[■]->[■]->[●]-->|
+       |                                  |
+       |<---   filter: only ■ and ●   --->|
+       |                                  |
+     start:                              end:
+    <token_1>                          <token_2>
+
+
+                |-->[★]->[★]->[★]->[★]-->|
+                |                          |
+                |<--   filter: only ★   -->|
+                |                          |
+              start:                      end:
+            <token_3>                    <token_4>
+```
+
+Realistically, there is currently no way of knowing without making more API
+calls. However, just making more API calls isn't a solution in Elm because of
+its architecture.
+
+> **SOLUTION:** As described in the **View** function, we may assume that
+overlapping timeline batches have overlapping events. If they overlap yet have
+no overlapping events, then their filters must be disjoint. If the filters are
+disjoint, we do not care whether they're overlapping.
+
+### Challenge 2: same filters, same spot
+
+Suppose there is a known timeline batch, and we're trying to **Update** the
+timeline to represent the timeline between `<token_1>` and `<token_2>` for a
+different filter:
+
+```
+       |-->[■]->[■]->[●]->[■]->[■]->[●]-->|
+       |                                  |
+       |<---   filter: only ■ and ●   --->|
+       |                                  |
+     start:                              end:
+    <token_1>                          <token_2>
+```
+
+If we wish to know what's in there for a different filter `f`, then:
+
+1. If `f` equals the filter from the timeline batch, we can copy the events.
+2. If `f` is a subfilter of the batch filter (for example: `only ■`) then we can
+    copy the events from the given batch, and then locally filter the events
+    that do no match filter `f`.
+3. If the batch filter is a subfilter of `f`, then we can use an API call
+    between the same batch tokens `<token_1>` and `<token_2>`. In the worst
+    case, we receive the exact same list of events. In another scenario, we
+    might discover far more events and receive some new batch value `<token_3>`
+    in-between `<token_1>` and `<token_2>`.
+4. If neither filter is a subfilter of the other and the two are (at least
+    partially) disjoint, then they do not need to correlate and any other batch
+    values can be chosen.
+

elm.json

@@ -5,15 +5,38 @@
     "license": "EUPL-1.1",
     "version": "2.1.2",
     "exposed-modules": [
+        "Internal.Config.Default",
+        "Internal.Config.Leaks",
+        "Internal.Config.Log",
+        "Internal.Config.Phantom",
+        "Internal.Config.Text",
+        "Internal.Filter.Timeline",
+        "Internal.Tools.DecodeExtra",
+        "Internal.Tools.EncodeExtra",
+        "Internal.Tools.Hashdict",
+        "Internal.Tools.Iddict",
+        "Internal.Tools.Json",
+        "Internal.Tools.Mashdict",
+        "Internal.Tools.Timestamp",
+        "Internal.Tools.VersionControl",
+        "Internal.Values.Context",
+        "Internal.Values.Envelope",
+        "Internal.Values.Event",
+        "Internal.Values.Settings",
+        "Internal.Values.StateManager",
+        "Internal.Values.Timeline",
+        "Internal.Values.Vault",
         "Matrix",
         "Matrix.Event",
-        "Matrix.Settings"
+        "Matrix.Settings",
+        "Types"
     ],
     "elm-version": "0.19.0 <= v < 0.20.0",
     "dependencies": {
         "elm/core": "1.0.0 <= v < 2.0.0",
         "elm/json": "1.0.0 <= v < 2.0.0",
         "elm/time": "1.0.0 <= v < 2.0.0",
+        "micahhahn/elm-safe-recursion": "2.0.0 <= v < 3.0.0",
         "miniBill/elm-fast-dict": "1.0.0 <= v < 2.0.0"
     },
     "test-dependencies": {

src/Internal/Config/Text.elm

@@ -1,5 +1,5 @@
 module Internal.Config.Text exposing
-    ( docs, failures, fields
+    ( docs, failures, fields, mappings, logs
     , accessTokenFoundLocally, accessTokenExpired, accessTokenInvalid
     , versionsFoundLocally, versionsReceived, versionsFailedToDecode
     , unsupportedVersionForEndpoint
@@ -27,7 +27,7 @@ You should only do this if you know what you're doing.
 
 ## Type documentation
 
-@docs docs, failures, fields
+@docs docs, failures, fields, mappings, logs
 
 
 ## API Authentication
@@ -116,9 +116,14 @@ docs :
     , envelope : TypeDocs
     , event : TypeDocs
     , hashdict : TypeDocs
+    , ibatch : TypeDocs
+    , iddict : TypeDocs
+    , itoken : TypeDocs
     , mashdict : TypeDocs
     , settings : TypeDocs
     , stateManager : TypeDocs
+    , timeline : TypeDocs
+    , timelineFilter : TypeDocs
     , unsigned : TypeDocs
     }
 docs =
@@ -148,6 +153,24 @@ docs =
             , "For example, the hashdict can store events and use their event id as their key."
             ]
         }
+    , ibatch =
+        { name = "IBatch"
+        , description =
+            [ "The internal batch tracks a patch of events on the Matrix timeline."
+            ]
+        }
+    , iddict =
+        { name = "Iddict"
+        , description =
+            [ "An iddict automatically handles creating appropriate keys by incrementally assiging a new key to new values."
+            ]
+        }
+    , itoken =
+        { name = "IToken"
+        , description =
+            [ "The IToken connects batches in the timeline and maintains relative order."
+            ]
+        }
     , mashdict =
         { name = "Mashdict"
         , description =
@@ -167,6 +190,18 @@ docs =
             , "Instead of making the user loop through the room's timeline of events, the StateManager offers the user a dictionary-like experience to navigate through the Matrix room state."
             ]
         }
+    , timeline =
+        { name = "Timeline"
+        , description =
+            [ "The Timeline tracks events and orders them in a simple way for the user to view them."
+            ]
+        }
+    , timelineFilter =
+        { name = "Timeline Filter"
+        , description =
+            [ "The Timeline Filter allows the user to be very specific about which events they're interested in."
+            ]
+        }
     , unsigned =
         { name = "Unsigned Data"
         , description =
@@ -179,13 +214,14 @@ docs =
 
 {-| Description of all edge cases where a JSON decoder can fail.
 -}
-failures : { hashdict : Desc, mashdict : Desc }
+failures : { hashdict : Desc, listWithOne : String, mashdict : Desc }
 failures =
     { hashdict =
-        [ "Not all values map to thir respected hash with the given hash function."
+        [ "Not all values map to their respected hash with the given hash function."
         ]
+    , listWithOne = "Expected at least one value in the list - zero found."
     , mashdict =
-        [ "Not all values map to thir respected hash with the given hash function."
+        [ "Not all values map to their respected hash with the given hash function."
         ]
     }
 
@@ -218,11 +254,41 @@ fields :
         , eventType : Desc
         , unsigned : Desc
         }
+    , ibatch :
+        { end : Desc
+        , events : Desc
+        , filter : Desc
+        , start : Desc
+        }
+    , iddict :
+        { cursor : Desc
+        , dict : Desc
+        }
+    , itoken :
+        { behind : Desc
+        , ends : Desc
+        , inFrontOf : Desc
+        , name : Desc
+        , starts : Desc
+        }
     , settings :
         { currentVersion : Desc
         , deviceName : Desc
         , syncTime : Desc
         }
+    , timeline :
+        { batches : Desc
+        , events : Desc
+        , filledBatches : Desc
+        , mostRecentBatch : Desc
+        , tokens : Desc
+        }
+    , timelineFilter :
+        { senders : Desc
+        , sendersAllowOthers : Desc
+        , types : Desc
+        , typesAllowOthers : Desc
+        }
     , unsigned :
         { age : Desc
         , prevContent : Desc
@@ -293,6 +359,45 @@ fields =
             [ "Contains optional extra information about the event."
             ]
         }
+    , ibatch =
+        { end =
+            [ "Pointer to the token that ends the internal batch."
+            ]
+        , events =
+            [ "List of event IDs contained within the internal batch."
+            ]
+        , filter =
+            [ "Filter that indicates how strictly the homeserver has selected when resulting into the given list of events."
+            ]
+        , start =
+            [ "Pointer to the token that starts the internal batch."
+            ]
+        }
+    , iddict =
+        { cursor =
+            [ "To ensure uniqueness of all keys and to prevent the usage of keys that were previously assigned to older values, the iddict tracks which is the smallest non-negative integer that hasn't been used yet."
+            ]
+        , dict =
+            [ "Dictionary that contains all values stored in the iddict."
+            ]
+        }
+    , itoken =
+        { behind =
+            [ "This token is behind all tokens in this field."
+            ]
+        , ends =
+            [ "This token is in front of the batches in this field."
+            ]
+        , inFrontOf =
+            [ "This token is ahead of all tokens in this field."
+            ]
+        , name =
+            [ "Opaque value provided by the homeserver."
+            ]
+        , starts =
+            [ "This token is at the start of the batches in this field."
+            ]
+        }
     , settings =
         { currentVersion =
             [ "Indicates the current version of the Elm SDK."
@@ -304,6 +409,40 @@ fields =
             [ "Indicates the frequency in miliseconds with which the Elm SDK should long-poll the /sync endpoint."
             ]
         }
+    , timeline =
+        { batches =
+            [ "Dictionary storing all event batches in the timeline."
+            ]
+        , events =
+            [ "Mapping that allows us to quickly zoom in on an event."
+            ]
+        , filledBatches =
+            [ "Counter that tracks how many batches are kept by the timeline."
+            , "Batches are only counted if they are filled by at least one event."
+            ]
+        , mostRecentBatch =
+            [ "Tracks the most recent batch that was sent by the homeserver - usually through `/sync`"
+            ]
+        , tokens =
+            [ "Index of all the tokens used to connect event batches on the timeline."
+            ]
+        }
+    , timelineFilter =
+        { senders =
+            [ "A list of senders that is considered an exception to the infinite pool of \"other\" users"
+            ]
+        , sendersAllowOthers =
+            [ "Value that determines whether the infinite pool of others is included."
+            , "If False, only the users mentioned in `senders` are included. If True, then all users who aren't mentioned in `senders` are included."
+            ]
+        , types =
+            [ "A list of event types that is considered an exception to the infinite pool of \"other\" event types."
+            ]
+        , typesAllowOthers =
+            [ "Value that determines whether the infinite pool of others is included."
+            , "If False, only the event types mentioned in `types` are included. If True, then all users who aren't mentioned in `types` are included."
+            ]
+        }
     , unsigned =
         { age =
             [ "The time in milliseconds that has elapsed since the event was sent. This field is generated by the local homeserver, and may be incorrect if the local time on at least one of the two servers is out of sync, which can cause the age to either be negative or greater than it actually is."
@@ -347,6 +486,35 @@ leakingValueFound leaking_value =
     "Found leaking value : " ++ leaking_value
 
 
+{-| These logs might appear during a process where something unexpected has
+happened. Most of these unexpected results, are taken account of by the Elm SDK,
+but logged so that the programmer can do something about it.
+-}
+logs : { keyIsNotAnInt : String -> String }
+logs =
+    { keyIsNotAnInt =
+        \key ->
+            String.concat
+                [ "Encountered a key `"
+                , key
+                , "` that cannot be converted to an Int"
+                ]
+    }
+
+
+{-| Function descriptions
+-}
+mappings : { itokenPTR : TypeDocs }
+mappings =
+    { itokenPTR =
+        { name = "ITokenPTR init"
+        , description =
+            [ "Converts an optional string to an Itoken pointer."
+            ]
+        }
+    }
+
+
 {-| The Matrix homeserver can specify how it wishes to communicate, and the Elm
 SDK aims to communicate accordingly. This may fail in some scenarios, however,
 in which case it will throw this error.

src/Internal/Filter/Timeline.elm

@@ -0,0 +1,349 @@
+module Internal.Filter.Timeline exposing
+    ( Filter
+    , pass, onlySenders, allSendersExcept, onlyTypes, allTypesExcept, fail
+    , match, run
+    , and
+    , subsetOf
+    , coder, encode, decoder
+    )
+
+{-|
+
+
+# Timeline filter
+
+The timeline filter creates filters for looking through a timeline, as well as
+for interacting with the Matrix API.
+
+
+## Timeline
+
+@docs Filter
+
+
+## Create
+
+@docs pass, onlySenders, allSendersExcept, onlyTypes, allTypesExcept, fail
+
+
+## Filter
+
+@docs match, run
+
+
+## Combine
+
+@docs and
+
+
+## Compare
+
+@docs subsetOf
+
+
+## JSON coders
+
+@docs coder, encode, decoder
+
+-}
+
+import Internal.Config.Text as Text
+import Internal.Tools.Json as Json
+import Json.Decode as D
+import Json.Encode as E
+import Set exposing (Set)
+
+
+{-| Placeholder Event type so the real Event doesn't need to be imported.
+-}
+type alias Event a =
+    { a | eventType : String, sender : String }
+
+
+{-| The Timeline Filter filters events out of a timeline, guaranteeing that only
+the events that meet the given criteria, meet the requirements.
+-}
+type Filter
+    = Filter
+        { senders : Set String
+        , sendersAllowOthers : Bool
+        , types : Set String
+        , typesAllowOthers : Bool
+        }
+
+
+{-| Allow events from all senders, except if they are on the provided list.
+
+If the list is empty, all events are allowed.
+
+-}
+allSendersExcept : List String -> Filter
+allSendersExcept senders =
+    case senders of
+        [] ->
+            pass
+
+        _ :: _ ->
+            Filter
+                { senders = Set.fromList senders
+                , sendersAllowOthers = True
+                , types = Set.empty
+                , typesAllowOthers = True
+                }
+
+
+{-| Allow events of every event type, except if they are on the provided list.
+
+If the list is empty, all events are allowed.
+
+-}
+allTypesExcept : List String -> Filter
+allTypesExcept types =
+    case types of
+        [] ->
+            pass
+
+        _ :: _ ->
+            Filter
+                { senders = Set.empty
+                , sendersAllowOthers = True
+                , types = Set.fromList types
+                , typesAllowOthers = True
+                }
+
+
+{-| Only allow an event if it meets the criteria of two Filters.
+-}
+and : Filter -> Filter -> Filter
+and (Filter f1) (Filter f2) =
+    let
+        stdAnd : Filter
+        stdAnd =
+            Filter
+                { senders =
+                    case ( f1.sendersAllowOthers, f2.sendersAllowOthers ) of
+                        ( True, True ) ->
+                            Set.union f1.senders f2.senders
+
+                        ( True, False ) ->
+                            Set.diff f2.senders f1.senders
+
+                        ( False, True ) ->
+                            Set.diff f1.senders f2.senders
+
+                        ( False, False ) ->
+                            Set.intersect f1.senders f2.senders
+                , sendersAllowOthers = f1.sendersAllowOthers && f2.sendersAllowOthers
+                , types =
+                    case ( f1.typesAllowOthers, f2.typesAllowOthers ) of
+                        ( True, True ) ->
+                            Set.union f1.types f2.types
+
+                        ( True, False ) ->
+                            Set.diff f2.types f1.types
+
+                        ( False, True ) ->
+                            Set.diff f1.types f2.types
+
+                        ( False, False ) ->
+                            Set.intersect f1.types f2.types
+                , typesAllowOthers = f1.typesAllowOthers && f2.typesAllowOthers
+                }
+    in
+    case stdAnd of
+        Filter f ->
+            if Set.isEmpty f.senders && not f.sendersAllowOthers then
+                fail
+
+            else if Set.isEmpty f.types && not f.typesAllowOthers then
+                fail
+
+            else
+                stdAnd
+
+
+{-| Define how to encode and decode a Timeline Filter to and from a JSON value.
+-}
+coder : Json.Coder Filter
+coder =
+    Json.object4
+        { name = Text.docs.timelineFilter.name
+        , description = Text.docs.timelineFilter.description
+        , init =
+            \a b c d ->
+                Filter
+                    { senders = a
+                    , sendersAllowOthers = b
+                    , types = c
+                    , typesAllowOthers = d
+                    }
+        }
+        (Json.field.optional.withDefault
+            { fieldName = "senders"
+            , toField = \(Filter f) -> f.senders
+            , description = Text.fields.timelineFilter.senders
+            , coder = Json.set Json.string
+            , default = ( Set.empty, [] )
+            , defaultToString = always "[]"
+            }
+        )
+        (Json.field.required
+            { fieldName = "sendersAllowOthers"
+            , toField = \(Filter f) -> f.sendersAllowOthers
+            , description = Text.fields.timelineFilter.sendersAllowOthers
+            , coder = Json.bool
+            }
+        )
+        (Json.field.optional.withDefault
+            { fieldName = "types"
+            , toField = \(Filter f) -> f.types
+            , description = Text.fields.timelineFilter.types
+            , coder = Json.set Json.string
+            , default = ( Set.empty, [] )
+            , defaultToString = always "[]"
+            }
+        )
+        (Json.field.required
+            { fieldName = "typesAllowOthers"
+            , toField = \(Filter f) -> f.typesAllowOthers
+            , description = Text.fields.timelineFilter.typesAllowOthers
+            , coder = Json.bool
+            }
+        )
+
+
+{-| Decode a Filter from a JSON value.
+-}
+decoder : Json.Decoder Filter
+decoder =
+    Json.decode coder
+
+
+{-| Encode a Filter into a JSON value.
+-}
+encode : Json.Encoder Filter
+encode =
+    Json.encode coder
+
+
+{-| Allow no events. This filter is likely quite useless in practice, but it is
+used in the test environment for sanity checks and comparisons.
+-}
+fail : Filter
+fail =
+    Filter
+        { senders = Set.empty
+        , sendersAllowOthers = False
+        , types = Set.empty
+        , typesAllowOthers = False
+        }
+
+
+{-| Determine whether an event passes a filter.
+-}
+match : Filter -> Event a -> Bool
+match (Filter f) { eventType, sender } =
+    let
+        mentionedSender : Bool
+        mentionedSender =
+            Set.member sender f.senders
+
+        mentionedType : Bool
+        mentionedType =
+            Set.member eventType f.types
+    in
+    xor mentionedSender f.sendersAllowOthers
+        && xor mentionedType f.typesAllowOthers
+
+
+{-| Only allow event sent by given senders.
+
+If an empty list is given, no events are allowed.
+
+-}
+onlySenders : List String -> Filter
+onlySenders senders =
+    case senders of
+        [] ->
+            fail
+
+        _ :: _ ->
+            Filter
+                { senders = Set.fromList senders
+                , sendersAllowOthers = False
+                , types = Set.empty
+                , typesAllowOthers = True
+                }
+
+
+{-| Only allow events of a given event type.
+
+If an empty list is given, no events are allowed.
+
+-}
+onlyTypes : List String -> Filter
+onlyTypes types =
+    case types of
+        [] ->
+            fail
+
+        _ :: _ ->
+            Filter
+                { senders = Set.empty
+                , sendersAllowOthers = True
+                , types = Set.fromList types
+                , typesAllowOthers = False
+                }
+
+
+{-| Create a filter that allows all events. This can be useful when trying to
+combine multiple filters, or when simply all events are allowed.
+-}
+pass : Filter
+pass =
+    Filter
+        { senders = Set.empty
+        , sendersAllowOthers = True
+        , types = Set.empty
+        , typesAllowOthers = True
+        }
+
+
+{-| Use a filter on a list of events.
+-}
+run : Filter -> List (Event a) -> List (Event a)
+run f events =
+    List.filter (match f) events
+
+
+{-| Determine whether the second argument is a subset filter of the first
+argument.
+-}
+subsetOf : Filter -> Filter -> Bool
+subsetOf (Filter big) (Filter small) =
+    let
+        isSSof : Set String -> Set String -> Bool
+        isSSof b s =
+            Set.intersect b s == s
+
+        isSubsetFor : ( Bool, Set String ) -> ( Bool, Set String ) -> Bool
+        isSubsetFor ( bb, sb ) ( bs, ss ) =
+            case ( bb, bs ) of
+                ( True, True ) ->
+                    isSSof ss sb
+
+                ( True, False ) ->
+                    Set.isEmpty (Set.intersect sb ss)
+
+                ( False, True ) ->
+                    False
+
+                ( False, False ) ->
+                    isSSof sb ss
+    in
+    isSubsetFor
+        ( big.sendersAllowOthers, big.senders )
+        ( small.sendersAllowOthers, small.senders )
+        && isSubsetFor
+            ( big.typesAllowOthers, big.types )
+            ( small.typesAllowOthers, small.types )

src/Internal/Tools/Hashdict.elm

@@ -3,7 +3,7 @@ module Internal.Tools.Hashdict exposing
     , empty, singleton, insert, remove, removeKey
     , isEmpty, member, memberKey, get, size, isEqual
     , keys, values, toList, fromList
-    , rehash, union
+    , rehash, union, map
     , coder, encode, decoder, softDecoder
     )
 
@@ -35,7 +35,7 @@ This allows you to store values based on an externally defined identifier.
 
 ## Transform
 
-@docs rehash, union
+@docs rehash, union, map
 
 
 ## JSON coders
@@ -189,6 +189,34 @@ keys (Hashdict h) =
     Dict.keys h.values
 
 
+{-| Map a value on a given key. If the outcome of the function changes the hash,
+the operation does nothing.
+-}
+map : String -> (a -> a) -> Hashdict a -> Hashdict a
+map key f (Hashdict h) =
+    Hashdict
+        { h
+            | values =
+                Dict.update
+                    key
+                    (Maybe.map
+                        (\value ->
+                            let
+                                newValue : a
+                                newValue =
+                                    f value
+                            in
+                            if h.hash newValue == h.hash value then
+                                newValue
+
+                            else
+                                value
+                        )
+                    )
+                    h.values
+        }
+
+
 {-| Determine if a value's hash is in a hashdict.
 -}
 member : a -> Hashdict a -> Bool

src/Internal/Tools/Iddict.elm

@@ -3,7 +3,7 @@ module Internal.Tools.Iddict exposing
     , empty, singleton, insert, map, remove
     , isEmpty, member, get, size
     , keys, values
-    , encode, decoder
+    , coder, encode, decoder
     )
 
 {-| The id-dict is a data type that lets us store values in a dictionary using
@@ -36,13 +36,13 @@ do not need to generate identifiers yourself.
 
 ## JSON coders
 
-@docs encode, decoder
+@docs coder, encode, decoder
 
 -}
 
 import FastDict as Dict exposing (Dict)
-import Json.Decode as D
-import Json.Encode as E
+import Internal.Config.Text as Text
+import Internal.Tools.Json as Json
 
 
 {-| The Iddict data type.
@@ -54,41 +54,49 @@ type Iddict a
         }
 
 
+{-| Define how an Iddict can be encoded and decoded to and from a JSON value.
+-}
+coder : Json.Coder a -> Json.Coder (Iddict a)
+coder x =
+    Json.object2
+        { name = Text.docs.iddict.name
+        , description = Text.docs.iddict.description
+        , init =
+            \c d ->
+                Iddict
+                    { cursor =
+                        Dict.keys d
+                            |> List.maximum
+                            |> Maybe.map ((+) 1)
+                            |> Maybe.withDefault 0
+                            |> max (Dict.size d)
+                            |> max c
+                    , dict = d
+                    }
+        }
+        (Json.field.optional.withDefault
+            { fieldName = "cursor"
+            , toField = \(Iddict i) -> i.cursor
+            , description = Text.fields.iddict.cursor
+            , coder = Json.int
+            , default = ( 0, [] )
+            , defaultToString = String.fromInt
+            }
+        )
+        (Json.field.required
+            { fieldName = "dict"
+            , toField = \(Iddict i) -> i.dict
+            , description = Text.fields.iddict.dict
+            , coder = Json.fastIntDict x
+            }
+        )
+
+
 {-| Decode an id-dict from a JSON value.
 -}
-decoder : D.Decoder a -> D.Decoder (Iddict a)
-decoder xDecoder =
-    D.map2
-        (\c pairs ->
-            let
-                dict : Dict Int a
-                dict =
-                    pairs
-                        |> List.filterMap
-                            (\( k, v ) ->
-                                k
-                                    |> String.toInt
-                                    |> Maybe.map (\n -> ( n, v ))
-                            )
-                        |> Dict.fromList
-            in
-            Iddict
-                { cursor =
-                    Dict.keys dict
-                        -- Larger than all values in the list
-                        |> List.map ((+) 1)
-                        |> List.maximum
-                        |> Maybe.withDefault 0
-                        |> max (Dict.size dict)
-                        -- At least the dict size
-                        |> max c
-
-                -- At least the given value
-                , dict = dict
-                }
-        )
-        (D.field "cursor" D.int)
-        (D.field "dict" <| D.keyValuePairs xDecoder)
+decoder : Json.Coder a -> Json.Decoder (Iddict a)
+decoder x =
+    Json.decode (coder x)
 
 
 {-| Create an empty id-dict.
@@ -103,16 +111,9 @@ empty =
 
 {-| Encode an id-dict to a JSON value.
 -}
-encode : (a -> E.Value) -> Iddict a -> E.Value
-encode encodeX (Iddict d) =
-    E.object
-        [ ( "cursor", E.int d.cursor )
-        , ( "dict"
-          , d.dict
-                |> Dict.toCoreDict
-                |> E.dict String.fromInt encodeX
-          )
-        ]
+encode : Json.Coder a -> Json.Encoder (Iddict a)
+encode x =
+    Json.encode (coder x)
 
 
 {-| Get a value from the id-dict using its key.

src/Internal/Tools/Json.elm

@@ -3,7 +3,7 @@ module Internal.Tools.Json exposing
     , Encoder, encode, Decoder, decode, Value
     , succeed, fail, andThen, lazy, map
     , Docs(..), RequiredField(..), toDocs
-    , list, slowDict, fastDict, maybe
+    , list, listWithOne, slowDict, fastDict, fastIntDict, set, maybe
     , Field, field
     , object2, object3, object4, object5, object6, object7, object8, object9, object10, object11
     )
@@ -49,7 +49,7 @@ module to build its encoders and decoders.
 
 ## Data types
 
-@docs list, slowDict, fastDict, maybe
+@docs list, listWithOne, slowDict, fastDict, fastIntDict, set, maybe
 
 
 ## Objects
@@ -68,11 +68,13 @@ Once all fields are constructed, the user can create JSON objects.
 
 import Dict as SlowDict
 import FastDict
-import Internal.Config.Log exposing (Log)
+import Internal.Config.Log as Log exposing (Log)
+import Internal.Config.Text as Text
 import Internal.Tools.DecodeExtra as D
 import Internal.Tools.EncodeExtra as E
 import Json.Decode as D
 import Json.Encode as E
+import Set exposing (Set)
 
 
 {-| A field of type `a` as a subtype of an object `object`.
@@ -139,8 +141,10 @@ type Docs
     | DocsDict Docs
     | DocsFloat
     | DocsInt
+    | DocsIntDict Docs
     | DocsLazy (() -> Docs)
     | DocsList Docs
+    | DocsListWithOne Docs
     | DocsMap (Descriptive { content : Docs })
     | DocsObject
         (Descriptive
@@ -155,6 +159,7 @@ type Docs
         )
     | DocsOptional Docs
     | DocsRiskyMap (Descriptive { content : Docs, failure : List String })
+    | DocsSet Docs
     | DocsString
     | DocsValue
 
@@ -290,6 +295,48 @@ fastDict (Coder old) =
         }
 
 
+{-| Define a fast dict where the keys are integers, not strings.
+-}
+fastIntDict : Coder value -> Coder (FastDict.Dict Int value)
+fastIntDict (Coder old) =
+    Coder
+        { encoder = FastDict.toCoreDict >> E.dict String.fromInt old.encoder
+        , decoder =
+            old.decoder
+                |> D.keyValuePairs
+                |> D.map
+                    (\items ->
+                        ( items
+                            |> List.map (Tuple.mapSecond Tuple.first)
+                            |> List.filterMap
+                                (\( k, v ) ->
+                                    Maybe.map (\a -> ( a, v )) (String.toInt k)
+                                )
+                            |> FastDict.fromList
+                        , List.concat
+                            [ items
+                                |> List.map Tuple.first
+                                |> List.filter
+                                    (\k ->
+                                        case String.toInt k of
+                                            Just _ ->
+                                                True
+
+                                            Nothing ->
+                                                False
+                                    )
+                                |> List.map Text.logs.keyIsNotAnInt
+                                |> List.map Log.log.warn
+                            , items
+                                |> List.map Tuple.second
+                                |> List.concatMap Tuple.second
+                            ]
+                        )
+                    )
+        , docs = DocsIntDict old.docs
+        }
+
+
 {-| Create a new field using any of the three provided options.
 
 For example, suppose we are creating a `Field String User` to represent the
@@ -465,6 +512,32 @@ list (Coder old) =
         }
 
 
+{-| Define a list that has at least one value
+-}
+listWithOne : Coder a -> Coder ( a, List a )
+listWithOne (Coder old) =
+    Coder
+        { encoder = \( h, t ) -> E.list old.encoder (h :: t)
+        , decoder =
+            old.decoder
+                |> D.list
+                |> D.andThen
+                    (\items ->
+                        case items of
+                            [] ->
+                                D.fail Text.failures.listWithOne
+
+                            ( h, l1 ) :: t ->
+                                D.succeed
+                                    ( ( h, List.map Tuple.first items )
+                                    , List.concatMap Tuple.second t
+                                        |> List.append l1
+                                    )
+                    )
+        , docs = DocsListWithOne old.docs
+        }
+
+
 {-| Map a value.
 
 Given that the value needs to be both encoded and decoded, the map function
@@ -1079,6 +1152,28 @@ object11 { name, description, init } fa fb fc fd fe ff fg fh fi fj fk =
         }
 
 
+{-| Define a set.
+-}
+set : Coder comparable -> Coder (Set comparable)
+set (Coder data) =
+    Coder
+        { encoder = E.set data.encoder
+        , decoder =
+            data.decoder
+                |> D.list
+                |> D.map
+                    (\items ->
+                        ( items
+                            |> List.map Tuple.first
+                            |> Set.fromList
+                        , items
+                            |> List.concatMap Tuple.second
+                        )
+                    )
+        , docs = DocsSet data.docs
+        }
+
+
 {-| Define a slow dict from the `elm/core` library.
 -}
 slowDict : Coder value -> Coder (SlowDict.Dict String value)

src/Internal/Tools/Mashdict.elm

@@ -3,7 +3,7 @@ module Internal.Tools.Mashdict exposing
     , empty, singleton, insert, remove, removeKey
     , isEmpty, member, memberKey, get, size, isEqual
     , keys, values, toList, fromList
-    , rehash, union
+    , rehash, union, map
     , coder, encode, decoder, softDecoder
     )
 
@@ -43,7 +43,7 @@ In general, you are advised to learn more about the
 
 ## Transform
 
-@docs rehash, union
+@docs rehash, union, map
 
 
 ## JSON coders
@@ -205,6 +205,34 @@ keys (Mashdict h) =
     Dict.keys h.values
 
 
+{-| Map a value on a given key. If the outcome of the function changes the hash,
+the operation does nothing.
+-}
+map : String -> (a -> a) -> Mashdict a -> Mashdict a
+map key f (Mashdict h) =
+    Mashdict
+        { h
+            | values =
+                Dict.update
+                    key
+                    (Maybe.map
+                        (\value ->
+                            case h.hash (f value) of
+                                Just newHash ->
+                                    if newHash == key then
+                                        f value
+
+                                    else
+                                        value
+
+                                Nothing ->
+                                    value
+                        )
+                    )
+                    h.values
+        }
+
+
 {-| Determine if a value's hash is in a mashdict.
 -}
 member : a -> Mashdict a -> Bool

src/Internal/Values/Event.elm

@@ -2,6 +2,7 @@ module Internal.Values.Event exposing
     ( Event
     , UnsignedData(..), age, prevContent, redactedBecause, transactionId
     , coder, encode, decoder
+    , isEqual
     )
 
 {-|
@@ -24,11 +25,17 @@ of a room.
 
 @docs coder, encode, decoder
 
+
+## Test functions
+
+@docs isEqual
+
 -}
 
 import Internal.Config.Text as Text
 import Internal.Tools.Json as Json
 import Internal.Tools.Timestamp as Timestamp exposing (Timestamp)
+import Json.Encode as E
 
 
 {-| The Event type occurs everywhere on a user's timeline.
@@ -148,6 +155,65 @@ encode =
     Json.encode coder
 
 
+{-| Compare two events and determine whether they're identical. Used mostly for
+testing purposes.
+-}
+isEqual : Event -> Event -> Bool
+isEqual e1 e2 =
+    if e1.eventId /= e2.eventId then
+        False
+
+    else if e1.originServerTs /= e2.originServerTs then
+        False
+
+    else if e1.roomId /= e2.roomId then
+        False
+
+    else if e1.sender /= e2.sender then
+        False
+
+    else if e1.stateKey /= e2.stateKey then
+        False
+
+    else if e1.eventType /= e2.eventType then
+        False
+
+    else
+        case ( e1.unsigned, e2.unsigned ) of
+            ( Nothing, Nothing ) ->
+                True
+
+            ( Just _, Nothing ) ->
+                False
+
+            ( Nothing, Just _ ) ->
+                False
+
+            ( Just (UnsignedData d1), Just (UnsignedData d2) ) ->
+                if d1.age /= d2.age then
+                    False
+
+                else if d1.transactionId /= d2.transactionId then
+                    False
+
+                else if Maybe.map (E.encode 0) d1.prevContent /= Maybe.map (E.encode 0) d2.prevContent then
+                    False
+
+                else
+                    case ( d1.redactedBecause, d2.redactedBecause ) of
+                        ( Nothing, Nothing ) ->
+                            True
+
+                        ( Nothing, Just _ ) ->
+                            False
+
+                        ( Just _, Nothing ) ->
+                            False
+
+                        ( Just se1, Just se2 ) ->
+                            isEqual se1 se2
+
+
 {-| Determine the previous `content` value for this event. This field is only a
 `Just value` if the event is a state event, and the Matrix Vault has permission
 to see the previous content.

src/Internal/Values/Timeline.elm

@@ -0,0 +1,707 @@
+module Internal.Values.Timeline exposing
+    ( Batch, Timeline
+    , empty, singleton
+    , mostRecentEvents, mostRecentEventsFrom
+    , addSync, insert
+    , coder, encode, decoder
+    )
+
+{-|
+
+
+# Timeline
+
+The Timeline data type represents a timeline in the Matrix room. The Matrix room
+timeline is quite a complex data type, as it is constantly only partially known
+by the Matrix client. This module exposes a data type that helps explore, track
+and maintain this room state.
+
+This design of the timeline uses the batches as waypoints to maintain an order.
+The Matrix API often returns batches that have the following four pieces of
+information:
+
+1.  A list of events.
+2.  A filter for which all of the events meet the criteria.
+3.  An end batch token.
+4.  _(Optional)_ A start batch token. If it is not provided, it is the start of
+    the timeline.
+
+Here's an example of such a timeline batch:
+
+           |-->[■]->[■]->[●]->[■]->[■]->[●]-->|
+           |                                  |
+           |<-- filter: only ■ and ●, no ★ -->|
+           |                                  |
+         start:                              end:
+        <token_1>                          <token_2>
+
+When the Matrix API later returns a batch token that starts with `<token_2>`,
+we know that we can connect it to the batch above and make a longer list of
+events!
+
+
+## Batch
+
+@docs Batch, Timeline
+
+
+## Create
+
+@docs empty, singleton
+
+
+## Query
+
+@docs mostRecentEvents, mostRecentEventsFrom
+
+
+## Manipulate
+
+@docs addSync, insert
+
+
+## JSON coder
+
+@docs coder, encode, decoder
+
+-}
+
+import FastDict as Dict exposing (Dict)
+import Internal.Config.Text as Text
+import Internal.Filter.Timeline as Filter exposing (Filter)
+import Internal.Tools.Hashdict as Hashdict exposing (Hashdict)
+import Internal.Tools.Iddict as Iddict exposing (Iddict)
+import Internal.Tools.Json as Json
+import Recursion
+import Recursion.Traverse
+import Set exposing (Set)
+
+
+{-| A batch is a batch of events that is placed onto the Timeline. Functions
+that require an insertion, generally require this data type.
+
+If the `start` value is `Nothing`, it is either the start of the timeline or the
+start of the timeline part that the user is allowed to view.
+
+-}
+type alias Batch =
+    { events : List String
+    , filter : Filter
+    , start : Maybe TokenValue
+    , end : TokenValue
+    }
+
+
+{-| Internal batch that's being saved by the Timeline to track a list of events.
+-}
+type alias IBatch =
+    { events : List String
+    , filter : Filter
+    , start : ITokenPTR
+    , end : ITokenPTR
+    }
+
+
+{-| Pointer to an IBatch in the Timeline.
+-}
+type IBatchPTR
+    = IBatchPTR IBatchPTRValue
+
+
+{-| Location indicator of an IBatch in the Timeline.
+-}
+type alias IBatchPTRValue =
+    Int
+
+
+{-| Internal token value that's being stored by the Timeline.
+
+If name is `Nothing`, it indicates the start of the timeline.
+
+-}
+type alias IToken =
+    { name : TokenValue
+    , starts : Set IBatchPTRValue -- This itoken starts the following batches
+    , ends : Set IBatchPTRValue -- This itoken ends the following batches
+    , inFrontOf : Set ITokenPTRValue -- This itoken is in front of the following tokens
+    , behind : Set ITokenPTRValue -- This itoken is behind the following tokens
+    }
+
+
+{-| Pointer to an IToken in the Timeline.
+-}
+type ITokenPTR
+    = ITokenPTR ITokenPTRValue
+    | StartOfTimeline
+
+
+{-| Location indicator of an IToken in the Timeline.
+-}
+type alias ITokenPTRValue =
+    String
+
+
+{-| The Timeline type represents the timeline state in a Matrix room.
+
+Following the description of the Matrix spec, a timeline contains the following
+items:
+
+  - Events that indicate timeline events
+  - Batch values that can be used to paginate through the timeline
+
+The topological shape of the timeline makes older API responses somewhat
+unreliable - as a result,
+
+-}
+type Timeline
+    = Timeline
+        { batches : Iddict IBatch
+        , events : Dict String ( IBatchPTR, List IBatchPTR )
+        , filledBatches : Int
+        , mostRecentBatch : ITokenPTR
+        , tokens : Hashdict IToken
+        }
+
+
+{-| Opaque token value sent by the Matrix API
+-}
+type alias TokenValue =
+    String
+
+
+{-| Add a new batch as a sync
+-}
+addSync : Batch -> Timeline -> Timeline
+addSync batch timeline =
+    case insertBatch batch timeline of
+        ( Timeline t, { start, end } ) ->
+            let
+                old : ITokenPTR
+                old =
+                    t.mostRecentBatch
+            in
+            case Timeline { t | mostRecentBatch = end } of
+                tl ->
+                    if old == start then
+                        tl
+
+                    else
+                        connectITokenToIToken old start tl
+
+
+{-| Define how a Timeline can be encoded and decoded to and from a JSON value.
+-}
+coder : Json.Coder Timeline
+coder =
+    Json.object5
+        { name = Text.docs.timeline.name
+        , description = Text.docs.timeline.description
+        , init =
+            \a b c d e ->
+                Timeline
+                    { batches = a
+                    , events = b
+                    , filledBatches = c
+                    , mostRecentBatch = d
+                    , tokens = e
+                    }
+        }
+        (Json.field.required
+            { fieldName = "batches"
+            , toField = \(Timeline t) -> t.batches
+            , description = Text.fields.timeline.batches
+            , coder = Iddict.coder coderIBatch
+            }
+        )
+        (Json.field.required
+            { fieldName = "events"
+            , toField = \(Timeline t) -> t.events
+            , description = Text.fields.timeline.events
+            , coder = Json.fastDict (Json.listWithOne coderIBatchPTR)
+            }
+        )
+        (Json.field.optional.withDefault
+            { fieldName = "filledBatches"
+            , toField = \(Timeline t) -> t.filledBatches
+            , description = Text.fields.timeline.filledBatches
+            , coder = Json.int
+            , default = ( 0, [] )
+            , defaultToString = String.fromInt
+            }
+        )
+        (Json.field.required
+            { fieldName = "mostRecentBatch"
+            , toField = \(Timeline t) -> t.mostRecentBatch
+            , description = Text.fields.timeline.mostRecentBatch
+            , coder = coderITokenPTR
+            }
+        )
+        (Json.field.required
+            { fieldName = "tokens"
+            , toField = \(Timeline t) -> t.tokens
+            , description = Text.fields.timeline.tokens
+            , coder = Hashdict.coder .name coderIToken
+            }
+        )
+
+
+{-| Define how to encode and decode a IBatch to and from a JSON value.
+-}
+coderIBatch : Json.Coder IBatch
+coderIBatch =
+    Json.object4
+        { name = Text.docs.ibatch.name
+        , description = Text.docs.ibatch.description
+        , init = IBatch
+        }
+        (Json.field.required
+            { fieldName = "events"
+            , toField = .events
+            , description = Text.fields.ibatch.events
+            , coder = Json.list Json.string
+            }
+        )
+        (Json.field.required
+            { fieldName = "filter"
+            , toField = .filter
+            , description = Text.fields.ibatch.filter
+            , coder = Filter.coder
+            }
+        )
+        (Json.field.required
+            { fieldName = "start"
+            , toField = .start
+            , description = Text.fields.ibatch.start
+            , coder = coderITokenPTR
+            }
+        )
+        (Json.field.required
+            { fieldName = "end"
+            , toField = .end
+            , description = Text.fields.ibatch.end
+            , coder = coderITokenPTR
+            }
+        )
+
+
+{-| Define how to encode and decode a IBatchPTR to and from a JSON value.
+-}
+coderIBatchPTR : Json.Coder IBatchPTR
+coderIBatchPTR =
+    Json.map
+        { name = Text.docs.itoken.name
+        , description = Text.docs.itoken.description
+        , back = \(IBatchPTR value) -> value
+        , forth = IBatchPTR
+        }
+        coderIBatchPTRValue
+
+
+{-| Define how to encode and decode a IBatchPTRValue to and from a JSON value.
+-}
+coderIBatchPTRValue : Json.Coder IBatchPTRValue
+coderIBatchPTRValue =
+    Json.int
+
+
+{-| Define how to encode and decode a IToken to and from a JSON value.
+-}
+coderIToken : Json.Coder IToken
+coderIToken =
+    Json.object5
+        { name = Text.docs.itoken.name
+        , description = Text.docs.itoken.description
+        , init = IToken
+        }
+        (Json.field.required
+            { fieldName = "name"
+            , toField = .name
+            , description = Text.fields.itoken.name
+            , coder = coderTokenValue
+            }
+        )
+        (Json.field.optional.withDefault
+            { fieldName = "starts"
+            , toField = .starts
+            , description = Text.fields.itoken.starts
+            , coder = Json.set coderIBatchPTRValue
+            , default = ( Set.empty, [] )
+            , defaultToString = always "[]"
+            }
+        )
+        (Json.field.optional.withDefault
+            { fieldName = "ends"
+            , toField = .ends
+            , description = Text.fields.itoken.ends
+            , coder = Json.set coderIBatchPTRValue
+            , default = ( Set.empty, [] )
+            , defaultToString = always "[]"
+            }
+        )
+        (Json.field.optional.withDefault
+            { fieldName = "inFrontOf"
+            , toField = .inFrontOf
+            , description = Text.fields.itoken.inFrontOf
+            , coder = Json.set coderITokenPTRValue
+            , default = ( Set.empty, [] )
+            , defaultToString = always "[]"
+            }
+        )
+        (Json.field.optional.withDefault
+            { fieldName = "behind"
+            , toField = .behind
+            , description = Text.fields.itoken.behind
+            , coder = Json.set coderITokenPTRValue
+            , default = ( Set.empty, [] )
+            , defaultToString = always "[]"
+            }
+        )
+
+
+{-| Define how to encode and decode a ITokenPTR to and from a JSON value.
+-}
+coderITokenPTR : Json.Coder ITokenPTR
+coderITokenPTR =
+    Json.maybe coderITokenPTRValue
+        |> Json.map
+            { name = Text.mappings.itokenPTR.name
+            , description = Text.mappings.itokenPTR.description
+            , back =
+                \itokenptr ->
+                    case itokenptr of
+                        ITokenPTR name ->
+                            Just name
+
+                        StartOfTimeline ->
+                            Nothing
+            , forth =
+                \value ->
+                    case value of
+                        Just name ->
+                            ITokenPTR name
+
+                        Nothing ->
+                            StartOfTimeline
+            }
+
+
+{-| Define how to encode and decode a ITokenPTRValue to and from a JSON value.
+-}
+coderITokenPTRValue : Json.Coder ITokenPTRValue
+coderITokenPTRValue =
+    Json.string
+
+
+{-| Define how to encode and decode a TokenValue to and from a JSON value.
+-}
+coderTokenValue : Json.Coder TokenValue
+coderTokenValue =
+    Json.string
+
+
+{-| Append a token at the end of a batch.
+-}
+connectIBatchToIToken : IBatchPTR -> ITokenPTR -> Timeline -> Timeline
+connectIBatchToIToken (IBatchPTR bptr) pointer (Timeline tl) =
+    case pointer of
+        StartOfTimeline ->
+            Timeline tl
+
+        ITokenPTR tptr ->
+            Timeline
+                { tl
+                    | batches =
+                        Iddict.map bptr
+                            (\batch -> { batch | end = pointer })
+                            tl.batches
+                    , tokens =
+                        Hashdict.map tptr
+                            (\token -> { token | ends = Set.insert bptr token.ends })
+                            tl.tokens
+                }
+
+
+{-| Append a token at the start of a batch.
+-}
+connectITokenToIBatch : ITokenPTR -> IBatchPTR -> Timeline -> Timeline
+connectITokenToIBatch pointer (IBatchPTR bptr) (Timeline tl) =
+    case pointer of
+        StartOfTimeline ->
+            Timeline tl
+
+        ITokenPTR tptr ->
+            Timeline
+                { tl
+                    | tokens =
+                        Hashdict.map tptr
+                            (\token -> { token | starts = Set.insert bptr token.starts })
+                            tl.tokens
+                    , batches =
+                        Iddict.map bptr
+                            (\batch -> { batch | start = pointer })
+                            tl.batches
+                }
+
+
+{-| Connect two tokens to each other, revealing their relative location.
+-}
+connectITokenToIToken : ITokenPTR -> ITokenPTR -> Timeline -> Timeline
+connectITokenToIToken pointer1 pointer2 (Timeline tl) =
+    case ( pointer1, pointer2 ) of
+        ( ITokenPTR early, ITokenPTR late ) ->
+            if early == late then
+                Timeline tl
+
+            else
+                Timeline
+                    { tl
+                        | tokens =
+                            tl.tokens
+                                |> Hashdict.map early
+                                    (\data ->
+                                        { data | behind = Set.insert late data.behind }
+                                    )
+                                |> Hashdict.map late
+                                    (\data ->
+                                        { data | inFrontOf = Set.insert early data.inFrontOf }
+                                    )
+                    }
+
+        ( _, _ ) ->
+            Timeline tl
+
+
+{-| Timeline JSON decoder that helps decode a Timeline from JSON.
+-}
+decoder : Json.Decoder Timeline
+decoder =
+    Json.decode coder
+
+
+{-| Create a new empty timeline.
+-}
+empty : Timeline
+empty =
+    Timeline
+        { batches = Iddict.empty
+        , events = Dict.empty
+        , filledBatches = 0
+        , mostRecentBatch = StartOfTimeline
+        , tokens = Hashdict.empty .name
+        }
+
+
+{-| Directly encode a Timeline into a JSON value.
+-}
+encode : Json.Encoder Timeline
+encode =
+    Json.encode coder
+
+
+{-| Get an IBatch from the Timeline.
+-}
+getIBatch : IBatchPTR -> Timeline -> Maybe IBatch
+getIBatch (IBatchPTR ptr) (Timeline { batches }) =
+    Iddict.get ptr batches
+
+
+{-| Get an IToken from the Timeline.
+-}
+getITokenFromPTR : ITokenPTR -> Timeline -> Maybe IToken
+getITokenFromPTR pointer (Timeline { tokens }) =
+    case pointer of
+        ITokenPTR ptr ->
+            Hashdict.get ptr tokens
+
+        StartOfTimeline ->
+            Nothing
+
+
+{-| Insert a batch anywhere else in the timeline.
+-}
+insert : Batch -> Timeline -> Timeline
+insert batch timeline =
+    timeline
+        |> insertBatch batch
+        |> Tuple.first
+
+
+{-| Insert a batch into the timeline.
+-}
+insertBatch : Batch -> Timeline -> ( Timeline, { start : ITokenPTR, end : ITokenPTR } )
+insertBatch batch timeline =
+    case batch.start of
+        Just start ->
+            timeline
+                |> invokeIToken start
+                |> Tuple.mapSecond (invokeIToken batch.end)
+                |> (\( startPTR, ( endPTR, newTimeline ) ) ->
+                        ( insertIBatch
+                            { events = batch.events
+                            , filter = batch.filter
+                            , start = startPTR
+                            , end = endPTR
+                            }
+                            newTimeline
+                        , { start = startPTR, end = endPTR }
+                        )
+                   )
+
+        Nothing ->
+            timeline
+                |> invokeIToken batch.end
+                |> (\( endPTR, newTimeline ) ->
+                        ( insertIBatch
+                            { events = batch.events
+                            , filter = batch.filter
+                            , start = StartOfTimeline
+                            , end = endPTR
+                            }
+                            newTimeline
+                        , { start = StartOfTimeline, end = endPTR }
+                        )
+                   )
+
+
+{-| Insert an internal batch into the timeline, and determine its result.
+-}
+insertIBatch : IBatch -> Timeline -> Timeline
+insertIBatch ibatch (Timeline tl) =
+    case Iddict.insert ibatch tl.batches of
+        ( batchPTR, newBatches ) ->
+            { tl
+                | batches = newBatches
+                , events =
+                    List.foldl
+                        (\event dict ->
+                            Dict.update event
+                                (\value ->
+                                    case value of
+                                        Nothing ->
+                                            Just ( IBatchPTR batchPTR, [] )
+
+                                        Just ( head, tail ) ->
+                                            Just ( IBatchPTR batchPTR, head :: tail )
+                                )
+                                dict
+                        )
+                        tl.events
+                        ibatch.events
+                , filledBatches =
+                    if List.isEmpty ibatch.events then
+                        tl.filledBatches
+
+                    else
+                        tl.filledBatches + 1
+            }
+                |> Timeline
+                |> connectITokenToIBatch ibatch.start (IBatchPTR batchPTR)
+                |> connectIBatchToIToken (IBatchPTR batchPTR) ibatch.end
+
+
+{-| Invoke an itoken to guarantee that it exists.
+-}
+invokeIToken : TokenValue -> Timeline -> ( ITokenPTR, Timeline )
+invokeIToken value (Timeline tl) =
+    ( ITokenPTR value
+    , Timeline
+        { tl
+            | tokens =
+                case Hashdict.get value tl.tokens of
+                    Just _ ->
+                        tl.tokens
+
+                    Nothing ->
+                        Hashdict.insert
+                            { name = value
+                            , starts = Set.empty
+                            , ends = Set.empty
+                            , inFrontOf = Set.empty
+                            , behind = Set.empty
+                            }
+                            tl.tokens
+        }
+    )
+
+
+{-| Under a given filter, find the most recent events.
+-}
+mostRecentEvents : Filter -> Timeline -> List (List String)
+mostRecentEvents filter (Timeline timeline) =
+    mostRecentFrom filter (Timeline timeline) timeline.mostRecentBatch
+
+
+{-| Instead of finding the most recent events from the latest sync, users can
+also find the most recent events given a token value.
+-}
+mostRecentEventsFrom : Filter -> ITokenPTRValue -> Timeline -> List (List String)
+mostRecentEventsFrom filter tokenName timeline =
+    mostRecentFrom filter timeline (ITokenPTR tokenName)
+
+
+{-| Under a given filter, starting from a given ITokenPTR, find the most recent
+events.
+-}
+mostRecentFrom : Filter -> Timeline -> ITokenPTR -> List (List String)
+mostRecentFrom filter timeline ptr =
+    Recursion.runRecursion
+        (\p ->
+            case getITokenFromPTR p.ptr timeline of
+                Nothing ->
+                    Recursion.base []
+
+                Just token ->
+                    if Set.member token.name p.visited then
+                        Recursion.base []
+
+                    else
+                        token.ends
+                            |> Set.toList
+                            |> List.filterMap (\bptrv -> getIBatch (IBatchPTR bptrv) timeline)
+                            |> List.filter (\ibatch -> Filter.subsetOf ibatch.filter filter)
+                            |> Recursion.Traverse.traverseList
+                                (\ibatch ->
+                                    Recursion.recurseThen
+                                        { ptr = ibatch.start, visited = Set.insert token.name p.visited }
+                                        (\optionalTimelines ->
+                                            case optionalTimelines of
+                                                [] ->
+                                                    List.singleton ibatch.events
+                                                        |> Recursion.base
+
+                                                _ :: _ ->
+                                                    optionalTimelines
+                                                        |> List.map
+                                                            (\outTimeline ->
+                                                                List.append outTimeline ibatch.events
+                                                            )
+                                                        |> Recursion.base
+                                        )
+                                )
+                            |> Recursion.map List.concat
+        )
+        { ptr = ptr, visited = Set.empty }
+
+
+{-| Recount the Timeline's amount of filled batches. Since the Timeline
+automatically tracks the count on itself, this is generally exclusively used in
+specific scenarios like decoding JSON values.
+-}
+recountFilledBatches : Timeline -> Timeline
+recountFilledBatches (Timeline tl) =
+    Timeline
+        { tl
+            | filledBatches =
+                tl.batches
+                    |> Iddict.values
+                    |> List.filter (\v -> v.events /= [])
+                    |> List.length
+        }
+
+
+{-| Create a timeline with a single batch inserted. This batch is considered the
+most recent batch, as if created by a sync.
+-}
+singleton : Batch -> Timeline
+singleton b =
+    insert b empty

tests/Test/Filter/Timeline.elm

@@ -0,0 +1,434 @@
+module Test.Filter.Timeline exposing (..)
+
+import Expect
+import Fuzz exposing (Fuzzer)
+import Internal.Filter.Timeline as Filter exposing (Filter)
+import Internal.Values.Event as Event
+import Json.Decode as D
+import Json.Encode as E
+import Set
+import Test exposing (..)
+import Test.Values.Event as TestEvent
+
+
+fuzzer : Fuzzer Filter
+fuzzer =
+    Fuzz.map2 Filter.and
+        (Fuzz.oneOf
+            [ Fuzz.map Filter.allSendersExcept (Fuzz.list Fuzz.string)
+            , Fuzz.map Filter.onlySenders (Fuzz.list Fuzz.string)
+            , Fuzz.constant Filter.pass
+            ]
+        )
+        (Fuzz.oneOf
+            [ Fuzz.map Filter.allTypesExcept (Fuzz.list Fuzz.string)
+            , Fuzz.map Filter.onlyTypes (Fuzz.list Fuzz.string)
+            , Fuzz.constant Filter.pass
+            ]
+        )
+
+
+suite : Test
+suite =
+    describe "Timeline filter"
+        [ describe "Tautological equivalences"
+            [ test "Pass /= fail"
+                (Filter.pass
+                    |> Expect.notEqual Filter.fail
+                    |> always
+                )
+            , test "All senders == pass"
+                (Filter.allSendersExcept []
+                    |> Expect.equal Filter.pass
+                    |> always
+                )
+            , test "All types == pass"
+                (Filter.allTypesExcept []
+                    |> Expect.equal Filter.pass
+                    |> always
+                )
+            , test "No senders == fail"
+                (Filter.onlySenders []
+                    |> Expect.equal Filter.fail
+                    |> always
+                )
+            , test "No types == fail"
+                (Filter.onlyTypes []
+                    |> Expect.equal Filter.fail
+                    |> always
+                )
+            , fuzz2 Fuzz.string
+                (Fuzz.list Fuzz.string)
+                "Some types /= some senders"
+                (\head tail ->
+                    Expect.notEqual
+                        (Filter.onlyTypes (head :: tail))
+                        (Filter.onlySenders (head :: tail))
+                )
+            , fuzz2 fuzzer
+                fuzzer
+                "Filter.and f1 f2 == pass iff f1 == f2 == pass"
+                (\filter1 filter2 ->
+                    Expect.equal
+                        (Filter.and filter1 filter2 == Filter.pass)
+                        (filter1 == Filter.pass && filter2 == Filter.pass)
+                )
+            ]
+        , describe "Event filters"
+            [ fuzz TestEvent.fuzzer
+                "Only event type filter matches"
+                (\event ->
+                    event
+                        |> Filter.match (Filter.onlyTypes [ event.eventType ])
+                        |> Expect.equal True
+                )
+            , fuzz TestEvent.fuzzer
+                "Only event sender filter matches"
+                (\event ->
+                    event
+                        |> Filter.match (Filter.onlySenders [ event.sender ])
+                        |> Expect.equal True
+                )
+            , fuzz TestEvent.fuzzer
+                "Not event type filter doesn't match"
+                (\event ->
+                    event
+                        |> Filter.match (Filter.allTypesExcept [ event.eventType ])
+                        |> Expect.equal False
+                )
+            , fuzz TestEvent.fuzzer
+                "Not event sender filter doesn't match"
+                (\event ->
+                    event
+                        |> Filter.match (Filter.allSendersExcept [ event.sender ])
+                        |> Expect.equal False
+                )
+            , fuzz2 TestEvent.fuzzer
+                (Fuzz.list Fuzz.string)
+                "Only matches when in sender list"
+                (\event senders ->
+                    event
+                        |> Filter.match (Filter.onlySenders senders)
+                        |> Expect.equal (List.member event.sender senders)
+                )
+            , fuzz2 TestEvent.fuzzer
+                (Fuzz.list Fuzz.string)
+                "Only matches when in type list"
+                (\event types ->
+                    event
+                        |> Filter.match (Filter.onlyTypes types)
+                        |> Expect.equal (List.member event.eventType types)
+                )
+            , fuzz2 TestEvent.fuzzer
+                (Fuzz.list Fuzz.string)
+                "All except doesn't match when in sender list"
+                (\event senders ->
+                    event
+                        |> Filter.match (Filter.allSendersExcept senders)
+                        |> Expect.notEqual (List.member event.sender senders)
+                )
+            , fuzz2 TestEvent.fuzzer
+                (Fuzz.list Fuzz.string)
+                "All except doesn't match when in type list"
+                (\event types ->
+                    event
+                        |> Filter.match (Filter.allTypesExcept types)
+                        |> Expect.notEqual (List.member event.eventType types)
+                )
+            , fuzz (Fuzz.list Fuzz.string)
+                "Only list AND all except list = fail senders"
+                (\senders ->
+                    Filter.onlySenders senders
+                        |> Filter.and (Filter.allSendersExcept senders)
+                        |> Expect.equal Filter.fail
+                )
+            , fuzz (Fuzz.list Fuzz.string)
+                "Only list AND all except list = fail types"
+                (\types ->
+                    Filter.onlyTypes types
+                        |> Filter.and (Filter.allTypesExcept types)
+                        |> Expect.equal Filter.fail
+                )
+            , fuzz2 (Fuzz.list Fuzz.string)
+                (Fuzz.list Fuzz.string)
+                "Only list + all except list = common types"
+                (\t1 t2 ->
+                    Expect.equal
+                        (Filter.and
+                            (Filter.onlyTypes t1)
+                            (Filter.allTypesExcept t2)
+                        )
+                        (Set.diff (Set.fromList t1) (Set.fromList t2)
+                            |> Set.toList
+                            |> Filter.onlyTypes
+                        )
+                )
+            , fuzz2 (Fuzz.list Fuzz.string)
+                (Fuzz.list Fuzz.string)
+                "Only list + all except list = common senders"
+                (\t1 t2 ->
+                    Expect.equal
+                        (Filter.and
+                            (Filter.onlySenders t1)
+                            (Filter.allSendersExcept t2)
+                        )
+                        (Set.diff (Set.fromList t1) (Set.fromList t2)
+                            |> Set.toList
+                            |> Filter.onlySenders
+                        )
+                )
+            ]
+        , describe "Subset testing"
+            [ fuzz2 fuzzer
+                fuzzer
+                "Combining two filters is always a subset"
+                (\filter1 filter2 ->
+                    filter1
+                        |> Filter.and filter2
+                        |> Expect.all
+                            [ Filter.subsetOf filter1 >> Expect.equal True
+                            , Filter.subsetOf filter2 >> Expect.equal True
+                            ]
+                )
+            , fuzz
+                (Fuzz.bool
+                    |> Fuzz.andThen
+                        (\same ->
+                            if same then
+                                Fuzz.map (\a -> ( a, a )) fuzzer
+
+                            else
+                                Fuzz.map2 Tuple.pair fuzzer fuzzer
+                        )
+                )
+                "subset goes both way iff equal"
+                (\( filter1, filter2 ) ->
+                    Expect.equal
+                        (filter1 == filter2)
+                        (Filter.subsetOf filter1 filter2
+                            && Filter.subsetOf filter2 filter1
+                        )
+                )
+            , fuzz2 Fuzz.string
+                (Fuzz.list Fuzz.string)
+                "One more excluded sender is a subset"
+                (\head tail ->
+                    Filter.allSendersExcept (head :: tail)
+                        |> Filter.subsetOf (Filter.allSendersExcept tail)
+                        |> Expect.equal True
+                )
+            , fuzz2 Fuzz.string
+                (Fuzz.list Fuzz.string)
+                "One more excluded type is a subset"
+                (\head tail ->
+                    Filter.allTypesExcept (head :: tail)
+                        |> Filter.subsetOf (Filter.allTypesExcept tail)
+                        |> Expect.equal True
+                )
+            , fuzz2 Fuzz.string
+                (Fuzz.list Fuzz.string)
+                "One less included sender is a subset"
+                (\head tail ->
+                    Filter.onlySenders tail
+                        |> Filter.subsetOf (Filter.onlySenders (head :: tail))
+                        |> Expect.equal True
+                )
+            , fuzz2 Fuzz.string
+                (Fuzz.list Fuzz.string)
+                "One less included type is a subset"
+                (\head tail ->
+                    Filter.onlyTypes tail
+                        |> Filter.subsetOf (Filter.onlyTypes (head :: tail))
+                        |> Expect.equal True
+                )
+            , fuzz3 Fuzz.string
+                (Fuzz.list Fuzz.string)
+                fuzzer
+                "One more excluded sender is a subset - even when combined with another fuzzer"
+                (\head tail filter ->
+                    Filter.allSendersExcept (head :: tail)
+                        |> Filter.and filter
+                        |> Filter.subsetOf (Filter.and filter <| Filter.allSendersExcept tail)
+                        |> Expect.equal True
+                )
+            , fuzz3 Fuzz.string
+                (Fuzz.list Fuzz.string)
+                fuzzer
+                "One more excluded type is a subset - even when combined with another fuzzer"
+                (\head tail filter ->
+                    Filter.allTypesExcept (head :: tail)
+                        |> Filter.and filter
+                        |> Filter.subsetOf (Filter.and filter <| Filter.allTypesExcept tail)
+                        |> Expect.equal True
+                )
+            , fuzz3 Fuzz.string
+                (Fuzz.list Fuzz.string)
+                fuzzer
+                "One less included sender is a subset - even when combined with another fuzzer"
+                (\head tail filter ->
+                    Filter.onlySenders tail
+                        |> Filter.and filter
+                        |> Filter.subsetOf (Filter.and filter <| Filter.onlySenders (head :: tail))
+                        |> Expect.equal True
+                )
+            , fuzz3 Fuzz.string
+                (Fuzz.list Fuzz.string)
+                fuzzer
+                "One less included type is a subset - even when combined with another fuzzer"
+                (\head tail filter ->
+                    Filter.onlyTypes tail
+                        |> Filter.and filter
+                        |> Filter.subsetOf (Filter.and filter <| Filter.onlyTypes (head :: tail))
+                        |> Expect.equal True
+                )
+            ]
+        , describe "Use case testing"
+            [ fuzz3 (Fuzz.list TestEvent.fuzzer)
+                (Fuzz.list Fuzz.string)
+                (Fuzz.list Fuzz.string)
+                "Only senders + only type"
+                (\events senders types ->
+                    let
+                        l1 : List Event.Event
+                        l1 =
+                            events
+                                |> Filter.run
+                                    (Filter.and
+                                        (Filter.onlySenders senders)
+                                        (Filter.onlyTypes types)
+                                    )
+
+                        l2 : List Event.Event
+                        l2 =
+                            List.filter
+                                (\e ->
+                                    List.member e.sender senders
+                                        && List.member e.eventType types
+                                )
+                                events
+                    in
+                    Expect.all
+                        [ Expect.equal (List.length l1) (List.length l2)
+                            |> always
+                        , List.map2 Event.isEqual l1 l2
+                            |> List.all identity
+                            |> Expect.equal True
+                            |> always
+                        ]
+                        ()
+                )
+            , fuzz3 (Fuzz.list TestEvent.fuzzer)
+                (Fuzz.list Fuzz.string)
+                (Fuzz.list Fuzz.string)
+                "Only senders + all except type"
+                (\events senders types ->
+                    let
+                        l1 : List Event.Event
+                        l1 =
+                            events
+                                |> Filter.run
+                                    (Filter.and
+                                        (Filter.onlySenders senders)
+                                        (Filter.allTypesExcept types)
+                                    )
+
+                        l2 : List Event.Event
+                        l2 =
+                            List.filter
+                                (\e ->
+                                    List.member e.sender senders
+                                        && (not <| List.member e.eventType types)
+                                )
+                                events
+                    in
+                    Expect.all
+                        [ Expect.equal (List.length l1) (List.length l2)
+                            |> always
+                        , List.map2 Event.isEqual l1 l2
+                            |> List.all identity
+                            |> Expect.equal True
+                            |> always
+                        ]
+                        ()
+                )
+            , fuzz3 (Fuzz.list TestEvent.fuzzer)
+                (Fuzz.list Fuzz.string)
+                (Fuzz.list Fuzz.string)
+                "All except senders + only type"
+                (\events senders types ->
+                    let
+                        l1 : List Event.Event
+                        l1 =
+                            events
+                                |> Filter.run
+                                    (Filter.and
+                                        (Filter.allSendersExcept senders)
+                                        (Filter.onlyTypes types)
+                                    )
+
+                        l2 : List Event.Event
+                        l2 =
+                            List.filter
+                                (\e ->
+                                    (not <| List.member e.sender senders)
+                                        && List.member e.eventType types
+                                )
+                                events
+                    in
+                    Expect.all
+                        [ Expect.equal (List.length l1) (List.length l2)
+                            |> always
+                        , List.map2 Event.isEqual l1 l2
+                            |> List.all identity
+                            |> Expect.equal True
+                            |> always
+                        ]
+                        ()
+                )
+            , fuzz3 (Fuzz.list TestEvent.fuzzer)
+                (Fuzz.list Fuzz.string)
+                (Fuzz.list Fuzz.string)
+                "All except senders + all except type"
+                (\events senders types ->
+                    let
+                        l1 : List Event.Event
+                        l1 =
+                            events
+                                |> Filter.run
+                                    (Filter.and
+                                        (Filter.allSendersExcept senders)
+                                        (Filter.allTypesExcept types)
+                                    )
+
+                        l2 : List Event.Event
+                        l2 =
+                            List.filter
+                                (\e ->
+                                    (not <| List.member e.sender senders)
+                                        && (not <| List.member e.eventType types)
+                                )
+                                events
+                    in
+                    Expect.all
+                        [ Expect.equal (List.length l1) (List.length l2)
+                            |> always
+                        , List.map2 Event.isEqual l1 l2
+                            |> List.all identity
+                            |> Expect.equal True
+                            |> always
+                        ]
+                        ()
+                )
+            ]
+        , describe "JSON"
+            [ fuzz fuzzer
+                "encode -> decode is the same"
+                (\filter ->
+                    filter
+                        |> Filter.encode
+                        |> E.encode 0
+                        |> D.decodeString Filter.decoder
+                        |> Expect.equal (Ok ( filter, [] ))
+                )
+            ]
+        ]

tests/Test/Tools/Iddict.elm

@@ -3,6 +3,7 @@ module Test.Tools.Iddict exposing (..)
 import Expect
 import Fuzz exposing (Fuzzer)
 import Internal.Tools.Iddict as Iddict exposing (Iddict)
+import Internal.Tools.Json as Json
 import Json.Decode as D
 import Json.Encode as E
 import Test exposing (..)
@@ -73,21 +74,23 @@ empty =
             )
         , test "JSON encode -> decode -> empty"
             (Iddict.empty
-                |> Iddict.encode identity
-                |> D.decodeValue (Iddict.decoder D.value)
+                |> Iddict.encode Json.value
+                |> D.decodeValue (Iddict.decoder Json.value)
+                |> Result.map Tuple.first
                 |> Expect.equal (Ok Iddict.empty)
                 |> always
             )
         , test "JSON encode"
             (Iddict.empty
-                |> Iddict.encode identity
+                |> Iddict.encode Json.value
                 |> E.encode 0
-                |> Expect.equal "{\"cursor\":0,\"dict\":{}}"
+                |> Expect.equal "{\"dict\":{}}"
                 |> always
             )
         , test "JSON decode"
-            ("{\"cursor\":0,\"dict\":{}}"
-                |> D.decodeString (Iddict.decoder D.value)
+            ("{\"dict\":{}}"
+                |> D.decodeString (Iddict.decoder Json.value)
+                |> Result.map Tuple.first
                 |> Expect.equal (Ok Iddict.empty)
                 |> always
             )
@@ -170,8 +173,9 @@ singleton =
             "JSON encode -> decode -> singleton"
             (\single ->
                 single
-                    |> Iddict.encode E.int
-                    |> D.decodeValue (Iddict.decoder D.int)
+                    |> Iddict.encode Json.int
+                    |> D.decodeValue (Iddict.decoder Json.int)
+                    |> Result.map Tuple.first
                     |> Expect.equal (Ok single)
             )
         , fuzz Fuzz.int
@@ -179,7 +183,7 @@ singleton =
             (\i ->
                 Iddict.singleton i
                     |> Tuple.second
-                    |> Iddict.encode E.int
+                    |> Iddict.encode Json.int
                     |> E.encode 0
                     |> Expect.equal ("{\"cursor\":1,\"dict\":{\"0\":" ++ String.fromInt i ++ "}}")
             )
@@ -187,7 +191,8 @@ singleton =
             "JSON decode"
             (\i ->
                 ("{\"cursor\":1,\"dict\":{\"0\":" ++ String.fromInt i ++ "}}")
-                    |> D.decodeString (Iddict.decoder D.int)
+                    |> D.decodeString (Iddict.decoder Json.int)
+                    |> Result.map Tuple.first
                     |> Tuple.pair 0
                     |> Expect.equal (Iddict.singleton i |> Tuple.mapSecond Ok)
             )

tests/Test/Values/Event.elm

@@ -1,5 +1,6 @@
 module Test.Values.Event exposing (..)
 
+import Expect
 import Fuzz exposing (Fuzzer)
 import Internal.Values.Event as Event exposing (Event)
 import Json.Encode as E
@@ -65,3 +66,15 @@ valueFuzzer =
         , Fuzz.map (E.list E.string) (Fuzz.list Fuzz.string)
         , Fuzz.map Event.encode (Fuzz.lazy (\_ -> fuzzer))
         ]
+
+
+suite : Test
+suite =
+    describe "Sanity check"
+        [ fuzz fuzzer
+            "event = event"
+            (\event ->
+                Event.isEqual event event
+                    |> Expect.equal True
+            )
+        ]

tests/Test/Values/Timeline.elm

@@ -0,0 +1,383 @@
+module Test.Values.Timeline exposing (..)
+
+import Expect
+import Fuzz exposing (Fuzzer)
+import Internal.Filter.Timeline as Filter exposing (Filter)
+import Internal.Tools.Json as Json
+import Internal.Values.Timeline as Timeline exposing (Batch, Timeline)
+import Json.Decode as D
+import Test exposing (..)
+import Test.Filter.Timeline as TestFilter
+
+
+fuzzer : Fuzzer Timeline
+fuzzer =
+    TestFilter.fuzzer
+        |> Fuzz.andThen
+            (\globalFilter ->
+                Fuzz.oneOf
+                    [ Fuzz.map2
+                        (\start batches ->
+                            List.foldl
+                                (\b ( s, f ) ->
+                                    ( b.end
+                                    , f >> Timeline.insert { b | start = Just s, filter = globalFilter }
+                                    )
+                                )
+                                ( start, identity )
+                                batches
+                                |> Tuple.second
+                        )
+                        Fuzz.string
+                        (Fuzz.listOfLengthBetween 0 10 fuzzerBatch)
+                    , Fuzz.map2
+                        (\start batches ->
+                            List.foldl
+                                (\b ( s, f ) ->
+                                    ( b.end
+                                    , f >> Timeline.insert { b | start = Just s, filter = Filter.and globalFilter b.filter }
+                                    )
+                                )
+                                ( start, identity )
+                                batches
+                                |> Tuple.second
+                        )
+                        Fuzz.string
+                        (Fuzz.listOfLengthBetween 0 4 fuzzerBatch)
+                    , Fuzz.map2
+                        (\start batches ->
+                            List.foldl
+                                (\b ( s, f ) ->
+                                    ( b.end
+                                    , f >> Timeline.addSync { b | start = Just s, filter = globalFilter }
+                                    )
+                                )
+                                ( start, identity )
+                                batches
+                                |> Tuple.second
+                        )
+                        Fuzz.string
+                        (Fuzz.listOfLengthBetween 0 10 fuzzerBatch)
+                    , Fuzz.map2
+                        (\start batches ->
+                            List.foldl
+                                (\b ( s, f ) ->
+                                    ( b.end
+                                    , f >> Timeline.addSync { b | start = Just s, filter = Filter.and globalFilter b.filter }
+                                    )
+                                )
+                                ( start, identity )
+                                batches
+                                |> Tuple.second
+                        )
+                        Fuzz.string
+                        (Fuzz.listOfLengthBetween 0 4 fuzzerBatch)
+                    ]
+                    |> Fuzz.listOfLengthBetween 0 10
+                    |> Fuzz.map (List.foldl (<|) Timeline.empty)
+            )
+
+
+fuzzerBatch : Fuzzer Batch
+fuzzerBatch =
+    Fuzz.map4 Batch
+        (Fuzz.list Fuzz.string)
+        TestFilter.fuzzer
+        (Fuzz.maybe Fuzz.string)
+        Fuzz.string
+
+
+suite : Test
+suite =
+    describe "Timeline"
+        [ describe "most recent events with filters"
+            [ fuzz TestFilter.fuzzer
+                "Events are returned properly"
+                (\filter ->
+                    Timeline.empty
+                        |> Timeline.insert
+                            { events = [ "a", "b", "c" ]
+                            , filter = filter
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.insert
+                            { events = [ "d", "e", "f" ]
+                            , filter = filter
+                            , start = Just "token_2"
+                            , end = "token_3"
+                            }
+                        |> Timeline.mostRecentEventsFrom filter "token_3"
+                        |> Expect.equal
+                            [ [ "a", "b", "c", "d", "e", "f" ] ]
+                )
+            , fuzz2 TestFilter.fuzzer
+                TestFilter.fuzzer
+                "Sub-events get the same results"
+                (\f1 f2 ->
+                    let
+                        subFilter =
+                            Filter.and f1 f2
+                    in
+                    Timeline.empty
+                        |> Timeline.insert
+                            { events = [ "a", "b", "c" ]
+                            , filter = f1
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.insert
+                            { events = [ "d", "e", "f" ]
+                            , filter = f1
+                            , start = Just "token_2"
+                            , end = "token_3"
+                            }
+                        |> Timeline.mostRecentEventsFrom subFilter "token_3"
+                        |> Expect.equal
+                            [ [ "a", "b", "c", "d", "e", "f" ] ]
+                )
+            , fuzz2 TestFilter.fuzzer
+                TestFilter.fuzzer
+                "ONLY same result if sub-filter"
+                (\f1 f2 ->
+                    Timeline.empty
+                        |> Timeline.insert
+                            { events = [ "a", "b", "c" ]
+                            , filter = f1
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.insert
+                            { events = [ "d", "e", "f" ]
+                            , filter = f1
+                            , start = Just "token_2"
+                            , end = "token_3"
+                            }
+                        |> Timeline.mostRecentEventsFrom f2 "token_3"
+                        |> (\events ->
+                                Expect.equal
+                                    (Filter.subsetOf f1 f2)
+                                    (events == [ [ "a", "b", "c", "d", "e", "f" ] ])
+                           )
+                )
+            ]
+        , describe "Forks in the road"
+            [ fuzz2 TestFilter.fuzzer
+                TestFilter.fuzzer
+                "Two options returned"
+                (\f1 f2 ->
+                    let
+                        subFilter =
+                            Filter.and f1 f2
+                    in
+                    Timeline.empty
+                        |> Timeline.insert
+                            { events = [ "a", "b", "c" ]
+                            , filter = f1
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.insert
+                            { events = [ "d", "e", "f" ]
+                            , filter = f2
+                            , start = Just "token_3"
+                            , end = "token_2"
+                            }
+                        |> Timeline.insert
+                            { events = [ "g", "h", "i" ]
+                            , filter = subFilter
+                            , start = Just "token_2"
+                            , end = "token_4"
+                            }
+                        |> Timeline.mostRecentEventsFrom subFilter "token_4"
+                        |> Expect.equal
+                            [ [ "a", "b", "c", "g", "h", "i" ]
+                            , [ "d", "e", "f", "g", "h", "i" ]
+                            ]
+                )
+            ]
+        , describe "Gaps"
+            [ fuzz TestFilter.fuzzer
+                "Gaps leave behind old events"
+                (\filter ->
+                    Timeline.empty
+                        |> Timeline.insert
+                            { events = [ "a", "b", "c" ]
+                            , filter = filter
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.insert
+                            { events = [ "d", "e", "f" ]
+                            , filter = filter
+                            , start = Just "token_3"
+                            , end = "token_4"
+                            }
+                        |> Timeline.mostRecentEventsFrom filter "token_4"
+                        |> Expect.equal [ [ "d", "e", "f" ] ]
+                )
+            , fuzz3 TestFilter.fuzzer
+                (Fuzz.list Fuzz.string)
+                (Fuzz.pair (Fuzz.list Fuzz.string) (Fuzz.list Fuzz.string))
+                "Gaps can be bridged"
+                (\filter l1 ( l2, l3 ) ->
+                    Timeline.empty
+                        |> Timeline.insert
+                            { events = l1
+                            , filter = filter
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.insert
+                            { events = l3
+                            , filter = filter
+                            , start = Just "token_3"
+                            , end = "token_4"
+                            }
+                        |> Timeline.insert
+                            { events = l2
+                            , filter = filter
+                            , start = Just "token_2"
+                            , end = "token_3"
+                            }
+                        |> Timeline.mostRecentEventsFrom filter "token_4"
+                        |> Expect.equal [ List.concat [ l1, l2, l3 ] ]
+                )
+            ]
+        , describe "JSON"
+            [ fuzz fuzzer
+                "Encode + Decode gives same output"
+                (\timeline ->
+                    timeline
+                        |> Json.encode Timeline.coder
+                        |> D.decodeValue (Json.decode Timeline.coder)
+                        |> Result.map Tuple.first
+                        |> Result.map (Timeline.mostRecentEvents Filter.pass)
+                        |> Expect.equal (Ok <| Timeline.mostRecentEvents Filter.pass timeline)
+                )
+            ]
+        , describe "Weird loops"
+            [ fuzz TestFilter.fuzzer
+                "Weird loops stop looping"
+                (\filter ->
+                    Timeline.empty
+                        |> Timeline.insert
+                            { events = [ "a", "b", "c" ]
+                            , filter = filter
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.insert
+                            { events = [ "d", "e", "f" ]
+                            , filter = filter
+                            , start = Just "token_2"
+                            , end = "token_3"
+                            }
+                        |> Timeline.insert
+                            { events = [ "g", "h", "i" ]
+                            , filter = filter
+                            , start = Just "token_3"
+                            , end = "token_2"
+                            }
+                        |> Timeline.mostRecentEventsFrom filter "token_2"
+                        |> Expect.equal
+                            [ [ "a", "b", "c" ]
+                            , [ "d", "e", "f", "g", "h", "i" ]
+                            ]
+                )
+            ]
+        , describe "Sync"
+            [ fuzz TestFilter.fuzzer
+                "Sync fills gaps"
+                (\filter ->
+                    Timeline.empty
+                        |> Timeline.addSync
+                            { events = [ "a", "b", "c" ]
+                            , filter = filter
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.addSync
+                            { events = [ "f", "g", "h" ]
+                            , filter = filter
+                            , start = Just "token_3"
+                            , end = "token_4"
+                            }
+                        |> Timeline.insert
+                            { events = [ "d", "e" ]
+                            , filter = filter
+                            , start = Just "token_2"
+                            , end = "token_3"
+                            }
+                        |> Timeline.mostRecentEvents filter
+                        |> Expect.equal [ [ "a", "b", "c", "d", "e", "f", "g", "h" ] ]
+                )
+            , fuzz TestFilter.fuzzer
+                "Sync doesn't fill open gaps"
+                (\filter ->
+                    Timeline.empty
+                        |> Timeline.addSync
+                            { events = [ "a", "b", "c" ]
+                            , filter = filter
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.addSync
+                            { events = [ "f", "g", "h" ]
+                            , filter = filter
+                            , start = Just "token_3"
+                            , end = "token_4"
+                            }
+                        |> Timeline.mostRecentEvents filter
+                        |> Expect.equal [ [ "f", "g", "h" ] ]
+                )
+            , fuzz3 (Fuzz.pair Fuzz.string Fuzz.string)
+                fuzzer
+                TestFilter.fuzzer
+                "Getting /sync is the same as getting from the token"
+                (\( start, end ) timeline filter ->
+                    let
+                        t : Timeline
+                        t =
+                            Timeline.addSync
+                                { events = [ "a", "b", "c" ]
+                                , filter = filter
+                                , start = Just start
+                                , end = end
+                                }
+                                timeline
+                    in
+                    Expect.equal
+                        (Timeline.mostRecentEvents filter t)
+                        (Timeline.mostRecentEventsFrom filter end t)
+                )
+            , fuzz TestFilter.fuzzer
+                "Weird loops stop looping"
+                (\filter ->
+                    Timeline.empty
+                        |> Timeline.insert
+                            { events = [ "a", "b", "c" ]
+                            , filter = filter
+                            , start = Just "token_1"
+                            , end = "token_2"
+                            }
+                        |> Timeline.insert
+                            { events = [ "d", "e", "f" ]
+                            , filter = filter
+                            , start = Just "token_2"
+                            , end = "token_3"
+                            }
+                        |> Timeline.insert
+                            { events = [ "g", "h", "i" ]
+                            , filter = filter
+                            , start = Just "token_3"
+                            , end = "token_2"
+                            }
+                        |> Timeline.mostRecentEventsFrom filter "token_2"
+                        |> Expect.equal
+                            [ [ "a", "b", "c" ]
+                            , [ "d", "e", "f", "g", "h", "i" ]
+                            ]
+                )
+            ]
+        ]