plan to refactor to support storing /context responses

doc/QUESTIONS.md

@@ -1,4 +1,19 @@
 remaining problems to resolve:
-how to store timelime fragments that we don't yet know how they should be sorted wrt the other events and gaps. the case with event permalinks and showing the replied to event when rendering a reply.
 
-room state... 
+how to store timelime fragments that we don't yet know how they should be sorted wrt the other events and gaps. the case with event permalinks and showing the replied to event when rendering a reply (anything from /context).
+    
+    either we could put timeline pieces that were the result of /context in something that is not the timeline. Gaps also don't really make sense there ... You can just paginate backwards and forwards. Or maybe still in the timeline but in a different scope not part of the sortKey, scope: live, or scope: piece-1204. While paginating, we could keep the start and end event_id of all the scopes in memory, and set a marker on them to stitch them together?
+
+    Hmmm, I can see the usefullness of the concept of timeline set with multiple timelines in it for this. for the live timeline it's less convenient as you're not bothered so much by the stitching up, but for /context pieces that run into the live timeline while paginating it seems more useful... we could have a marker entry that refers to the next or previous scope ... this way we could also use gap entries for /context timelines, just one on either end.
+
+    the start and end event_id of a scope, keeping that in memory, how do we make sure this is safe taking transactions into account? our preferred strategy so far has been to read everything from store inside a txn to make sure we don't have any stale caches or races. Would be nice to keep this.
+
+    so while paginating, you'd check the event_id of the event against the start/end event_id of every scope to see if stitching is in order, and add marker entries if so. Perhaps marker entries could also be used to stitch up rooms that have changed versioning?
+
+    What does all of this mean for using sortKey as an identifier? Will we need to take scope into account as well everywhere?
+
+    we'll need to at least contemplate how room state will be handled with all of the above.
+    
+how do we deal with the fact that an event can be rendered (and updated) multiple times in the timeline as part of replies. 
+
+room state... 

doc/architecture.md

@@ -7,6 +7,24 @@ The matrix layer assumes a transaction-based storage layer, modelled much to how
 
 For this reason a `Room` processes a sync response in two phases: `persistSync` & `emitSync`, with the return value of the former being passed into the latter to avoid double processing.
 
+## Timeline, fragments & event indices.
+
+A room in matrix is a DAG (directed, acyclic graph) of events, also known as the timeline. Morpheus is only aware of fragments of this graph, and can be unaware how these fragments relate to each other until a common event is found while paginating a fragment. After doing an initial sync, you start with one fragment. When looking up an event with the `/context` endpoint (for fetching a replied to message, or navigating to a given event id, e.g. through a permalink), a new, unconnected, fragment is created. Also, when receiving a limited sync response during incremental sync, a new fragment is created. Here, the relationship is clear, so they are immediately linked up at creation. Events in morpheus are identified within a room by `[fragment_id, event_index]`. The `event_index` is an unique number within a fragment to sort events in chronological order in the timeline. `fragment_id` cannot be directly compared for sorting (as the relationship may be unknown), but with help of the `FragmentIndex`, one can attempt to sort events by their `FragmentIndex([fragment_id, event_index])`.
+
+A fragment is the following data structure:
+```
+let fragment := {
+    roomId: string
+    id: number
+    previousId: number?
+    nextId: number?
+    prevToken: string?
+    nextToken: string?
+}
+```
+
+## Observing the session
+
 `Room`s on the `Session` are exposed as an `ObservableMap` collection, which is like an ordinary `Map` but emits events when it is modified (here when a room is added, removed, or the properties of a room change). `ObservableMap` can have different operators applied to it like `mapValues()`, `filterValues()` each returning a new `ObservableMap`-like object, and also `sortValues()` returning an `ObservableList` (emitting events when a room at an index is added, removed, moved or changes properties).
 
 So example, for the room list, `Room` objects from `Session.rooms` are mapped to a `RoomTileViewModel` and then sorted. This gives us fine-grained events at the end of the collection chain that can be easily and efficiently rendered by the `ListView` component.