diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md new file mode 100644 index 00000000..621e155a --- /dev/null +++ b/doc/FRAGMENTS.md @@ -0,0 +1,88 @@ + - DONE: write FragmentIndex + - DONE: adapt SortKey ... naming! : + - FragmentIdIndex (index as in db index) + - compare(idA, idB) + - SortKey + - FragmentId + - EventIndex + - DONE: write fragmentStore + - load all fragments + - add a fragment (live on limited sync, or /context) + - connect two fragments + - update token on fragment (when filling gap or connecting two fragments) + + fragments can need connecting when filling a gap or creating a new /context fragment + - DONE: adapt timelineStore + + how will fragments be exposed in timeline store? + - all read operations are passed a fragment id + - adapt persister + - DONE: persist fragments in /sync + - DONE: fill gaps / fragment filling + - DONE: load n items before and after key, + - DONE: need to add fragments as we come across boundaries + - DONE: also cache fragments? not for now ... + - DONE: not doing any of the above, just reloading and rebuilding for now + + - DONE: adapt Timeline + - DONE: turn ObservableArray into ObservableSortedArray + - upsert already sorted sections + - DONE: upsert single entry + - adapt TilesCollection & Tile to entry changes + + - add live fragment id optimization if we haven't done so already + - lets try to not have to have the fragmentindex in memory if the timeline isn't loaded + - could do this by only loading all fragments into index when filling gaps, backpaginating, ... and on persister load only load the last fragment. This wouldn't even need a FragmentIndex? + +# Leftover items + +implement SortedArray::setManySorted in a performant manner +implement FragmentIdComparator::add in a performant manner +there is some duplication (also in memory) between SortedArray and TilesCollection. Both keep a sorted list based on fragmentId/eventIndex... TilesCollection doesn't use the index in the event handlers at all. we could allow timeline to export a structure that just emits "these entries are a thing (now)" and not have to go through sorting twice. Timeline would have to keep track of the earliest key so it can use it in loadAtTop, but that should be easy. Hmmm. also, Timeline might want to be in charge of unloading parts of the loaded timeline, and for that it would need to know the order of entries. So maybe not ... we'll see. + +check: do /sync events not have a room_id and /messages do??? + +so a gap is two connected fragments where either the first fragment has a nextToken and/or the second fragment has a previousToken. It can be both, so we can have a gap where you can fill in from the top, from the bottom (like when limited sync) or both. + + + + +also, filling gaps and storing /context, how do we find the fragment we could potentially merge with to look for overlapping events? + +with /sync this is all fine and dandy, but with /context is there a way where we don't need to look up every event_id in the store to see if it's already there? + we can do a anyOf(event_id) on timelineStore.index("by_index") by sorting the event ids according to IndexedDb.cmp and passing the next value to cursor.continue(nextId). + +so we'll need to remove previous/nextEvent on the timeline store and come up with a method to find the first matched event in a list of eventIds. + so we'll need to map all event ids to an event and return the first one that is not null. If we haven't read all events but we know that all the previous ones are null, then we can already return the result. + + we can call this findFirstEventIn(roomId, [event ids]) + +thoughts: + - ranges in timeline store with fragmentId might not make sense anymore as doing queries over multiple fragment ids doesn't make sense anymore ... still makes sense to have them part of SortKey though ... + - we need a test for querytarget::lookup, or make sure it works well ... + + +# Reading the timeline with fragments + +- what format does the persister return newEntries after persisting sync or a gap fill? + - a new fragment can be created during a limited sync + - when doing a /context or /messages call, we could have joined with another fragment + - don't think we need to describe a result spanning multiple fragments here + so: + + in case of limited sync, we just say there was a limited sync, this is the fragment that was created for it so we can show a gap in the timeline + + in case of a gap fill, we need to return what was changed to the fragment (was it joined with another fragment, what's the new token), and which events were actually added. + +we return entries! fragmentboundaryentry(start or end) or evententry. so looks much like the gaps we had before, but now they are not stored in the timeline store, but based on fragments. + +- where do we translate from fragments to gap entries? and back? in the timeline object? + that would make sense, that seems to be the only place we need that translation + +# SortKey + +so, it feels simpler to store fragmentId and eventIndex as fields on the entry instead of an array/arraybuffer in the field sortKey. Currently, the tiles code somewhat relies on having sortKeys but nothing too hard to change. + +so, what we could do: + - we create EventKey(fragmentId, eventIndex) that has the nextKey methods. + - we create a class EventEntry that wraps what is stored in the timeline store. This has a reference to the fragmentindex and has an opaque compare method. Tiles delegate to this method. EventEntry could later on also contain methods like MatrixEvent has in the riot js-sdk, e.g. something to safely dig into the event object. diff --git a/doc/QUESTIONS.md b/doc/QUESTIONS.md new file mode 100644 index 00000000..2bad1bae --- /dev/null +++ b/doc/QUESTIONS.md @@ -0,0 +1,19 @@ +remaining problems to resolve: + +how to store timelime fragments that we don't yet know how they should be sorted wrt the other events and gaps. the case with event permalinks and showing the replied to event when rendering a reply (anything from /context). + + either we could put timeline pieces that were the result of /context in something that is not the timeline. Gaps also don't really make sense there ... You can just paginate backwards and forwards. Or maybe still in the timeline but in a different scope not part of the sortKey, scope: live, or scope: piece-1204. While paginating, we could keep the start and end event_id of all the scopes in memory, and set a marker on them to stitch them together? + + Hmmm, I can see the usefullness of the concept of timeline set with multiple timelines in it for this. for the live timeline it's less convenient as you're not bothered so much by the stitching up, but for /context pieces that run into the live timeline while paginating it seems more useful... we could have a marker entry that refers to the next or previous scope ... this way we could also use gap entries for /context timelines, just one on either end. + + the start and end event_id of a scope, keeping that in memory, how do we make sure this is safe taking transactions into account? our preferred strategy so far has been to read everything from store inside a txn to make sure we don't have any stale caches or races. Would be nice to keep this. + + so while paginating, you'd check the event_id of the event against the start/end event_id of every scope to see if stitching is in order, and add marker entries if so. Perhaps marker entries could also be used to stitch up rooms that have changed versioning? + + What does all of this mean for using sortKey as an identifier? Will we need to take scope into account as well everywhere? + + we'll need to at least contemplate how room state will be handled with all of the above. + +how do we deal with the fact that an event can be rendered (and updated) multiple times in the timeline as part of replies. + +room state... diff --git a/doc/TODO.md b/doc/TODO.md index abf03663..f3e41d42 100644 --- a/doc/TODO.md +++ b/doc/TODO.md @@ -25,6 +25,11 @@ - DONE: support timeline - DONE: clicking on a room list, you see messages (userId -> body) - DONE: style minimal UI + - DONE: implement gap filling and fragments (see FRAGMENTS.md) + - allow collection items (especially tiles) to self-update + - improve fragmentidcomparer::add + - send messages + - better UI - fix MappedMap update mechanism - see if in BaseObservableMap we need to change ...params - put sync button and status label inside SessionView @@ -32,6 +37,8 @@ - find out if `(this._emitCollectionUpdate)(this)` is different than `this._emitCollectionUpdate(this)` - got "database tried to mutate when not allowed" or something error as well - find out why when RoomPersister.(\_createGapEntry/\_createEventEntry) we remove .buffer the transaction fails (good), but upon fixing and refreshing is missing a message! syncToken should not be saved, so why isn't this again in the sync response and now the txn does succeed? + - DONE: take access token out of IDB? this way it can be stored in a more secure thing for non-web clients, together wit encryption key for olm sessions ... ? like macos keychain, gnome keyring, ... maybe using https://www.npmjs.com/package/keytar + - experiment with using just a normal array with 2 numbers for sortkeys, to work in Edge as well. - send messages - fill gaps with call to /messages - create sync filter diff --git a/doc/architecture.md b/doc/architecture.md new file mode 100644 index 00000000..286113d5 --- /dev/null +++ b/doc/architecture.md @@ -0,0 +1,47 @@ +The matrix layer consists of a `Session`, which represents a logged in user session. It's the root object you can get rooms off. It can persist and load itself from storage, at which point it's ready to be displayed. It doesn't sync it's own though, and you need to create and start a Sync object for updates to be pushed and persisted to the session. `Sync` is the thing (although not the only thing) that mutates the `Session`, with `Session` being unaware of `Sync`. + +The matrix layer assumes a transaction-based storage layer, modelled much to how IndexedDB works. The idea is that any logical operation like process sync response, send a message, ... runs completely in a transaction that gets aborted if anything goes wrong. This helps the storage to always be in a consistent state. For this reason you'll often see transactions (txn) being passed in the code. Also, the idea is to not emit any events until readwrite transactions have been committed. + + - Reduce the chance that errors (in the event handlers) abort the transaction. You *could* catch & rethrow but it can get messy. + - Try to keep transactions as short-lived as possible, to not block other transactions. + +For this reason a `Room` processes a sync response in two phases: `persistSync` & `emitSync`, with the return value of the former being passed into the latter to avoid double processing. + +## Timeline, fragments & event indices. + +A room in matrix is a DAG (directed, acyclic graph) of events, also known as the timeline. Morpheus is only aware of fragments of this graph, and can be unaware how these fragments relate to each other until a common event is found while paginating a fragment. After doing an initial sync, you start with one fragment. When looking up an event with the `/context` endpoint (for fetching a replied to message, or navigating to a given event id, e.g. through a permalink), a new, unconnected, fragment is created. Also, when receiving a limited sync response during incremental sync, a new fragment is created. Here, the relationship is clear, so they are immediately linked up at creation. Events in morpheus are identified within a room by `[fragment_id, event_index]`. The `event_index` is an unique number within a fragment to sort events in chronological order in the timeline. `fragment_id` cannot be directly compared for sorting (as the relationship may be unknown), but with help of the `FragmentIndex`, one can attempt to sort events by their `FragmentIndex([fragment_id, event_index])`. + +A fragment is the following data structure: +``` +let fragment := { + roomId: string + id: number + previousId: number? + nextId: number? + previousToken: string? + nextToken: string? +} +``` + +## Observing the session + +`Room`s on the `Session` are exposed as an `ObservableMap` collection, which is like an ordinary `Map` but emits events when it is modified (here when a room is added, removed, or the properties of a room change). `ObservableMap` can have different operators applied to it like `mapValues()`, `filterValues()` each returning a new `ObservableMap`-like object, and also `sortValues()` returning an `ObservableList` (emitting events when a room at an index is added, removed, moved or changes properties). + +So for example, the room list, `Room` objects from `Session.rooms` are mapped to a `RoomTileViewModel` and then sorted. This gives us fine-grained events at the end of the collection chain that can be easily and efficiently rendered by the `ListView` component. + +On that note, view components are just a simple convention, having these methods: + + - `mount()` - prepare to become part of the document and interactive, ensure `root()` returns a valid DOM node. + - `root()` - the room DOM node for the component. Only valid to be called between `mount()` and `unmount()`. + - `update(attributes)` (to be renamed to `setAttributes(attributes)`) - update the attributes for this component. Not all components support all attributes to be updated. For example most components expect a viewModel, but if you want a component with a different view model, you'd just create a new one. + - `unmount()` - tear down after having been removed from the document. + +The initial attributes are usually received by the constructor in the first argument. Other arguments are usually freeform, `ListView` accepting a closure to create a child component from a collection value. + +Templating and one-way databinding are neccesary improvements, but not assumed by the component contract. + +Updates from view models can come in two ways. View models emit a change event, that can be listened to from a view. This usually includes the name of the property that changed. This is the mechanism used to update the room name in the room header of the currently active room for example. + +For view models part of an observable collection (and to be rendered by a ListView), updates can also propagate through the collection and delivered by the ListView to the view in question. This avoids every child component in a ListView having to attach a listener to it's viewModel. This is the mechanism to update the room name in a RoomTile in the room list for example. + +TODO: specify how the collection based updates work. (not specified yet, we'd need a way to derive a key from a value to emit an update from within a collection, but haven't found a nice way of specifying that in an api) diff --git a/doc/sync-updates.md b/doc/sync-updates.md index 1d3f0410..45333c3e 100644 --- a/doc/sync-updates.md +++ b/doc/sync-updates.md @@ -1,7 +1,7 @@ # persistance vs model update of a room ## persist first, return update object, update model with update object - - + - we went with this ## update model first, return update object, persist with update object - not all models exist at all times (timeline only when room is "open"), so model to create timeline update object might not exist for persistence need @@ -15,4 +15,4 @@ ## persist first, read from storage to update model + guaranteed consistency between what is on screen and in storage - - slower as we need to reread what was just synced every time (big accounts with frequent updates) \ No newline at end of file + - slower as we need to reread what was just synced every time (big accounts with frequent updates) diff --git a/index.html b/index.html index facaa4d3..477d7c05 100644 --- a/index.html +++ b/index.html @@ -45,6 +45,10 @@ flex: 1; overflow-y: scroll; } + + .RoomView_error { + color: red; + } diff --git a/package-lock.json b/package-lock.json index 6253d97e..56f65bc8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -95,9 +95,9 @@ } }, "impunity": { - "version": "0.0.5", - "resolved": "https://registry.npmjs.org/impunity/-/impunity-0.0.5.tgz", - "integrity": "sha512-ro+enrZPFTyY2U1sV9NytsyejE2tS5theAArM95iPYYQHUvO9YN0VjgfXP0KJfxwh4Xb6vBTRBmHIgx9GUx2Xg==", + "version": "0.0.7", + "resolved": "https://registry.npmjs.org/impunity/-/impunity-0.0.7.tgz", + "integrity": "sha512-+DhzXSWrzqI1KNroKt3y1LkLTn/aoJpt4DzxWN+hair+Jfb+iJAbTEsSFkYUG7kASP9TF9GvI0hIBUul6PjpKg==", "dev": true, "requires": { "colors": "^1.3.3", diff --git a/package.json b/package.json index 4b5db04a..6ae62e9c 100644 --- a/package.json +++ b/package.json @@ -22,7 +22,7 @@ "homepage": "https://github.com/bwindels/morpheusjs#readme", "devDependencies": { "finalhandler": "^1.1.1", - "impunity": "^0.0.5", + "impunity": "^0.0.7", "serve-static": "^1.13.2" } } diff --git a/prototypes/idb-continue-key.html b/prototypes/idb-continue-key.html new file mode 100644 index 00000000..dc488deb --- /dev/null +++ b/prototypes/idb-continue-key.html @@ -0,0 +1,165 @@ + + + + + + diff --git a/prototypes/manifest.appcache b/prototypes/manifest.appcache new file mode 100644 index 00000000..ff420219 --- /dev/null +++ b/prototypes/manifest.appcache @@ -0,0 +1,4 @@ +CACHE MANIFEST +# v1 +/responsive-layout-flex.html +/me.jpg diff --git a/prototypes/me.jpg b/prototypes/me.jpg new file mode 100644 index 00000000..615e10c8 Binary files /dev/null and b/prototypes/me.jpg differ diff --git a/prototypes/responsive-layout-flex.html b/prototypes/responsive-layout-flex.html new file mode 100644 index 00000000..c822ac08 --- /dev/null +++ b/prototypes/responsive-layout-flex.html @@ -0,0 +1,505 @@ + + + + + + + + + + + +
+
+ +
+
+

Select a room on the left side

+
+
+
+
+ +
Avatar for some room
+
+

Talk to Bruno

+
The room to talk to Bruno
+
+ +
+
    +
  • Message 1, message 1, message 1, message 1, message 1, message 1, message 1, message 1
    • +
  • Message 2, message 2, message 2, message 2, message 2, message 2, message 2, message 2
    • +
  • Message 3, message 3, message 3, message 3, message 3, message 3, message 3, message 3
    • +
  • Message 4, message 4, message 4, message 4, message 4, message 4, message 4, message 4
    • +
  • Message 5, message 5, message 5, message 5, message 5, message 5, message 5, message 5
    • +
  • Message 6, message 6, message 6, message 6, message 6, message 6, message 6, message 6
    • +
  • Message 7, message 7, message 7, message 7, message 7, message 7, message 7, message 7
    • +
  • Message 8, message 8, message 8, message 8, message 8, message 8, message 8, message 8
    • +
  • Message 9, message 9, message 9, message 9, message 9, message 9, message 9, message 9
    • +
  • Message 10, message 10, message 10, message 10, message 10, message 10, message 10, message 10
    • +
  • Message 11, message 11, message 11, message 11, message 11, message 11, message 11, message 11
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
  • Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12
    • +
+
+
+
+

Bruno

+

Ban | Kick | Mock

+
+
+
+ + + + diff --git a/prototypes/responsive-layout-grid.html b/prototypes/responsive-layout-grid.html new file mode 100644 index 00000000..f8ded23f --- /dev/null +++ b/prototypes/responsive-layout-grid.html @@ -0,0 +1,133 @@ + + + + + + + +
+ +
+
+

Select a room on the left side

+
+
+ +
+
+

Bruno

+

Ban | Kick | Mock

+
+ + + + diff --git a/src/ui/viewmodels/SessionViewModel.js b/src/domain/session/SessionViewModel.js similarity index 89% rename from src/ui/viewmodels/SessionViewModel.js rename to src/domain/session/SessionViewModel.js index 470ee517..03a985ce 100644 --- a/src/ui/viewmodels/SessionViewModel.js +++ b/src/domain/session/SessionViewModel.js @@ -1,6 +1,6 @@ import EventEmitter from "../../EventEmitter.js"; -import RoomTileViewModel from "./RoomTileViewModel.js"; -import RoomViewModel from "./RoomViewModel.js"; +import RoomTileViewModel from "./roomlist/RoomTileViewModel.js"; +import RoomViewModel from "./room/RoomViewModel.js"; export default class SessionViewModel extends EventEmitter { constructor(session) { diff --git a/src/domain/session/room/RoomViewModel.js b/src/domain/session/room/RoomViewModel.js new file mode 100644 index 00000000..0e5f9b1c --- /dev/null +++ b/src/domain/session/room/RoomViewModel.js @@ -0,0 +1,54 @@ +import EventEmitter from "../../../EventEmitter.js"; +import TimelineViewModel from "./timeline/TimelineViewModel.js"; + +export default class RoomViewModel extends EventEmitter { + constructor(room) { + super(); + this._room = room; + this._timeline = null; + this._timelineVM = null; + this._onRoomChange = this._onRoomChange.bind(this); + this._timelineError = null; + } + + async enable() { + this._room.on("change", this._onRoomChange); + try { + this._timeline = await this._room.openTimeline(); + this._timelineVM = new TimelineViewModel(this._timeline); + this.emit("change", "timelineViewModel"); + } catch (err) { + console.error(`room.openTimeline(): ${err.message}:\n${err.stack}`); + this._timelineError = err; + this.emit("change", "error"); + } + } + + disable() { + if (this._timeline) { + // will stop the timeline from delivering updates on entries + this._timeline.close(); + } + } + + // room doesn't tell us yet which fields changed, + // so emit all fields originating from summary + _onRoomChange() { + this.emit("change", "name"); + } + + get name() { + return this._room.name; + } + + get timelineViewModel() { + return this._timelineVM; + } + + get error() { + if (this._timelineError) { + return `Something went wrong loading the timeline: ${this._timelineError.message}`; + } + return null; + } +} diff --git a/src/domain/session/room/timeline/TilesCollection.js b/src/domain/session/room/timeline/TilesCollection.js new file mode 100644 index 00000000..8af2879d --- /dev/null +++ b/src/domain/session/room/timeline/TilesCollection.js @@ -0,0 +1,153 @@ +import BaseObservableList from "../../../../observable/list/BaseObservableList.js"; +import sortedIndex from "../../../../utils/sortedIndex.js"; + +// maps 1..n entries to 0..1 tile. Entries are what is stored in the timeline, either an event or fragmentboundary +export default class TilesCollection extends BaseObservableList { + constructor(entries, tileCreator) { + super(); + this._entries = entries; + this._tiles = null; + this._entrySubscription = null; + this._tileCreator = tileCreator; + } + + onSubscribeFirst() { + this._entrySubscription = this._entries.subscribe(this); + this._populateTiles(); + } + + _populateTiles() { + this._tiles = []; + let currentTile = null; + for (let entry of this._entries) { + if (!currentTile || !currentTile.tryIncludeEntry(entry)) { + currentTile = this._tileCreator(entry); + if (currentTile) { + this._tiles.push(currentTile); + } + } + } + let prevTile = null; + for (let tile of this._tiles) { + if (prevTile) { + prevTile.updateNextSibling(tile); + } + tile.updatePreviousSibling(prevTile); + prevTile = tile; + } + if (prevTile) { + prevTile.updateNextSibling(null); + } + } + + _findTileIdx(entry) { + return sortedIndex(this._tiles, entry, (entry, tile) => { + // negate result because we're switching the order of the params + return -tile.compareEntry(entry); + }); + } + + _findTileAtIdx(entry, idx) { + const tile = this._getTileAtIdx(idx); + if (tile && tile.compareEntry(entry) === 0) { + return tile; + } + } + + _getTileAtIdx(tileIdx) { + if (tileIdx >= 0 && tileIdx < this._tiles.length) { + return this._tiles[tileIdx]; + } + return null; + } + + onUnsubscribeLast() { + this._entrySubscription = this._entrySubscription(); + this._tiles = null; + } + + onReset() { + // if TileViewModel were disposable, dispose here, or is that for views to do? views I suppose ... + this._buildInitialTiles(); + this.emitReset(); + } + + onAdd(index, entry) { + const tileIdx = this._findTileIdx(entry); + const prevTile = this._getTileAtIdx(tileIdx - 1); + if (prevTile && prevTile.tryIncludeEntry(entry)) { + this.emitUpdate(tileIdx - 1, prevTile); + return; + } + // not + 1 because this entry hasn't been added yet + const nextTile = this._getTileAtIdx(tileIdx); + if (nextTile && nextTile.tryIncludeEntry(entry)) { + this.emitUpdate(tileIdx, nextTile); + return; + } + + const newTile = this._tileCreator(entry); + if (newTile) { + prevTile && prevTile.updateNextSibling(newTile); + nextTile && nextTile.updatePreviousSibling(newTile); + this._tiles.splice(tileIdx, 0, newTile); + this.emitAdd(tileIdx, newTile); + } + // find position by sort key + // ask siblings to be included? both? yes, twice: a (insert c here) b, ask a(c), if yes ask b(a), else ask b(c)? if yes then b(a)? + } + + onUpdate(index, entry, params) { + const tileIdx = this._findTileIdx(entry); + const tile = this._findTileAtIdx(entry, tileIdx); + if (tile) { + const newParams = tile.updateEntry(entry, params); + if (newParams) { + this.emitUpdate(tileIdx, tile, newParams); + } + } + // technically we should handle adding a tile here as well + // in case before we didn't have a tile for it but now we do + // but in reality we don't have this use case as the type and msgtype + // doesn't change. Decryption maybe is the exception? + + + // outcomes here can be + // tiles should be removed (got redacted and we don't want it in the timeline) + // tile should be added where there was none before ... ? + // entry should get it's own tile now + // merge with neighbours? ... hard to imagine use case for this ... + } + + // would also be called when unloading a part of the timeline + onRemove(index, entry) { + const tileIdx = this._findTileIdx(entry); + const tile = this._findTileAtIdx(entry, tileIdx); + if (tile) { + const removeTile = tile.removeEntry(entry); + if (removeTile) { + const prevTile = this._getTileAtIdx(tileIdx - 1); + const nextTile = this._getTileAtIdx(tileIdx + 1); + this._tiles.splice(tileIdx, 1); + prevTile && prevTile.updateNextSibling(nextTile); + nextTile && nextTile.updatePreviousSibling(prevTile); + this.emitRemove(tileIdx, tile); + } else { + this.emitUpdate(tileIdx, tile); + } + } + } + + onMove(fromIdx, toIdx, value) { + // this ... cannot happen in the timeline? + // should be sorted by sortKey and sortKey is immutable + } + + [Symbol.iterator]() { + return this._tiles.values(); + } + + get length() { + return this._tiles.length; + } +} diff --git a/src/domain/session/room/timeline/TimelineViewModel.js b/src/domain/session/room/timeline/TimelineViewModel.js new file mode 100644 index 00000000..5c179673 --- /dev/null +++ b/src/domain/session/room/timeline/TimelineViewModel.js @@ -0,0 +1,52 @@ +/* +need better naming, but +entry = event or gap from matrix layer +tile = item on visual timeline like event, date separator?, group of joined events + + +shall we put date separators as marker in EventViewItem or separate item? binary search will be complicated ... + + +pagination ... + +on the timeline viewmodel (containing the TilesCollection?) we'll have a method to (un)load a tail or head of +the timeline (counted in tiles), which results to a range in sortKeys we want on the screen. We pass that range +to the room timeline, which unload entries from memory. +when loading, it just reads events from a sortkey backwards or forwards... +*/ +import TilesCollection from "./TilesCollection.js"; +import tilesCreator from "./tilesCreator.js"; + +export default class TimelineViewModel { + constructor(timeline) { + this._timeline = timeline; + // once we support sending messages we could do + // timeline.entries.concat(timeline.pendingEvents) + // for an ObservableList that also contains local echos + this._tiles = new TilesCollection(timeline.entries, tilesCreator({timeline})); + } + + // doesn't fill gaps, only loads stored entries/tiles + loadAtTop() { + // load 100 entries, which may result in 0..100 tiles + return this._timeline.loadAtTop(100); + } + + unloadAtTop(tileAmount) { + // get lowerSortKey for tile at index tileAmount - 1 + // tell timeline to unload till there (included given key) + } + + loadAtBottom() { + + } + + unloadAtBottom(tileAmount) { + // get upperSortKey for tile at index tiles.length - tileAmount + // tell timeline to unload till there (included given key) + } + + get tiles() { + return this._tiles; + } +} diff --git a/src/domain/session/room/timeline/tiles/GapTile.js b/src/domain/session/room/timeline/tiles/GapTile.js new file mode 100644 index 00000000..c063db25 --- /dev/null +++ b/src/domain/session/room/timeline/tiles/GapTile.js @@ -0,0 +1,52 @@ +import SimpleTile from "./SimpleTile.js"; + +export default class GapTile extends SimpleTile { + constructor(options, timeline) { + super(options); + this._timeline = timeline; + this._loading = false; + this._error = null; + } + + async fill() { + // prevent doing this twice + if (!this._loading) { + this._loading = true; + // this._emitUpdate("isLoading"); + try { + await this._timeline.fillGap(this._entry, 10); + } catch (err) { + console.error(`timeline.fillGap(): ${err.message}:\n${err.stack}`); + this._error = err; + // this._emitUpdate("error"); + } finally { + this._loading = false; + // this._emitUpdate("isLoading"); + } + } + } + + get shape() { + return "gap"; + } + + get isLoading() { + return this._loading; + } + + get isUp() { + return this._entry.direction.isBackward; + } + + get isDown() { + return this._entry.direction.isForward; + } + + get error() { + if (this._error) { + const dir = this._entry.prev_batch ? "previous" : "next"; + return `Could not load ${dir} messages: ${this._error.message}`; + } + return null; + } +} diff --git a/src/domain/session/room/timeline/tiles/ImageTile.js b/src/domain/session/room/timeline/tiles/ImageTile.js new file mode 100644 index 00000000..ebe8d022 --- /dev/null +++ b/src/domain/session/room/timeline/tiles/ImageTile.js @@ -0,0 +1,26 @@ +import MessageTile from "./MessageTile.js"; + +export default class ImageTile extends MessageTile { + constructor(options) { + super(options); + + // we start loading the image here, + // and call this._emitUpdate once it's loaded? + // or maybe we have an becameVisible() callback on tiles where we start loading it? + } + get src() { + return ""; + } + + get width() { + return 200; + } + + get height() { + return 200; + } + + get label() { + return "this is an image"; + } +} diff --git a/src/domain/session/room/timeline/tiles/LocationTile.js b/src/domain/session/room/timeline/tiles/LocationTile.js new file mode 100644 index 00000000..69dbc629 --- /dev/null +++ b/src/domain/session/room/timeline/tiles/LocationTile.js @@ -0,0 +1,20 @@ +import MessageTile from "./MessageTile.js"; + +/* +map urls: +apple: https://developer.apple.com/library/archive/featuredarticles/iPhoneURLScheme_Reference/MapLinks/MapLinks.html +android: https://developers.google.com/maps/documentation/urls/guide +wp: maps:49.275267 -122.988617 +https://www.habaneroconsulting.com/stories/insights/2011/opening-native-map-apps-from-the-mobile-browser +*/ +export default class LocationTile extends MessageTile { + get mapsLink() { + const geoUri = this._getContent().geo_uri; + const [lat, long] = geoUri.split(":")[1].split(","); + return `maps:${lat} ${long}`; + } + + get label() { + return `${this.sender} sent their location, click to see it in maps.`; + } +} diff --git a/src/domain/session/room/timeline/tiles/MessageTile.js b/src/domain/session/room/timeline/tiles/MessageTile.js new file mode 100644 index 00000000..8ada2310 --- /dev/null +++ b/src/domain/session/room/timeline/tiles/MessageTile.js @@ -0,0 +1,30 @@ +import SimpleTile from "./SimpleTile.js"; + +export default class MessageTile extends SimpleTile { + + constructor(options) { + super(options); + this._date = new Date(this._entry.event.origin_server_ts); + } + + get shape() { + return "message"; + } + + get sender() { + return this._entry.event.sender; + } + + get date() { + return this._date.toLocaleDateString(); + } + + get time() { + return this._date.toLocaleTimeString(); + } + + _getContent() { + const event = this._entry.event; + return event && event.content; + } +} diff --git a/src/domain/session/room/timeline/tiles/RoomMemberTile.js b/src/domain/session/room/timeline/tiles/RoomMemberTile.js new file mode 100644 index 00000000..cc9796e9 --- /dev/null +++ b/src/domain/session/room/timeline/tiles/RoomMemberTile.js @@ -0,0 +1,14 @@ +import SimpleTile from "./SimpleTile.js"; + +export default class RoomNameTile extends SimpleTile { + + get shape() { + return "announcement"; + } + + get label() { + const event = this._entry.event; + const content = event.content; + return `${event.sender} changed membership to ${content.membership}`; + } +} diff --git a/src/domain/session/room/timeline/tiles/RoomNameTile.js b/src/domain/session/room/timeline/tiles/RoomNameTile.js new file mode 100644 index 00000000..f415efe6 --- /dev/null +++ b/src/domain/session/room/timeline/tiles/RoomNameTile.js @@ -0,0 +1,14 @@ +import SimpleTile from "./SimpleTile.js"; + +export default class RoomNameTile extends SimpleTile { + + get shape() { + return "announcement"; + } + + get label() { + const event = this._entry.event; + const content = event.content; + return `${event.sender} changed the room name to "${content.name}"` + } +} diff --git a/src/domain/session/room/timeline/tiles/SimpleTile.js b/src/domain/session/room/timeline/tiles/SimpleTile.js new file mode 100644 index 00000000..605fc1c1 --- /dev/null +++ b/src/domain/session/room/timeline/tiles/SimpleTile.js @@ -0,0 +1,65 @@ +export default class SimpleTile { + constructor({entry, emitUpdate}) { + this._entry = entry; + this._emitUpdate = emitUpdate; + } + // view model props for all subclasses + // hmmm, could also do instanceof ... ? + get shape() { + // "gap" | "message" | "image" | ... ? + } + + // don't show display name / avatar + // probably only for MessageTiles of some sort? + get isContinuation() { + return false; + } + + get hasDateSeparator() { + return false; + } + // TilesCollection contract? unused atm + get upperEntry() { + return this._entry; + } + + // TilesCollection contract? unused atm + get lowerEntry() { + return this._entry; + } + + // TilesCollection contract + compareEntry(entry) { + return this._entry.compare(entry); + } + + // update received for already included (falls within sort keys) entry + updateEntry(entry) { + // return names of props updated, or true for all, or null for no changes caused + return true; + } + + // return whether the tile should be removed + // as SimpleTile only has one entry, the tile should be removed + removeEntry(entry) { + return true; + } + + // SimpleTile can only contain 1 entry + tryIncludeEntry() { + return false; + } + // let item know it has a new sibling + updatePreviousSibling(prev) { + + } + + // let item know it has a new sibling + updateNextSibling(next) { + + } + + get internalId() { + return this._entry.asEventKey().toString(); + } +} diff --git a/src/domain/session/room/timeline/tiles/TextTile.js b/src/domain/session/room/timeline/tiles/TextTile.js new file mode 100644 index 00000000..03485ea0 --- /dev/null +++ b/src/domain/session/room/timeline/tiles/TextTile.js @@ -0,0 +1,14 @@ +import MessageTile from "./MessageTile.js"; + +export default class TextTile extends MessageTile { + get label() { + const content = this._getContent(); + const body = content && content.body; + const sender = this._entry.event.sender; + if (this._entry.type === "m.emote") { + return `* ${sender} ${body}`; + } else { + return `${sender}: ${body}`; + } + } +} diff --git a/src/domain/session/room/timeline/tilesCreator.js b/src/domain/session/room/timeline/tilesCreator.js new file mode 100644 index 00000000..96638aa0 --- /dev/null +++ b/src/domain/session/room/timeline/tilesCreator.js @@ -0,0 +1,43 @@ +import GapTile from "./tiles/GapTile.js"; +import TextTile from "./tiles/TextTile.js"; +import ImageTile from "./tiles/ImageTile.js"; +import LocationTile from "./tiles/LocationTile.js"; +import RoomNameTile from "./tiles/RoomNameTile.js"; +import RoomMemberTile from "./tiles/RoomMemberTile.js"; + +export default function ({timeline, emitUpdate}) { + return function tilesCreator(entry) { + const options = {entry, emitUpdate}; + if (entry.isGap) { + return new GapTile(options, timeline); + } else if (entry.event) { + const event = entry.event; + switch (event.type) { + case "m.room.message": { + const content = event.content; + const msgtype = content && content.msgtype; + switch (msgtype) { + case "m.text": + case "m.notice": + case "m.emote": + return new TextTile(options); + case "m.image": + return new ImageTile(options); + case "m.location": + return new LocationTile(options); + default: + // unknown msgtype not rendered + return null; + } + } + case "m.room.name": + return new RoomNameTile(options); + case "m.room.member": + return new RoomMemberTile(options); + default: + // unknown type not rendered + return null; + } + } + } +} diff --git a/src/ui/viewmodels/RoomTileViewModel.js b/src/domain/session/roomlist/RoomTileViewModel.js similarity index 100% rename from src/ui/viewmodels/RoomTileViewModel.js rename to src/domain/session/roomlist/RoomTileViewModel.js diff --git a/src/main.js b/src/main.js index 7c165d73..f0787673 100644 --- a/src/main.js +++ b/src/main.js @@ -3,7 +3,7 @@ import Session from "./matrix/session.js"; import createIdbStorage from "./matrix/storage/idb/create.js"; import Sync from "./matrix/sync.js"; import SessionView from "./ui/web/SessionView.js"; -import SessionViewModel from "./ui/viewmodels/SessionViewModel.js"; +import SessionViewModel from "./domain/session/SessionViewModel.js"; const HOST = "localhost"; const HOMESERVER = `http://${HOST}:8008`; @@ -11,50 +11,61 @@ const USERNAME = "bruno1"; const USER_ID = `@${USERNAME}:${HOST}`; const PASSWORD = "testtest"; -function getSessionId(userId) { +function getSessionInfo(userId) { const sessionsJson = localStorage.getItem("morpheus_sessions_v1"); if (sessionsJson) { const sessions = JSON.parse(sessionsJson); const session = sessions.find(session => session.userId === userId); if (session) { - return session.id; + return session; } } } +function storeSessionInfo(loginData) { + const sessionsJson = localStorage.getItem("morpheus_sessions_v1"); + const sessions = sessionsJson ? JSON.parse(sessionsJson) : []; + const sessionId = (Math.floor(Math.random() * Number.MAX_SAFE_INTEGER)).toString(); + const sessionInfo = { + id: sessionId, + deviceId: loginData.device_id, + userId: loginData.user_id, + homeServer: loginData.home_server, + accessToken: loginData.access_token, + }; + sessions.push(sessionInfo); + localStorage.setItem("morpheus_sessions_v1", JSON.stringify(sessions)); + return sessionInfo; +} + async function login(username, password, homeserver) { const hsApi = new HomeServerApi(homeserver); const loginData = await hsApi.passwordLogin(username, password).response(); - const sessionsJson = localStorage.getItem("morpheus_sessions_v1"); - const sessions = sessionsJson ? JSON.parse(sessionsJson) : []; - const sessionId = (Math.floor(Math.random() * Number.MAX_SAFE_INTEGER)).toString(); - console.log(loginData); - sessions.push({userId: loginData.user_id, id: sessionId}); - localStorage.setItem("morpheus_sessions_v1", JSON.stringify(sessions)); - return {sessionId, loginData}; + return storeSessionInfo(loginData); } function showSession(container, session) { const vm = new SessionViewModel(session); const view = new SessionView(vm); - container.appendChild(view.mount()); + view.mount(); + container.appendChild(view.root()); } // eslint-disable-next-line no-unused-vars export default async function main(label, button, container) { try { - let sessionId = getSessionId(USER_ID); - let loginData; - if (!sessionId) { - ({sessionId, loginData} = await login(USERNAME, PASSWORD, HOMESERVER)); + let sessionInfo = getSessionInfo(USER_ID); + if (!sessionInfo) { + sessionInfo = await login(USERNAME, PASSWORD, HOMESERVER); } - const storage = await createIdbStorage(`morpheus_session_${sessionId}`); - const session = new Session(storage); - if (loginData) { - await session.setLoginData(loginData); - } - await session.load(); - const hsApi = new HomeServerApi(HOMESERVER, session.accessToken); + const storage = await createIdbStorage(`morpheus_session_${sessionInfo.id}`); + const hsApi = new HomeServerApi(HOMESERVER, sessionInfo.accessToken); + const session = new Session({storage, hsApi, sessionInfo: { + deviceId: sessionInfo.deviceId, + userId: sessionInfo.userId, + homeServer: sessionInfo.homeServer, //only pass relevant fields to Session + }}); + await session.load(); console.log("session loaded"); const needsInitialSync = !session.syncToken; if (needsInitialSync) { @@ -77,6 +88,6 @@ export default async function main(label, button, container) { label.innerText = "sync stopped"; }); } catch(err) { - console.error(err); + console.error(`${err.message}:\n${err.stack}`); } } diff --git a/src/matrix/error.js b/src/matrix/error.js index 83cc8652..520698d9 100644 --- a/src/matrix/error.js +++ b/src/matrix/error.js @@ -9,5 +9,7 @@ export class StorageError extends Error { } export class RequestAbortError extends Error { +} -} \ No newline at end of file +export class NetworkError extends Error { +} diff --git a/src/matrix/hs-api.js b/src/matrix/hs-api.js index 2eb2a1f1..4c6ebf86 100644 --- a/src/matrix/hs-api.js +++ b/src/matrix/hs-api.js @@ -1,6 +1,7 @@ import { HomeServerError, - RequestAbortError + RequestAbortError, + NetworkError } from "./error.js"; class RequestWrapper { @@ -20,7 +21,9 @@ class RequestWrapper { export default class HomeServerApi { constructor(homeserver, accessToken) { - this._homeserver = homeserver; + // store these both in a closure somehow so it's harder to get at in case of XSS? + // one could change the homeserver as well so the token gets sent there, so both must be protected from read/write + this._homeserver = homeserver; this._accessToken = accessToken; } @@ -30,7 +33,7 @@ export default class HomeServerApi { _request(method, csPath, queryParams = {}, body) { const queryString = Object.entries(queryParams) - .filter(([name, value]) => value !== undefined) + .filter(([, value]) => value !== undefined) .map(([name, value]) => `${encodeURIComponent(name)}=${encodeURIComponent(value)}`) .join("&"); const url = this._url(`${csPath}?${queryString}`); @@ -62,10 +65,18 @@ export default class HomeServerApi { } } }, err => { - switch (err.name) { - case "AbortError": throw new RequestAbortError(); - default: throw err; //new Error(`Unrecognized DOMException: ${err.name}`); - } + if (err.name === "AbortError") { + throw new RequestAbortError(); + } else if (err instanceof TypeError) { + // Network errors are reported as TypeErrors, see + // https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#Checking_that_the_fetch_was_successful + // this can either mean user is offline, server is offline, or a CORS error (server misconfiguration). + // + // One could check navigator.onLine to rule out the first + // but the 2 later ones are indistinguishable from javascript. + throw new NetworkError(err.message); + } + throw err; }); return new RequestWrapper(promise, controller); } @@ -82,6 +93,11 @@ export default class HomeServerApi { return this._get("/sync", {since, timeout, filter}); } + // params is from, dir and optionally to, limit, filter. + messages(roomId, params) { + return this._get(`/rooms/${roomId}/messages`, params); + } + passwordLogin(username, password) { return this._post("/login", undefined, { "type": "m.login.password", @@ -92,4 +108,4 @@ export default class HomeServerApi { "password": password }); } -} \ No newline at end of file +} diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js deleted file mode 100644 index c08fbc92..00000000 --- a/src/matrix/room/persister.js +++ /dev/null @@ -1,88 +0,0 @@ -import SortKey from "../storage/sortkey.js"; - -export default class RoomPersister { - constructor(roomId) { - this._roomId = roomId; - this._lastSortKey = new SortKey(); - } - - async load(txn) { - //fetch key here instead? - const [lastEvent] = await txn.roomTimeline.lastEvents(this._roomId, 1); - if (lastEvent) { - this._lastSortKey = new SortKey(lastEvent.sortKey); - console.log("room persister load", this._roomId, this._lastSortKey.toString()); - } else { - console.warn("could not recover last sort key for ", this._roomId); - } - } - - // async persistGapFill(...) { - - // } - - persistSync(roomResponse, txn) { - let nextKey = this._lastSortKey; - const timeline = roomResponse.timeline; - const entries = []; - // is limited true for initial sync???? or do we need to handle that as a special case? - // I suppose it will, yes - if (timeline.limited) { - nextKey = nextKey.nextKeyWithGap(); - entries.push(this._createGapEntry(nextKey, timeline.prev_batch)); - } - // const startOfChunkSortKey = nextKey; - if (timeline.events) { - for(const event of timeline.events) { - nextKey = nextKey.nextKey(); - entries.push(this._createEventEntry(nextKey, event)); - } - } - // write to store - for(const entry of entries) { - txn.roomTimeline.append(entry); - } - // right thing to do? if the txn fails, not sure we'll continue anyways ... - // only advance the key once the transaction has - // succeeded - txn.complete().then(() => { - console.log("txn complete, setting key"); - this._lastSortKey = nextKey; - }); - - // persist state - const state = roomResponse.state; - if (state.events) { - for (const event of state.events) { - txn.roomState.setStateEvent(this._roomId, event) - } - } - - if (timeline.events) { - for (const event of timeline.events) { - if (typeof event.state_key === "string") { - txn.roomState.setStateEvent(this._roomId, event); - } - } - } - return entries; - } - - _createGapEntry(sortKey, prevBatch) { - return { - roomId: this._roomId, - sortKey: sortKey.buffer, - event: null, - gap: {prev_batch: prevBatch} - }; - } - - _createEventEntry(sortKey, event) { - return { - roomId: this._roomId, - sortKey: sortKey.buffer, - event: event, - gap: null - }; - } -} diff --git a/src/matrix/room/room.js b/src/matrix/room/room.js index a581fbcb..7c17b3b1 100644 --- a/src/matrix/room/room.js +++ b/src/matrix/room/room.js @@ -1,22 +1,25 @@ import EventEmitter from "../../EventEmitter.js"; import RoomSummary from "./summary.js"; -import RoomPersister from "./persister.js"; -import Timeline from "./timeline.js"; +import SyncWriter from "./timeline/persistence/SyncWriter.js"; +import Timeline from "./timeline/Timeline.js"; +import FragmentIdComparer from "./timeline/FragmentIdComparer.js"; export default class Room extends EventEmitter { - constructor(roomId, storage, emitCollectionChange) { + constructor({roomId, storage, hsApi, emitCollectionChange}) { super(); - this._roomId = roomId; - this._storage = storage; + this._roomId = roomId; + this._storage = storage; + this._hsApi = hsApi; this._summary = new RoomSummary(roomId); - this._persister = new RoomPersister(roomId); + this._fragmentIdComparer = new FragmentIdComparer([]); + this._syncWriter = new SyncWriter({roomId, storage, fragmentIdComparer: this._fragmentIdComparer}); this._emitCollectionChange = emitCollectionChange; this._timeline = null; } - persistSync(roomResponse, membership, txn) { + async persistSync(roomResponse, membership, txn) { const summaryChanged = this._summary.applySync(roomResponse, membership, txn); - const newTimelineEntries = this._persister.persistSync(roomResponse, txn); + const newTimelineEntries = await this._syncWriter.writeSync(roomResponse, txn); return {summaryChanged, newTimelineEntries}; } @@ -32,7 +35,7 @@ export default class Room extends EventEmitter { load(summary, txn) { this._summary.load(summary); - return this._persister.load(txn); + return this._syncWriter.load(txn); } get name() { @@ -50,6 +53,8 @@ export default class Room extends EventEmitter { this._timeline = new Timeline({ roomId: this.id, storage: this._storage, + hsApi: this._hsApi, + fragmentIdComparer: this._fragmentIdComparer, closeCallback: () => this._timeline = null, }); await this._timeline.load(); diff --git a/src/matrix/room/timeline.js b/src/matrix/room/timeline.js deleted file mode 100644 index 44f5f83d..00000000 --- a/src/matrix/room/timeline.js +++ /dev/null @@ -1,36 +0,0 @@ -import { ObservableArray } from "../../observable/index.js"; - -export default class Timeline { - constructor({roomId, storage, closeCallback}) { - this._roomId = roomId; - this._storage = storage; - this._closeCallback = closeCallback; - this._entriesList = new ObservableArray(); - } - - /** @package */ - async load() { - const txn = await this._storage.readTxn([this._storage.storeNames.roomTimeline]); - const entries = await txn.roomTimeline.lastEvents(this._roomId, 100); - for (const entry of entries) { - this._entriesList.append(entry); - } - } - - /** @package */ - appendLiveEntries(newEntries) { - for (const entry of newEntries) { - this._entriesList.append(entry); - } - } - - /** @public */ - get entries() { - return this._entriesList; - } - - /** @public */ - close() { - this._closeCallback(); - } -} diff --git a/src/matrix/room/timeline/Direction.js b/src/matrix/room/timeline/Direction.js new file mode 100644 index 00000000..6e527e88 --- /dev/null +++ b/src/matrix/room/timeline/Direction.js @@ -0,0 +1,30 @@ + + +export default class Direction { + constructor(isForward) { + this._isForward = isForward; + } + + get isForward() { + return this._isForward; + } + + get isBackward() { + return !this.isForward; + } + + asApiString() { + return this.isForward ? "f" : "b"; + } + + static get Forward() { + return _forward; + } + + static get Backward() { + return _backward; + } +} + +const _forward = Object.freeze(new Direction(true)); +const _backward = Object.freeze(new Direction(false)); diff --git a/src/matrix/room/timeline/EventKey.js b/src/matrix/room/timeline/EventKey.js new file mode 100644 index 00000000..e05c38f4 --- /dev/null +++ b/src/matrix/room/timeline/EventKey.js @@ -0,0 +1,158 @@ +const DEFAULT_LIVE_FRAGMENT_ID = 0; +const MIN_EVENT_INDEX = Number.MIN_SAFE_INTEGER + 1; +const MAX_EVENT_INDEX = Number.MAX_SAFE_INTEGER - 1; +const MID_EVENT_INDEX = 0; + +// key for events in the timelineEvents store +export default class EventKey { + constructor(fragmentId, eventIndex) { + this.fragmentId = fragmentId; + this.eventIndex = eventIndex; + } + + nextFragmentKey() { + // could take MIN_EVENT_INDEX here if it can't be paged back + return new EventKey(this.fragmentId + 1, MID_EVENT_INDEX); + } + + nextKeyForDirection(direction) { + if (direction.isForward) { + return this.nextKey(); + } else { + return this.previousKey(); + } + } + + previousKey() { + return new EventKey(this.fragmentId, this.eventIndex - 1); + } + + nextKey() { + return new EventKey(this.fragmentId, this.eventIndex + 1); + } + + static get maxKey() { + return new EventKey(Number.MAX_SAFE_INTEGER, Number.MAX_SAFE_INTEGER); + } + + static get minKey() { + return new EventKey(Number.MIN_SAFE_INTEGER, Number.MIN_SAFE_INTEGER); + } + + static get defaultLiveKey() { + return new EventKey(DEFAULT_LIVE_FRAGMENT_ID, MID_EVENT_INDEX); + } + + toString() { + return `[${this.fragmentId}/${this.eventIndex}]`; + } +} + +//#ifdef TESTS +export function xtests() { + const fragmentIdComparer = {compare: (a, b) => a - b}; + + return { + test_no_fragment_index(assert) { + const min = EventKey.minKey; + const max = EventKey.maxKey; + const a = new EventKey(); + a.eventIndex = 1; + a.fragmentId = 1; + + assert(min.compare(min) === 0); + assert(max.compare(max) === 0); + assert(a.compare(a) === 0); + + assert(min.compare(max) < 0); + assert(max.compare(min) > 0); + + assert(min.compare(a) < 0); + assert(a.compare(min) > 0); + + assert(max.compare(a) > 0); + assert(a.compare(max) < 0); + }, + + test_default_key(assert) { + const k = new EventKey(fragmentIdComparer); + assert.equal(k.fragmentId, MID); + assert.equal(k.eventIndex, MID); + }, + + test_inc(assert) { + const a = new EventKey(fragmentIdComparer); + const b = a.nextKey(); + assert.equal(a.fragmentId, b.fragmentId); + assert.equal(a.eventIndex + 1, b.eventIndex); + const c = b.previousKey(); + assert.equal(b.fragmentId, c.fragmentId); + assert.equal(c.eventIndex + 1, b.eventIndex); + assert.equal(a.eventIndex, c.eventIndex); + }, + + test_min_key(assert) { + const minKey = EventKey.minKey; + const k = new EventKey(fragmentIdComparer); + assert(minKey.fragmentId <= k.fragmentId); + assert(minKey.eventIndex <= k.eventIndex); + assert(k.compare(minKey) > 0); + assert(minKey.compare(k) < 0); + }, + + test_max_key(assert) { + const maxKey = EventKey.maxKey; + const k = new EventKey(fragmentIdComparer); + assert(maxKey.fragmentId >= k.fragmentId); + assert(maxKey.eventIndex >= k.eventIndex); + assert(k.compare(maxKey) < 0); + assert(maxKey.compare(k) > 0); + }, + + test_immutable(assert) { + const a = new EventKey(fragmentIdComparer); + const fragmentId = a.fragmentId; + const eventIndex = a.eventIndex; + a.nextFragmentKey(); + assert.equal(a.fragmentId, fragmentId); + assert.equal(a.eventIndex, eventIndex); + }, + + test_cmp_fragmentid_first(assert) { + const a = new EventKey(fragmentIdComparer); + const b = new EventKey(fragmentIdComparer); + a.fragmentId = 2; + a.eventIndex = 1; + b.fragmentId = 1; + b.eventIndex = 100000; + assert(a.compare(b) > 0); + }, + + test_cmp_eventindex_second(assert) { + const a = new EventKey(fragmentIdComparer); + const b = new EventKey(fragmentIdComparer); + a.fragmentId = 1; + a.eventIndex = 100000; + b.fragmentId = 1; + b.eventIndex = 2; + assert(a.compare(b) > 0); + assert(b.compare(a) < 0); + }, + + test_cmp_max_larger_than_min(assert) { + assert(EventKey.minKey.compare(EventKey.maxKey) < 0); + }, + + test_cmp_fragmentid_first_large(assert) { + const a = new EventKey(fragmentIdComparer); + const b = new EventKey(fragmentIdComparer); + a.fragmentId = MAX; + a.eventIndex = MIN; + b.fragmentId = MIN; + b.eventIndex = MAX; + assert(b < a); + assert(a > b); + } + }; +} +//#endif diff --git a/src/matrix/room/timeline/FragmentIdComparer.js b/src/matrix/room/timeline/FragmentIdComparer.js new file mode 100644 index 00000000..c26aa043 --- /dev/null +++ b/src/matrix/room/timeline/FragmentIdComparer.js @@ -0,0 +1,252 @@ +/* +lookups will be far more frequent than changing fragment order, +so data structure should be optimized for fast lookup + +we can have a Map: fragmentId to sortIndex + +changing the order, we would need to rebuild the index +lets do this the stupid way for now, changing any fragment rebuilds all islands + +to build this: +first load all fragments +put them in a map by id +now iterate through them + +until no more fragments + get the first + create an island array, and add to list with islands + going backwards and forwards + get and remove sibling and prepend/append it to island array + stop when no more previous/next + return list with islands + +*/ + +import {isValidFragmentId} from "./common.js"; + +function findBackwardSiblingFragments(current, byId) { + const sortedSiblings = []; + while (isValidFragmentId(current.previousId)) { + const previous = byId.get(current.previousId); + if (!previous) { + break; + } + if (previous.nextId !== current.id) { + throw new Error(`Previous fragment ${previous.id} doesn't point back to ${current.id}`); + } + byId.delete(current.previousId); + sortedSiblings.unshift(previous); + current = previous; + } + return sortedSiblings; +} + +function findForwardSiblingFragments(current, byId) { + const sortedSiblings = []; + while (isValidFragmentId(current.nextId)) { + const next = byId.get(current.nextId); + if (!next) { + break; + } + if (next.previousId !== current.id) { + throw new Error(`Next fragment ${next.id} doesn't point back to ${current.id}`); + } + byId.delete(current.nextId); + sortedSiblings.push(next); + current = next; + } + return sortedSiblings; +} + + +function createIslands(fragments) { + const byId = new Map(); + for(let f of fragments) { + byId.set(f.id, f); + } + + const islands = []; + while(byId.size) { + const current = byId.values().next().value; + byId.delete(current.id); + // new island + const previousSiblings = findBackwardSiblingFragments(current, byId); + const nextSiblings = findForwardSiblingFragments(current, byId); + const island = previousSiblings.concat(current, nextSiblings); + islands.push(island); + } + return islands.map(a => new Island(a)); +} + +class Island { + constructor(sortedFragments) { + this._idToSortIndex = new Map(); + sortedFragments.forEach((f, i) => { + this._idToSortIndex.set(f.id, i); + }); + } + + compare(idA, idB) { + const sortIndexA = this._idToSortIndex.get(idA); + if (sortIndexA === undefined) { + throw new Error(`first id ${idA} isn't part of this island`); + } + const sortIndexB = this._idToSortIndex.get(idB); + if (sortIndexB === undefined) { + throw new Error(`second id ${idB} isn't part of this island`); + } + return sortIndexA - sortIndexB; + } + + get fragmentIds() { + return this._idToSortIndex.keys(); + } +} + +/* +index for fast lookup of how two fragments can be sorted +*/ +export default class FragmentIdComparer { + constructor(fragments) { + this._fragmentsById = fragments.reduce((map, f) => {map.set(f.id, f); return map;}, new Map()); + this.rebuild(fragments); + } + + _getIsland(id) { + const island = this._idToIsland.get(id); + if (island === undefined) { + throw new Error(`Unknown fragment id ${id}`); + } + return island; + } + + compare(idA, idB) { + if (idA === idB) { + return 0; + } + const islandA = this._getIsland(idA); + const islandB = this._getIsland(idB); + if (islandA !== islandB) { + throw new Error(`${idA} and ${idB} are on different islands, can't tell order`); + } + return islandA.compare(idA, idB); + } + + rebuild(fragments) { + const islands = createIslands(fragments); + this._idToIsland = new Map(); + for(let island of islands) { + for(let id of island.fragmentIds) { + this._idToIsland.set(id, island); + } + } + } + + add(fragment) { + this._fragmentsById.set(fragment.id, fragment); + this.rebuild(this._fragmentsById.values()); + } +} + +//#ifdef TESTS +export function tests() { + return { + test_1_island_3_fragments(assert) { + const index = new FragmentIdComparer([ + {id: 3, previousId: 2}, + {id: 1, nextId: 2}, + {id: 2, nextId: 3, previousId: 1}, + ]); + assert(index.compare(1, 2) < 0); + assert(index.compare(2, 1) > 0); + + assert(index.compare(1, 3) < 0); + assert(index.compare(3, 1) > 0); + + assert(index.compare(2, 3) < 0); + assert(index.compare(3, 2) > 0); + + assert.equal(index.compare(1, 1), 0); + }, + test_falsy_id(assert) { + const index = new FragmentIdComparer([ + {id: 0, nextId: 1}, + {id: 1, previousId: 0}, + ]); + assert(index.compare(0, 1) < 0); + assert(index.compare(1, 0) > 0); + }, + test_falsy_id_reverse(assert) { + const index = new FragmentIdComparer([ + {id: 1, previousId: 0}, + {id: 0, nextId: 1}, + ]); + assert(index.compare(0, 1) < 0); + assert(index.compare(1, 0) > 0); + }, + test_allow_unknown_id(assert) { + // as we tend to load fragments incrementally + // as events come into view, we need to allow + // unknown previousId/nextId in the fragments that we do load + assert.doesNotThrow(() => { + new FragmentIdComparer([ + {id: 1, previousId: 2}, + {id: 0, nextId: 3}, + ]); + }); + }, + test_throw_on_link_mismatch(assert) { + // as we tend to load fragments incrementally + // as events come into view, we need to allow + // unknown previousId/nextId in the fragments that we do load + assert.throws(() => { + new FragmentIdComparer([ + {id: 1, previousId: 0}, + {id: 0, nextId: 2}, + ]); + }); + }, + test_2_island_dont_compare(assert) { + const index = new FragmentIdComparer([ + {id: 1}, + {id: 2}, + ]); + assert.throws(() => index.compare(1, 2)); + assert.throws(() => index.compare(2, 1)); + }, + test_2_island_compare_internally(assert) { + const index = new FragmentIdComparer([ + {id: 1, nextId: 2}, + {id: 2, previousId: 1}, + {id: 11, nextId: 12}, + {id: 12, previousId: 11}, + + ]); + + assert(index.compare(1, 2) < 0); + assert(index.compare(11, 12) < 0); + + assert.throws(() => index.compare(1, 11)); + assert.throws(() => index.compare(12, 2)); + }, + test_unknown_id(assert) { + const index = new FragmentIdComparer([{id: 1}]); + assert.throws(() => index.compare(1, 2)); + assert.throws(() => index.compare(2, 1)); + }, + test_rebuild_flushes_old_state(assert) { + const index = new FragmentIdComparer([ + {id: 1, nextId: 2}, + {id: 2, previousId: 1}, + ]); + index.rebuild([ + {id: 11, nextId: 12}, + {id: 12, previousId: 11}, + ]); + + assert.throws(() => index.compare(1, 2)); + assert(index.compare(11, 12) < 0); + }, + } +} +//#endif diff --git a/src/matrix/room/timeline/Timeline.js b/src/matrix/room/timeline/Timeline.js new file mode 100644 index 00000000..52900777 --- /dev/null +++ b/src/matrix/room/timeline/Timeline.js @@ -0,0 +1,71 @@ +import { SortedArray } from "../../../observable/index.js"; +import Direction from "./Direction.js"; +import GapWriter from "./persistence/GapWriter.js"; +import TimelineReader from "./persistence/TimelineReader.js"; + +export default class Timeline { + constructor({roomId, storage, closeCallback, fragmentIdComparer, hsApi}) { + this._roomId = roomId; + this._storage = storage; + this._closeCallback = closeCallback; + this._fragmentIdComparer = fragmentIdComparer; + this._hsApi = hsApi; + this._entriesList = new SortedArray((a, b) => a.compare(b)); + this._timelineReader = new TimelineReader({ + roomId: this._roomId, + storage: this._storage, + fragmentIdComparer: this._fragmentIdComparer + }); + } + + /** @package */ + async load() { + const entries = await this._timelineReader.readFromEnd(100); + this._entriesList.setManySorted(entries); + } + + /** @package */ + appendLiveEntries(newEntries) { + this._entriesList.setManySorted(newEntries); + } + + /** @public */ + async fillGap(fragmentEntry, amount) { + const response = await this._hsApi.messages(this._roomId, { + from: fragmentEntry.token, + dir: fragmentEntry.direction.asApiString(), + limit: amount + }).response(); + const gapWriter = new GapWriter({ + roomId: this._roomId, + storage: this._storage, + fragmentIdComparer: this._fragmentIdComparer + }); + const newEntries = await gapWriter.writeFragmentFill(fragmentEntry, response); + this._entriesList.setManySorted(newEntries); + } + + // tries to prepend `amount` entries to the `entries` list. + async loadAtTop(amount) { + if (this._entriesList.length() === 0) { + return; + } + const firstEntry = this._entriesList.array()[0]; + const entries = await this._timelineReader.readFrom( + firstEntry.asEventKey(), + Direction.Backward, + amount + ); + this._entriesList.setManySorted(entries); + } + + /** @public */ + get entries() { + return this._entriesList; + } + + /** @public */ + close() { + this._closeCallback(); + } +} diff --git a/src/matrix/room/timeline/common.js b/src/matrix/room/timeline/common.js new file mode 100644 index 00000000..7565d8a4 --- /dev/null +++ b/src/matrix/room/timeline/common.js @@ -0,0 +1,3 @@ +export function isValidFragmentId(id) { + return typeof id === "number"; +} diff --git a/src/matrix/room/timeline/entries/BaseEntry.js b/src/matrix/room/timeline/entries/BaseEntry.js new file mode 100644 index 00000000..3ef00862 --- /dev/null +++ b/src/matrix/room/timeline/entries/BaseEntry.js @@ -0,0 +1,29 @@ +//entries can be sorted, first by fragment, then by entry index. +import EventKey from "../EventKey.js"; + +export default class BaseEntry { + constructor(fragmentIdComparer) { + this._fragmentIdComparer = fragmentIdComparer; + } + + get fragmentId() { + throw new Error("unimplemented"); + } + + get entryIndex() { + throw new Error("unimplemented"); + } + + compare(otherEntry) { + if (this.fragmentId === otherEntry.fragmentId) { + return this.entryIndex - otherEntry.entryIndex; + } else { + // This might throw if the relation of two fragments is unknown. + return this._fragmentIdComparer.compare(this.fragmentId, otherEntry.fragmentId); + } + } + + asEventKey() { + return new EventKey(this.fragmentId, this.entryIndex); + } +} diff --git a/src/matrix/room/timeline/entries/EventEntry.js b/src/matrix/room/timeline/entries/EventEntry.js new file mode 100644 index 00000000..ce3697fa --- /dev/null +++ b/src/matrix/room/timeline/entries/EventEntry.js @@ -0,0 +1,32 @@ +import BaseEntry from "./BaseEntry.js"; + +export default class EventEntry extends BaseEntry { + constructor(eventEntry, fragmentIdComparer) { + super(fragmentIdComparer); + this._eventEntry = eventEntry; + } + + get fragmentId() { + return this._eventEntry.fragmentId; + } + + get entryIndex() { + return this._eventEntry.eventIndex; + } + + get content() { + return this._eventEntry.event.content; + } + + get event() { + return this._eventEntry.event; + } + + get type() { + return this._eventEntry.event.type; + } + + get id() { + return this._eventEntry.event.event_id; + } +} diff --git a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js new file mode 100644 index 00000000..5eb772f3 --- /dev/null +++ b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js @@ -0,0 +1,100 @@ +import BaseEntry from "./BaseEntry.js"; +import Direction from "../Direction.js"; +import {isValidFragmentId} from "../common.js"; + +export default class FragmentBoundaryEntry extends BaseEntry { + constructor(fragment, isFragmentStart, fragmentIdComparer) { + super(fragmentIdComparer); + this._fragment = fragment; + // TODO: should isFragmentStart be Direction instead of bool? + this._isFragmentStart = isFragmentStart; + } + + static start(fragment, fragmentIdComparer) { + return new FragmentBoundaryEntry(fragment, true, fragmentIdComparer); + } + + static end(fragment, fragmentIdComparer) { + return new FragmentBoundaryEntry(fragment, false, fragmentIdComparer); + } + + get started() { + return this._isFragmentStart; + } + + get hasEnded() { + return !this.started; + } + + get fragment() { + return this._fragment; + } + + get fragmentId() { + return this._fragment.id; + } + + get entryIndex() { + if (this.started) { + return Number.MIN_SAFE_INTEGER; + } else { + return Number.MAX_SAFE_INTEGER; + } + } + + get isGap() { + return !!this.token; + } + + get token() { + if (this.started) { + return this.fragment.previousToken; + } else { + return this.fragment.nextToken; + } + } + + set token(token) { + if (this.started) { + this.fragment.previousToken = token; + } else { + this.fragment.nextToken = token; + } + } + + get linkedFragmentId() { + if (this.started) { + return this.fragment.previousId; + } else { + return this.fragment.nextId; + } + } + + set linkedFragmentId(id) { + if (this.started) { + this.fragment.previousId = id; + } else { + this.fragment.nextId = id; + } + } + + get hasLinkedFragment() { + return isValidFragmentId(this.linkedFragmentId); + } + + get direction() { + if (this.started) { + return Direction.Backward; + } else { + return Direction.Forward; + } + } + + withUpdatedFragment(fragment) { + return new FragmentBoundaryEntry(fragment, this._isFragmentStart, this._fragmentIdComparer); + } + + createNeighbourEntry(neighbour) { + return new FragmentBoundaryEntry(neighbour, !this._isFragmentStart, this._fragmentIdComparer); + } +} diff --git a/src/matrix/room/timeline/persistence/GapWriter.js b/src/matrix/room/timeline/persistence/GapWriter.js new file mode 100644 index 00000000..9941c712 --- /dev/null +++ b/src/matrix/room/timeline/persistence/GapWriter.js @@ -0,0 +1,229 @@ +import EventKey from "../EventKey.js"; +import EventEntry from "../entries/EventEntry.js"; +import {createEventEntry, directionalAppend} from "./common.js"; + +export default class GapWriter { + constructor({roomId, storage, fragmentIdComparer}) { + this._roomId = roomId; + this._storage = storage; + this._fragmentIdComparer = fragmentIdComparer; + } + // events is in reverse-chronological order (last event comes at index 0) if backwards + async _findOverlappingEvents(fragmentEntry, events, txn) { + const eventIds = events.map(e => e.event_id); + let nonOverlappingEvents = events; + let neighbourFragmentEntry; + const neighbourEventId = await txn.timelineEvents.findFirstOccurringEventId(this._roomId, eventIds); + if (neighbourEventId) { + // trim overlapping events + const neighbourEventIndex = events.findIndex(e => e.event_id === neighbourEventId); + nonOverlappingEvents = events.slice(0, neighbourEventIndex); + // get neighbour fragment to link it up later on + const neighbourEvent = await txn.timelineEvents.getByEventId(this._roomId, neighbourEventId); + const neighbourFragment = await txn.timelineFragments.get(this._roomId, neighbourEvent.fragmentId); + neighbourFragmentEntry = fragmentEntry.createNeighbourEntry(neighbourFragment); + } + return {nonOverlappingEvents, neighbourFragmentEntry}; + } + + async _findLastFragmentEventKey(fragmentEntry, txn) { + const {fragmentId, direction} = fragmentEntry; + if (direction.isBackward) { + const [firstEvent] = await txn.timelineEvents.firstEvents(this._roomId, fragmentId, 1); + return new EventKey(firstEvent.fragmentId, firstEvent.eventIndex); + } else { + const [lastEvent] = await txn.timelineEvents.lastEvents(this._roomId, fragmentId, 1); + return new EventKey(lastEvent.fragmentId, lastEvent.eventIndex); + } + } + + _storeEvents(events, startKey, direction, txn) { + const entries = []; + // events is in reverse chronological order for backwards pagination, + // e.g. order is moving away from the `from` point. + let key = startKey; + for(let event of events) { + key = key.nextKeyForDirection(direction); + const eventStorageEntry = createEventEntry(key, this._roomId, event); + txn.timelineEvents.insert(eventStorageEntry); + const eventEntry = new EventEntry(eventStorageEntry, this._fragmentIdComparer); + directionalAppend(entries, eventEntry, direction); + } + return entries; + } + + async _updateFragments(fragmentEntry, neighbourFragmentEntry, end, entries, txn) { + const {direction} = fragmentEntry; + directionalAppend(entries, fragmentEntry, direction); + // set `end` as token, and if we found an event in the step before, link up the fragments in the fragment entry + if (neighbourFragmentEntry) { + fragmentEntry.linkedFragmentId = neighbourFragmentEntry.fragmentId; + neighbourFragmentEntry.linkedFragmentId = fragmentEntry.fragmentId; + // if neighbourFragmentEntry was found, it means the events were overlapping, + // so no pagination should happen anymore. + neighbourFragmentEntry.token = null; + fragmentEntry.token = null; + + txn.timelineFragments.update(neighbourFragmentEntry.fragment); + directionalAppend(entries, neighbourFragmentEntry, direction); + + // update fragmentIdComparer here after linking up fragments + this._fragmentIdComparer.add(fragmentEntry.fragment); + this._fragmentIdComparer.add(neighbourFragmentEntry.fragment); + } else { + fragmentEntry.token = end; + } + txn.timelineFragments.update(fragmentEntry.fragment); + } + + async writeFragmentFill(fragmentEntry, response) { + const {fragmentId, direction} = fragmentEntry; + // chunk is in reverse-chronological order when backwards + const {chunk, start, end} = response; + let entries; + + if (!Array.isArray(chunk)) { + throw new Error("Invalid chunk in response"); + } + if (typeof end !== "string") { + throw new Error("Invalid end token in response"); + } + + const txn = await this._storage.readWriteTxn([ + this._storage.storeNames.timelineEvents, + this._storage.storeNames.timelineFragments, + ]); + + try { + // make sure we have the latest fragment from the store + const fragment = await txn.timelineFragments.get(this._roomId, fragmentId); + if (!fragment) { + throw new Error(`Unknown fragment: ${fragmentId}`); + } + fragmentEntry = fragmentEntry.withUpdatedFragment(fragment); + // check that the request was done with the token we are aware of (extra care to avoid timeline corruption) + if (fragmentEntry.token !== start) { + throw new Error("start is not equal to prev_batch or next_batch"); + } + // find last event in fragment so we get the eventIndex to begin creating keys at + let lastKey = await this._findLastFragmentEventKey(fragmentEntry, txn); + // find out if any event in chunk is already present using findFirstOrLastOccurringEventId + const { + nonOverlappingEvents, + neighbourFragmentEntry + } = await this._findOverlappingEvents(fragmentEntry, chunk, txn); + + // create entries for all events in chunk, add them to entries + entries = this._storeEvents(nonOverlappingEvents, lastKey, direction, txn); + await this._updateFragments(fragmentEntry, neighbourFragmentEntry, end, entries, txn); + } catch (err) { + txn.abort(); + throw err; + } + + await txn.complete(); + + return entries; + } +} + +//#ifdef TESTS +//import MemoryStorage from "../storage/memory/MemoryStorage.js"; + +export function xtests() { + const roomId = "!abc:hs.tld"; + + // sets sortKey and roomId on an array of entries + function createTimeline(roomId, entries) { + let key = new SortKey(); + for (let entry of entries) { + if (entry.gap && entry.gap.prev_batch) { + key = key.nextKeyWithGap(); + } + entry.sortKey = key; + if (entry.gap && entry.gap.next_batch) { + key = key.nextKeyWithGap(); + } else if (!entry.gap) { + key = key.nextKey(); + } + entry.roomId = roomId; + } + } + + function areSorted(entries) { + for (var i = 1; i < entries.length; i++) { + const isSorted = entries[i - 1].sortKey.compare(entries[i].sortKey) < 0; + if(!isSorted) { + return false + } + } + return true; + } + + return { + "test backwards gap fill with overlapping neighbouring event": async function(assert) { + const currentPaginationToken = "abc"; + const gap = {gap: {prev_batch: currentPaginationToken}}; + const storage = new MemoryStorage({roomTimeline: createTimeline(roomId, [ + {event: {event_id: "b"}}, + {gap: {next_batch: "ghi"}}, + gap, + ])}); + const persister = new RoomPersister({roomId, storage}); + const response = { + start: currentPaginationToken, + end: "def", + chunk: [ + {event_id: "a"}, + {event_id: "b"}, + {event_id: "c"}, + {event_id: "d"}, + ] + }; + const {newEntries, replacedEntries} = await persister.persistGapFill(gap, response); + // should only have taken events up till existing event + assert.equal(newEntries.length, 2); + assert.equal(newEntries[0].event.event_id, "c"); + assert.equal(newEntries[1].event.event_id, "d"); + assert.equal(replacedEntries.length, 2); + assert.equal(replacedEntries[0].gap.next_batch, "hij"); + assert.equal(replacedEntries[1].gap.prev_batch, currentPaginationToken); + assert(areSorted(newEntries)); + assert(areSorted(replacedEntries)); + }, + "test backwards gap fill with non-overlapping neighbouring event": async function(assert) { + const currentPaginationToken = "abc"; + const newPaginationToken = "def"; + const gap = {gap: {prev_batch: currentPaginationToken}}; + const storage = new MemoryStorage({roomTimeline: createTimeline(roomId, [ + {event: {event_id: "a"}}, + {gap: {next_batch: "ghi"}}, + gap, + ])}); + const persister = new RoomPersister({roomId, storage}); + const response = { + start: currentPaginationToken, + end: newPaginationToken, + chunk: [ + {event_id: "c"}, + {event_id: "d"}, + {event_id: "e"}, + {event_id: "f"}, + ] + }; + const {newEntries, replacedEntries} = await persister.persistGapFill(gap, response); + // should only have taken events up till existing event + assert.equal(newEntries.length, 5); + assert.equal(newEntries[0].gap.prev_batch, newPaginationToken); + assert.equal(newEntries[1].event.event_id, "c"); + assert.equal(newEntries[2].event.event_id, "d"); + assert.equal(newEntries[3].event.event_id, "e"); + assert.equal(newEntries[4].event.event_id, "f"); + assert(areSorted(newEntries)); + + assert.equal(replacedEntries.length, 1); + assert.equal(replacedEntries[0].gap.prev_batch, currentPaginationToken); + }, + } +} +//#endif diff --git a/src/matrix/room/timeline/persistence/SyncWriter.js b/src/matrix/room/timeline/persistence/SyncWriter.js new file mode 100644 index 00000000..550df584 --- /dev/null +++ b/src/matrix/room/timeline/persistence/SyncWriter.js @@ -0,0 +1,224 @@ +import EventKey from "../EventKey.js"; +import EventEntry from "../entries/EventEntry.js"; +import FragmentBoundaryEntry from "../entries/FragmentBoundaryEntry.js"; +import {createEventEntry} from "./common.js"; + +export default class SyncWriter { + constructor({roomId, storage, fragmentIdComparer}) { + this._roomId = roomId; + this._storage = storage; + this._fragmentIdComparer = fragmentIdComparer; + this._lastLiveKey = null; + } + + async load(txn) { + const liveFragment = await txn.timelineFragments.liveFragment(this._roomId); + if (liveFragment) { + const [lastEvent] = await txn.timelineEvents.lastEvents(this._roomId, liveFragment.id, 1); + // sorting and identifying (e.g. sort key and pk to insert) are a bit intertwined here + // we could split it up into a SortKey (only with compare) and + // a EventKey (no compare or fragment index) with nextkey methods and getters/setters for eventIndex/fragmentId + // we probably need to convert from one to the other though, so bother? + this._lastLiveKey = new EventKey(liveFragment.id, lastEvent.eventIndex); + } + // if there is no live fragment, we don't create it here because load gets a readonly txn. + // this is on purpose, load shouldn't modify the store + console.log("room persister load", this._roomId, this._lastLiveKey && this._lastLiveKey.toString()); + } + + async _createLiveFragment(txn, previousToken) { + const liveFragment = await txn.timelineFragments.liveFragment(this._roomId); + if (!liveFragment) { + if (!previousToken) { + previousToken = null; + } + const fragment = { + roomId: this._roomId, + id: EventKey.defaultLiveKey.fragmentId, + previousId: null, + nextId: null, + previousToken: previousToken, + nextToken: null + }; + txn.timelineFragments.add(fragment); + this._fragmentIdComparer.add(fragment); + return fragment; + } else { + return liveFragment; + } + } + + async _replaceLiveFragment(oldFragmentId, newFragmentId, previousToken, txn) { + const oldFragment = await txn.timelineFragments.get(this._roomId, oldFragmentId); + if (!oldFragment) { + throw new Error(`old live fragment doesn't exist: ${oldFragmentId}`); + } + oldFragment.nextId = newFragmentId; + txn.timelineFragments.update(oldFragment); + const newFragment = { + roomId: this._roomId, + id: newFragmentId, + previousId: oldFragmentId, + nextId: null, + previousToken: previousToken, + nextToken: null + }; + txn.timelineFragments.add(newFragment); + this._fragmentIdComparer.add(newFragment); + return {oldFragment, newFragment}; + } + + async writeSync(roomResponse, txn) { + const entries = []; + const timeline = roomResponse.timeline; + if (!this._lastLiveKey) { + // means we haven't synced this room yet (just joined or did initial sync) + + // as this is probably a limited sync, prev_batch should be there + // (but don't fail if it isn't, we won't be able to back-paginate though) + let liveFragment = await this._createLiveFragment(txn, timeline.prev_batch); + this._lastLiveKey = new EventKey(liveFragment.id, EventKey.defaultLiveKey.eventIndex); + entries.push(FragmentBoundaryEntry.start(liveFragment, this._fragmentIdComparer)); + } else if (timeline.limited) { + // replace live fragment for limited sync, *only* if we had a live fragment already + const oldFragmentId = this._lastLiveKey.fragmentId; + this._lastLiveKey = this._lastLiveKey.nextFragmentKey(); + const {oldFragment, newFragment} = await this._replaceLiveFragment(oldFragmentId, this._lastLiveKey.fragmentId, timeline.prev_batch, txn); + entries.push(FragmentBoundaryEntry.end(oldFragment, this._fragmentIdComparer)); + entries.push(FragmentBoundaryEntry.start(newFragment, this._fragmentIdComparer)); + } + let currentKey = this._lastLiveKey; + if (timeline.events) { + for(const event of timeline.events) { + currentKey = currentKey.nextKey(); + const entry = createEventEntry(currentKey, this._roomId, event); + txn.timelineEvents.insert(entry); + entries.push(new EventEntry(entry, this._fragmentIdComparer)); + } + } + // right thing to do? if the txn fails, not sure we'll continue anyways ... + // only advance the key once the transaction has succeeded + txn.complete().then(() => { + this._lastLiveKey = currentKey; + }) + + // persist state + const state = roomResponse.state; + if (state.events) { + for (const event of state.events) { + txn.roomState.setStateEvent(this._roomId, event); + } + } + // persist live state events in timeline + if (timeline.events) { + for (const event of timeline.events) { + if (typeof event.state_key === "string") { + txn.roomState.setStateEvent(this._roomId, event); + } + } + } + + return entries; + } +} + +//#ifdef TESTS +//import MemoryStorage from "../storage/memory/MemoryStorage.js"; + +export function xtests() { + const roomId = "!abc:hs.tld"; + + // sets sortKey and roomId on an array of entries + function createTimeline(roomId, entries) { + let key = new SortKey(); + for (let entry of entries) { + if (entry.gap && entry.gap.prev_batch) { + key = key.nextKeyWithGap(); + } + entry.sortKey = key; + if (entry.gap && entry.gap.next_batch) { + key = key.nextKeyWithGap(); + } else if (!entry.gap) { + key = key.nextKey(); + } + entry.roomId = roomId; + } + } + + function areSorted(entries) { + for (var i = 1; i < entries.length; i++) { + const isSorted = entries[i - 1].sortKey.compare(entries[i].sortKey) < 0; + if(!isSorted) { + return false + } + } + return true; + } + + return { + "test backwards gap fill with overlapping neighbouring event": async function(assert) { + const currentPaginationToken = "abc"; + const gap = {gap: {prev_batch: currentPaginationToken}}; + const storage = new MemoryStorage({roomTimeline: createTimeline(roomId, [ + {event: {event_id: "b"}}, + {gap: {next_batch: "ghi"}}, + gap, + ])}); + const persister = new RoomPersister({roomId, storage}); + const response = { + start: currentPaginationToken, + end: "def", + chunk: [ + {event_id: "a"}, + {event_id: "b"}, + {event_id: "c"}, + {event_id: "d"}, + ] + }; + const {newEntries, replacedEntries} = await persister.persistGapFill(gap, response); + // should only have taken events up till existing event + assert.equal(newEntries.length, 2); + assert.equal(newEntries[0].event.event_id, "c"); + assert.equal(newEntries[1].event.event_id, "d"); + assert.equal(replacedEntries.length, 2); + assert.equal(replacedEntries[0].gap.next_batch, "hij"); + assert.equal(replacedEntries[1].gap.prev_batch, currentPaginationToken); + assert(areSorted(newEntries)); + assert(areSorted(replacedEntries)); + }, + "test backwards gap fill with non-overlapping neighbouring event": async function(assert) { + const currentPaginationToken = "abc"; + const newPaginationToken = "def"; + const gap = {gap: {prev_batch: currentPaginationToken}}; + const storage = new MemoryStorage({roomTimeline: createTimeline(roomId, [ + {event: {event_id: "a"}}, + {gap: {next_batch: "ghi"}}, + gap, + ])}); + const persister = new RoomPersister({roomId, storage}); + const response = { + start: currentPaginationToken, + end: newPaginationToken, + chunk: [ + {event_id: "c"}, + {event_id: "d"}, + {event_id: "e"}, + {event_id: "f"}, + ] + }; + const {newEntries, replacedEntries} = await persister.persistGapFill(gap, response); + // should only have taken events up till existing event + assert.equal(newEntries.length, 5); + assert.equal(newEntries[0].gap.prev_batch, newPaginationToken); + assert.equal(newEntries[1].event.event_id, "c"); + assert.equal(newEntries[2].event.event_id, "d"); + assert.equal(newEntries[3].event.event_id, "e"); + assert.equal(newEntries[4].event.event_id, "f"); + assert(areSorted(newEntries)); + + assert.equal(replacedEntries.length, 1); + assert.equal(replacedEntries[0].gap.prev_batch, currentPaginationToken); + }, + } +} +//#endif diff --git a/src/matrix/room/timeline/persistence/TimelineReader.js b/src/matrix/room/timeline/persistence/TimelineReader.js new file mode 100644 index 00000000..7d86115b --- /dev/null +++ b/src/matrix/room/timeline/persistence/TimelineReader.js @@ -0,0 +1,84 @@ +import {directionalConcat, directionalAppend} from "./common.js"; +import EventKey from "../EventKey.js"; +import Direction from "../Direction.js"; +import EventEntry from "../entries/EventEntry.js"; +import FragmentBoundaryEntry from "../entries/FragmentBoundaryEntry.js"; + +export default class TimelineReader { + constructor({roomId, storage, fragmentIdComparer}) { + this._roomId = roomId; + this._storage = storage; + this._fragmentIdComparer = fragmentIdComparer; + } + + _openTxn() { + return this._storage.readTxn([ + this._storage.storeNames.timelineEvents, + this._storage.storeNames.timelineFragments, + ]); + } + + async readFrom(eventKey, direction, amount) { + const txn = await this._openTxn(); + return this._readFrom(eventKey, direction, amount, txn); + } + + async _readFrom(eventKey, direction, amount, txn) { + let entries = []; + const timelineStore = txn.timelineEvents; + const fragmentStore = txn.timelineFragments; + + while (entries.length < amount && eventKey) { + let eventsWithinFragment; + if (direction.isForward) { + eventsWithinFragment = await timelineStore.eventsAfter(this._roomId, eventKey, amount); + } else { + eventsWithinFragment = await timelineStore.eventsBefore(this._roomId, eventKey, amount); + } + const eventEntries = eventsWithinFragment.map(e => new EventEntry(e, this._fragmentIdComparer)); + entries = directionalConcat(entries, eventEntries, direction); + // prepend or append eventsWithinFragment to entries, and wrap them in EventEntry + + if (entries.length < amount) { + const fragment = await fragmentStore.get(this._roomId, eventKey.fragmentId); + // this._fragmentIdComparer.addFragment(fragment); + let fragmentEntry = new FragmentBoundaryEntry(fragment, direction.isBackward, this._fragmentIdComparer); + // append or prepend fragmentEntry, reuse func from GapWriter? + directionalAppend(entries, fragmentEntry, direction); + // don't count it in amount perhaps? or do? + if (fragmentEntry.hasLinkedFragment) { + const nextFragment = await fragmentStore.get(this._roomId, fragmentEntry.linkedFragmentId); + this._fragmentIdComparer.add(nextFragment); + const nextFragmentEntry = new FragmentBoundaryEntry(nextFragment, direction.isForward, this._fragmentIdComparer); + directionalAppend(entries, nextFragmentEntry, direction); + eventKey = nextFragmentEntry.asEventKey(); + } else { + eventKey = null; + } + } + } + + return entries; + } + + async readFromEnd(amount) { + const txn = await this._openTxn(); + const liveFragment = await txn.timelineFragments.liveFragment(this._roomId); + // room hasn't been synced yet + if (!liveFragment) { + return []; + } + this._fragmentIdComparer.add(liveFragment); + const liveFragmentEntry = FragmentBoundaryEntry.end(liveFragment, this._fragmentIdComparer); + const eventKey = liveFragmentEntry.asEventKey(); + const entries = await this._readFrom(eventKey, Direction.Backward, amount, txn); + entries.unshift(liveFragmentEntry); + return entries; + } + + // reads distance up and down from eventId + // or just expose eventIdToKey? + readAtEventId(eventId, distance) { + return null; + } +} diff --git a/src/matrix/room/timeline/persistence/common.js b/src/matrix/room/timeline/persistence/common.js new file mode 100644 index 00000000..93d96f94 --- /dev/null +++ b/src/matrix/room/timeline/persistence/common.js @@ -0,0 +1,24 @@ +export function createEventEntry(key, roomId, event) { + return { + fragmentId: key.fragmentId, + eventIndex: key.eventIndex, + roomId, + event: event, + }; +} + +export function directionalAppend(array, value, direction) { + if (direction.isForward) { + array.push(value); + } else { + array.unshift(value); + } +} + +export function directionalConcat(array, otherArray, direction) { + if (direction.isForward) { + return array.concat(otherArray); + } else { + return otherArray.concat(array); + } +} diff --git a/src/matrix/session.js b/src/matrix/session.js index e030a3dd..763d0265 100644 --- a/src/matrix/session.js +++ b/src/matrix/session.js @@ -2,64 +2,61 @@ import Room from "./room/room.js"; import { ObservableMap } from "../observable/index.js"; export default class Session { - constructor(storage) { - this._storage = storage; - this._session = null; - this._rooms = new ObservableMap(); + // sessionInfo contains deviceId, userId and homeServer + constructor({storage, hsApi, sessionInfo}) { + this._storage = storage; + this._hsApi = hsApi; + this._session = null; + this._sessionInfo = sessionInfo; + this._rooms = new ObservableMap(); this._roomUpdateCallback = (room, params) => this._rooms.update(room.id, params); - } - // should be called before load - // loginData has device_id, user_id, home_server, access_token - async setLoginData(loginData) { - console.log("session.setLoginData"); - const txn = await this._storage.readWriteTxn([this._storage.storeNames.session]); - const session = {loginData}; - txn.session.set(session); - await txn.complete(); - } + } - async load() { - const txn = await this._storage.readTxn([ - this._storage.storeNames.session, - this._storage.storeNames.roomSummary, - this._storage.storeNames.roomState, - this._storage.storeNames.roomTimeline, - ]); - // restore session object - this._session = await txn.session.get(); - if (!this._session) { - throw new Error("session store is empty"); - } - // load rooms - const rooms = await txn.roomSummary.getAll(); - await Promise.all(rooms.map(summary => { - const room = this.createRoom(summary.roomId); - return room.load(summary, txn); - })); - } + async load() { + const txn = await this._storage.readTxn([ + this._storage.storeNames.session, + this._storage.storeNames.roomSummary, + this._storage.storeNames.roomState, + this._storage.storeNames.timelineEvents, + this._storage.storeNames.timelineFragments, + ]); + // restore session object + this._session = await txn.session.get(); + if (!this._session) { + this._session = {}; + return; + } + // load rooms + const rooms = await txn.roomSummary.getAll(); + await Promise.all(rooms.map(summary => { + const room = this.createRoom(summary.roomId); + return room.load(summary, txn); + })); + } get rooms() { return this._rooms; } - createRoom(roomId) { - const room = new Room(roomId, this._storage, this._roomUpdateCallback); - this._rooms.add(roomId, room); - return room; - } + createRoom(roomId) { + const room = new Room({ + roomId, + storage: this._storage, + emitCollectionChange: this._roomUpdateCallback, + hsApi: this._hsApi, + }); + this._rooms.add(roomId, room); + return room; + } - applySync(syncToken, accountData, txn) { - if (syncToken !== this._session.syncToken) { - this._session.syncToken = syncToken; - txn.session.set(this._session); - } - } + persistSync(syncToken, accountData, txn) { + if (syncToken !== this._session.syncToken) { + this._session.syncToken = syncToken; + txn.session.set(this._session); + } + } - get syncToken() { - return this._session.syncToken; - } - - get accessToken() { - return this._session.loginData.access_token; - } + get syncToken() { + return this._session.syncToken; + } } diff --git a/src/matrix/storage/common.js b/src/matrix/storage/common.js new file mode 100644 index 00000000..d1050e5a --- /dev/null +++ b/src/matrix/storage/common.js @@ -0,0 +1,6 @@ +export const STORE_NAMES = Object.freeze(["session", "roomState", "roomSummary", "timelineEvents", "timelineFragments"]); + +export const STORE_MAP = Object.freeze(STORE_NAMES.reduce((nameMap, name) => { + nameMap[name] = name; + return nameMap; +}, {})); diff --git a/src/matrix/storage/idb/create.js b/src/matrix/storage/idb/create.js index 42f66e04..ce8b6aca 100644 --- a/src/matrix/storage/idb/create.js +++ b/src/matrix/storage/idb/create.js @@ -2,31 +2,33 @@ import Storage from "./storage.js"; import { openDatabase } from "./utils.js"; export default async function createIdbStorage(databaseName) { - const db = await openDatabase(databaseName, createStores, 1); - return new Storage(db); + const db = await openDatabase(databaseName, createStores, 1); + return new Storage(db); } function createStores(db) { - db.createObjectStore("session", {keyPath: "key"}); - // any way to make keys unique here? (just use put?) - db.createObjectStore("roomSummary", {keyPath: "roomId"}); - // needs roomId separate because it might hold a gap and no event - const timeline = db.createObjectStore("roomTimeline", {keyPath: ["roomId", "sortKey"]}); - timeline.createIndex("byEventId", [ - "roomId", - "event.event_id" - ], {unique: true}); + db.createObjectStore("session", {keyPath: "key"}); + // any way to make keys unique here? (just use put?) + db.createObjectStore("roomSummary", {keyPath: "roomId"}); - db.createObjectStore("roomState", {keyPath: [ - "roomId", - "event.type", - "event.state_key" - ]}); - - // const roomMembers = db.createObjectStore("roomMembers", {keyPath: [ - // "event.room_id", - // "event.content.membership", - // "event.state_key" - // ]}); - // roomMembers.createIndex("byName", ["room_id", "content.name"]); -} \ No newline at end of file + // need index to find live fragment? prooobably ok without for now + db.createObjectStore("timelineFragments", {keyPath: ["roomId", "id"]}); + const timelineEvents = db.createObjectStore("timelineEvents", {keyPath: ["roomId", "fragmentId", "eventIndex"]}); + timelineEvents.createIndex("byEventId", [ + "roomId", + "event.event_id" + ], {unique: true}); + + db.createObjectStore("roomState", {keyPath: [ + "roomId", + "event.type", + "event.state_key" + ]}); + + // const roomMembers = db.createObjectStore("roomMembers", {keyPath: [ + // "event.room_id", + // "event.content.membership", + // "event.state_key" + // ]}); + // roomMembers.createIndex("byName", ["room_id", "content.name"]); +} diff --git a/src/matrix/storage/idb/query-target.js b/src/matrix/storage/idb/query-target.js index 51b20c7b..ec5c4530 100644 --- a/src/matrix/storage/idb/query-target.js +++ b/src/matrix/storage/idb/query-target.js @@ -1,16 +1,20 @@ -import {iterateCursor} from "./utils.js"; +import {iterateCursor, reqAsPromise} from "./utils.js"; export default class QueryTarget { constructor(target) { this._target = target; } + get(key) { + return reqAsPromise(this._target.get(key)); + } + reduce(range, reducer, initialValue) { return this._reduce(range, reducer, initialValue, "next"); } reduceReverse(range, reducer, initialValue) { - return this._reduce(range, reducer, initialValue, "next"); + return this._reduce(range, reducer, initialValue, "prev"); } selectLimit(range, amount) { @@ -34,7 +38,7 @@ export default class QueryTarget { const results = []; await iterateCursor(cursor, (value) => { results.push(value); - return false; + return {done: false}; }); return results; } @@ -55,12 +59,48 @@ export default class QueryTarget { return this._find(range, predicate, "prev"); } + /** + * Checks if a given set of keys exist. + * Calls `callback(key, found)` for each key in `keys`, in key sorting order (or reversed if backwards=true). + * If the callback returns true, the search is halted and callback won't be called again. + * `callback` is called with the same instances of the key as given in `keys`, so direct comparison can be used. + */ + async findExistingKeys(keys, backwards, callback) { + const direction = backwards ? "prev" : "next"; + const compareKeys = (a, b) => backwards ? -indexedDB.cmp(a, b) : indexedDB.cmp(a, b); + const sortedKeys = keys.slice().sort(compareKeys); + const firstKey = backwards ? sortedKeys[sortedKeys.length - 1] : sortedKeys[0]; + const lastKey = backwards ? sortedKeys[0] : sortedKeys[sortedKeys.length - 1]; + const cursor = this._target.openKeyCursor(IDBKeyRange.bound(firstKey, lastKey), direction); + let i = 0; + let consumerDone = false; + await iterateCursor(cursor, (value, key) => { + // while key is larger than next key, advance and report false + while(i < sortedKeys.length && compareKeys(sortedKeys[i], key) < 0 && !consumerDone) { + consumerDone = callback(sortedKeys[i], false); + ++i; + } + if (i < sortedKeys.length && compareKeys(sortedKeys[i], key) === 0 && !consumerDone) { + consumerDone = callback(sortedKeys[i], true); + ++i; + } + const done = consumerDone || i >= sortedKeys.length; + const jumpTo = !done && sortedKeys[i]; + return {done, jumpTo}; + }); + // report null for keys we didn't to at the end + while (!consumerDone && i < sortedKeys.length) { + consumerDone = callback(sortedKeys[i], false); + ++i; + } + } + _reduce(range, reducer, initialValue, direction) { let reducedValue = initialValue; const cursor = this._target.openCursor(range, direction); return iterateCursor(cursor, (value) => { reducedValue = reducer(reducedValue, value); - return true; + return {done: false}; }); } @@ -75,7 +115,7 @@ export default class QueryTarget { const results = []; await iterateCursor(cursor, (value) => { results.push(value); - return predicate(results); + return {done: predicate(results)}; }); return results; } @@ -88,10 +128,10 @@ export default class QueryTarget { if (found) { result = value; } - return found; + return {done: found}; }); if (found) { return result; } } -} \ No newline at end of file +} diff --git a/src/matrix/storage/idb/storage.js b/src/matrix/storage/idb/storage.js index b4a6659c..ca7e2256 100644 --- a/src/matrix/storage/idb/storage.js +++ b/src/matrix/storage/idb/storage.js @@ -1,6 +1,5 @@ import Transaction from "./transaction.js"; - -export const STORE_NAMES = ["session", "roomState", "roomSummary", "roomTimeline"]; +import { STORE_NAMES } from "../common.js"; export default class Storage { constructor(idbDatabase) { @@ -30,4 +29,4 @@ export default class Storage { const txn = this._db.transaction(storeNames, "readwrite"); return new Transaction(txn, storeNames); } -} \ No newline at end of file +} diff --git a/src/matrix/storage/idb/stores/RoomTimelineStore.js b/src/matrix/storage/idb/stores/RoomTimelineStore.js deleted file mode 100644 index cd5ad4e5..00000000 --- a/src/matrix/storage/idb/stores/RoomTimelineStore.js +++ /dev/null @@ -1,64 +0,0 @@ -import SortKey from "../../sortkey.js"; - -export default class RoomTimelineStore { - constructor(timelineStore) { - this._timelineStore = timelineStore; - } - - async lastEvents(roomId, amount) { - return this.eventsBefore(roomId, SortKey.maxKey, amount); - } - - async firstEvents(roomId, amount) { - return this.eventsAfter(roomId, SortKey.minKey, amount); - } - - eventsAfter(roomId, sortKey, amount) { - const range = IDBKeyRange.bound([roomId, sortKey.buffer], [roomId, SortKey.maxKey.buffer], true, false); - return this._timelineStore.selectLimit(range, amount); - } - - async eventsBefore(roomId, sortKey, amount) { - const range = IDBKeyRange.bound([roomId, SortKey.minKey.buffer], [roomId, sortKey.buffer], false, true); - const events = await this._timelineStore.selectLimitReverse(range, amount); - events.reverse(); // because we fetched them backwards - return events; - } - - // entry should have roomId, sortKey, event & gap keys - append(entry) { - this._timelineStore.add(entry); - } - // should this happen as part of a transaction that stores all synced in changes? - // e.g.: - // - timeline events for all rooms - // - latest sync token - // - new members - // - new room state - // - updated/new account data - - - - appendGap(roomId, sortKey, gap) { - this._timelineStore.add({ - roomId: roomId, - sortKey: sortKey.buffer, - event: null, - gap: gap, - }); - } - - appendEvent(roomId, sortKey, event) { - console.info(`appending event for room ${roomId} with key ${sortKey}`); - this._timelineStore.add({ - roomId: roomId, - sortKey: sortKey.buffer, - event: event, - gap: null, - }); - } - // could be gap or event - async removeEntry(roomId, sortKey) { - this._timelineStore.delete([roomId, sortKey.buffer]); - } -} diff --git a/src/matrix/storage/idb/stores/TimelineEventStore.js b/src/matrix/storage/idb/stores/TimelineEventStore.js new file mode 100644 index 00000000..7c333245 --- /dev/null +++ b/src/matrix/storage/idb/stores/TimelineEventStore.js @@ -0,0 +1,228 @@ +import EventKey from "../../../room/timeline/EventKey.js"; + +class Range { + constructor(only, lower, upper, lowerOpen, upperOpen) { + this._only = only; + this._lower = lower; + this._upper = upper; + this._lowerOpen = lowerOpen; + this._upperOpen = upperOpen; + } + + asIDBKeyRange(roomId) { + // only + if (this._only) { + return IDBKeyRange.only([roomId, this._only.fragmentId, this._only.eventIndex]); + } + // lowerBound + // also bound as we don't want to move into another roomId + if (this._lower && !this._upper) { + return IDBKeyRange.bound( + [roomId, this._lower.fragmentId, this._lower.eventIndex], + [roomId, this._lower.fragmentId, EventKey.maxKey.eventIndex], + this._lowerOpen, + false + ); + } + // upperBound + // also bound as we don't want to move into another roomId + if (!this._lower && this._upper) { + return IDBKeyRange.bound( + [roomId, this._upper.fragmentId, EventKey.minKey.eventIndex], + [roomId, this._upper.fragmentId, this._upper.eventIndex], + false, + this._upperOpen + ); + } + // bound + if (this._lower && this._upper) { + return IDBKeyRange.bound( + [roomId, this._lower.fragmentId, this._lower.eventIndex], + [roomId, this._upper.fragmentId, this._upper.eventIndex], + this._lowerOpen, + this._upperOpen + ); + } + } +} +/* + * @typedef {Object} Gap + * @property {?string} prev_batch the pagination token for this backwards facing gap + * @property {?string} next_batch the pagination token for this forwards facing gap + * + * @typedef {Object} Event + * @property {string} event_id the id of the event + * @property {string} type the + * @property {?string} state_key the state key of this state event + * + * @typedef {Object} Entry + * @property {string} roomId + * @property {EventKey} eventKey + * @property {?Event} event if an event entry, the event + * @property {?Gap} gap if a gap entry, the gap +*/ +export default class TimelineEventStore { + constructor(timelineStore) { + this._timelineStore = timelineStore; + } + + /** Creates a range that only includes the given key + * @param {EventKey} eventKey the key + * @return {Range} the created range + */ + onlyRange(eventKey) { + return new Range(eventKey); + } + + /** Creates a range that includes all keys before eventKey, and optionally also the key itself. + * @param {EventKey} eventKey the key + * @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the upper end. + * @return {Range} the created range + */ + upperBoundRange(eventKey, open=false) { + return new Range(undefined, undefined, eventKey, undefined, open); + } + + /** Creates a range that includes all keys after eventKey, and optionally also the key itself. + * @param {EventKey} eventKey the key + * @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the lower end. + * @return {Range} the created range + */ + lowerBoundRange(eventKey, open=false) { + return new Range(undefined, eventKey, undefined, open); + } + + /** Creates a range that includes all keys between `lower` and `upper`, and optionally the given keys as well. + * @param {EventKey} lower the lower key + * @param {EventKey} upper the upper key + * @param {boolean} [lowerOpen=false] whether the lower key is included (false) or excluded (true) from the range. + * @param {boolean} [upperOpen=false] whether the upper key is included (false) or excluded (true) from the range. + * @return {Range} the created range + */ + boundRange(lower, upper, lowerOpen=false, upperOpen=false) { + return new Range(undefined, lower, upper, lowerOpen, upperOpen); + } + + /** Looks up the last `amount` entries in the timeline for `roomId`. + * @param {string} roomId + * @param {number} fragmentId + * @param {number} amount + * @return {Promise} a promise resolving to an array with 0 or more entries, in ascending order. + */ + async lastEvents(roomId, fragmentId, amount) { + const eventKey = EventKey.maxKey; + eventKey.fragmentId = fragmentId; + return this.eventsBefore(roomId, eventKey, amount); + } + + /** Looks up the first `amount` entries in the timeline for `roomId`. + * @param {string} roomId + * @param {number} fragmentId + * @param {number} amount + * @return {Promise} a promise resolving to an array with 0 or more entries, in ascending order. + */ + async firstEvents(roomId, fragmentId, amount) { + const eventKey = EventKey.minKey; + eventKey.fragmentId = fragmentId; + return this.eventsAfter(roomId, eventKey, amount); + } + + /** Looks up `amount` entries after `eventKey` in the timeline for `roomId` within the same fragment. + * The entry for `eventKey` is not included. + * @param {string} roomId + * @param {EventKey} eventKey + * @param {number} amount + * @return {Promise} a promise resolving to an array with 0 or more entries, in ascending order. + */ + eventsAfter(roomId, eventKey, amount) { + const idbRange = this.lowerBoundRange(eventKey, true).asIDBKeyRange(roomId); + return this._timelineStore.selectLimit(idbRange, amount); + } + + /** Looks up `amount` entries before `eventKey` in the timeline for `roomId` within the same fragment. + * The entry for `eventKey` is not included. + * @param {string} roomId + * @param {EventKey} eventKey + * @param {number} amount + * @return {Promise} a promise resolving to an array with 0 or more entries, in ascending order. + */ + async eventsBefore(roomId, eventKey, amount) { + const range = this.upperBoundRange(eventKey, true).asIDBKeyRange(roomId); + const events = await this._timelineStore.selectLimitReverse(range, amount); + events.reverse(); // because we fetched them backwards + return events; + } + + /** Finds the first eventId that occurs in the store, if any. + * For optimal performance, `eventIds` should be in chronological order. + * + * The order in which results are returned might be different than `eventIds`. + * Call the return value to obtain the next {id, event} pair. + * @param {string} roomId + * @param {string[]} eventIds + * @return {Function} + */ + // performance comment from above refers to the fact that there *might* + // be a correlation between event_id sorting order and chronology. + // In that case we could avoid running over all eventIds, as the reported order by findExistingKeys + // would match the order of eventIds. That's why findLast is also passed as backwards to keysExist. + // also passing them in chronological order makes sense as that's how we'll receive them almost always. + async findFirstOccurringEventId(roomId, eventIds) { + const byEventId = this._timelineStore.index("byEventId"); + const keys = eventIds.map(eventId => [roomId, eventId]); + const results = new Array(keys.length); + let firstFoundKey; + + // find first result that is found and has no undefined results before it + function firstFoundAndPrecedingResolved() { + for(let i = 0; i < results.length; ++i) { + if (results[i] === undefined) { + return; + } else if(results[i] === true) { + return keys[i]; + } + } + } + + await byEventId.findExistingKeys(keys, false, (key, found) => { + const index = keys.indexOf(key); + results[index] = found; + firstFoundKey = firstFoundAndPrecedingResolved(); + return !!firstFoundKey; + }); + // key of index is [roomId, eventId], so pick out eventId + return firstFoundKey && firstFoundKey[1]; + } + + /** Inserts a new entry into the store. The combination of roomId and eventKey should not exist yet, or an error is thrown. + * @param {Entry} entry the entry to insert + * @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not. + * @throws {StorageError} ... + */ + insert(entry) { + // TODO: map error? or in idb/store? + return this._timelineStore.add(entry); + } + + /** Updates the entry into the store with the given [roomId, eventKey] combination. + * If not yet present, will insert. Might be slower than add. + * @param {Entry} entry the entry to update. + * @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not. + */ + update(entry) { + return this._timelineStore.put(entry); + } + + get(roomId, eventKey) { + return this._timelineStore.get([roomId, eventKey.fragmentId, eventKey.eventIndex]); + } + // returns the entries as well!! (or not always needed? I guess not always needed, so extra method) + removeRange(roomId, range) { + // TODO: read the entries! + return this._timelineStore.delete(range.asIDBKeyRange(roomId)); + } + + getByEventId(roomId, eventId) { + return this._timelineStore.index("byEventId").get([roomId, eventId]); + } +} diff --git a/src/matrix/storage/idb/stores/TimelineFragmentStore.js b/src/matrix/storage/idb/stores/TimelineFragmentStore.js new file mode 100644 index 00000000..86bd243f --- /dev/null +++ b/src/matrix/storage/idb/stores/TimelineFragmentStore.js @@ -0,0 +1,46 @@ +export default class RoomFragmentStore { + constructor(store) { + this._store = store; + } + + _allRange(roomId) { + return IDBKeyRange.bound( + [roomId, Number.MIN_SAFE_INTEGER], + [roomId, Number.MAX_SAFE_INTEGER] + ); + } + + all(roomId) { + return this._store.selectAll(this._allRange(roomId)); + } + + /** Returns the fragment without a nextToken and without nextId, + if any, with the largest id if there are multiple (which should not happen) */ + liveFragment(roomId) { + // why do we need this? + // Ok, take the case where you've got a /context fragment and a /sync fragment + // They are not connected. So, upon loading the persister, which one do we take? We can't sort them ... + // we assume that the one without a nextToken and without a nextId is a live one + // there should really be only one like this + + // reverse because assuming live fragment has bigger id than non-live ones + return this._store.findReverse(this._allRange(roomId), fragment => { + return typeof fragment.nextId !== "number" && typeof fragment.nextToken !== "string"; + }); + } + + // should generate an id an return it? + // depends if we want to do anything smart with fragment ids, + // like give them meaning depending on range. not for now probably ... + add(fragment) { + return this._store.add(fragment); + } + + update(fragment) { + return this._store.put(fragment); + } + + get(roomId, fragmentId) { + return this._store.get([roomId, fragmentId]); + } +} diff --git a/src/matrix/storage/idb/transaction.js b/src/matrix/storage/idb/transaction.js index 78883964..1a16d08d 100644 --- a/src/matrix/storage/idb/transaction.js +++ b/src/matrix/storage/idb/transaction.js @@ -2,8 +2,9 @@ import {txnAsPromise} from "./utils.js"; import Store from "./store.js"; import SessionStore from "./stores/SessionStore.js"; import RoomSummaryStore from "./stores/RoomSummaryStore.js"; -import RoomTimelineStore from "./stores/RoomTimelineStore.js"; +import TimelineEventStore from "./stores/TimelineEventStore.js"; import RoomStateStore from "./stores/RoomStateStore.js"; +import TimelineFragmentStore from "./stores/TimelineFragmentStore.js"; export default class Transaction { constructor(txn, allowedStoreNames) { @@ -41,8 +42,12 @@ export default class Transaction { return this._store("roomSummary", idbStore => new RoomSummaryStore(idbStore)); } - get roomTimeline() { - return this._store("roomTimeline", idbStore => new RoomTimelineStore(idbStore)); + get timelineFragments() { + return this._store("timelineFragments", idbStore => new TimelineFragmentStore(idbStore)); + } + + get timelineEvents() { + return this._store("timelineEvents", idbStore => new TimelineEventStore(idbStore)); } get roomState() { @@ -56,4 +61,4 @@ export default class Transaction { abort() { this._txn.abort(); } -} \ No newline at end of file +} diff --git a/src/matrix/storage/idb/utils.js b/src/matrix/storage/idb/utils.js index 1d522f24..83ef3411 100644 --- a/src/matrix/storage/idb/utils.js +++ b/src/matrix/storage/idb/utils.js @@ -35,11 +35,11 @@ export function iterateCursor(cursor, processValue) { resolve(false); return; // end of results } - const isDone = processValue(cursor.value); - if (isDone) { + const {done, jumpTo} = processValue(cursor.value, cursor.key); + if (done) { resolve(true); } else { - cursor.continue(); + cursor.continue(jumpTo); } }; }); @@ -49,7 +49,7 @@ export async function fetchResults(cursor, isDone) { const results = []; await iterateCursor(cursor, (value) => { results.push(value); - return isDone(results); + return {done: isDone(results)}; }); return results; } @@ -100,4 +100,4 @@ export async function findStoreValue(db, storeName, toCursor, matchesValue) { throw new Error("Value not found"); } return match; -} \ No newline at end of file +} diff --git a/src/matrix/storage/memory/Storage.js b/src/matrix/storage/memory/Storage.js new file mode 100644 index 00000000..fb178b01 --- /dev/null +++ b/src/matrix/storage/memory/Storage.js @@ -0,0 +1,37 @@ +import Transaction from "./transaction.js"; +import { STORE_MAP, STORE_NAMES } from "../common.js"; + +export default class Storage { + constructor(initialStoreValues = {}) { + this._validateStoreNames(Object.keys(initialStoreValues)); + this.storeNames = STORE_MAP; + this._storeValues = STORE_NAMES.reduce((values, name) => { + values[name] = initialStoreValues[name] || null; + }, {}); + } + + _validateStoreNames(storeNames) { + const idx = storeNames.findIndex(name => !STORE_MAP.hasOwnProperty(name)); + if (idx !== -1) { + throw new Error(`Invalid store name ${storeNames[idx]}`); + } + } + + _createTxn(storeNames, writable) { + this._validateStoreNames(storeNames); + const storeValues = storeNames.reduce((values, name) => { + return values[name] = this._storeValues[name]; + }, {}); + return Promise.resolve(new Transaction(storeValues, writable)); + } + + readTxn(storeNames) { + // TODO: avoid concurrency + return this._createTxn(storeNames, false); + } + + readWriteTxn(storeNames) { + // TODO: avoid concurrency + return this._createTxn(storeNames, true); + } +} diff --git a/src/matrix/storage/memory/Transaction.js b/src/matrix/storage/memory/Transaction.js new file mode 100644 index 00000000..437962da --- /dev/null +++ b/src/matrix/storage/memory/Transaction.js @@ -0,0 +1,57 @@ +import RoomTimelineStore from "./stores/RoomTimelineStore.js"; + +export default class Transaction { + constructor(storeValues, writable) { + this._storeValues = storeValues; + this._txnStoreValues = {}; + this._writable = writable; + } + + _store(name, mapper) { + if (!this._txnStoreValues.hasOwnProperty(name)) { + if (!this._storeValues.hasOwnProperty(name)) { + throw new Error(`Transaction wasn't opened for store ${name}`); + } + const store = mapper(this._storeValues[name]); + const clone = store.cloneStoreValue(); + // extra prevention for writing + if (!this._writable) { + Object.freeze(clone); + } + this._txnStoreValues[name] = clone; + } + return mapper(this._txnStoreValues[name]); + } + + get session() { + throw new Error("not yet implemented"); + // return this._store("session", storeValue => new SessionStore(storeValue)); + } + + get roomSummary() { + throw new Error("not yet implemented"); + // return this._store("roomSummary", storeValue => new RoomSummaryStore(storeValue)); + } + + get roomTimeline() { + return this._store("roomTimeline", storeValue => new RoomTimelineStore(storeValue)); + } + + get roomState() { + throw new Error("not yet implemented"); + // return this._store("roomState", storeValue => new RoomStateStore(storeValue)); + } + + complete() { + for(let name of Object.keys(this._txnStoreValues)) { + this._storeValues[name] = this._txnStoreValues[name]; + } + this._txnStoreValues = null; + return Promise.resolve(); + } + + abort() { + this._txnStoreValues = null; + return Promise.resolve(); + } +} diff --git a/src/matrix/storage/memory/stores/RoomTimelineStore.js b/src/matrix/storage/memory/stores/RoomTimelineStore.js new file mode 100644 index 00000000..3c17c045 --- /dev/null +++ b/src/matrix/storage/memory/stores/RoomTimelineStore.js @@ -0,0 +1,228 @@ +import SortKey from "../../room/timeline/SortKey.js"; +import sortedIndex from "../../../utils/sortedIndex.js"; +import Store from "./Store.js"; + +function compareKeys(key, entry) { + if (key.roomId === entry.roomId) { + return key.sortKey.compare(entry.sortKey); + } else { + return key.roomId < entry.roomId ? -1 : 1; + } +} + +class Range { + constructor(timeline, lower, upper, lowerOpen, upperOpen) { + this._timeline = timeline; + this._lower = lower; + this._upper = upper; + this._lowerOpen = lowerOpen; + this._upperOpen = upperOpen; + } + + /** projects the range onto the timeline array */ + project(roomId, maxCount = Number.MAX_SAFE_INTEGER) { + // determine lowest and highest allowed index. + // Important not to bleed into other roomIds here. + const lowerKey = {roomId, sortKey: this._lower || SortKey.minKey }; + // apply lower key being open (excludes given key) + let minIndex = sortedIndex(this._timeline, lowerKey, compareKeys); + if (this._lowerOpen && minIndex < this._timeline.length && compareKeys(lowerKey, this._timeline[minIndex]) === 0) { + minIndex += 1; + } + const upperKey = {roomId, sortKey: this._upper || SortKey.maxKey }; + // apply upper key being open (excludes given key) + let maxIndex = sortedIndex(this._timeline, upperKey, compareKeys); + if (this._upperOpen && maxIndex < this._timeline.length && compareKeys(upperKey, this._timeline[maxIndex]) === 0) { + maxIndex -= 1; + } + // find out from which edge we should grow + // if upper or lower bound + // again, important not to go below minIndex or above maxIndex + // to avoid bleeding into other rooms + let startIndex, endIndex; + if (!this._lower && this._upper) { + startIndex = Math.max(minIndex, maxIndex - maxCount); + endIndex = maxIndex; + } else if (this._lower && !this._upper) { + startIndex = minIndex; + endIndex = Math.min(maxIndex, minIndex + maxCount); + } else { + startIndex = minIndex; + endIndex = maxIndex; + } + + // if startIndex is out of range, make range empty + if (startIndex === this._timeline.length) { + startIndex = endIndex = 0; + } + const count = endIndex - startIndex; + return {startIndex, count}; + } + + select(roomId, maxCount) { + const {startIndex, count} = this.project(roomId, this._timeline, maxCount); + return this._timeline.slice(startIndex, startIndex + count); + } +} + +export default class RoomTimelineStore extends Store { + constructor(timeline, writable) { + super(timeline || [], writable); + } + + get _timeline() { + return this._storeValue; + } + + /** Creates a range that only includes the given key + * @param {SortKey} sortKey the key + * @return {Range} the created range + */ + onlyRange(sortKey) { + return new Range(this._timeline, sortKey, sortKey); + } + + /** Creates a range that includes all keys before sortKey, and optionally also the key itself. + * @param {SortKey} sortKey the key + * @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the upper end. + * @return {Range} the created range + */ + upperBoundRange(sortKey, open=false) { + return new Range(this._timeline, undefined, sortKey, undefined, open); + } + + /** Creates a range that includes all keys after sortKey, and optionally also the key itself. + * @param {SortKey} sortKey the key + * @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the lower end. + * @return {Range} the created range + */ + lowerBoundRange(sortKey, open=false) { + return new Range(this._timeline, sortKey, undefined, open); + } + + /** Creates a range that includes all keys between `lower` and `upper`, and optionally the given keys as well. + * @param {SortKey} lower the lower key + * @param {SortKey} upper the upper key + * @param {boolean} [lowerOpen=false] whether the lower key is included (false) or excluded (true) from the range. + * @param {boolean} [upperOpen=false] whether the upper key is included (false) or excluded (true) from the range. + * @return {Range} the created range + */ + boundRange(lower, upper, lowerOpen=false, upperOpen=false) { + return new Range(this._timeline, lower, upper, lowerOpen, upperOpen); + } + + /** Looks up the last `amount` entries in the timeline for `roomId`. + * @param {string} roomId + * @param {number} amount + * @return {Promise} a promise resolving to an array with 0 or more entries, in ascending order. + */ + lastEvents(roomId, amount) { + return this.eventsBefore(roomId, SortKey.maxKey, amount); + } + + /** Looks up the first `amount` entries in the timeline for `roomId`. + * @param {string} roomId + * @param {number} amount + * @return {Promise} a promise resolving to an array with 0 or more entries, in ascending order. + */ + firstEvents(roomId, amount) { + return this.eventsAfter(roomId, SortKey.minKey, amount); + } + + /** Looks up `amount` entries after `sortKey` in the timeline for `roomId`. + * The entry for `sortKey` is not included. + * @param {string} roomId + * @param {SortKey} sortKey + * @param {number} amount + * @return {Promise} a promise resolving to an array with 0 or more entries, in ascending order. + */ + eventsAfter(roomId, sortKey, amount) { + const events = this.lowerBoundRange(sortKey, true).select(roomId, amount); + return Promise.resolve(events); + } + + /** Looks up `amount` entries before `sortKey` in the timeline for `roomId`. + * The entry for `sortKey` is not included. + * @param {string} roomId + * @param {SortKey} sortKey + * @param {number} amount + * @return {Promise} a promise resolving to an array with 0 or more entries, in ascending order. + */ + eventsBefore(roomId, sortKey, amount) { + const events = this.upperBoundRange(sortKey, true).select(roomId, amount); + return Promise.resolve(events); + } + + /** Looks up the first, if any, event entry (so excluding gap entries) after `sortKey`. + * @param {string} roomId + * @param {SortKey} sortKey + * @return {Promise<(?Entry)>} a promise resolving to entry, if any. + */ + nextEvent(roomId, sortKey) { + const searchSpace = this.lowerBoundRange(sortKey, true).select(roomId); + const event = searchSpace.find(entry => !!entry.event); + return Promise.resolve(event); + } + + /** Looks up the first, if any, event entry (so excluding gap entries) before `sortKey`. + * @param {string} roomId + * @param {SortKey} sortKey + * @return {Promise<(?Entry)>} a promise resolving to entry, if any. + */ + previousEvent(roomId, sortKey) { + const searchSpace = this.upperBoundRange(sortKey, true).select(roomId); + const event = searchSpace.reverse().find(entry => !!entry.event); + return Promise.resolve(event); + } + + /** Inserts a new entry into the store. The combination of roomId and sortKey should not exist yet, or an error is thrown. + * @param {Entry} entry the entry to insert + * @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not. + * @throws {StorageError} ... + */ + insert(entry) { + this.assertWritable(); + const insertIndex = sortedIndex(this._timeline, entry, compareKeys); + if (insertIndex < this._timeline.length) { + const existingEntry = this._timeline[insertIndex]; + if (compareKeys(entry, existingEntry) === 0) { + return Promise.reject(new Error("entry already exists")); + } + } + this._timeline.splice(insertIndex, 0, entry); + return Promise.resolve(); + } + + /** Updates the entry into the store with the given [roomId, sortKey] combination. + * If not yet present, will insert. Might be slower than add. + * @param {Entry} entry the entry to update. + * @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not. + */ + update(entry) { + this.assertWritable(); + let update = false; + const updateIndex = sortedIndex(this._timeline, entry, compareKeys); + if (updateIndex < this._timeline.length) { + const existingEntry = this._timeline[updateIndex]; + if (compareKeys(entry, existingEntry) === 0) { + update = true; + } + } + this._timeline.splice(updateIndex, update ? 1 : 0, entry); + return Promise.resolve(); + } + + get(roomId, sortKey) { + const range = this.onlyRange(sortKey); + const {startIndex, count} = range.project(roomId); + const event = count ? this._timeline[startIndex] : undefined; + return Promise.resolve(event); + } + + removeRange(roomId, range) { + this.assertWritable(); + const {startIndex, count} = range.project(roomId); + const removedEntries = this._timeline.splice(startIndex, count); + return Promise.resolve(removedEntries); + } +} diff --git a/src/matrix/storage/memory/stores/Store.js b/src/matrix/storage/memory/stores/Store.js new file mode 100644 index 00000000..8028a783 --- /dev/null +++ b/src/matrix/storage/memory/stores/Store.js @@ -0,0 +1,27 @@ +export default class Store { + constructor(storeValue, writable) { + this._storeValue = storeValue; + this._writable = writable; + } + + // makes a copy deep enough that any modifications in the store + // won't affect the original + // used for transactions + cloneStoreValue() { + // assumes 1 level deep is enough, and that values will be replaced + // rather than updated. + if (Array.isArray(this._storeValue)) { + return this._storeValue.slice(); + } else if (typeof this._storeValue === "object") { + return Object.assign({}, this._storeValue); + } else { + return this._storeValue; + } + } + + assertWritable() { + if (!this._writable) { + throw new Error("Tried to write in read-only transaction"); + } + } +} diff --git a/src/matrix/storage/sortkey.js b/src/matrix/storage/sortkey.js deleted file mode 100644 index bfc15a5f..00000000 --- a/src/matrix/storage/sortkey.js +++ /dev/null @@ -1,181 +0,0 @@ -const MIN_INT32 = -2147483648; -const MID_INT32 = 0; -const MAX_INT32 = 2147483647; - -const MIN_UINT32 = 0; -const MID_UINT32 = 2147483647; -const MAX_UINT32 = 4294967295; - -const MIN = MIN_UINT32; -const MID = MID_UINT32; -const MAX = MAX_UINT32; - -export default class SortKey { - constructor(buffer) { - if (buffer) { - this._keys = new DataView(buffer); - } else { - this._keys = new DataView(new ArrayBuffer(8)); - // start default key right at the middle gap key, min event key - // so we have the same amount of key address space either way - this.gapKey = MID; - this.eventKey = MIN; - } - } - - get gapKey() { - return this._keys.getUint32(0, false); - } - - set gapKey(value) { - return this._keys.setUint32(0, value, false); - } - - get eventKey() { - return this._keys.getUint32(4, false); - } - - set eventKey(value) { - return this._keys.setUint32(4, value, false); - } - - get buffer() { - return this._keys.buffer; - } - - nextKeyWithGap() { - const k = new SortKey(); - k.gapKey = this.gapKey + 1; - k.eventKey = MIN; - return k; - } - - nextKey() { - const k = new SortKey(); - k.gapKey = this.gapKey; - k.eventKey = this.eventKey + 1; - return k; - } - - previousKey() { - const k = new SortKey(); - k.gapKey = this.gapKey; - k.eventKey = this.eventKey - 1; - return k; - } - - clone() { - const k = new SortKey(); - k.gapKey = this.gapKey; - k.eventKey = this.eventKey; - return k; - } - - static get maxKey() { - const maxKey = new SortKey(); - maxKey.gapKey = MAX; - maxKey.eventKey = MAX; - return maxKey; - } - - static get minKey() { - const minKey = new SortKey(); - minKey.gapKey = MIN; - minKey.eventKey = MIN; - return minKey; - } - - compare(otherKey) { - const gapDiff = this.gapKey - otherKey.gapKey; - if (gapDiff === 0) { - return this.eventKey - otherKey.eventKey; - } else { - return gapDiff; - } - } - - toString() { - return `[${this.gapKey}/${this.eventKey}]`; - } -} - -//#ifdef TESTS -export function tests() { - return { - test_default_key(assert) { - const k = new SortKey(); - assert.equal(k.gapKey, MID); - assert.equal(k.eventKey, MIN); - }, - - test_inc(assert) { - const a = new SortKey(); - const b = a.nextKey(); - assert.equal(a.gapKey, b.gapKey); - assert.equal(a.eventKey + 1, b.eventKey); - const c = b.previousKey(); - assert.equal(b.gapKey, c.gapKey); - assert.equal(c.eventKey + 1, b.eventKey); - assert.equal(a.eventKey, c.eventKey); - }, - - test_min_key(assert) { - const minKey = SortKey.minKey; - const k = new SortKey(); - assert(minKey.gapKey <= k.gapKey); - assert(minKey.eventKey <= k.eventKey); - }, - - test_max_key(assert) { - const maxKey = SortKey.maxKey; - const k = new SortKey(); - assert(maxKey.gapKey >= k.gapKey); - assert(maxKey.eventKey >= k.eventKey); - }, - - test_immutable(assert) { - const a = new SortKey(); - const gapKey = a.gapKey; - const eventKey = a.eventKey; - a.nextKeyWithGap(); - assert.equal(a.gapKey, gapKey); - assert.equal(a.eventKey, eventKey); - }, - - test_cmp_gapkey_first(assert) { - const a = new SortKey(); - const b = new SortKey(); - a.gapKey = 2; - a.eventKey = 1; - b.gapKey = 1; - b.eventKey = 100000; - assert(a.compare(b) > 0); - }, - - test_cmp_eventkey_second(assert) { - const a = new SortKey(); - const b = new SortKey(); - a.gapKey = 1; - a.eventKey = 100000; - b.gapKey = 1; - b.eventKey = 2; - assert(a.compare(b) > 0); - }, - - test_cmp_max_larger_than_min(assert) { - assert(SortKey.minKey.compare(SortKey.maxKey) < 0); - }, - - test_cmp_gapkey_first_large(assert) { - const a = new SortKey(); - const b = new SortKey(); - a.gapKey = MAX; - a.eventKey = MIN; - b.gapKey = MIN; - b.eventKey = MAX; - assert(b < a); - assert(a > b); - } - }; -} -//#endif diff --git a/src/matrix/sync.js b/src/matrix/sync.js index aae1fa0a..37425090 100644 --- a/src/matrix/sync.js +++ b/src/matrix/sync.js @@ -1,7 +1,7 @@ import { - RequestAbortError, - HomeServerError, - StorageError + RequestAbortError, + HomeServerError, + StorageError } from "./error.js"; import EventEmitter from "../EventEmitter.js"; @@ -9,119 +9,120 @@ const INCREMENTAL_TIMEOUT = 30000; const SYNC_EVENT_LIMIT = 10; function parseRooms(roomsSection, roomCallback) { - if (!roomsSection) { - return; - } - const allMemberships = ["join", "invite", "leave"]; - for(const membership of allMemberships) { - const membershipSection = roomsSection[membership]; - if (membershipSection) { - const rooms = Object.entries(membershipSection) - for (const [roomId, roomResponse] of rooms) { - roomCallback(roomId, roomResponse, membership); - } - } - } + if (roomsSection) { + const allMemberships = ["join", "invite", "leave"]; + for(const membership of allMemberships) { + const membershipSection = roomsSection[membership]; + if (membershipSection) { + return Object.entries(membershipSection).map(([roomId, roomResponse]) => { + return roomCallback(roomId, roomResponse, membership); + }); + } + } + } + return []; } export default class Sync extends EventEmitter { - constructor(hsApi, session, storage) { - super(); - this._hsApi = hsApi; - this._session = session; - this._storage = storage; - this._isSyncing = false; - this._currentRequest = null; - } - // returns when initial sync is done - async start() { - if (this._isSyncing) { - return; - } - this._isSyncing = true; - let syncToken = this._session.syncToken; - // do initial sync if needed - if (!syncToken) { - // need to create limit filter here - syncToken = await this._syncRequest(); - } - this._syncLoop(syncToken); - } + constructor(hsApi, session, storage) { + super(); + this._hsApi = hsApi; + this._session = session; + this._storage = storage; + this._isSyncing = false; + this._currentRequest = null; + } + // returns when initial sync is done + async start() { + if (this._isSyncing) { + return; + } + this._isSyncing = true; + let syncToken = this._session.syncToken; + // do initial sync if needed + if (!syncToken) { + // need to create limit filter here + syncToken = await this._syncRequest(); + } + this._syncLoop(syncToken); + } - async _syncLoop(syncToken) { - // if syncToken is falsy, it will first do an initial sync ... - while(this._isSyncing) { - try { - console.log(`starting sync request with since ${syncToken} ...`); - syncToken = await this._syncRequest(syncToken, INCREMENTAL_TIMEOUT); - } catch (err) { - this._isSyncing = false; - if (!(err instanceof RequestAbortError)) { - console.warn("stopping sync because of error"); - this.emit("error", err); - } - } - } - this.emit("stopped"); - } + async _syncLoop(syncToken) { + // if syncToken is falsy, it will first do an initial sync ... + while(this._isSyncing) { + try { + console.log(`starting sync request with since ${syncToken} ...`); + syncToken = await this._syncRequest(syncToken, INCREMENTAL_TIMEOUT); + } catch (err) { + this._isSyncing = false; + if (!(err instanceof RequestAbortError)) { + console.warn("stopping sync because of error"); + this.emit("error", err); + } + } + } + this.emit("stopped"); + } - async _syncRequest(syncToken, timeout) { - this._currentRequest = this._hsApi.sync(syncToken, undefined, timeout); - const response = await this._currentRequest.response(); - syncToken = response.next_batch; - const storeNames = this._storage.storeNames; - const syncTxn = await this._storage.readWriteTxn([ - storeNames.session, - storeNames.roomSummary, - storeNames.roomTimeline, - storeNames.roomState, - ]); + async _syncRequest(syncToken, timeout) { + this._currentRequest = this._hsApi.sync(syncToken, undefined, timeout); + const response = await this._currentRequest.response(); + syncToken = response.next_batch; + const storeNames = this._storage.storeNames; + const syncTxn = await this._storage.readWriteTxn([ + storeNames.session, + storeNames.roomSummary, + storeNames.roomState, + storeNames.timelineEvents, + storeNames.timelineFragments, + ]); const roomChanges = []; try { - this._session.applySync(syncToken, response.account_data, syncTxn); + this._session.persistSync(syncToken, response.account_data, syncTxn); // to_device // presence - if (response.rooms) { - parseRooms(response.rooms, (roomId, roomResponse, membership) => { - let room = this._session.rooms.get(roomId); - if (!room) { - room = this._session.createRoom(roomId); - } - console.log(` * applying sync response to room ${roomId} ...`); - const changes = room.persistSync(roomResponse, membership, syncTxn); + if (response.rooms) { + const promises = parseRooms(response.rooms, async (roomId, roomResponse, membership) => { + let room = this._session.rooms.get(roomId); + if (!room) { + room = this._session.createRoom(roomId); + } + console.log(` * applying sync response to room ${roomId} ...`); + const changes = await room.persistSync(roomResponse, membership, syncTxn); roomChanges.push({room, changes}); - }); - } - } catch(err) { - console.warn("aborting syncTxn because of error"); - // avoid corrupting state by only - // storing the sync up till the point - // the exception occurred - syncTxn.abort(); - throw err; - } - try { - await syncTxn.complete(); - console.info("syncTxn committed!!"); - } catch (err) { - throw new StorageError("unable to commit sync tranaction", err); - } + }); + await Promise.all(promises); + } + } catch(err) { + console.warn("aborting syncTxn because of error"); + // avoid corrupting state by only + // storing the sync up till the point + // the exception occurred + syncTxn.abort(); + throw err; + } + try { + await syncTxn.complete(); + console.info("syncTxn committed!!"); + } catch (err) { + throw new StorageError("unable to commit sync tranaction", err); + } // emit room related events after txn has been closed for(let {room, changes} of roomChanges) { room.emitSync(changes); } - return syncToken; - } + return syncToken; + } - stop() { - if (!this._isSyncing) { - return; - } - this._isSyncing = false; - if (this._currentRequest) { - this._currentRequest.abort(); - this._currentRequest = null; - } - } + stop() { + if (!this._isSyncing) { + return; + } + this._isSyncing = false; + if (this._currentRequest) { + this._currentRequest.abort(); + this._currentRequest = null; + } + } } diff --git a/src/observable/index.js b/src/observable/index.js index 36e037cd..828a6aea 100644 --- a/src/observable/index.js +++ b/src/observable/index.js @@ -3,7 +3,8 @@ import FilteredMap from "./map/FilteredMap.js"; import MappedMap from "./map/MappedMap.js"; import BaseObservableMap from "./map/BaseObservableMap.js"; // re-export "root" (of chain) collections -export { default as ObservableArray} from "./list/ObservableArray.js"; +export { default as ObservableArray } from "./list/ObservableArray.js"; +export { default as SortedArray } from "./list/SortedArray.js"; export { default as ObservableMap } from "./map/ObservableMap.js"; // avoid circular dependency between these classes diff --git a/src/observable/list/BaseObservableList.js b/src/observable/list/BaseObservableList.js index a1c38e91..97e50b20 100644 --- a/src/observable/list/BaseObservableList.js +++ b/src/observable/list/BaseObservableList.js @@ -34,4 +34,11 @@ export default class BaseObservableList extends BaseObservableCollection { } } + [Symbol.iterator]() { + throw new Error("unimplemented"); + } + + get length() { + throw new Error("unimplemented"); + } } diff --git a/src/observable/list/ObservableArray.js b/src/observable/list/ObservableArray.js index e33dfe1a..bb5c7758 100644 --- a/src/observable/list/ObservableArray.js +++ b/src/observable/list/ObservableArray.js @@ -11,6 +11,28 @@ export default class ObservableArray extends BaseObservableList { this.emitAdd(this._items.length - 1, item); } + insertMany(idx, items) { + for(let item of items) { + this.insert(idx, item); + idx += 1; + } + } + + insert(idx, item) { + this._items.splice(idx, 0, item); + this.emitAdd(idx, item); + } + + get array() { + return this._items; + } + + at(idx) { + if (this._items && idx >= 0 && idx < this._items.length) { + return this._items[idx]; + } + } + get length() { return this._items.length; } diff --git a/src/observable/list/SortedArray.js b/src/observable/list/SortedArray.js new file mode 100644 index 00000000..8c90ce15 --- /dev/null +++ b/src/observable/list/SortedArray.js @@ -0,0 +1,50 @@ +import BaseObservableList from "./BaseObservableList.js"; +import sortedIndex from "../../utils/sortedIndex.js"; + +export default class SortedArray extends BaseObservableList { + constructor(comparator) { + super(); + this._comparator = comparator; + this._items = []; + } + + setManySorted(items) { + // TODO: we can make this way faster by only looking up the first and last key, + // and merging whatever is inbetween with items + // if items is not sorted, 💩🌀 will follow! + // should we check? + // Also, once bulk events are supported in collections, + // we can do a bulk add event here probably if there are no updates + // BAD CODE! + for(let item of items) { + this.set(item); + } + } + + set(item, updateParams = null) { + const idx = sortedIndex(this._items, item, this._comparator); + if (idx >= this._items.length || this._comparator(this._items[idx], item) !== 0) { + this._items.splice(idx, 0, item); + this.emitAdd(idx, item) + } else { + this._items[idx] = item; + this.emitUpdate(idx, item, updateParams); + } + } + + remove(item) { + throw new Error("unimplemented"); + } + + get array() { + return this._items; + } + + get length() { + return this._items.length; + } + + [Symbol.iterator]() { + return this._items.values(); + } +} diff --git a/src/observable/list/SortedMapList.js b/src/observable/list/SortedMapList.js index 7f525fff..ebd7e86a 100644 --- a/src/observable/list/SortedMapList.js +++ b/src/observable/list/SortedMapList.js @@ -1,5 +1,5 @@ import BaseObservableList from "./BaseObservableList.js"; - +import sortedIndex from "../../utils/sortedIndex.js"; /* @@ -25,35 +25,6 @@ so key -> Map -> value -> node -> *parentNode -> rootNode with a node containing {value, leftCount, rightCount, leftNode, rightNode, parentNode} */ - -/** - * @license - * Based off baseSortedIndex function in Lodash - * Copyright JS Foundation and other contributors - * Released under MIT license - * Based on Underscore.js 1.8.3 - * Copyright Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors - */ -function sortedIndex(array, value, comparator) { - let low = 0; - let high = array.length; - - while (low < high) { - let mid = (low + high) >>> 1; - let cmpResult = comparator(value, array[mid]); - - if (cmpResult > 0) { - low = mid + 1; - } else if (cmpResult < 0) { - high = mid; - } else { - low = high = mid; - } - } - return high; -} - - // does not assume whether or not the values are reference // types modified outside of the collection (and affecting sort order) or not diff --git a/src/observable/map/MappedMap.js b/src/observable/map/MappedMap.js index 6d10325f..4b16373b 100644 --- a/src/observable/map/MappedMap.js +++ b/src/observable/map/MappedMap.js @@ -1,5 +1,8 @@ import BaseObservableMap from "./BaseObservableMap.js"; - +/* +so a mapped value can emit updates on it's own with this._updater that is passed in the mapping function +how should the mapped value be notified of an update though? and can it then decide to not propagate the update? +*/ export default class MappedMap extends BaseObservableMap { constructor(source, mapper) { super(); diff --git a/src/ui/viewmodels/RoomViewModel.js b/src/ui/viewmodels/RoomViewModel.js deleted file mode 100644 index bc75dea6..00000000 --- a/src/ui/viewmodels/RoomViewModel.js +++ /dev/null @@ -1,36 +0,0 @@ -import EventEmitter from "../../EventEmitter.js"; - -export default class RoomViewModel extends EventEmitter { - constructor(room) { - super(); - this._room = room; - this._timeline = null; - this._onRoomChange = this._onRoomChange.bind(this); - } - - async enable() { - this._room.on("change", this._onRoomChange); - this._timeline = await this._room.openTimeline(); - this.emit("change", "timelineEntries"); - } - - disable() { - if (this._timeline) { - this._timeline.close(); - } - } - - // room doesn't tell us yet which fields changed, - // so emit all fields originating from summary - _onRoomChange() { - this.emit("change", "name"); - } - - get name() { - return this._room.name; - } - - get timelineEntries() { - return this._timeline && this._timeline.entries; - } -} diff --git a/src/ui/web/RoomView.js b/src/ui/web/RoomView.js index 987bdeeb..d26cf5df 100644 --- a/src/ui/web/RoomView.js +++ b/src/ui/web/RoomView.js @@ -14,13 +14,14 @@ export default class RoomView { mount() { this._viewModel.on("change", this._onViewModelUpdate); this._nameLabel = html.h2(null, this._viewModel.name); - this._timelineList = new ListView({ - list: this._viewModel.timelineEntries - }, entry => new TimelineTile(entry)); + this._errorLabel = html.div({className: "RoomView_error"}); + + this._timelineList = new ListView({}, entry => new TimelineTile(entry)); this._timelineList.mount(); this._root = html.div({className: "RoomView"}, [ this._nameLabel, + this._errorLabel, this._timelineList.root() ]); @@ -40,8 +41,10 @@ export default class RoomView { if (prop === "name") { this._nameLabel.innerText = this._viewModel.name; } - else if (prop === "timelineEntries") { - this._timelineList.update({list: this._viewModel.timelineEntries}); + else if (prop === "timelineViewModel") { + this._timelineList.update({list: this._viewModel.timelineViewModel.tiles}); + } else if (prop === "error") { + this._errorLabel.innerText = this._viewModel.error; } } diff --git a/src/ui/web/TimelineTile.js b/src/ui/web/TimelineTile.js index 7c362d78..4c0ff45d 100644 --- a/src/ui/web/TimelineTile.js +++ b/src/ui/web/TimelineTile.js @@ -1,29 +1,8 @@ import * as html from "./html.js"; -function tileText(event) { - const content = event.content; - switch (event.type) { - case "m.room.message": { - const msgtype = content.msgtype; - switch (msgtype) { - case "m.text": - return content.body; - default: - return `unsupported msgtype: ${msgtype}`; - } - } - case "m.room.name": - return `changed the room name to "${content.name}"`; - case "m.room.member": - return `changed membership to ${content.membership}`; - default: - return `unsupported event type: ${event.type}`; - } -} - export default class TimelineTile { - constructor(entry) { - this._entry = entry; + constructor(tileVM) { + this._tileVM = tileVM; this._root = null; } @@ -32,21 +11,7 @@ export default class TimelineTile { } mount() { - let children; - if (this._entry.gap) { - children = [ - html.strong(null, "Gap"), - " with prev_batch ", - html.strong(null, this._entry.gap.prev_batch) - ]; - } else if (this._entry.event) { - const event = this._entry.event; - children = [ - html.strong(null, event.sender), - `: ${tileText(event)}`, - ]; - } - this._root = html.li(null, children); + this._root = renderTile(this._tileVM); return this._root; } @@ -54,3 +19,23 @@ export default class TimelineTile { update() {} } + +function renderTile(tile) { + switch (tile.shape) { + case "message": + return html.li(null, [html.strong(null, tile.internalId+" "), tile.label]); + case "gap": { + const button = html.button(null, (tile.isUp ? "🠝" : "🠟") + " fill gap"); + const handler = () => { + tile.fill(); + button.removeEventListener("click", handler); + }; + button.addEventListener("click", handler); + return html.li(null, [html.strong(null, tile.internalId+" "), button]); + } + case "announcement": + return html.li(null, [html.strong(null, tile.internalId+" "), tile.label]); + default: + return html.li(null, [html.strong(null, tile.internalId+" "), "unknown tile shape: " + tile.shape]); + } +} diff --git a/src/ui/web/html.js b/src/ui/web/html.js index 754e3ebe..ebf57091 100644 --- a/src/ui/web/html.js +++ b/src/ui/web/html.js @@ -51,3 +51,4 @@ export function main(... params) { return el("main", ... params); } export function article(... params) { return el("article", ... params); } export function aside(... params) { return el("aside", ... params); } export function pre(... params) { return el("pre", ... params); } +export function button(... params) { return el("button", ... params); } diff --git a/src/utils/sortedIndex.js b/src/utils/sortedIndex.js new file mode 100644 index 00000000..70eaa9b8 --- /dev/null +++ b/src/utils/sortedIndex.js @@ -0,0 +1,26 @@ +/** + * @license + * Based off baseSortedIndex function in Lodash + * Copyright JS Foundation and other contributors + * Released under MIT license + * Based on Underscore.js 1.8.3 + * Copyright Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors + */ +export default function sortedIndex(array, value, comparator) { + let low = 0; + let high = array.length; + + while (low < high) { + let mid = (low + high) >>> 1; + let cmpResult = comparator(value, array[mid]); + + if (cmpResult > 0) { + low = mid + 1; + } else if (cmpResult < 0) { + high = mid; + } else { + low = high = mid; + } + } + return high; +}