From f3034800ae7d94028d7e032ff0972e5b9e729dd4 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 12:26:08 +0100
Subject: [PATCH 01/83] update docs, write down architecture

---
 doc/architecture.md | 29 +++++++++++++++++++++++++++++
 doc/sync-updates.md |  4 ++--
 2 files changed, 31 insertions(+), 2 deletions(-)
 create mode 100644 doc/architecture.md

diff --git a/doc/architecture.md b/doc/architecture.md
new file mode 100644
index 00000000..1a2f4c3a
--- /dev/null
+++ b/doc/architecture.md
@@ -0,0 +1,29 @@
+The matrix layer consists of a `Session`, which represents a logged in user session. It's the root object you can get rooms off. It can persist and load itself from storage, at which point it's ready to be displayed. It doesn't sync it's own though, and you need to create and start a Sync object for updates to be pushed and persisted to the session. `Sync` is the thing (although not the only thing) that mutates the `Session`, with `Session` being unaware of `Sync`.
+
+The matrix layer assumes a transaction-based storage layer, modelled much to how IndexedDB works. The idea is that any logical operation like process sync response, send a message, ... runs completely in a transaction that gets aborted if anything goes wrong. This helps the storage to always be in a consistent state. For this reason you'll often see transactions (txn) being passed in the code. Also, the idea is to not emit any events until readwrite transactions have been committed. 
+
+ - Reduce the chance that errors (in the event handlers) abort the transaction. You *could* catch & rethrow but it can get messy.
+ - Try to keep transactions as short-lived as possible, to not block other transactions.
+
+For this reason a `Room` processes a sync response in two phases: `persistSync` & `emitSync`, with the return value of the former being passed into the latter to avoid double processing.
+
+`Room`s on the `Session` are exposed as an `ObservableMap` collection, which is like an ordinary `Map` but emits events when it is modified (here when a room is added, removed, or the properties of a room change). `ObservableMap` can have different operators applied to it like `mapValues()`, `filterValues()` each returning a new `ObservableMap`-like object, and also `sortValues()` returning an `ObservableList` (emitting events when a room at an index is added, removed, moved or changes properties).
+
+So example, for the room list, `Room` objects from `Session.rooms` are mapped to a `RoomTileViewModel` and then sorted. This gives us fine-grained events at the end of the collection chain that can be easily and efficiently rendered by the `ListView` component.
+
+On that note, view components are just a simple convention, having these methods:
+
+    - `mount()` - prepare to become part of the document and interactive, ensure `root()` returns a valid DOM node.
+    - `root()` - the room DOM node for the component. Only valid to be called between `mount()` and `unmount()`.
+    - `update(attributes)` (to be renamed to `setAttributes(attributes)`) - update the attributes for this component. Not all components support all attributes to be updated. For example most components expect a viewModel, but if you want a component with a different view model, you'd just create a new one.
+    - `unmount()` - tear down after having been removed from the document.
+
+The initial attributes are usually received by the constructor in the first argument. Other arguments are usually freeform, `ListView` accepting a closure to create a child component from a collection value.
+
+Templating and one-way databinding are neccesary improvements, but not assumed by the component contract.
+
+Updates from view models can come in two ways. View models emit a change event, that can be listened to from a view. This usually includes the name of the property that changed. This is the mechanism used to update the room name in the room header of the currently active room for example.
+
+For view models part of an observable collection (and to be rendered by a ListView), updates can also propagate through the collection and delivered by the ListView to the view in question. This avoids every child component in a ListView having to attach a listener to it's viewModel. This is the mechanism to update the room name in a RoomTile in the room list for example.
+
+TODO: specify how the collection based updates work.
diff --git a/doc/sync-updates.md b/doc/sync-updates.md
index 1d3f0410..45333c3e 100644
--- a/doc/sync-updates.md
+++ b/doc/sync-updates.md
@@ -1,7 +1,7 @@
 # persistance vs model update of a room
 
 ## persist first, return update object, update model with update object
- - 
+ - we went with this
 ## update model first, return update object, persist with update object
  - not all models exist at all times (timeline only when room is "open"),
  	so model to create timeline update object might not exist for persistence need
@@ -15,4 +15,4 @@
 
 ## persist first, read from storage to update model
  + guaranteed consistency between what is on screen and in storage
- - slower as we need to reread what was just synced every time (big accounts with frequent updates)
\ No newline at end of file
+ - slower as we need to reread what was just synced every time (big accounts with frequent updates)

From 049e70e312af7e7ac755b50eb3ad101c4899d78d Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 12:26:59 +0100
Subject: [PATCH 02/83] throw NetworkError from HomeServerApi

---
 src/matrix/error.js  |  4 +++-
 src/matrix/hs-api.js | 21 +++++++++++++++------
 2 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/src/matrix/error.js b/src/matrix/error.js
index 83cc8652..520698d9 100644
--- a/src/matrix/error.js
+++ b/src/matrix/error.js
@@ -9,5 +9,7 @@ export class StorageError extends Error {
 }
 
 export class RequestAbortError extends Error {
+}
 
-}
\ No newline at end of file
+export class NetworkError extends Error { 
+}
diff --git a/src/matrix/hs-api.js b/src/matrix/hs-api.js
index 2eb2a1f1..e54b66f6 100644
--- a/src/matrix/hs-api.js
+++ b/src/matrix/hs-api.js
@@ -1,6 +1,7 @@
 import {
 	HomeServerError,
-	RequestAbortError
+	RequestAbortError,
+    NetworkError
 } from "./error.js";
 
 class RequestWrapper {
@@ -62,10 +63,18 @@ export default class HomeServerApi {
 				}
 			}
 		}, err => {
-			switch (err.name) {
-				case "AbortError": throw new RequestAbortError();
-				default: throw err; //new Error(`Unrecognized DOMException: ${err.name}`);
-			}
+            if (err.name === "AbortError") {
+                throw new RequestAbortError();
+            } else if (err instanceof TypeError) {
+                // Network errors are reported as TypeErrors, see
+                // https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#Checking_that_the_fetch_was_successful
+                // this can either mean user is offline, server is offline, or a CORS error (server misconfiguration).
+                // 
+                // One could check navigator.onLine to rule out the first
+                // but the 2 later ones are indistinguishable from javascript.
+                throw new NetworkError(err.message);
+            }
+			throw err;
 		});
 		return new RequestWrapper(promise, controller);
 	}
@@ -92,4 +101,4 @@ export default class HomeServerApi {
           "password": password
         });
 	}
-}
\ No newline at end of file
+}

From e3d931b9660547f009c7a87714b15d66eb7a3457 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 12:28:05 +0100
Subject: [PATCH 03/83] responsive layout prototypes

---
 prototypes/manifest.appcache           |   4 +
 prototypes/me.jpg                      | Bin 0 -> 22631 bytes
 prototypes/responsive-layout-flex.html | 506 +++++++++++++++++++++++++
 prototypes/responsive-layout-grid.html | 133 +++++++
 4 files changed, 643 insertions(+)
 create mode 100644 prototypes/manifest.appcache
 create mode 100644 prototypes/me.jpg
 create mode 100644 prototypes/responsive-layout-flex.html
 create mode 100644 prototypes/responsive-layout-grid.html

diff --git a/prototypes/manifest.appcache b/prototypes/manifest.appcache
new file mode 100644
index 00000000..ff420219
--- /dev/null
+++ b/prototypes/manifest.appcache
@@ -0,0 +1,4 @@
+CACHE MANIFEST
+# v1
+/responsive-layout-flex.html
+/me.jpg
diff --git a/prototypes/me.jpg b/prototypes/me.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..615e10c8a0fb67e92319eee369b14fa9e73c59d2
GIT binary patch
literal 22631
zcmbSyWl&sQ*JY!PTYvy*+#3n*PH=BrgKKbi3j`Y1AdS<wCO9D>xVu9L?hrHt0%SVR
z^L}5=OwEs(xmAaI>(WrQ`>bti?ek~p&nn<800SKzh>nH<1OhQJF|crmadEJ*amb0D
z;S*C*P*YJ-fIu`1oXj+I?DQZIiy$jI7dIatA2qYExDbyRCodll5+)`l4mJ)ME-o1l
zEr^!q|M~jU4Isoo8bcaGL81d76C$AyBK_$@Dh2?M(2)MQ0ROp=kWo<4(1943Sl9#r
zWF!<6WK<M1G*nc?Cxa2c1E3P35z%r>qd(KK0MfY=^Mt1qVbIGoekFlSpEK}UdPHDi
zkv=CQr(k4aW?=>M@e2qF35&?Sl9N|ZR8rQ~(bdy8Ff_8Vwz0LdcX0Id^7ird^ACu8
z8x{R7CKj5SmY$LMJ}Wz?xTF+bR$ftA_2FYvb4zPmdq+=iU;n`1(D2C2?A-jq;?nZU
z=GOMk?%w{v;nBtA)%DHK+h2G0NC1?7%|ZP9*P#E&9zujY$f&3&sKCGWAR+rA778IM
z8Z9?Ek+c@j!u=T?PdEm#OiEGXS4?_d$T^9n$21lx1K%d&#b1;DvFQKKposr3i~if7
z|F-AP3IGQM39(@)gaApvjS)+6g|r;8CX$LAF~GMN$^!VWRnmocE5}G{E|LRK2rkKq
z@XYdw-?&iWJ?B1NFHBCQM+zr@O3#|gTuNBs?xvpFet2}?#56uO1dRM(z1LwAiKSHP
z#Z#1c_Xkj~!uMpYh1^MQ4caT-HJID3W3j{F(ICO`#hP|yKCi)bHC18c-schgrg7xB
z`k|fGG)E+{7K9nLmj6Y=aqmZviSz8J5($tJQ0J;OSL*j|lr+e^K=#z+65fB(&B`oh
z4YRwbF<Ir;R6JfCc%K%OdCH8s+eQu7OX`2lGi05L#EX)_yZeR4k2<X8*J<$w#ci@w
zbrm8Bv%mr)hqv75hB`D)pg(|uA#<H?@2$P2{T2?@87$)-r>YBja{mA@Z`My6WSGc~
zKQdoC?!DsLI8AcD6_?~y!EwSF{kStJj2D^#Ix(p@EV;<6cJs-Ipz{L^hTrGRw9W@9
z(M1yuGFa8kn}y;;I8pWX;W}K3*ZmswD14Uq$YzUAc<4!DJBI4*oJB3$4|(sSqIcj$
zVOzpDg4Pi?`kpcz@COj(fahNmCoVq{^d^TC2#a9T_DF$k9~zZe4`XgYV}=B9lCgC>
z&3ch^8JZGqwE_yRQdH$~z(U^xP$GDj=F+=ailKo2^GXdPO{@L8&;KrfeF<MW;E3f|
z4nVB+@8V!2)<m9Gk^^v{r$PZntSpm!fM-y(63~@G(5%Loza+~xRTGlgu|s%+$Me1J
zAJDgB6x7^qLPN}l`k=5+Ntqw?xMrKkR|)RYL;V7DS}ju)X;byGpI&bcXrMH1pYzk?
zNWYNZA;80!bomyNVYyRd=RdG!4)dwf$3d}kUq_b)R!nNl(w|q@m&yfK$6;7oX1jD>
z10Q%zVy`%#pZenTb&}F{JgOOfPF1==u91G5SI!*fnY63=WvJEO@}SoNun~Tw0GG9a
zTXE0VGZ7{u4=HIadF{-%SwxdX4#2bh*5onw>suRsmK!(8jXC!2m<4N)RdCjIGnNc=
zyP^9X0jZNNE4r>Z`{mH~CbrZA@b(`7Pde2+L_Pyp%;Yzc`}fn%SV>^<zK;<T>x1>%
zPkcFfV*x=(VOWE^uMS%ioBG{n*J#27rb>L7zd!8XlrfIUurs#%KHX;Ve!qCqjcL{A
zB{$qzcZU_sKK!1^E|XIJSTuwqXm?$1VX_+GkZ-QX1n^LyA`U2UY<q931fv~^|7gCk
zz5dZTWQtvxihSoQL-3SHL9NrWVRNq~4Iytfs30pTC?tcu_~RPihgi4smo!}4*M~{9
zrcLwBFXAA{W7Edf&O9!o)E2tUsEP7Arv1Q?s!wP40gsM-gC`dXIB!S@%sul13kL78
z^wHyv_eVB!o_w(9eL>WVOwA|!7_<e?!9fiI(IN7Zk9OB}5-Wx|m%I!m<{?sK+ig$Y
z{68y2Un?&B0eHDzo@;FD<i2Vv3xIdcf0~X&d8d8gsV6<@co`>d3~B94$&~*ww0}{{
z7VPtL|4H&k9bq5_hH9{TL@2jWRkTa4I58FPN@iQk*_Jt$LdDqVf<gB8RyWT_r}Ehw
zjJi$iz4|O6mJsbb-Aa@9FDQs1ka?p2!tOtJh?W$#9BnZI=zLmIXvF_64EyoFutvl@
z$h6W*7%5bl1<C%emy9zK1DX)fM-O8u4kK(QBhDJ&RxfMp9tX(LdyE8&=&4=D;*B-h
zG4BkD{l<H&8sF`a!-+Q#{7rMYb4@b2s=@JXtawEi<%M;p&{xTC(g)&>)(}pOU^bao
z38>ts6Nkn({hqVyo5&nH9!y+td4-u4#@WplTvl(PjZbZsp(5RL^%F(p20PLXHd4qR
zY-4?Ri$8t$2J=SJ{G{i*+YOGt1G;aty6k-<8{T6HuBN;Ch1Kj!E1;=HmvQIW9QZp_
zB-%da<eOGm76B&tcKcCihBf=Tm4{-26#ZJ{AHY$ySwLwV0W&>VJ|I={Q8(dP3V955
zmiztQ+puq$uW5C4vD#l*<?~=nNNO3Z)a-?RT4NEm7@j0Uxh(#z>0piOIb}761_%X5
z`HB(V!DU$1m-o~D02ccB*D?i`_YyI-5_u$J_CeGpx02h&6y9B3o!Mv4y}~xN{D*IA
zUblc#kJBb80S{FMXEussQK>v@=whMy*Gu%JfkW_g<`?wzq6)xz<tA_I#PLm4ngboY
z$SzL~KlbAFUa$I0-2nhqSyiv-r_fKk#*CeIWoUBH_UW%)Ce6K#`?TCcRORQOU^Q9(
z@&3MO8B!q)>`y=MQ5HVb^V4R;Z|d<Mh+rk~Q;VX?3$Pq17?r+V{?Mul;~vG69@+?1
zZpJ;T=%M9+AusRTFLG0kep}1LS(-qum^?Hh{ARQwvGASm+uV@MlTh(j&yw{kPbL|=
z!msSXLgj$FH7Q--*%vK?%0Px0Hsw7b9e0ZJwp&Uy)s8r!KCO5>;;0EpI2+DdYdh%-
zrj=D#pj+rbhl;7k%)!8W0+#n;Jlh4q-=xRgkoO+K8_>yr*G4{c=*Br!>ZyLRqeQ|H
zE)sY<V^_CAZi=+l8E927S$<h9wH=}Elzjg4b0X@q5tM6G*s~}8g_39~X*3OClIiDO
z7L2wka)QF}DhB*o4%$PhLKMEg0lg``36TDu$c}J@LS!iGKN<gD@c-{=0^uY7WCob7
zf2=K*(B$18fTR)ZQcm`(5A#)_EmbvnfzF2}a>5Fp7SW%70LLMGAC#T>=CS}$^r+j5
zl~3|zpS|KVRo>fRRq6X~n&^G0k5%qPp=J;srtY7-+<D4?<;6kUCJB#SIVP8#mol{T
z*2@cK8Ws+*T_6{u1$CkKbB|JkpN6U@rd#dz$^HO93okR<IJ#6?skwv)%QfIufjKJ!
zO{F{0QoWah^v%X!yizA5`e=kC<|ERuy&g}(cEVfJ28$lSzOZ3HxA}9nK~+03hz(uF
zD9Kwc$y)6oo=)>i;X&7{C;ocI%KS+`;XzF&0efp1MPOnH2x?EkMxqqQoz4nmrb${P
zA=zm6P{&q)Rm7k5KkJ^U&&nP;)<sGs!eB2K|A0}g7|qY%;~~(1Jz?Bz*n-P2O23;D
z;_}5mN&2Bu=dt*?dEtn{c-n2)Q?rAWK~ZZGnWZmv1pw=f5`l6*+@Dc+keRtw0Tmvb
z(Y^v7Isw9;W5*~gGC(k(2Es?Lu^+zNU#F5Q<tTxypZf^^9Dn`hD^?$#jl~t!h`2*$
zOqJmhL3JV!5}=bso9`bT3^<%E-&IviR^`|t?2)h(zPS%pGiq{2PTg>&7ecvP;Dab$
zZu*NWRZDiIK+DHFe}KXe-?o8ms^iP{=z|hSz&5*NPtBk=Hi56J_{7Zj!tO^ktJNj8
zkyv`ykk03lLwbd+t*R{TibF!HFRY&;lO{5fP4d;`w`f#J>l5?e3`wfb;{+sJNeEU`
zOD$I?J7`IcHH>=FQv0>AGBo7y=+w9)J+S{8+(6?11nCE4?Ff=j-uge_x;g}$RX#%#
zOdU*HkOEQnLAxV3N}m@18<p^X>A@^n6|eBK6oT-l=GJ`Lk3`gKT&((TNl!L#-U{(I
z)Yb5m^Y4kUw8no}+ns$f948YdJL%RpwfThnrhZ0x8MKpBfk3I=blw>J1#i({Er({L
z=2#qVpDinX=%%FXQ>e(7oTf8dMB3oOVkzMRXd&{0mK-DCpBiZ9zucxN{ojHJ@v(oZ
zBo5+qMEU%;LShb@mIII|?;VMX>gDLKRR8)}4dE$HRY!9-`dR3W-2z26A{&@dxVzUG
z(F21|1{b|s=1QIWu)QTa<;XbBG+yr-V4|3<S6Q(V0EM>s9WS{O$DW^Er9DbcIwq0}
zyyDn?oi|}jsW_;-=g~e@Vo5aW77t`3iyDN8yX!EuvgBJ@F((E>zKuB7T~YJYoB6%I
z1mBnHSMD3PER%VZhiP=Zz#GtQ$-mWSjn7{~ssA$5VaS(}I3-`0mdLyI2arE6k@1FO
z*iNc;$~-jXPOm|l?uLvG{?SiX77F>fbLWULIsU?%g!Q*rn1!Ot!60fdo5ybrC66U+
z@a;{_@SE&tJSkuar_?<}6~u0>QC_AZk1?sWG_TOCKqC?n_yfZGEB>juowBr{jdql|
zP>A2$xc4QaXZ}WbQms)+TI^(S1Wnbn9oxLYAAs5QsmvJDHI_%1$abCR9{_mmX=rWT
znxP(a{-usJ&4?Qtb_eC+;^8h(4(hx7nAQ>2Mz-TyuJHVxOHY`pB!9luvpg$|>NVNn
zxPElsN3eOSA9{6`I{SDcny8k}ws6(#z)T5v{|j{@TgiLXwpMKx)yM%2Gkks{He`6|
z(c2vW>zYZKvSDpImATL$IL|oc4oBzErIOHg)s=v(hbknSzns5?xZ~wmZ-sb*t*HV%
zvD_pGi<1qxgFQr<AxeXZulw86g;-7wd2v+tC5420rqJN7&U=-VrzahsZT{lqwVrTm
zb8Qg0@T16+SJpV0T}KMJmD+`SUVSTH5=b3c#d=6VhI#U7#4+8cA?O49*jtdZEX;1o
zoX}Wcra|+AYMGdI1YAjCU+jryrt6&*>nXd|D7Coa;XB;PS(KYf%Y0PnpFyXG9WGp<
zlg$n$3P`5M_Q23=*-$#k3@vX#9Kb7Es==hO84JssmU2E+GB?G+2_W+UzT;wCA3jBS
zUODDOh)GZP9{{HlI*ENLcSjY>Jj!Zu@z{7OYvudN6;WEEJjPNmUg)GFL5_$DWx%K=
zHg<_Tb@}?{7x3zA-8VY{&lVHh=`Xj74|1}_<2d(0p7P<oNbR05kO$aVPHd<+hm$c{
z>c}nRt=_Qsm}v_?(y5BpAdTOAhqYZ!u)S2%Ul+6ePg7cma4ur+VnoXVRW_MIhV1>@
zj})R6LuC;q9&z^HC)KHctGZ|5D!Q(&bWJfI3vz*uiRFI)wNhmH^z(8x=ro&{<hFy-
zCMd`1+Hb&a8vAF#N3H~>YF-j{qoN*22O^7-ox@Uv8?A>Xvl+|NWXlw{&-`N=qlani
zfJv{<i<wE!#UfuUnRkNB{{U{HEI*x$kr)Hg(vF|x)Gt3qLHzasIQ!>+0MyF46okXn
zuww$z))vE<<NSjC*_?LUcc-+AZzlj$>6x13pbzyUWBv@V`>VCzi2qd`f=#08jwsUL
z%$A4!AHJ2int<F77n<m4BBLZ@N}C%VXEuVJWJF22@8s;-=dC2{$0EteDWhI>s3x7T
z6+x%bdB7`DSu`x=7N(#Xd)oJVl;ZFBd4A~ug&EU-yPJah!^6<j8!u)_ln8b_WqWBj
zQHaD3HKLz`QW86*>Iyj(YK53Yph2acAhqvNm%L<Y?kC{n^K^LPb~GrmHQ8xe!1sq5
zt%fEKIF)W6?J<F;Y<V#tzZCyai#a-8ZI6ZHMbOgi57TSt+u&*4PhrAZNfuHkZj+mC
z701IeJnfXYTFyuur5s=vtq2Wx0C8io{0pj2y27V3CaIVbr88}FT0LtHG<}o+UHWhs
zp+V}Q3nLAy@NwiL+YCbs5(ZrZznY*&qt>L6{>$YucTXdAl8ChY{%A$Sm%aZ>kLz<s
zYXOxDK!{#!(d-A~$KCgSXXW)9PS5)&g4D>mngP}BbWy00df9I`Em~IcCdPYgy~V<G
z{T{*YzHSCTV5~qg(0(Et@_;{m&?Wu1$neP7l~&Rv#gXF&*$8~x!@E#r3R)h&X418i
zJ=$n_WAfTgOo}7TNUR+T>9b(&_4Z6zvZUfB7nz8F#NyT(rpsmP_>@77*N%EpKKJCQ
z<I_9>7oeDL9jB6J-!r_Txxx90E_r_ZqYk#%?Hpx#v~!yu`hFhXh4*{AC(9!p57llr
zzHOzX$eKzlY2DREN$KZ;B%&Fj9F!tbQ#-sA(bD|Ww)}la{g01~BXSY)w1T5DqElzJ
z{|}eLErqRw$YhJo)de%{FNw@3!+B67g(fYgLJI<pb~Hyq@{T7Tw77!hezsz9i8pF#
zTW+?GeQz)rP#hMs>#`)}RU7U;9E6+9Q7Cax>2HI;iSyaE1|{4>TMdFQOH~*uD!BV!
z{kY0d_uufTkeF)Oj2UoqAFlMuqPpD9_eagaGCJ_e;qN>lYF`81ZFP`Yj)jL+)Ns7j
zjIURoNtDde)6o7v_<^$<$pjCm+!EJK8D4e1fn!-yhd(B1MD&7LD?hRvZXicCE>(wA
znbL-GH(w&4UIm<+J1V@0)SZsm7p^88UQ29NoY1FimIlq79u7UQP<rt2F2tt>JEWqi
zy^<PNUIshT=d<Esbi=TsqIu)z;2%PjUC>2*w~|NBFpux&4T-}7bngJGv&Xt7?kf>n
zc0vP=QDfHezO};cSGwdK;l~h1kClwJH7U?Gbia%aw?~cxJE1Q7NIax@;rqjG<02?f
zUo;TlMNdpGTn(#TlUY%J5kQVY?=`Yr>vZ_6y2pYYQ|}-cSI*O6UwSt_Q@dS~2IG;9
zbH*-G`-)J=bosSben#0oy&m7qvk!`M4frXOmI~SrRj27uc*MyV6ZnpCdUDB3yWJDA
zp#?ofjTs-vzr3pifvoO5?;$9yGp3R3jSXxWR|MCS^;dO03%ltNBT~#j(8eNC@oS7a
zdVR^zA8x`_&Ff2wWNh-AE%6{<l2V*9N49s`t#GAyU%oypiGzMg{>J3u%-kH4DU^z`
zz4$4VkMa(@^7crcDml&^fWM{AKV%T)LFqWh|H={VAzBF}`_jT3fc0t1bBX!goJtN>
zhGeC|*BTe~nr(jokls7}G{=Ov%(#q|-B7HFvVrt5<2wW6QkKpJLH9dP!*W__b^uAh
z1rpgLN~*%$PyRSo3!=UfEYoz@&EXS&M#3wb!1joj!>ws&yh&F#b&I_%($C50#V!`w
z6GWEyMk7Kz3woOaNYmaBK0C~T*v@6P@@*wKhN0tknI?+iY!zu;;pj%~gc~+_(|H5o
ziulrphgy+$a}|Yziom)PXs$a-197+%4Q#H+a}qh{nZ=9D1HUKTmswxkjFSQ^%hrLf
zjeOQ9ijCynraObuen8l>#GcBk01C>QOGlpSJ3{Zp`h_d<(r+tT_lrt}2JG9?r5b>d
zIoLN98tX%4++aVxI^Ebf8%1V*!zGl+egP$n2{6-pW`--d5iKG!bva1x9JJxbWI!$J
z<1%x@INgcWHx^F(dZVFFZ;6>fSH?4iH2a{1;2o<Grcw>7_n5hnLp%Et1^XoGN7Hm2
zS1|z5s}m(e9$Dz)uFwDa4o&e9S36s`(7D%Fa;8aveHv}`u1eEGMF1}VRPYC&zX4`J
z(zNd0jF#GNtO%!_P~U{@gQ!FhJg9Q6-{qV+38mk@vv~Zd64T5W-kPBHYR;(`ch%l&
z)+N`;Zxe#<tKYla&Nk<$_J0J6MC!A{PH}iIVP~L>+hWd@*y@s{5^oz+?PJ)Kw^krP
z&+TJjFOz2{^?t2uNd!8c6I`?!c53jR&o(zD$7Gh^WM<Y=4j_`~z%mXH`8TY=5j`uS
z)<%+}fg>_IBH8o)Q)Clo!x3$(>dX|9JQAW|g}RBAqz34xI1ws2SFOnh3T(K4{{xse
zYnKr4;Nc$kw~K?=H{){`bN7Qc4Q{uJon#QfE!ha<gQY}OAovGR_mMvCR_z6Dzd=FY
zjb!<;F@bZMN9AtZONlOp>naK9_z`X9U6;=fdh%t_?=h})SeWmT5+x_$zL~MR1Hgct
z2T6(vD~qpyc&qhJjKR?#Gaobebs3ACrMFSeE4O~{%wesfJN7XE1bs`LJbIAg=VQKh
ziysvh?Dd3!d(=y?G#Ivt4fHu^gm-j~!nUE&!aA^L_qwID?<<H;WRn44wXo|_fYnN=
z37(oL=1_PVWo5475%_jQK}<*e3u|!?1pKof6$dMwc7XL1A32U>3>x2z1J!6YxqPe=
zeh#<~fHW#)pK-&|py6F2SRDuDaG8rhog5}DtMK6+9oHf8Dt~<!;bUDJAu-!hU$!=3
z;hNw5BiqgP+-oB9FH$ZqROMO7GSU+<YH}-)hql9bg#MBME7m}tfO|c>!fr)hoWO;c
zz3O(=aAM~{?aQQnxwSG@`W1hGd;TYFJ#DV9zo%i_3nf{ZX+lHY8@K&hvHmu?x-_^^
zo;Ujs+}cJaYB=B9C4PYxJT@&cQ+4H~2OT>tmogZDl3c@-@&P%?{OINUwOb}NmgK~n
z5fQ+~!Mj6K_`6uMGn!-U%qJ$Tj2ibc@%}H&G}}%uMHPQel1RL>5YPDWUK9n}j>V59
zkh<s(0BeS|n9-`DC3A1*ET}ktsg`k5s=`t#vO;1Dj@}~AkpwTZeb3h?iGU-537peL
zggbzdk&A;pl7puj0&nrtDTw~=pXisvf`A|IKU|6?^53SAf8Y#Xd0Q;lY2N|5oy2lR
zb`*f}aQx9K+@Ry*!1<dD-AIrX`Pb5)6l)hBLdIfgh9>YAE2u|`oEmmZWl;8=+#kMx
z8*2y&+FqenOo)03*ZI)DVD-1dNSIjjaAGrA*>^5_y|U69ZAq@dQl(T|f0|oqz3gmD
zhdX_Se+|4}d~Q|oBg(v~*)8^~r7-j2L(Z4|bqT><+ClZXT9Rr$NX2~e-OU%GH4~TS
zbh5l<I)iCU(SVDHU?;rofq<13wcpX8;C||vsv{?X0?)k?C9?PKWLC)<TGy4=36gJ|
z+=g&2J>Xx$14Ec=Uw34j*tW0LdW^S1gLDpXAZw>L)^gOjSo0q*@MDab0=sJyqZOqk
zf1PSr74m;2Qb{{q>3H!P*LVYkVtY*wl5^q(hMdX49q%(%h+b=QetJgOoE~DFyiopg
zpv0wfq7ZP#*|fZca&&PQXZ-oabZsrp1)8P_GMnSEQlereN7yT2AsX|AlnlWOHqh06
z+y^^Gr>*16V>`;3q|f)kJtl-d8r3g;A&Br_<?5e9IadxTG~K2PGUk3)?xo(!&f3II
z*A!V#cno1q*1C7Bhoi>JCG<hH!)k)&fXl`-q1|o{Z8Gb}V1CAVs>D`k_cknW_TeP{
zww`e|NVeuzfLCj0UrXPqt`8$3%-X~bNsKAp*4=nlbHywQa(7P@MX34^iA#w)u`(r;
zFpPf}+GoD3%nYhyBjGI0la&T5ey&)qrqlk20=pq$%@L}8&hQ@9vc#61_tbF4tS{xG
z5?vt>wQ)e)Q^&ZU;?*nGIp<iEU)I1o?KLBMC-RCRVc`chf?tC&xJjoNpW%&=`q*p#
z$+H_a?bE&ZnTEMK(+s3j!>hqaeTjJ-D21Rtsy9%$!rI&W5HjtMT^=8)a~618-HP2j
z*`=9&#xDTg3yO3SrQUjEgUzQWkD)7HI#s6dk^?_G<$Zo3n_rS4nPFl$snest%9HHg
zMhF?fyh}%V@~?0$xh#pVXQ<mw6vg_G?S&(~V@0*qWG9T(=JFOG60i`RJ<Jf9yfki|
zdAYk?BurNKPWtUP=hml(cd*~?4~GW6;z0Q}C64)^7Mj9IAoYIIO!ET6dR+|f)D!>M
zfq*_Y&rVkN#s-YqO(J7{fJ=zsQQgUYeSl{zTZ58YeVmuirJ5$0!pE=TD;zzHVVr}P
z4@C+eUN$X-U!0p`1n?%FR)HbPlo{yXdZeTo;kUN`>R(2@2$qIeOVNL*5TlNf@Rdnq
z)IN06JJ@60Fdq}41<i=yTba64vm_F=4G$=Y?Kis#{w!_3(q_2Tf>8(bl;)izo`_TX
z=E0u2K6v925j+mn`EYOw=IKO?^*NItnxa>_Oh;7Zb!_MGiJC7u(Z5vnzMm^QVU<5Q
zyiTB6Ul=DO2!!bs{M3;HjzU(-M*w77&D$}2R?Io$EiNV2{tA6;gl{r!<ata&B5L4G
zK5*=FXd%*d?d#Y~xh8vNtu*3oNg4b<0He}C1f4qs=mkhWfa}jnZA*SMay0WR3W~xz
zeAame&nwC<gqo$M2fc;FF=>hCm&62ds+bDc^;6OcW@;Q)>{iTMt;0m+v69hwaCPWO
zZ%7pQx@LL4540u#I!zfDP;-X`vFWWsc$*rxh3fKW2V6bhkgcjbXO#cBEs=d{7VNqi
zEQ4M0Y)zeSTM#xyxx@*Cd~W*zz0M<DnZX6iwPXM8QSad$l<s!|y9t-D3Rblj1lJtt
z(^J#D>!{jny!oMPF}y4Klkc>^wvE=45A8`)q!1CL7ZQ6gH=Y^q*j0S3sYzLO;)r*a
zA9?WzJ~R^}d`6XrK@PmC=bS&=5fqrf<|e#8PU`6j)VFDxc4cB<HTI>}j>lgl8SrCP
zYFuzVgSW>cs8Vp`EZoFql+;4VbDX~5%Z2r#0JqG!acLuq7FW2|x#S9f!W4f%WWK4$
zdd@cW`>#y-Z<qJT9txXgxs`qj%FCC=#W<n3{{2r04mMlHfr`+qQtvBbuK|xx)evv3
zoV1UxM$#fW?BMn*w&fK|3V|Nxv#&-dShk{U+Q9xelT4@W$MMuXwJL>lCByEOjvweE
z42J!N?-iQ#_>3MmZ7tmVcXn)xbM^8MPTJc2fwv03u-erat(8kxg;tLZ89P$85<_3Q
zuTX8t3!TCm_dYVuGFbYuZyz8?87`+8=g&hAB8YnJ)7wzx&n*Ja@UTL><0{5882n90
zKgeW7{Pbg>d;v=8Bh)XM7c;c%EbcU3dgT4|^yx`=ZQU<mIeJ(62EF~bmPqAIlj?WX
zIZu+4m8G%BEx9?@-PealrU6lwZ$p6mA8oeA!B0}8p^TGq?{^;qLYfmi(Ym(od|QrA
zgw}ddgK^zE@^kLJ52WrQ)UA^51E}3~O2`TJh5Yk78M39Mhi*tF969@91~2TG?-%g0
z-s|ylwayi_MKS1J&S-qrD0~Dz?mit1gSdrgZm9Qj@)bx0b=8RBxj|o(U9h9HP9*R5
zudAoZ-<RLT3k|!5WY-K44D-m6W0_nAqd9&br&x(ePsTia8rpsUXQ!kP*cRA7-AXE-
z1yS4Wl{B6{@%N`SL}}dAA(7RqpL^#TFCq87M+zrVa#YQKr{PLb^|GRxZBRKXB!h;(
zF@?!Nj*CLmo{)yGa25$Z4v>I>#b(0`fqR)*xT1W3lbOBT99}6rbg1gI6p_|n))%3p
z00YeC5X#Cwe6f;(BY;=Y!2uUw>1c1)t}}t;CUT9~ubTzcg^)LwZ+4$Yi62}3z-DV;
z2Ja-*5=b=|MSCtb))~4?TIM$MCGy;AFuuzoi%*$Ut}N@e`fPo&wAEqyTDuYY-KyeB
z&`o%9Q-|)OL7krTRJ_hwtU_%<oOjPQNhQHb7~#0>@@2b3b0LUA)m9<)1h2g4+e#t2
zf@-;BpD=Yr&1-Dax}uU5BK&Nkm>pCvdgyU!Ov8jw9nao`EB+CW*)nbFj2?#=4bhNi
z>WFBP)|_?98fj_(x02JCHybv+jS#S65WTFY;clkV;eMf@LZ+5v#IyTxAN3yP_;k(}
zxyk2T;N#T~`3jQ!UY!-Xht*pv>cEHlc~;yuW(4snGC%9|>5AU!yewCUU?)uD3%^op
zTdl2LYcf4PCn!nVNdCk>PzNv=39XqZB&Y(qumSt1=vvPovwdjKOhS%m{F&!|YlXz9
zdE4UmahMawy=oGzvo)VVS_;K@I1fcSHhmuDmE{#o#{zh~*pp2xT@>ClMRxk#a)DlZ
zWum$|tvDc#Cz;{X{jg>oFq1YjRHfOuXWjZ%Bs?B3yPkQ0BY$y_r5>jEb@2kiHdh98
z3Hd3apZ9A&NdKn@v@Yl!(&w-p1ZBe7r4aM3J!By76X`+GS>8|Tju;+;h24bCHE`Su
z8IovO$q|evZ>rRvCjy_G*zr(U=^KBT;=)T!UMN9nhL>Rg$5~fT=HjXSAZgC>qP{4m
zw@OW&&hapo2md;s9|N~O6fq5Lft1Pu2?i>Tzu%|8>?nifd^;AhIj+fbU632T#Xpoc
z4$gc!lSIenkR3NL>=u^(L^%7Mq_%i^>=Ay}EbylJx6>98In{dT-7d}cA!OLUYfKQk
zMScBkAjITA^K+)m;8Utnn#V<RSAs<-JXUZ0_MP^n;^*eKTWM2=Z%Y?H#h#r;?u4<W
zUA%f_pwRgu(;T>k7n)oz9kDbL)-c@fVaD9@T;y9~HuZ(T?S1I|$5&LPx<;V0T>PKb
z4eMi_6TWZ6bQ4teEPP#-qakjAmDdu{EzQOZzdI##>66~*Q@5Ymrg999T1Ax4w(c6q
z>vN~GJ3H*s#;1%5z78G8<?Gop@7rIm?xis}FCnAmWIgZg(9GWPzMr-*X+{mKFSeR;
ze#$rrP$sJ>S&~N62mRL1JN!v7?7}o*?1Mjb@o?@`HSJ>CXyGhMYF6PsyZUe+%sMEq
zn%J(?FUoHuEC}$gjMOW$E}Z?Sg`$ot70atmLdJJ8g&H;!sTHYCjKt(1U527g4A@j=
zZSNNeBZka+DvM~Mrb8f8h?-voF-MSh!1wQ<0F+f!6G=@BGD89A&t>uf7JR2#zkbyC
zrKK)(4Y&Gzr{V44(Av3!$+BbF&fF;u_4&X`NrB{Omkjj0^&HphY!SHVQM_v$cYUwW
z7~6Iy62IWm)cq+`bxL(-m9ANm@wI9Mw_(fpO8wxiu-F`wG1D~-Zf9cb-#z~|lXzaW
z4M=lS=0u67t$BX0ICkpdTX5&}j5zf7p}G^F@sUrE@zFCDN>aaDs|%Sg+2a8(UXBiK
zt8vLP<CI7$``R;<a2N^E);cBQXZ5~phb2vw-;_KNC0G?T$pn|Zeqva3yf>|eWeteC
z%Q%oaUHLoRK|E}OQqG>PBA=!%ILlNu2O8Mi)GtpY-l9=YG&O~yenp#H^%ii-zJEI3
z&p9?UZ+ZQ#ko<>kj)$?3&?#hRiG}lR&2%3!%L@fnZGzoc6!Nk+oWHYfPW5YRb7Z@>
zSMm~cQ|XtZb~bKF27i@nJkR9qk0&*$yyjn3$I~FEUuYk<Q7*B|5%2~6kRRI?W2}12
z*PW?flH&=j<aDD9M*0Jg$gBPgo<@uo=u^6{x5Yi;Q5qp9mWo+tSpV)Rovja2De0zN
zrmA=BB5oN@4-pSFnz<EDx6T~&d4JMf_Y@d8HRr+3?#uO6T+LJK*AIDLf+M}#fS^Bs
z>vj4^;YT)kTssBWS)_iG`>I#C#*Tv^1K*aZg-c}o{*+hEYOcYVF&%fmnbWBl?!eLH
z5u>(_tC&i$AuZ1lL1-uWS}3iy&%lX61Y=8!T~?G1ImZ&#Zj+jb@u}W#<2?1Aabj{x
z|GSubE9yM^uN9_##T*@Pcve<W5CpeJnCOkxgTibBqAvBK(h+EBH0lnwjFn-bib#qw
zu4MHTy;fiib=ng}HxmzBB&;%sWFtTb?Fj*<TZtMZJ*MIOa~CPyA>cwgT@rS_ySnKV
zt~zl!4Obfu_=HW|enPwKgNp8E{=O0{l1k}dyXb+fwAJ5R8Z_#%`gF)(&Q<-n%!Ms;
zueL75*&i#a&AFz;;!Hfr^QLx{Qgk@>g^vjiH+{iC_2I&IyWe9-*UVkcwEd1Lk=7S5
zr0cyhAhfOqEae!>jH`}bgkGvGMTD}R52kUrD}0q*j)L&H|5WlD_DI+5Whg3A_~R^M
z+Hx<I0h!_Q_6YvxFTwp+@Ij2_;P?EcyGx6cr_fDCKWYU%E&{Hef~)%6HWHTERD#;D
z1tj)g!$sJdxDm2+`(DkY3~a5iM%uHS_u)cfVt@UEH@jduz9yHNOK^3hK}YfCbcajL
zIOTnE4+aW}2lMfIb<Y=;^5XP2So#zk{;=>tBah+XH#_t4Zuqsr)*Mq$$a&*4Q_XvM
zgQy#@xqN$+#Aa1NN><b;SmC~SluB}5oUN_ZAm6f$jS0#mT!Ilp_*gbsLlbDF7}cHw
zT2qSf1Cmw0ur#rb4`XA4WYie$kaOqz;tjM{k4x$=q<;#CTkR9USc(W~TC|ewyn051
z%&k}JO7-otwwZ7F((wwz_s?yDKHI4bZ(xB`?R6E7D!b9B=2y@_LZ=d=tqsCV{>Y%M
zpbFvPs*dQ7Al42g8GO%iYlkP+8Ribc#|m=HZk*Ck5eu&-{*z*ZA8wCi=8;C8hl=vQ
zqF=03ny6t9eCU&+FRG1WOOp6z64J~bNd9bjXgaGr!A9e(BN}+lpUd>Fq?4Lgv5EXh
zkEZ;(ZgSTJpC8oHFoM|)Z04!!C`o#ab0GR<hXmuEH#EWUo$NFnZJKiPIzC!4j>3M<
zY9OP+od!S3YO=g|p!S?P_rW{=?<_hScoEf|=Suu6GC-883T7IF@>SMH?)#y}y)z#<
z!YMjhgU26~XxbY2>kr_*y6l2@OH`gx@Z1Tnhh?2X`0lQjG-f}ilWG3+D;F1l6qIuy
zCXxM7N<(!UTIBDP5h)f=mC+2B+cB!TSes3(rO;7=*mj_4r!17|RYB*$2%oZWHY(Mt
zszN+-u}4Bsbmyxy?UBtLHs+)*G|<Bs(bJ!)qQh8_rPasfafLK#H)f-_w4(rG5U7iT
zr_?j890YaAL7a^cXPO|k@&C~w{|cDtkY`dD-8{+i(#Hmin#0yAG#o&a8@8L;9A_1%
zZTeA%%fWC<h5@n$)n4V1zWdr5Sl)yFNI^AcC36Nyeo4mx+bQJFtz50qtds(dC!({l
zYPXc@FA#sLk##}B)*}Z5)V~RKT!t7Bj(hT2R=$SSiqj7+F67aj!tj4OvXv4(J#->h
z$J;@3bIKIh6ZSERkzo13RcrPwhr?_8l{E)tTqWEl+&gJ2)su*MJy@m?=>E3I*D@f-
z(=4hd@iqBYVd{+$k8ILpmyN07(D82mYZSMAx~z)=ftR^r`;y<PRZB-deiuyF2bf=r
zYXo>bllRQf72ZavwKHj9ydnwFl(`fQnBf?WW|m}@LrL^2W2X089rQ-wJOR0nd-^Uc
zP6^P*4~a*;PV{sYVfvZ!i(y~rw%;`MG3IfSApk~EESx<sZYMrUn!i?!6s7^!Op>21
zD)3sPx~uECR6@a26jj<m>bGNDiN+!p9#HtD<zCMdX%-bX)H0eMwrE&PPT!;((sh(>
z_G*M-sl->E+o(mk1mIPX%J=fFR8U!q^$Js*TX)Zu)gZhru_yP%57Tc4ajcA%{$$UO
z$7$K(FEnG3UjOVmVotV5FeD<$Q;xvcFREEBM_~-5$W&@f=Hg-#*-{%{gmacb>{z#k
zvtpzUhb7_GhmcO?u>Bu;WncL8@ptM)MIIeF@TNYEhELt=c13Wv&rV`khv8hVJ?WMf
z(5xN+Doi+U_M?b*UCo+x7Fy|_vB~tq9W#uoDl2ss_VbPO##L)|b=xY_uM6ce_`%J3
z+S;sTXi@w$!WBAmP<v!84fGOi4fJp#y!66ZG$BotujN=ah})q<;mlqbI|PBne<@6a
z7-;)<@C!c9$Ou5_<};c|>#3Q!QW$EsM|Z!u!~qsYOdp#=e_dJ~$f>q)7W6@Ve$j*)
zbx~UxE+k#zD|bCd#T#fdJjC?iI&fdf^>ndr+1aqRo@=$2O8D*7Z<v7kv!;9&nkl-p
z$IrhC9w>~mxC*$biRGl^Y^idwHNm-n9{u20JeA}^B@8$D9@Au;+2a3Z&)R#}Z-TYo
zbSW^6Rsu0;>~nEVBBc5J#d|;ma0<~t9Z#N?Ebhj}z%mt9q>PUEC|~G*wl>@alaq(D
z9k@Rno9VO?r3dD65UK7ybYpHKDAq_tL2V3(S~zz!B_e3tgy<|a1zSBMh@*MC5yqva
z;rv*ZT+=h)mgiBS22q^8Y%bwkKESm!-Erk=MS<3OTd|{_Y7Zj4du>8pR9SdIJeBvM
zUi#L+92)2s<${T(6;c-G^Za8<Hqr2o#5v&GRAO@;V>MfCPeK(QhPXJJQQ*>!CKrIY
zRxdAOb_6%871`0q9uft`WYwXR$ETYASZ-P!)Q-9j?Kc!)6JFs>D+n1Vu<(FINQHjl
za>Rr*&X07a3;6%6h{%@@1v@u=w^z@r>KTwhxtkLor{`eTwIopz3s%!fbt%-%{R9Yp
zM*>mwDMq=?n*T0i+-{QUe^WIJDEwI~pJ0_l_LV=)Cj=s52X*JQ!Nvt)N2e+fp$iL>
ze58Eq)_P{COI2u<lQ_6bCRD4Jq9^uSm$c4SHCzgq8ZfJKp(IwU#KTIw_=+T@H>Ps(
zrHnT}IH#_xOk2;~0D-~lsu4#UJ|r7l<#OoXx2ms>9L*g&^1-@%NIW^oIy7@(Q>cZr
z;T3~MeucouaZ$dn>ckj+hz1)?RHzV}4^dSAX8_%|5UmuU>9FE*2!x@iA=Dp$wu2fY
zfN!2siTsrDt^;R7cUIFBcj40RcUjs}#laZ1{?J!(Hf2G^cW`Ut9SPxDKG$?}uR2!x
znYgK`Jm1xwMEw+qAs6?hBsc-nDa1XMwdIYTDn*#}d&1~9De*cUGWD3Kc62qvSJ4c3
z_$1FjcoNN8(~0}^{E-HxvBi)n($QM40M2qJngo<L=Czmk_(q%-M6^Q3CO6+I4Fz#7
zjLi<<%KSmuYYqWjQP<Cu>_ikv_4MfHK%r^tEzWf%5u~~a>QeP&xvCLv(s37Nk{J#M
zAIuCJ!ug$Y?cJLRHm5?rOKl}+<(a&IcKC{Ov)8<=+s!<}SoKvKEx4Ln6tDiJt&&f!
zN0!Qb@PTGP{8)m6x=-+#TBu#UFnvUPTtoQNr1Z_P$@LT6g<R9>*;^u+c%2GWHrV|Q
z+}AnYiv0Dr6f4Mi@ET7~<}=>2VYjsnyqi(S=lTJZtAp-CNvI8n<K9NDN$}Kry{;yK
z6b$7`)Xy40umF9+`tNFWY#hx^QDv566P@m-X{RzqIhFH9e8!RbEpxzivw*GgL`>C-
z6SQ<+U_D8A@h`e9!`o5O>iq)t4gHd(U%?z4p$nZN@%!zSIVzSXE1i0@GuAn*Q%ZTI
z1may^ln=zZ7j`d&C7O-Y#6KlV-SzyHq$!64T7H2@kMJvN<YSb0zcC=B2uW#;Jl#5=
zTQkR7QJU1vK?ZXj#V}Teo4vPaPMI{@wVo^S$DowTBV&m?$w%)_kEP+`oZhw=_3l;;
z<rSFAP$fUEl4S>gbFIJv!r$BQ)dk#9RoD{-Y$*4sog)%HvII3Hz73oGT9KI#p78MI
z=gFz-0^5|)dGm`<)OA(a6k^WJjX0uJ>V=aMUqLEKkmssZF&9cnu;vO8E#u!ZU{D$0
zSB_;~j)f|u-Bt!Q0yHB|R3Mb{zn$XWiF1f1l9C*gJ%DdTTTz1;ke*#Qt5gV><Bjjx
z00ou7`H8<DeV2$1ubnV07|l(UswD&E2R!l1AAtZHIMWVXKAVju9tuUzrw%M8-$y+<
zvNw6+;hU8-vlErNrH%<rGUco90m;U#wXB6xC4|Rc^=6u7RcbnuQJX+SY$EO8d=OeG
zjgj60qiTQq32g9Q-3M6Yq1Lbv5Lk~0^@O=`9b7o{<1|bd;$ZK_XdUfOE|cIj-Q_(g
zMH+NUHLZ-Kbu)#EYKVxg(;b#gS9APVv{Cjq5+@!o!b#VO#xp!Ag<SgI@DJd*md_6C
zo#9pAmuqZBi5#xz6@^F^FtSLt&nq`(b=SD1kgCrOP{l=jt1I!=0%knqb-_N^Y&!9q
z`Xpr)peU`BL^Wx$)g*2G4JiWHr4zX*>=r+jpIw8-i+|E|$faJUVq&`cw$@qR9rOwr
zv*w#P`d)U@x5JxCA`W%2HX=X(o?|*(#=|`)Uw?h~3%JCa1*Ohjj$Y(Q)rGuf%fnG6
zTgfVEqIi;IUda%YvFmGIocf->azcnA#gQi1;m95tvG){p%_6ge7LL07$T``FH<dbb
zU`O3TL`?hpCQBRHhMz7c3h@S0PE5xMv6&Ydz<&Z-FsDOP;e`}w1tNkWvR5J6_}{H(
z<Upj@XsC=mpe)Bo8KA6;Sz3t7kF!tE{s&+^S2`yMyyXe)aKIDI=6yx=ET=nF+u5(W
zEs7cTv#!9Ld%xA)<Y#+%O(W$yt7mg~B{S_B(kCV@ZNJq|OjrihJruf3TKtK&N0QsS
zM_MG#>+F=SXxS;dm(TpJYgYqf%Q}{pZ9tVYfcac$Pk8KtVlVn+7b_yV$%{8uXmkwb
zQ`8_h9gR8%(pcUm&~4SnzL`tl=!tOUs(8|-=4YAKZwzViv#NF4cn^FmB*MdYF+X%k
z{jHC~A{$7$#7jo0=vUhkYuIS{j@wK>trL6rPXu@9;uw@MN^^WAuXa>q;MxG$R{5iv
zsFymc;)%1#v*>x_plh}7Hx}=NPt538<vYs7Cwk_j&|oKfCUQf-q=(S=w5W=T<i~oW
zHJZuz>JAko=%GgNV4hMt*U*JhIqb@yW$p>q(I{2C4CK^-OhdqHIEsvlM0aoeR^ty&
z3<i{(;N5OOlna_IE04s9oA~{FKW+oN(jfVDA%wb1#Z<|V$}8_%50rPDM20lKX}Vun
z4xn0AW|o0GH!Kyq2v==Jo*PAYH{2-v@2($1YGs;v5Sw19S7ek<0Z4I_)`~=I`nkJ0
zdNzU}I!EZk#7C$`#qrZp*b2(Zyi(Zm4v4)k71L6*2b3~pCL<;l5u=J>1UF?l?qxXu
zhpJlz(jV8==+$P_*R~ImdCZMkzfurmu7SF22Pi%Ayr+6wJMIt>SM5Pcr}S%l$$W*`
zx;z`yzHcfXO5#U-zNxHoyc>2B8-YIk_I}W><!@tV_fY8R`A7Qs4&V&Q)rE&pHKJ?L
zIMGS~3wbGx9Wx+lq)Ba93c$?pxL&AF0)JgY2p})b7#h}|m1u5v^<e&FZ92G5&ai<e
zIZ#wo`J(4I8EI{KC%jvzQks3o+rlu;Exb?G9^#qoc%qSMmADb0nNnq&&YjoJV7E}f
z?l_{`ab2-3>XZ6dIRnZYaa(WpWxBPo3)dDVXRf3xf)F^XTD5?cmghhPTiyJIr8c;9
z@f%(UxhI>n#!$-O(jA(`^;OGI{z$Uxi^Esiuwy-|JUqOM$n}pc#Xiiov~C(C8GgS@
zSMZf{E{`o+@E!XEPi8*`)nM(DPu?A?_fZga^pZ2+N#<EkhA7c&i}UK%|Cs0{_a5H+
zkf{(95t3b{i2Y2}THK9yN~bcP*bQ8xXpqpC(@BxaYeW-B2tsyO%E8=&9*SLfsvxv}
zAXbWBzdTru4+%V1ty!zugpuv2Taj62HrJivXj7qAW(3GU+%w@umP|$|YOl&l&C-!r
z%ApBrVwx!ZVmjut{QxoT%E($#K8z+-2Q<VHqZz~zq<&(AKmd8BqG*rAtE7@6D}_{+
zpI(N*#g!W0kwYPRPIW-&^w14mEx-0>$qwnG?^AQ5R?1H6vd(h^+&9&te%sJ`pmgGm
z`*X2KlR+omy25pQxEsY;rWI_BRW$s;-RwQjnNi<d4Uzp27=ICt)V?=uMwSh8HF4{s
ze(HMT(2=}ih&~5M=)1YD8F%bn9y2A0d1BA!Hf&JW^}zZaZ$%9kHJP;aCG|b3sq1sr
zp7Yu#X?6<S8;ufIdN`i6nuE$WO47O~l$48w@HdvND{6=4gehd+C1Q$)-p>ngD4x-F
z;ppl{pYurMXNaors^FlDcOAcLo@eCcD?JrjL6Y1*Hi9_u1m`gfsH0e25Wn+u{H+<&
zU8Mwh2&}BS{KgZ764@$rTO*pFWU7)jXP4MF?^%;g^pa~cIjFax^h`7dVp0RQ!i~_$
zU~ta|OG+3#pPZokc^{f|StIP)qw<2Xm^*pIfAH=pqrkH^6-#~@ZTRc$>2+DLZ1&W5
zCnx?+f9E-icW~R)*KG2u-}h0&Jp5~3b#O@LB@~(yh(Zm%1T$kN20-g{7*=17oQhA`
z=N1j$eblp%O$>e1Mf;?T0l>N6W>}~pu!is}8v6-Ph|Kqr8{AP=8CY~vHQni;_M`C^
zCENc{49H=txH4!e&P4=@O$Z0Kqb@PvrZQbeN~eM~<}Y~u0g;??aj_91!{}D(6(fMe
zO$1HR=#nqXXF=9<=*hz8yO1Kna^w5@H^VHz(D<~&x57#75yPI8a*T*GYJZQlDgB*A
zNuF|G`I}p0XTk_(D2!yKaQNkazmKt7yw|<bc^SGWhv<Bivmcn6KJ$st##6Mt^{q<z
zJ)a@G@kBD_gZCyp|GV;dXWW>)kkF3RLNn6P1Gpx|EdU0aMJpO{6tLGyiD;znSOiub
zoTTz)E_0T%0}y|Cr0>Qammy@{j{bqFhc#PXKKOlQ@y}=aV*M@*NvuLFy@m8B{OtOi
zpVBUqRI!emlW<EPpE0(LKzBKtjH{IlWCl?h$@?J=*+;RPU398kLqDEK{BLZ9fV<lu
zioF$;Eb7~2uQS$4KeE)3k#D0o(<A)!UPrkE@fPpMugUuLL{if8U{*zkhN<fDUxG5l
z_Nn3eSDMzI`Mt#Lua+yl+1~<Al$fcF!dm(*F~lwklw%oFZICokrfloJmNmuK3^H5-
z$}mlW+a2ip{F%i`n@%})u&`mRMW=ckHJJo5iTFad)#t{k%XDRXjWI1#R;fa@aeflI
zG{LONQg}!vS2InSL?yNKIyKzdlnVT)+jSzk8-+NCh6@=<s8qo7^mgdBv|O2+E!d%b
zs4m?<bxliy_$!3vRP|MuC#0f`rBWXb5Jn&%tGHB?H6{R>Ehpa$o<fa<)XEpbwNx=l
zOO4D)=%#3o7fOi{5I~|ShsNtfgi+y&=~$K`P^B^1%m>-~7Z9ZJGmyQ%!@!8-^A{Tw
z$}EMbrF>0qr;sUR?8Wpcw=~Fc#TgR&s(FGB-BSvrkF{AVsy{mNF&pWHFFw<$i*b5O
zet_qM-P<ZbGl%@mktk+^tuMqV+BF2G_w^4T31-)91BQ(ZeLp&b-S}R!zeLrklK&nM
z<HOHPR>4O-=UtxI<|18&r!G`pII>OLJ<%Cy!6=fSq52gi6$5ay-~2{BtP)3(VmybD
zU6AVeC0!bF#Vg#aOt2qw#bg&lj_#*T<578tFe+Bd*CO{OYidXDPbU*v#aJZ^;bb0u
zM$6%;TveF6Qd=MP&fzuiJS(llQarX#EAd%z5$-7~u|-%CHZP<e#*wEke<xvrUyi)Z
zcZZG^^u}R|5dn>Eu(P(=-G~cnZPebR=4(1%AIc4mBr;Jl5+gTqi(f&f5{DI0zI3^$
zIAT9QiSx6_ir5E5)Aq!0uH*(%!NyWGZUh%|KWj0Z1CiEijg5oCNrWhmE{&tifP4vr
zs6Jbgo+n^$v(8SPBC^En>k?bc0paVyVkWZyU8j_0H8XGQU-K_|V`2}(rO1b!D7$;{
zbLYHkR)35BW*??(Ym58bdxhWoWz+blRvvVz)|ExfzjEJ6MwmXZ*>jSknZ<>yo@KkL
zRm(^za!qHQX?8OKZA$Y)WGnrbX7gUZX)mz=3db*OO^H$O!Pj?(_P@OZb#$uO2pnhn
zZzI|ss@9C=D|_0lUENcR20oDwYjgI=o*+_-QHGo}&h%JkiMRWSW6wh6cdMxW{^cZO
zxRzF}a#NpTGY>S@QMZgvt{iGkLQ@E=D2H0kBAT2Ohpnup3n_f!3pL^ksVL%;t<?9#
zNaBe3xW6YbG?5S`E&_x~icW%>i%qAzbbGt9iB%=c*M$`W5FcoJxspk4ko1h;X#%J9
zt<Lrp(}y3QLU|Kf^p3Khm_?Iz4JS&i+UFHe%&MC#eRmejD(ej$98o9CWM_o#T%Vf#
zy9OF7%wXS4M|ulQHbP`7$dyuM^Ap0@@(uF%6w3cc08JIL>Psfq_mcaLvt991Ut6pe
zqDWBv@=q=8#b)?=${j~mMV$krV}8@qBc)<Igqq#(mBPiWrtSl&T>ih(x%u3rn>`8P
zqeBlU(?_B(Sfaq`R1lMJTiUJMHe>5m;#|ny)zFUwunFgbPg-x#(q{*npe5(1{c1!z
zxCfEzQGx*KKD8q7X@L0@5mXvA-;SB>S_huLl|tw<o`!)He2^4{9C}m%3b$MzQ&-M#
z4mtLyCIF9^bLuDrNo=SSr_4t*`Sy{??cS=#I7P=?dQ?)}Gn@>cPf7-r$fZEd!uP0N
zA%YJ|w1`>0Vm^YW+JSfgnm}bN3|O3X>z>r2*aj@eB=9lXuSmq7lni?c7B$Dc1tJKb
z1E2s3u(4rzse)j1#XoL4Vu9SQ7oI+p!nizkt0(T%4}sIIA>4vr*0b-9V=6hT?T`+7
z_V%h1DQCyM45HPnVPRjO{A%)y4{DAG!3*0IU))D!X!FR}RT-?>1e<n8CF2{>zAIpV
z+ds&ETIH1S%|8C=uWsa>$t=gESNwSESW8_`q{Gqnbks+?{H@1PQ8yzw<Wq^yUfk4V
z@M$A%T}kb(Ww(9tIaVF(L;eyQ_|%n1rN{7*gZ%3nGI~?8SDxOrq$)}q9QE*&t6!OF
zUjG1u{o-PiSg3K^aTTSe__|BGt&a>!n9c(o*#1?_+{G2jZIuIzagoxi+u93eGOVns
z7YplMu&kw_;ykO8PX6-pJ!;x`t`IzI4;jaCR)u+Gj;DQLCivtox%IB1&&H53Rk?_v
zJ;iEym2k3ywXN!P;+{FI%WZQ`xW|`s9#3XCtt63H0th48na?OnJEBofT3{g5i$NpQ
zQ^hf9;(#70;+QGJfk20fTv7u{4FF9`NksrH6$JNCTOoO-NYr;`09P-nc>e%Rv)q>F
z?Gg@z94=@s6y-gU*XFY|PY~&PGUod3IGBut&RG5fn)2;W$2y(cFc)SR^TNgQ{-dAP
zxvB3&(w6etNe@?YSf9>=r|PMEQ(vFtdM>N+5v~J8r9e_LG4`2wKkya*02<)63-7k>
zjhUFUWVY^l)iyKBfX%fC=~6%zCL2IF?rA%9JxnGRrL9UfU0dtui?IBtv7q1%)nx!e
zJD*ylvfzd3@9S1kyX4oKQZ6r3-b%{FiC6%<_v=x)0%O6$jl!D8a+*cv0Dp-;m1a_i
zrTOVr>~4*{%yTc^Mh#Tk1X3&QJ!ll-nzKEA2}JhEDC9T<RFI|^lU%Qa^=~#mu}c2{
zGK{8uJu9(6#dSgRE1xZ1lchHXC?Dt6n0Pon>V23UO-9O2YeNf>_T!OL#@WXmD!Iy3
zh}@33@69pXwZS7Fog?53^`y!E6OVcu7&HM99A_02iVspMUOJD;ntp<ijH%&~Nh?f=
z9h_w1wB12CsC=NOBN_ToT9$&wrFkPXr;exdqYaGq9=&Oq9Fy17^`JtG9Ot*KNPy$E
zDiC?e{Hns7a%mJl01iPvg*A^I4k#~>4@!%7T(CUxKn{O#P)YLgMrrsU<e$Qkl%C%I
z07?dm&N4HPtwO;AsXP-_ijp|(QBQ1-BvLs)yF0KwYFSzrQ);GtD~a)LwwDqc7*EL?
ze(|%^j^Bk_@ehc;%B^i`<ITs+<M==N^}@54Y@D2)^^Hk%N2`FsJNsGuh3}eh>5_Qw
zNw?pM3iHQqwUO+kCPsbfjz>8-?ae9a?@C#Eu>9y9h?j%Gp_?5>d8JW|jw%UUsmg+C
z;}<7nXE}2F%$7o;Jh{bIGYoRVs!T2>Vq1~eW}^EvpzsE3zMUl%t30V@75Sef7xVuB
zhBYI&UB2hikVpafhDB!!X#(&^=SnT*?%4cmPF37%8D2}?p#K1`k*Ko9R{=uv*wyPT
zR^rHyBI;<ydTz>}z}G$Yc{{y4LoR>ToPB8dn944H79a6Pp;~y#bYQm;$1nRzcAx88
z)_y3{?l2vUEP8M0`PY%E$_M)9%}X234g}cx)A^jUz(!y4>PzOC?oiJhsv0>Jf#elA
zsl|Auz5VPsF<tFoeMqf~PZsNT#1@cFo{V$<0N12()WA+VC-YzBbJc*SjMpEgc<RdF
zi0@IYrS&|i{A;MTytleg&`lhJ)mR#56sgKPqR6LGQqX2|_ue?tEZL(+lI6!}Mn~t;
zxeZU^2CH&eEK)mE=op>N{uRfy@HpEW9DWrGT+N(_@y!~ZyfD}}{8DoCMZ3$3%jpc4
zmyauFp(OtR`l=5u6<z@+JY>`i^EN(J9-XN%FLE<aUgumRhF=XCN%^AAV%vbA9QDUq
zZ<wgr9)wj>42P_rl{f6iscKeT&ZyU{w5LMffn}~!fLITqs9NnoIRu`aDW7Fi)v4=l
zihf{Nd(!t%Ic77_MMwJikUh+da4-k8Xjxvd`6N@5{9ofyc?b{?qpuuO@-xR3Gv%7I
zMNq+FBU9Q`si~?C6P~n@zR3?6$i-5Q<i@x>k8{$c7ct1(NCD47Tr;C6?0RldkHqX*
z3$e<73FKtcPs)C7l%=<>IuX#)ef0O}MPfEjYjpP43nXk+)Z}8lKS0-E)ot)dS<rmR
z{Y7~zWE|A?nvAw{ERe`i)bLk|xJkv`99XO@YsT9&dx!+B1$gOPc8BALE;d7aJf}Q)
zgYKWG?OlROlAv{9%0@v4AlFo6rLpsss8piuG*^*FQA}KqrC5=0GoQk%jtx<oWrR39
z_o4X|z{PmQNuDrA<3PwuV}dxO^8w<X%hNRxKsi!#??A^8wB84EOx#8|rQOXS5xNnN
z&ZEG{?VoCqh|V~mL{C67??BTSI^Z0AY2X~-15&I(d2h&ZPGQGd2XeWLxB<8n1;;=I
zNuN_q$sGq23hV;(q+m{X<Gnv1Wtial`c@vT;+riz6pe)P_f$L&sifLlnp3Abj?h;$
zWq9C?<#q~yWR^I<u0LM!K#_jWV8@@mCNcR|Hm9%M>as9)*$_M;^#1_s)>Agt_vu{o
zsNRV7@VRn>=BumhBW0U&o=<8}az@?-FapOlCNOd<D;}+bv0k6jkdSyEg(I9O=L6K7
zW}taOgNzgPtBpBJVMvvjV2*R`P&LF!!#MhNHCjO=nNuMH&<s^$XbJoLS5z?3`%K`i
zTNbpXPw#8|%Z+-3e69z+d8o|s=O#f_&99@<8#WmmhJAY0>M)Lo!QwLor?Ovv{1S6K
zf8QR|dGQVlimoh=c`1Iy{{T%tN~=e>{k~yu;V-}b2|wB*{`jEWL_d%8s6NHl=lv;7
zkjLNsD71Tv+vZ>M?S20M@JT+?7hT__C)(oAQhy4JSo47A@}ulP{{R{MsP{?sxug3F
z`uUQ5t~_zNnr{=Im8lJqXO=X-Vvn&jUL?QmS$1D=3)NCSQb(p~{{U}NI=9l9_B0>F
z)P~4*bNN%eNw#Yq&+aXy?s)SLUPWHDzi4DxA$NvQcxcWm39TG`(hVY+p=^w<DcN)}
zsfwz!#nJhr(zM?jY8JR#wOge1`>sEsu9goQ=+6*oW_Vw^I-H8}h_BsveB;>ktI^*s
z!Se(4IHhK{IsUDKudO-T$(8qm_l+l*2RR)*Ydk?ScKqq$+D>zU_|vlPcC5z|{u)1@
z`4;@tB#q;rdQ)<T2XNh07S+M(DMg{`T9u>RudU<dwJ+=MC-dX}@SxnpKa2FJzQ#MZ
z=}s1j+?qYom0ZeSVK3{{Lw6rI%_Cke-V?Waf7v6)L;2GEo*~+ox*pFmm)b~&?W{iP
zflO<MegpB>inaE5!9O6S5yn4(N8v@+Vjc$@Q%aR|@+s|_L7XwASS2L!kEyHKAknyI
znq1|qnv>RYZHaH^#z6CWo}Fs?Tm#GS!;FrGsdno}nyNV<X0wzO_eY~jg*-fLpWe+Q
zwo&q&`wD_c&+wj=OYJDX$N5r!X>Aw<i1!tj)t^R1OlBGyN@%#H*B-xG*3)(C?KQ!-
zxpe;kXFn<Y4;9XTY2zeCnnS3Ak>wxGrQxa^)68&kR&u||{{WHdmLCuykybrIEv^3m
zJtOJ(irlu=tS;PnH&Hn~0XRRAuMRhg-I#tf;d;@X$nHIjQiWB}^Xe=N^^EL&3XG=*
zAk^Swe>(6>UlHHh^A*I)?5seot8a)J%pe0RNTai;HAzpbOdc99;GO<Y{EuLA2cW2m
zgOmDKn#J)d><}ihiT!qxzt*geiAnr=duRQbNBC81*q^FZ=&pBdj&L~?-N4{v16)b?
zi)?z>%1`^^ntme02g_i<d&&O*)}>}OgsuFicD*_0*i+F%9#1vrPw^UD6BUvVP0mmB
z6-r--I;;Nxs}aV3*<AksJkZnZX&en_y0PlbfI14NaeHlUq|G#O{{X(gSCQR#uU5MM
z09kh~Jr(djO3Zutq$II9KBkk5`XhK{&+RwzN37iVt4g;S`$%)>gE{_{gLmVT%Gz6x
z_vAVK2l&^YH1ZM&BaYdm^W{I?1L;!f)O}8f;b}wcHg8{R_xHq!Ze5h0yPOZmR&d>%
zxvb`EkKvSl0~HfTE1#NE_?k-%ec7|b)TXUYi~JF(B<cn?Ao{g8?H0;DX@-3VHJEM5
zJP>F$q#wL7S}Z+2tij^*2|sq1_4k@5QiK2(2e&4od-W-e_yg05plz6REA=$jw~&qq
zHKb|NTN3eEg8u-Qf7jkyH&KzxQhlN${9>v7l2hCDpxMfLZ6A+Xw;1;Yj>rE1&#%G%
z0GV^oj~{$~bfQ!|?_bWSeTrW}`O^K8QP_TTaxUdO#~8om%lu2;WPjfuogQp|-yfY&
z`z*QpzpWo-mmhQWqmg$P_LeXHe80re5+VL2{OLTz{__5Ho%VS_{{U8hIzG=S{sTuM
z?kAe!SNyquiKV@LMhCqcfbJ_d$&emUieZXF{0;e0?ALN$ab0gq^Zx)NS9<jfKRR8*
z`@*s|K7Q+dRBdeZH2s?HFYUUEdS8$H16;866l8X+kDK0%u<mH~eqsH+-k;<D007sb
zvOL*8zTTZ_!s1c<XY-}(*=s(g{wOnBr6CoZF69HRY3X>uj_?Shk(<Xl!plc>@-93`
z-BeUwKp(_Fp>CsW1CjNqmDG%aF-7Eyl`IY`T^NI!20t31_KC0;KKu$i%Yr($<wqkh
zS<M%;TQ8sMNuTFg<*q;4rj(eB{M2%JlJk${rp+?SI*<XT8PmJ3=UL(jPw<+MT1s)1
z{b};9tdH$HRK5y7nP}Za0r~#`Axvw?@NtrA5osqMck`z$lVoLu7KpyHna^lj<Uj4I
zdMUxCJ==}~3}>x4SuYu5QZ1}&+rO<V#66yAJ>-|%RyUGl;{%^cVY-#E$o#7E+eANr
zQ_>YU3WKi{b4Rgj0gac%RL%1xaz1AF1Bz(h(`i3ZTE#Y;V;s_d_2}p9CzHy5#HoLm
z{1Y}ckdOk9J;xPcZAt3OinW1Bw0@MGMr}_fp&yC1KrF}islwJV)DnKRCeC;!ntZ{?
zG`ZQ(vivkZ<=^T87>Dpv@St$D8IhFW?fmJ6@kdfKMcXU=nwLbr;%wu!8TP9G0B?`e
zr3U^i<YtR8{c{Yy+AMAzPG~F*RsPVAJgKBN>(g&<S}e$T{5tz(zO=cjU)lq(ADuMn
z2RtYrojz7(^Gq`PMeuSnPTzyjA6m`+(0@QZg$;ivW83?*axxxohh5DeM{s)3eSNDd
z?UEewdgrY-&zS!J9X>`w&ZsVo*CY<&kh+@5^JK@~Y*FS&Io?GajF0Wgf6C2YF%PX9
z!RRY6PbmI2rSs+f`%&!J)^|&)W|1st0G_p-_L+zAoYD4)K>OtOrR>=M0Ji9Me}Sd9
z-jrjtW__k_@0`;8qGQJMN3vu7+@bue)Aps#Yb;GK_s^v-+GQWd`cw977JEaiU*Kud
zaaq4?oBQ9TA8VVrAC)g=%|EvA-JjfPQr2hNCG{VbD7=%=X!dLtb%yJIxY6op?Nmkc
z0m=`~o%a6#SpNW8IV5Ww-w6KK;LEg}RWW|J`-k(T7s>tBrOL}$o)P`B-53U|i`cqh
zhv829WG5dsKMFZnqQ_xn_SfD@R36lZH2&cHsh?<w{{R@`m(Pg*02=;uYG1X(%k3|~
zJc#l4MJL(f{@|y1(c|xr&W}1W3EnvOqf%PJzvcV}{gyB8W9Vs(Wf|#^YJV{?_lM^|
zV0sZn*%HCwC-ztM=u>MZ{ua-%sLT?Bji(;fqB_#&uTR8PDZZyAOqUNQcs7W_s2?cB
zFG^0-6H#;1jAI>;ykyV^G}mGUK-0L-PAR0>ooeu;uOiPGqtdgD+`Wh#Q(EF&{qt4G
z+2>0+pq-LtpQkiurDd6<&);v(nq9d0kLO3TW-`xUwu3c;FC2<tl>q>-AB|;#Nam9q
z{uKS0lh0-LVfdPVXozDkO*gS%07?1^&OJphLbUywFqU<QdnAjd^mbBeGkm~sgY&2n
zL2xR<+o2gbqn0Qs;_=npqbm&iY?&AXr=>LA%tu^tR>iDbaDO^?+2S9-I#O;cq1NzO
zjJ_(rizu6kPCj!?XOV_7Pad_TA&d{Wzm+^9j(}43i?F}4E6scLe={p<pW!s3-d6qt
ziqycz;Y--j?Fv~GKGfNmEhXu;kFiUh)u*1clzT>OvRpddwr2|&4^k<Av)G<kR*(Mx
zT7yfSnk=6Rx_%`Ar~d%A`csCUzuq-^l9xL)SzZ-=pT(60gk!5UFW93Q>sI>;Tnagv
zqRX(W>>{$-L_dc=Q%?IdS^J`<KhlhR)UM4VgP^v8`;iaX;{NER7V%^53Yd;XC#4r<
zeOiz5FS!x<5r1$|<VF3#QW`)_7i0RhALL(h8X$}Nf{!9E?h2BVG_K0}y)WfoaL=>F
z{lQ8hjDHBH_oF<~b}y^Z{#ExOh6u>}i%t7HB>llso|&SHu==$Z_7~iUe$5y6X+Fyq
zKIp0G2Q>MaKU2g103!R557{EeL;2Hw#~)&9&wNowGa=wGuXr!GsrD59@uW5Y<K{JU
zy&dsSo<?sg!auS3l#PJ?{{ZPtHVuv+C+SzSOY28FA>|lH_J0;<8##LsQJZ;JsvoUt
z;}n%%(VX+h=s$(DWEk8YRQ^<Ke0|gTR=zVqQN=%KVf}+1ll%;QtPev$1a$_s02G)9
zsH55m<SY6A00Srk=xKgolUkcbdJ1!>0~i#0L4RQOpW*)i1kFpC5dH2yI<U9NkC+gB
hsJVqaqQGE}O3h+H#!XU6TAY<?;_~_XK6|gc|Jfm68l(UK

literal 0
HcmV?d00001

diff --git a/prototypes/responsive-layout-flex.html b/prototypes/responsive-layout-flex.html
new file mode 100644
index 00000000..97995aac
--- /dev/null
+++ b/prototypes/responsive-layout-flex.html
@@ -0,0 +1,506 @@
+<!DOCTYPE html>
+<html manifest="manifest.appcache">
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, user-scalable=no">
+    <meta name="application-name" content="Talk to Bruno"/>
+    <meta name="msapplication-navbutton-color" content="red"/>
+    <style type="text/css">
+        html {
+            height: 100%;
+            display: flex;
+            min-height: 0;
+        }
+        body {
+            flex: 1;
+            margin: 0;
+            display: flex;
+            flex-direction: column;
+            font-family: sans-serif;
+        }
+
+        .log {
+            flex: 0 0 50px;
+            overflow-y: scroll;
+            margin: 0;
+            background: white;
+            color: black;
+            word-wrap: anywhere;
+            display: none;
+        }
+
+        .container {
+            flex: 1;
+            display: flex;
+            min-height: 0;
+            min-width: 0;
+            width: 100vw;
+        }
+        
+        /* mobile layout */
+        @media screen and (max-width: 800px) {
+            .back { display: block !important; }
+            .room-panel, .room-panel-placeholder { display: none; }
+            .room-shown .room-panel { display: unset; }
+            .room-shown .left-panel { display: none; }
+            .right-shown .timeline-panel { display: none; }
+        }
+
+        .left-panel {
+            flex: 1;
+            background: red;
+            color: white;
+            overflow-y: auto;
+        }
+
+        .left-panel ul {
+            list-style: none;
+            padding: 0;
+            margin: 0;
+        }
+
+        .left-panel li {
+            margin: 5px;
+            padding: 10px;
+            background: darkred;
+            display: flex;
+            align-items: center;
+        }
+
+        .left-panel li > * {
+            margin-right: 10px;
+        }
+
+        .left-panel div.description {
+            margin: 0;
+            flex: 1 1 0;
+            min-width: 0;
+        }
+
+        .description > * {
+            overflow: hidden;
+            white-space: nowrap;
+            text-overflow: ellipsis;
+        }
+
+        .description .last-message {
+            font-size: 0.8em;
+
+        }
+
+        .room-panel-placeholder, .room-panel {
+            flex: 3;
+            color: white;
+            background: black;
+        }
+
+        .room-panel {
+            min-width: 0;
+            display: none;
+        }
+
+        .room-shown .room-panel {
+            display: flex;
+        }
+
+        .room-shown .room-panel-placeholder {
+            display: none;
+        }
+
+        .right-panel {
+            flex: 1;
+            background: green;
+            color: white;
+            display: none;
+        }
+
+        .timeline-panel {
+            flex: 3;
+            min-height: 0;
+            display: flex;
+            flex-direction: column;
+        }
+
+        .right-shown .right-panel {
+            display: block;
+        }
+
+        .room-header {
+            display: flex;
+            padding: 10px;
+        }
+
+        .room-header *:last-child {
+            margin-right: 0 !important;
+        }
+
+        .room-header > * {
+            margin-right: 10px !important;
+        }
+
+        .room-header button {
+            width: 40px;
+            height: 40px;
+            display: none;
+            font-size: 1.5em;
+            padding: 0;
+            display: block;
+            background: white;
+            border: none;
+            font-weight: bolder;
+            line-height: 40px;
+        }
+
+        .room-header button.back {
+            display: none;
+        }
+
+        .room-header .topic {
+            font-size: 0.8em;
+            overflow: hidden;
+            white-space: nowrap;
+            text-overflow: ellipsis;
+        }
+
+        .back::before {
+            content: "☰";
+        }
+
+        .more::before {
+            content: "⋮";
+        }
+
+        .room-header {
+            align-items: center;
+        }
+
+        .room-header .room-description {
+            flex: 1 1 auto;
+            min-width: 0;
+        }
+
+        .room-header h2 {
+            overflow: hidden;
+            white-space: nowrap;
+            text-overflow: ellipsis;
+            margin: 0;
+        }
+
+        .timeline-panel ul {
+            flex: 1;
+            overflow-y: auto;
+            list-style: none;
+            padding: 0;
+            margin: 0;
+        }
+
+        .timeline-panel li {
+            background: darkblue;
+            padding: 10px;
+            margin: 10px;
+        }
+
+        .composer {
+            display: flex;
+        }
+
+        .composer input {
+            display: block;
+            flex: 1;
+            font-size: 1.2em;
+            border: none;
+            padding: 10px;
+        }
+
+        .avatar {
+            --avatar-size: 32px;
+            width: var(--avatar-size);
+            height: var(--avatar-size);
+            border-radius: 100px;
+            overflow: hidden;
+            flex-shrink: 0;
+            line-height: var(--avatar-size);
+            font-size: calc(var(--avatar-size) * 0.6);
+            text-align: center;
+            letter-spacing: calc(var(--avatar-size) * -0.05);
+            background: white;
+            color: black;
+        }
+
+        .avatar.large {
+            --avatar-size: 40px;
+        }
+
+        .avatar img {
+            width: 100%;
+            height: 100%;
+        }
+    </style>
+</head>
+<body>
+    <ul class="log"></ul>
+    <div class="container">
+        <div class="left-panel">
+            <ul>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 1</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 2</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium">R3</div>
+                    <div class="description">
+                        <div class="name">Room 3</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 4</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 5</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 6</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 7</div>
+                        <div class="last-message">Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium">BW</div>
+                    <div class="description">
+                        <div class="name">Room 8</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 9</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 10</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 11</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium">🍔</div>
+                    <div class="description">
+                        <div class="name">Room 12</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 13</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>
+                <li>
+                    <div class="avatar medium"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="description">
+                        <div class="name">Room 14</div>
+                        <div class="last-message">Message 12, message 12, message 12</div>
+                    </div>
+                </li>   
+            </ul>
+        </div>
+        <div class="room-panel-placeholder">
+            <h2>Select a room on the left side</h2>
+        </div>
+        <div class="room-panel">
+            <div class="timeline-panel">
+                <div class="room-header">
+                    <button class="back"></button>
+                    <div class="avatar large"><img src="me.jpg" alt="Avatar for some room"/></div>
+                    <div class="room-description">
+                        <h2>Talk to Bruno</h2>
+                        <div class="topic">The room to talk to Bruno</div>
+                    </div>
+                    <button class="more"></button>
+                </div>
+                <ul>
+                    <li>Message 1, message 1, message 1, message 1, message 1, message 1, message 1, message 1</li>
+                    <li>Message 2, message 2, message 2, message 2, message 2, message 2, message 2, message 2</li>
+                    <li>Message 3, message 3, message 3, message 3, message 3, message 3, message 3, message 3</li>
+                    <li>Message 4, message 4, message 4, message 4, message 4, message 4, message 4, message 4</li>
+                    <li>Message 5, message 5, message 5, message 5, message 5, message 5, message 5, message 5</li>
+                    <li>Message 6, message 6, message 6, message 6, message 6, message 6, message 6, message 6</li>
+                    <li>Message 7, message 7, message 7, message 7, message 7, message 7, message 7, message 7</li>
+                    <li>Message 8, message 8, message 8, message 8, message 8, message 8, message 8, message 8</li>
+                    <li>Message 9, message 9, message 9, message 9, message 9, message 9, message 9, message 9</li>
+                    <li>Message 10, message 10, message 10, message 10, message 10, message 10, message 10, message 10</li>
+                    <li>Message 11, message 11, message 11, message 11, message 11, message 11, message 11, message 11</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                    <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+                </ul>
+                <div class="composer"><input type="text" placeholder="Send a message" /></div>
+            </div>
+            <div class="right-panel">
+                <h2>Bruno</h2>
+                <p>Ban | Kick | Mock</p>
+            </div>
+        </div>
+    </div>
+
+    <script type="text/javascript">
+        const left = document.querySelector(".left-panel");
+        const room = document.querySelector(".room-panel");
+        const timeline = document.querySelector(".timeline-panel ul");
+        const right = document.querySelector(".right-panel");
+        const back = document.querySelector(".back");
+        const more = document.querySelector(".more");
+        const container = document.querySelector(".container");
+        const logNode = document.querySelector(".log");
+        const composer = document.querySelector(".composer input");
+
+        function fullscreen() {
+            try {
+                document.body.webkitRequestFullscreen();
+            } catch (err) {
+                log(`could not set fullscreen: ${err.message}`);
+            }
+        }
+
+        function log(text) {
+            return;
+            const li = document.createElement("li");
+            li.appendChild(document.createTextNode(text));
+            if (logNode.childElementCount) {
+                logNode.insertBefore(li, logNode.firstChild);
+            } else {
+                logNode.appendChild(li);
+            }
+        }
+
+        left.addEventListener("click", (e) => {
+            container.classList.toggle("room-shown");
+            e.preventDefault();
+            e.stopPropagation();
+        });
+
+        timeline.addEventListener("click", (e) => {
+            room.classList.add("right-shown");
+            e.preventDefault();
+            e.stopPropagation();
+        });
+
+        right.addEventListener("click", (e) => {
+            room.classList.remove("right-shown");
+            e.preventDefault();
+            e.stopPropagation();
+        });
+
+        back.addEventListener("click", () => {
+            container.classList.remove("room-shown");
+        });
+
+        more.addEventListener("click", fullscreen);
+
+        const isMyPhone = navigator.userAgent.match(/Windows Phone/i);
+
+        if (isMyPhone) {
+            log("on my phone");
+        } else {
+            log("not on my phone");
+        }
+
+        window.addEventListener("load", () => {
+            fullscreen();
+            setTimeout(() => {
+                log("hello!");
+                fullscreen();
+            }, 100);
+        });
+
+        if (window.applicationCache) {
+            window.applicationCache.oncached = () => {
+                log("app is cached now!");
+            };
+            window.applicationCache.onupdateready = () => {
+                log("app has update ready!");
+            };
+        }
+
+        if (isMyPhone) {
+            composer.addEventListener("blur", () => {
+                log(`composer blurred, clearing maxHeight`);
+                container.style.removeProperty("maxHeight");
+                container.style.maxHeight = "";
+                container.style.maxHeight = null;
+            });
+            composer.addEventListener("focus", () => {
+                const h = 640 - 260;
+                log(`composer focused, setting container height to ${h}`);
+                container.style.maxHeight = `${h}px`;
+            });
+        }
+
+        document.scrollingElement.addEventListener("scroll", () => {
+            log(`window scrolled to ${document.scrollingElement.scrollTop}`);
+        });
+        window.addEventListener("resize", () => {
+            // document.body.style.height = `${window.innerHeight - 100}px`;
+            // document.scrollingElement.style.background = "red";
+            // window.scrollTo(0, 0);
+            log(`window resize ${window.innerHeight}`);
+        });
+    </script>
+</body>
+</html>
diff --git a/prototypes/responsive-layout-grid.html b/prototypes/responsive-layout-grid.html
new file mode 100644
index 00000000..f8ded23f
--- /dev/null
+++ b/prototypes/responsive-layout-grid.html
@@ -0,0 +1,133 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <style type="text/css">
+        body {
+            height: 100vh;
+            margin: 0;
+            display: grid;
+        }
+
+        body              { grid-template-columns: 1fr 0 0; }
+        body.middle-shown { grid-template-columns: 0 1fr 0; }
+        body.right-shown  { grid-template-columns: 0 0 1fr; }
+
+        @media(min-width: 800px) {
+            /*
+            body without middle-shown is used when having middle-panel-placeholder,
+            which shouldn't be the panel shown on < 800px)
+            */
+            body, body.middle-shown { grid-template-columns: 2fr 3fr ; }
+            body.right-shown        { grid-template-columns: 0 3fr 2fr; }
+        }
+
+        @media(min-width: 1024px) {
+            body {
+                grid-template-columns: 1fr 3fr 0;
+            }
+            
+            body.right-shown {
+                grid-template-columns: 1fr 3fr 1fr;
+            }
+        }
+
+        .left-panel {
+            grid-column: 1;
+            background: red;
+            color: white;
+        }
+
+        .middle-panel-placeholder, .middle-panel {
+            color: white;
+            background: blue;
+            grid-column: 2;
+        }
+
+        .middle-panel {
+            display: none;
+        }
+
+        .right-panel {
+            grid-column: 3;
+            background: green;
+            color: white;
+            display: none;
+        }
+    </style>
+</head>
+<body>
+    <div class="left-panel">
+        <ul>
+            <li>Room 1</li>
+            <li>Room 2</li>
+            <li>Room 3</li>
+            <li>Room 4</li>
+            <li>Room 5</li>
+            <li>Room 6</li>
+            <li>Room 7</li>
+            <li>Room 8</li>
+            <li>Room 9</li>
+            <li>Room 10</li>
+            <li>Room 11</li>
+            <li>Room 12</li>
+        </ul>
+    </div>
+    <div class="middle-panel-placeholder">
+        <h2>Select a room on the left side</h2>
+    </div>
+    <div class="middle-panel">
+        <ul>
+            <li>Message 1, message 1, message 1, message 1, message 1, message 1, message 1, message 1</li>
+            <li>Message 2, message 2, message 2, message 2, message 2, message 2, message 2, message 2</li>
+            <li>Message 3, message 3, message 3, message 3, message 3, message 3, message 3, message 3</li>
+            <li>Message 4, message 4, message 4, message 4, message 4, message 4, message 4, message 4</li>
+            <li>Message 5, message 5, message 5, message 5, message 5, message 5, message 5, message 5</li>
+            <li>Message 6, message 6, message 6, message 6, message 6, message 6, message 6, message 6</li>
+            <li>Message 7, message 7, message 7, message 7, message 7, message 7, message 7, message 7</li>
+            <li>Message 8, message 8, message 8, message 8, message 8, message 8, message 8, message 8</li>
+            <li>Message 9, message 9, message 9, message 9, message 9, message 9, message 9, message 9</li>
+            <li>Message 10, message 10, message 10, message 10, message 10, message 10, message 10, message 10</li>
+            <li>Message 11, message 11, message 11, message 11, message 11, message 11, message 11, message 11</li>
+            <li>Message 12, message 12, message 12, message 12, message 12, message 12, message 12, message 12</li>
+        </ul>
+    </div>
+    <div class="right-panel">
+        <h2>Bruno</h2>
+        <p>Ban | Kick | Mock</p>
+    </div>
+
+    <script type="text/javascript">
+        const left = document.querySelector(".left-panel");
+        const middle = document.querySelector(".middle-panel");
+        const middlePlaceholder = document.querySelector(".middle-panel-placeholder");
+        const right = document.querySelector(".right-panel");
+        const container = document.body;
+
+        left.addEventListener("click", () => {
+            // middle or right shown
+            if (container.className) {
+                container.className = "";
+                middle.style.display = "";
+                right.style.display = "";
+                middlePlaceholder.style.display = "block";
+            } else {
+                container.className = "middle-shown";
+                middle.style.display = "block";
+                middlePlaceholder.style.display = "none";
+                right.style.display = "";
+            }
+        });
+
+        middle.addEventListener("click", () => {
+            if (container.className === "right-shown") {
+                container.className = "middle-shown";
+                right.style.display = "";
+            } else {
+                container.className = "right-shown";
+                right.style.display = "block";
+            }
+        });
+    </script>
+</body>
+</html>

From 2dbd0fb9dc1dee2f085478926b00ba08766f25c4 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 19:58:54 +0100
Subject: [PATCH 04/83] move view models over to domain

---
 src/{ui/viewmodels => domain/session}/SessionViewModel.js     | 4 ++--
 src/{ui/viewmodels => domain/session/room}/RoomViewModel.js   | 2 +-
 .../session/roomlist}/RoomTileViewModel.js                    | 0
 src/main.js                                                   | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)
 rename src/{ui/viewmodels => domain/session}/SessionViewModel.js (89%)
 rename src/{ui/viewmodels => domain/session/room}/RoomViewModel.js (94%)
 rename src/{ui/viewmodels => domain/session/roomlist}/RoomTileViewModel.js (100%)

diff --git a/src/ui/viewmodels/SessionViewModel.js b/src/domain/session/SessionViewModel.js
similarity index 89%
rename from src/ui/viewmodels/SessionViewModel.js
rename to src/domain/session/SessionViewModel.js
index 470ee517..03a985ce 100644
--- a/src/ui/viewmodels/SessionViewModel.js
+++ b/src/domain/session/SessionViewModel.js
@@ -1,6 +1,6 @@
 import EventEmitter from "../../EventEmitter.js";
-import RoomTileViewModel from "./RoomTileViewModel.js";
-import RoomViewModel from "./RoomViewModel.js";
+import RoomTileViewModel from "./roomlist/RoomTileViewModel.js";
+import RoomViewModel from "./room/RoomViewModel.js";
 
 export default class SessionViewModel extends EventEmitter {
     constructor(session) {
diff --git a/src/ui/viewmodels/RoomViewModel.js b/src/domain/session/room/RoomViewModel.js
similarity index 94%
rename from src/ui/viewmodels/RoomViewModel.js
rename to src/domain/session/room/RoomViewModel.js
index bc75dea6..cec084b7 100644
--- a/src/ui/viewmodels/RoomViewModel.js
+++ b/src/domain/session/room/RoomViewModel.js
@@ -1,4 +1,4 @@
-import EventEmitter from "../../EventEmitter.js";
+import EventEmitter from "../../../EventEmitter.js";
 
 export default class RoomViewModel extends EventEmitter {
     constructor(room) {
diff --git a/src/ui/viewmodels/RoomTileViewModel.js b/src/domain/session/roomlist/RoomTileViewModel.js
similarity index 100%
rename from src/ui/viewmodels/RoomTileViewModel.js
rename to src/domain/session/roomlist/RoomTileViewModel.js
diff --git a/src/main.js b/src/main.js
index 7c165d73..96d38873 100644
--- a/src/main.js
+++ b/src/main.js
@@ -3,7 +3,7 @@ import Session from "./matrix/session.js";
 import createIdbStorage from "./matrix/storage/idb/create.js";
 import Sync from "./matrix/sync.js";
 import SessionView from "./ui/web/SessionView.js";
-import SessionViewModel from "./ui/viewmodels/SessionViewModel.js";
+import SessionViewModel from "./domain/session/SessionViewModel.js";
 
 const HOST = "localhost";
 const HOMESERVER = `http://${HOST}:8008`;

From 994f1c57d3b058b08fddb673ad051a6fcfcdc6f1 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 20:00:37 +0100
Subject: [PATCH 05/83] store all logout data outside of the session storage

so we could store it in gnome keyring, macOs keychain, ... on non-webclients,
also better separation
---
 src/main.js           | 53 ++++++++++++++++++++++++++-----------------
 src/matrix/session.js | 19 ++++------------
 2 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/src/main.js b/src/main.js
index 96d38873..4f840c4b 100644
--- a/src/main.js
+++ b/src/main.js
@@ -11,50 +11,61 @@ const USERNAME = "bruno1";
 const USER_ID = `@${USERNAME}:${HOST}`;
 const PASSWORD = "testtest";
 
-function getSessionId(userId) {
+function getSessionInfo(userId) {
 	const sessionsJson = localStorage.getItem("morpheus_sessions_v1");
 	if (sessionsJson) {
 		const sessions = JSON.parse(sessionsJson);
 		const session = sessions.find(session => session.userId === userId);
 		if (session) {
-			return session.id;
+			return session;
 		}
 	}
 }
 
+function storeSessionInfo(loginData) {
+    const sessionsJson = localStorage.getItem("morpheus_sessions_v1");
+    const sessions = sessionsJson ? JSON.parse(sessionsJson) : [];
+    const sessionId = (Math.floor(Math.random() * Number.MAX_SAFE_INTEGER)).toString();
+    const sessionInfo = {
+        id: sessionId,
+        deviceId: loginData.device_id,
+        userId: loginData.user_id,
+        homeServer: loginData.home_server,
+        accessToken: loginData.access_token,
+    };
+    sessions.push(sessionInfo);
+    localStorage.setItem("morpheus_sessions_v1", JSON.stringify(sessions));
+    return sessionInfo;
+}
+
 async function login(username, password, homeserver) {
 	const hsApi = new HomeServerApi(homeserver);
 	const loginData = await hsApi.passwordLogin(username, password).response();
-	const sessionsJson = localStorage.getItem("morpheus_sessions_v1");
-	const sessions = sessionsJson ? JSON.parse(sessionsJson) : [];
-	const sessionId = (Math.floor(Math.random() * Number.MAX_SAFE_INTEGER)).toString();
-	console.log(loginData);
-	sessions.push({userId: loginData.user_id, id: sessionId});
-	localStorage.setItem("morpheus_sessions_v1", JSON.stringify(sessions));
-	return {sessionId, loginData};
+	return storeSessionInfo(loginData);
 }
 
 function showSession(container, session) {
     const vm = new SessionViewModel(session);
     const view = new SessionView(vm);
-    container.appendChild(view.mount());
+    view.mount();
+    container.appendChild(view.root());
 }
 
 // eslint-disable-next-line no-unused-vars
 export default async function main(label, button, container) {
 	try {
-		let sessionId = getSessionId(USER_ID);
-		let loginData;
-		if (!sessionId) {
-			({sessionId, loginData} = await login(USERNAME, PASSWORD, HOMESERVER));
+		let sessionInfo = getSessionInfo(USER_ID);
+		if (!sessionInfo) {
+			sessionInfo = await login(USERNAME, PASSWORD, HOMESERVER);
 		}
-		const storage = await createIdbStorage(`morpheus_session_${sessionId}`);
-		const session = new Session(storage);
-		if (loginData) {
-			await session.setLoginData(loginData);
-		}
-		await session.load();
-        const hsApi = new HomeServerApi(HOMESERVER, session.accessToken);
+        const storage = await createIdbStorage(`morpheus_session_${sessionInfo.id}`);
+        const hsApi = new HomeServerApi(HOMESERVER, sessionInfo.accessToken);
+        const session = new Session({storage, hsApi, sessionInfo: {
+            deviceId: sessionInfo.deviceId,
+            userId: sessionInfo.userId,
+            homeServer: sessionInfo.homeServer, //only pass relevant fields to Session
+        }});
+        await session.load();
         console.log("session loaded");
         const needsInitialSync = !session.syncToken;
         if (needsInitialSync) {
diff --git a/src/matrix/session.js b/src/matrix/session.js
index e030a3dd..e80ea6ae 100644
--- a/src/matrix/session.js
+++ b/src/matrix/session.js
@@ -2,21 +2,13 @@ import Room from "./room/room.js";
 import { ObservableMap } from "../observable/index.js";
 
 export default class Session {
-	constructor(storage) {
+	constructor({storage, sessionInfo}) {
 		this._storage = storage;
 		this._session = null;
+        this._sessionInfo = sessionInfo;
 		this._rooms = new ObservableMap();
         this._roomUpdateCallback = (room, params) => this._rooms.update(room.id, params);
 	}
-	// should be called before load
-	// loginData has device_id, user_id, home_server, access_token
-	async setLoginData(loginData) {
-		console.log("session.setLoginData");
-		const txn = await this._storage.readWriteTxn([this._storage.storeNames.session]);
-		const session = {loginData};
-		txn.session.set(session);
-		await txn.complete();
-	}
 
 	async load() {
 		const txn = await this._storage.readTxn([
@@ -28,7 +20,8 @@ export default class Session {
 		// restore session object
 		this._session = await txn.session.get();
 		if (!this._session) {
-			throw new Error("session store is empty");
+            this._session = {};
+			return;
 		}
 		// load rooms
 		const rooms = await txn.roomSummary.getAll();
@@ -58,8 +51,4 @@ export default class Session {
 	get syncToken() {
 		return this._session.syncToken;
 	}
-
-	get accessToken() {
-		return this._session.loginData.access_token;
-	}
 }

From 1757a27475c7f43bd7bad5773a0ca28dd7a28f60 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 20:01:28 +0100
Subject: [PATCH 06/83] consistent naming

---
 src/matrix/session.js | 2 +-
 src/matrix/sync.js    | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/matrix/session.js b/src/matrix/session.js
index e80ea6ae..1f9a3d7e 100644
--- a/src/matrix/session.js
+++ b/src/matrix/session.js
@@ -41,7 +41,7 @@ export default class Session {
 		return room;
 	}
 
-	applySync(syncToken, accountData, txn) {
+	persistSync(syncToken, accountData, txn) {
 		if (syncToken !== this._session.syncToken) {
 			this._session.syncToken = syncToken;
 			txn.session.set(this._session);
diff --git a/src/matrix/sync.js b/src/matrix/sync.js
index aae1fa0a..4ad89466 100644
--- a/src/matrix/sync.js
+++ b/src/matrix/sync.js
@@ -78,7 +78,7 @@ export default class Sync extends EventEmitter {
 		]);
         const roomChanges = [];
         try {
-            this._session.applySync(syncToken, response.account_data, syncTxn);
+            this._session.persistSync(syncToken, response.account_data, syncTxn);
             // to_device
             // presence
 			if (response.rooms) {

From ec925d7c4955f29d903faac9aede25b8ec14af11 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 20:03:18 +0100
Subject: [PATCH 07/83] draft of how to implement filling a timeline gap

---
 src/matrix/room/persister.js |  6 +++---
 src/matrix/room/room.js      |  9 ++++++---
 src/matrix/room/timeline.js  | 24 ++++++++++++++++++++++++
 src/matrix/session.js        | 11 +++++++++--
 4 files changed, 42 insertions(+), 8 deletions(-)

diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js
index c08fbc92..6ed72ece 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/persister.js
@@ -17,9 +17,9 @@ export default class RoomPersister {
 		}
 	}
 
-	// async persistGapFill(...) {
-
-	// }
+	async persistGapFill(gapEntry, response) {
+        throw new Error("not yet implemented");
+	}
 
 	persistSync(roomResponse, txn) {
 		let nextKey = this._lastSortKey;
diff --git a/src/matrix/room/room.js b/src/matrix/room/room.js
index a581fbcb..014ea529 100644
--- a/src/matrix/room/room.js
+++ b/src/matrix/room/room.js
@@ -4,10 +4,11 @@ import RoomPersister from "./persister.js";
 import Timeline from "./timeline.js";
 
 export default class Room extends EventEmitter {
-	constructor(roomId, storage, emitCollectionChange) {
+	constructor({roomId, storage, hsApi, emitCollectionChange}) {
         super();
-		this._roomId = roomId;
-		this._storage = storage;
+        this._roomId = roomId;
+        this._storage = storage;
+        this._hsApi = hsApi;
 		this._summary = new RoomSummary(roomId);
 		this._persister = new RoomPersister(roomId);
         this._emitCollectionChange = emitCollectionChange;
@@ -50,6 +51,8 @@ export default class Room extends EventEmitter {
         this._timeline = new Timeline({
             roomId: this.id,
             storage: this._storage,
+            persister: this._persister,
+            hsApi: this._hsApi,
             closeCallback: () => this._timeline = null,
         });
         await this._timeline.load();
diff --git a/src/matrix/room/timeline.js b/src/matrix/room/timeline.js
index 44f5f83d..27abee37 100644
--- a/src/matrix/room/timeline.js
+++ b/src/matrix/room/timeline.js
@@ -24,6 +24,30 @@ export default class Timeline {
         }
     }
 
+    /** @public */
+    async fillGap(gapEntry, amount) {
+        const gap = gapEntry.gap;
+        let direction;
+        if (gap.prev_batch) {
+            direction = "b";
+        } else if (gap.next_batch) {
+            direction = "f";
+        } else {
+            throw new Error("Invalid gap, no prev_batch or next_batch field: " + JSON.stringify(gapEntry.gap));
+        }
+        const token = gap.prev_batch || gap.next_batch;
+
+        const response = await this._hsApi.messages({
+            roomId: this._roomId,
+            from: token,
+            dir: direction,
+            limit: amount
+        });
+        const newEntries = await this._persister.persistGapFill(gapEntry, response);
+        // find where to replace existing gap with newEntries by doing binary search
+
+    }
+
     /** @public */
     get entries() {
         return this._entriesList;
diff --git a/src/matrix/session.js b/src/matrix/session.js
index 1f9a3d7e..9dbeb010 100644
--- a/src/matrix/session.js
+++ b/src/matrix/session.js
@@ -2,8 +2,10 @@ import Room from "./room/room.js";
 import { ObservableMap } from "../observable/index.js";
 
 export default class Session {
-	constructor({storage, sessionInfo}) {
+    // sessionInfo contains deviceId, userId and homeServer
+	constructor({storage, hsApi, sessionInfo}) {
 		this._storage = storage;
+        this._hsApi = hsApi;
 		this._session = null;
         this._sessionInfo = sessionInfo;
 		this._rooms = new ObservableMap();
@@ -36,7 +38,12 @@ export default class Session {
     }
 
 	createRoom(roomId) {
-		const room = new Room(roomId, this._storage, this._roomUpdateCallback);
+		const room = new Room({
+            roomId,
+            storage: this._storage,
+            emitCollectionChange: this._roomUpdateCallback,
+            hsApi: this._hsApi,
+        });
 		this._rooms.add(roomId, room);
 		return room;
 	}

From 95bef00054ccdf46b940f8a7a6f93b011366c733 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 20:03:47 +0100
Subject: [PATCH 08/83] some comments

---
 src/matrix/hs-api.js | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/matrix/hs-api.js b/src/matrix/hs-api.js
index e54b66f6..07e9babc 100644
--- a/src/matrix/hs-api.js
+++ b/src/matrix/hs-api.js
@@ -21,7 +21,9 @@ class RequestWrapper {
 
 export default class HomeServerApi {
 	constructor(homeserver, accessToken) {
-		this._homeserver = homeserver;
+        // store these both in a closure somehow so it's harder to get at in case of XSS?
+        // one could change the homeserver as well so the token gets sent there, so both must be protected from read/write
+        this._homeserver = homeserver;
 		this._accessToken = accessToken;
 	}
 

From 6940e14b1817d6f39fd46520a69b1b1583cab880 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 20:04:28 +0100
Subject: [PATCH 09/83] move sortedIndex out of observable as other code will
 want to use it too

---
 src/observable/list/SortedMapList.js | 31 +---------------------------
 src/utils/sortedIndex.js             | 26 +++++++++++++++++++++++
 2 files changed, 27 insertions(+), 30 deletions(-)
 create mode 100644 src/utils/sortedIndex.js

diff --git a/src/observable/list/SortedMapList.js b/src/observable/list/SortedMapList.js
index 7f525fff..a3479be0 100644
--- a/src/observable/list/SortedMapList.js
+++ b/src/observable/list/SortedMapList.js
@@ -1,5 +1,5 @@
 import BaseObservableList from "./BaseObservableList.js";
-
+import sortedIndex from "../../utils/sortedIndex";
 
 /*
 
@@ -25,35 +25,6 @@ so key -> Map<key,value> -> value -> node -> *parentNode -> rootNode
 with a node containing {value, leftCount, rightCount, leftNode, rightNode, parentNode}
 */
 
-
-/**
- * @license
- * Based off baseSortedIndex function in Lodash <https://lodash.com/>
- * Copyright JS Foundation and other contributors <https://js.foundation/>
- * Released under MIT license <https://lodash.com/license>
- * Based on Underscore.js 1.8.3 <http://underscorejs.org/LICENSE>
- * Copyright Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
- */
-function sortedIndex(array, value, comparator) {
-    let low = 0;
-    let high = array.length;
-
-    while (low < high) {
-        let mid = (low + high) >>> 1;
-        let cmpResult = comparator(value, array[mid]);
-
-        if (cmpResult > 0) {
-            low = mid + 1;
-        } else if (cmpResult < 0) {
-            high = mid;
-        } else {
-            low = high = mid;
-        }
-    }
-    return high;
-}
-
-
 // does not assume whether or not the values are reference
 // types modified outside of the collection (and affecting sort order) or not
 
diff --git a/src/utils/sortedIndex.js b/src/utils/sortedIndex.js
new file mode 100644
index 00000000..70eaa9b8
--- /dev/null
+++ b/src/utils/sortedIndex.js
@@ -0,0 +1,26 @@
+/**
+ * @license
+ * Based off baseSortedIndex function in Lodash <https://lodash.com/>
+ * Copyright JS Foundation and other contributors <https://js.foundation/>
+ * Released under MIT license <https://lodash.com/license>
+ * Based on Underscore.js 1.8.3 <http://underscorejs.org/LICENSE>
+ * Copyright Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
+ */
+export default function sortedIndex(array, value, comparator) {
+    let low = 0;
+    let high = array.length;
+
+    while (low < high) {
+        let mid = (low + high) >>> 1;
+        let cmpResult = comparator(value, array[mid]);
+
+        if (cmpResult > 0) {
+            low = mid + 1;
+        } else if (cmpResult < 0) {
+            high = mid;
+        } else {
+            low = high = mid;
+        }
+    }
+    return high;
+}

From 1f5d488105085620a0ccf60234735bb430f5ddde Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 20:04:56 +0100
Subject: [PATCH 10/83] draft of timeline tiles support

---
 .../session/room/timeline/TilesCollection.js  | 86 +++++++++++++++++++
 .../room/timeline/TimelineViewModel.js        | 52 +++++++++++
 .../session/room/timeline/tiles/GapTile.js    | 13 +++
 .../session/room/timeline/tiles/SimpleTile.js | 52 +++++++++++
 .../session/room/timeline/tilesCreator.js     | 35 ++++++++
 5 files changed, 238 insertions(+)
 create mode 100644 src/domain/session/room/timeline/TilesCollection.js
 create mode 100644 src/domain/session/room/timeline/TimelineViewModel.js
 create mode 100644 src/domain/session/room/timeline/tiles/GapTile.js
 create mode 100644 src/domain/session/room/timeline/tiles/SimpleTile.js
 create mode 100644 src/domain/session/room/timeline/tilesCreator.js

diff --git a/src/domain/session/room/timeline/TilesCollection.js b/src/domain/session/room/timeline/TilesCollection.js
new file mode 100644
index 00000000..f4ef2236
--- /dev/null
+++ b/src/domain/session/room/timeline/TilesCollection.js
@@ -0,0 +1,86 @@
+import BaseObservableList from "../../../../observable/list/BaseObservableList.js";
+
+// maps 1..n entries to 0..1 tile. Entries are what is stored in the timeline, either an event or gap
+export default class TilesCollection extends BaseObservableList {
+    constructor(entries, tileCreator) {
+        super();
+        this._entries = entries;
+        this._tiles = null;
+        this._entrySubscription = null;
+        this._tileCreator = tileCreator;
+    }
+
+    onSubscribeFirst() {
+        this._entrySubscription = this._entries.subscribe(this);
+        this._populateTiles();
+    }
+
+    _populateTiles() {
+        this._tiles = [];
+        let currentTile = null;
+        for (let entry of this._entries) {
+            if (!currentTile || !currentTile.tryIncludeEntry(entry)) {
+                currentTile = this._tileCreator(entry);
+                this._tiles.push(currentTile);
+            }
+        }
+        let prevTile = null;
+        for (let tile of this._tiles) {
+            if (prevTile) {
+                prevTile.updateNextSibling(tile);
+            }
+            tile.updatePreviousSibling(prevTile);
+            prevTile = tile;
+        }
+        if (prevTile) {
+            prevTile.updateNextSibling(null);
+        }
+    }
+
+    _findTileIndex(sortKey) {
+        return sortedIndex(this._tiles, sortKey, (key, tile) => {
+            return tile.compareSortKey(key);
+        });
+    }
+
+    onUnsubscribeLast() {
+        this._entrySubscription = this._entrySubscription();
+        this._tiles = null;
+    }
+
+    onReset() {
+        // if TileViewModel were disposable, dispose here, or is that for views to do? views I suppose ...
+        this._buildInitialTiles();
+        this.emitReset();
+    }
+
+    onAdd(index, value) {
+        // find position by sort key
+        // ask siblings to be included? both? yes, twice: a (insert c here) b, ask a(c), if yes ask b(a), else ask b(c)? if yes then b(a)?
+    }
+
+    onUpdate(index, value, params) {
+        // outcomes here can be
+        //   tiles should be removed (got redacted and we don't want it in the timeline)
+        //   tile should be added where there was none before ... ?
+        //   entry should get it's own tile now
+        //   merge with neighbours? ... hard to imagine for this use case ...
+
+        // also emit update for tile
+    }
+
+    onRemove(index, value) {
+        // find tile, if any
+        // remove entry from tile
+        // emit update or remove (if empty now) on tile
+    }
+
+    onMove(fromIdx, toIdx, value) {
+        // this ... cannot happen in the timeline?
+        // should be sorted by sortKey and sortKey is immutable
+    }
+
+    [Symbol.iterator]() {
+        return this._tiles.values();
+    }
+}
diff --git a/src/domain/session/room/timeline/TimelineViewModel.js b/src/domain/session/room/timeline/TimelineViewModel.js
new file mode 100644
index 00000000..5c179673
--- /dev/null
+++ b/src/domain/session/room/timeline/TimelineViewModel.js
@@ -0,0 +1,52 @@
+/*
+need better naming, but
+entry = event or gap from matrix layer
+tile = item on visual timeline like event, date separator?, group of joined events
+
+
+shall we put date separators as marker in EventViewItem or separate item? binary search will be complicated ...
+
+
+pagination ...
+
+on the timeline viewmodel (containing the TilesCollection?) we'll have a method to (un)load a tail or head of
+the timeline (counted in tiles), which results to a range in sortKeys we want on the screen. We pass that range
+to the room timeline, which unload entries from memory.
+when loading, it just reads events from a sortkey backwards or forwards...
+*/
+import TilesCollection from "./TilesCollection.js";
+import tilesCreator from "./tilesCreator.js";
+
+export default class TimelineViewModel {
+    constructor(timeline) {
+        this._timeline = timeline;
+        // once we support sending messages we could do
+        // timeline.entries.concat(timeline.pendingEvents)
+        // for an ObservableList that also contains local echos
+        this._tiles = new TilesCollection(timeline.entries, tilesCreator({timeline}));
+    }
+
+    // doesn't fill gaps, only loads stored entries/tiles
+    loadAtTop() {
+        // load 100 entries, which may result in 0..100 tiles
+        return this._timeline.loadAtTop(100);
+    }
+
+    unloadAtTop(tileAmount) {
+        // get lowerSortKey for tile at index tileAmount - 1
+        // tell timeline to unload till there (included given key)
+    }
+
+    loadAtBottom() {
+
+    }
+
+    unloadAtBottom(tileAmount) {
+        // get upperSortKey for tile at index tiles.length - tileAmount
+        // tell timeline to unload till there (included given key)
+    }
+
+    get tiles() {
+        return this._tiles;
+    }
+}
diff --git a/src/domain/session/room/timeline/tiles/GapTile.js b/src/domain/session/room/timeline/tiles/GapTile.js
new file mode 100644
index 00000000..95e59a27
--- /dev/null
+++ b/src/domain/session/room/timeline/tiles/GapTile.js
@@ -0,0 +1,13 @@
+import SimpleTile from "./SimpleTile";
+
+export default class GapTile extends SimpleTile {
+    constructor(entry, timeline) {
+        super(entry);
+        this._timeline = timeline;
+    }
+
+    // GapTile specific behaviour
+    fill() {
+        return this._timeline.fillGap(this._entry, 10);
+    }
+}
diff --git a/src/domain/session/room/timeline/tiles/SimpleTile.js b/src/domain/session/room/timeline/tiles/SimpleTile.js
new file mode 100644
index 00000000..018bacbd
--- /dev/null
+++ b/src/domain/session/room/timeline/tiles/SimpleTile.js
@@ -0,0 +1,52 @@
+export default class SimpleTile {
+    constructor(entry) {
+        this._entry = entry;
+    }
+    // view model props for all subclasses
+    // hmmm, could also do instanceof ... ?
+    get shape() {
+        // "gap" | "message" | "image" | ... ?
+    }
+
+    // don't show display name / avatar
+    // probably only for MessageTiles of some sort?
+    get isContinuation() {
+        return false;
+    }
+
+    get hasDateSeparator() {
+        return false;
+    }
+
+    get upperSortKey() {
+        return this._entry.sortKey;
+    }
+
+    get lowerSortKey() {
+        return this._entry.sortKey;
+    }
+
+    // TilesCollection contract
+    compareSortKey(key) {
+        return this._entry.sortKey.compare(key);
+    }
+
+    // update received for already included (falls within sort keys) entry
+    updateEntry(entry) {
+
+    }
+
+    // simple entry can only contain 1 entry
+    tryIncludeEntry() {
+        return false;
+    }
+    // let item know it has a new sibling
+    updatePreviousSibling(prev) {
+
+    }
+
+    // let item know it has a new sibling
+    updateNextSibling(next) {
+    
+    }
+}
diff --git a/src/domain/session/room/timeline/tilesCreator.js b/src/domain/session/room/timeline/tilesCreator.js
new file mode 100644
index 00000000..74375f1c
--- /dev/null
+++ b/src/domain/session/room/timeline/tilesCreator.js
@@ -0,0 +1,35 @@
+import GapTile from "./tiles/GapTile.js";
+import TextTile from "./tiles/TextTile.js";
+import ImageTile from "./tiles/ImageTile.js";
+import RoomNameTile from "./tiles/RoomNameTile.js";
+import RoomMemberTile from "./tiles/RoomMemberTile.js";
+
+export default function ({timeline}) {
+    return function tilesCreator(entry) {
+        if (entry.gap) {
+            return new GapTile(entry, timeline);
+        } else if (entry.event) {
+            const event = entry.event;
+            switch (event.type) {
+                case "m.room.message": {
+                    const content = event.content;
+                    const msgtype = content && content.msgtype;
+                    switch (msgtype) {
+                        case "m.text":
+                            return new TextTile(entry);
+                        case "m.image":
+                            return new ImageTile(entry);
+                        default:
+                            return null;    // unknown tile types are not rendered?
+                    }
+                }
+                case "m.room.name":
+                    return new RoomNameTile(entry);
+                case "m.room.member":
+                    return new RoomMemberTile(entry);
+                default:
+                    return null;
+            }
+        }
+    }   
+}

From d6e357cc225642409b02e598eaef433f04d82cb0 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 8 Mar 2019 20:05:21 +0100
Subject: [PATCH 11/83] update TODO

---
 doc/TODO.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/doc/TODO.md b/doc/TODO.md
index abf03663..48be2e5a 100644
--- a/doc/TODO.md
+++ b/doc/TODO.md
@@ -32,6 +32,8 @@
     - find out if `(this._emitCollectionUpdate)(this)` is different than `this._emitCollectionUpdate(this)` 
     - got "database tried to mutate when not allowed" or something error as well
     - find out why when RoomPersister.(\_createGapEntry/\_createEventEntry) we remove .buffer the transaction fails (good), but upon fixing and refreshing is missing a message! syncToken should not be saved, so why isn't this again in the sync response and now the txn does succeed?
+ - DONE: take access token out of IDB? this way it can be stored in a more secure thing for non-web clients, together wit encryption key for olm sessions ... ? like macos keychain, gnome keyring, ... maybe using https://www.npmjs.com/package/keytar
+ - experiment with using just a normal array with 2 numbers for sortkeys, to work in Edge as well.
  - send messages
  - fill gaps with call to /messages
  - create sync filter

From 153d54a285ab5c605e2834c2a8432bb0331b0121 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 9 Mar 2019 00:40:03 +0100
Subject: [PATCH 12/83] work on tile view models

---
 .../session/room/timeline/tiles/GapTile.js    | 39 ++++++++++++++++---
 .../session/room/timeline/tiles/ImageTile.js  | 22 +++++++++++
 .../room/timeline/tiles/LocationTile.js       | 20 ++++++++++
 .../room/timeline/tiles/MessageTile.js        | 26 +++++++++++++
 .../room/timeline/tiles/RoomMemberTile.js     |  9 +++++
 .../room/timeline/tiles/RoomNameTile.js       |  9 +++++
 .../session/room/timeline/tiles/SimpleTile.js | 14 +++++--
 .../session/room/timeline/tiles/TextTile.js   |  8 ++++
 .../session/room/timeline/tilesCreator.js     | 21 ++++++----
 9 files changed, 153 insertions(+), 15 deletions(-)
 create mode 100644 src/domain/session/room/timeline/tiles/ImageTile.js
 create mode 100644 src/domain/session/room/timeline/tiles/LocationTile.js
 create mode 100644 src/domain/session/room/timeline/tiles/MessageTile.js
 create mode 100644 src/domain/session/room/timeline/tiles/RoomMemberTile.js
 create mode 100644 src/domain/session/room/timeline/tiles/RoomNameTile.js
 create mode 100644 src/domain/session/room/timeline/tiles/TextTile.js

diff --git a/src/domain/session/room/timeline/tiles/GapTile.js b/src/domain/session/room/timeline/tiles/GapTile.js
index 95e59a27..fe0cb6fe 100644
--- a/src/domain/session/room/timeline/tiles/GapTile.js
+++ b/src/domain/session/room/timeline/tiles/GapTile.js
@@ -1,13 +1,42 @@
 import SimpleTile from "./SimpleTile";
 
 export default class GapTile extends SimpleTile {
-    constructor(entry, timeline) {
-        super(entry);
+    constructor(options, timeline) {
+        super(options);
         this._timeline = timeline;
+        this._loading = false;
+        this._error = null;
     }
 
-    // GapTile specific behaviour
-    fill() {
-        return this._timeline.fillGap(this._entry, 10);
+    async fill() {
+        // prevent doing this twice
+        if (!this._loading) {
+            this._loading = true;
+            this._emitUpdate("isLoading");
+            try {
+                return await this._timeline.fillGap(this._entry, 10);
+            } catch (err) {
+                this._loading = false;
+                this._error = err;
+                this._emitUpdate("isLoading");
+                this._emitUpdate("error");
+            }
+        }
+    }
+
+    get isLoading() {
+        return this._loading;
+    }
+
+    get direction() {
+        return this._entry.prev_batch ? "backward" : "forward";
+    }
+
+    get error() {
+        if (this._error) {
+            const dir = this._entry.prev_batch ? "previous" : "next";
+            return `Could not load ${dir} messages: ${this._error.message}`;
+        }
+        return null;
     }
 }
diff --git a/src/domain/session/room/timeline/tiles/ImageTile.js b/src/domain/session/room/timeline/tiles/ImageTile.js
new file mode 100644
index 00000000..8c18491e
--- /dev/null
+++ b/src/domain/session/room/timeline/tiles/ImageTile.js
@@ -0,0 +1,22 @@
+import MessageTile from "./MessageTile.js";
+
+export default class ImageTile extends MessageTile {
+    constructor(options) {
+        super(options);
+
+        // we start loading the image here,
+        // and call this._emitUpdate once it's loaded?
+        // or maybe we have an becameVisible() callback on tiles where we start loading it?
+    }
+    get src() {
+        return "";
+    }
+
+    get width() {
+        return 200;
+    }
+
+    get height() {
+        return 200;
+    }
+}
diff --git a/src/domain/session/room/timeline/tiles/LocationTile.js b/src/domain/session/room/timeline/tiles/LocationTile.js
new file mode 100644
index 00000000..69dbc629
--- /dev/null
+++ b/src/domain/session/room/timeline/tiles/LocationTile.js
@@ -0,0 +1,20 @@
+import MessageTile from "./MessageTile.js";
+
+/*
+map urls:
+apple:   https://developer.apple.com/library/archive/featuredarticles/iPhoneURLScheme_Reference/MapLinks/MapLinks.html
+android: https://developers.google.com/maps/documentation/urls/guide
+wp:      maps:49.275267 -122.988617
+https://www.habaneroconsulting.com/stories/insights/2011/opening-native-map-apps-from-the-mobile-browser
+*/
+export default class LocationTile extends MessageTile {
+    get mapsLink() {
+        const geoUri = this._getContent().geo_uri;
+        const [lat, long] = geoUri.split(":")[1].split(",");
+        return `maps:${lat} ${long}`;
+    }
+
+    get label() {
+        return `${this.sender} sent their location, click to see it in maps.`;
+    }
+}
diff --git a/src/domain/session/room/timeline/tiles/MessageTile.js b/src/domain/session/room/timeline/tiles/MessageTile.js
new file mode 100644
index 00000000..f57f8379
--- /dev/null
+++ b/src/domain/session/room/timeline/tiles/MessageTile.js
@@ -0,0 +1,26 @@
+import SimpleTile from "./SimpleTile.js";
+
+export default class MessageTile extends SimpleTile {
+
+    constructor(options) {
+        super(options);
+        this._date = new Date(this._entry.event.origin_server_ts);
+    }
+
+    get sender() {
+        return this._entry.event.sender;
+    }
+
+    get date() {
+        return this._date.toLocaleDateString();
+    }
+
+    get time() {
+        return this._date.toLocaleTimeString();
+    }
+
+    _getContent() {
+        const event = this._entry.event;
+        return event && event.content;
+    }
+}
diff --git a/src/domain/session/room/timeline/tiles/RoomMemberTile.js b/src/domain/session/room/timeline/tiles/RoomMemberTile.js
new file mode 100644
index 00000000..df31945c
--- /dev/null
+++ b/src/domain/session/room/timeline/tiles/RoomMemberTile.js
@@ -0,0 +1,9 @@
+import SimpleTile from "./SimpleTile.js";
+
+export default class RoomNameTile extends SimpleTile {
+    get label() {
+        const event = this._entry.event;
+        const content = event.content;
+        return `${event.sender} changed membership to ${content.membership}`;
+    }
+}
diff --git a/src/domain/session/room/timeline/tiles/RoomNameTile.js b/src/domain/session/room/timeline/tiles/RoomNameTile.js
new file mode 100644
index 00000000..e352b168
--- /dev/null
+++ b/src/domain/session/room/timeline/tiles/RoomNameTile.js
@@ -0,0 +1,9 @@
+import SimpleTile from "./SimpleTile.js";
+
+export default class RoomNameTile extends SimpleTile {
+    get label() {
+        const event = this._entry.event;
+        const content = event.content;
+        return `${event.sender} changed the room name to "${content.name}"`
+    }
+}
diff --git a/src/domain/session/room/timeline/tiles/SimpleTile.js b/src/domain/session/room/timeline/tiles/SimpleTile.js
index 018bacbd..bacb8259 100644
--- a/src/domain/session/room/timeline/tiles/SimpleTile.js
+++ b/src/domain/session/room/timeline/tiles/SimpleTile.js
@@ -1,6 +1,7 @@
 export default class SimpleTile {
-    constructor(entry) {
+    constructor({entry, emitUpdate}) {
         this._entry = entry;
+        this._emitUpdate = emitUpdate;
     }
     // view model props for all subclasses
     // hmmm, could also do instanceof ... ?
@@ -33,10 +34,17 @@ export default class SimpleTile {
 
     // update received for already included (falls within sort keys) entry
     updateEntry(entry) {
-
+        // return names of props updated, or true for all, or null for no changes caused
+        return true;
     }
 
-    // simple entry can only contain 1 entry
+    // return whether the tile should be removed
+    // as SimpleTile only has one entry, the tile should be removed
+    removeEntry(entry) {
+        return true;
+    }
+
+    // SimpleTile can only contain 1 entry
     tryIncludeEntry() {
         return false;
     }
diff --git a/src/domain/session/room/timeline/tiles/TextTile.js b/src/domain/session/room/timeline/tiles/TextTile.js
new file mode 100644
index 00000000..411b1a55
--- /dev/null
+++ b/src/domain/session/room/timeline/tiles/TextTile.js
@@ -0,0 +1,8 @@
+import MessageTile from "./MessageTile.js";
+
+export default class TextTile extends MessageTile {
+    get text() {
+        const content = this._getContent();
+        return content && content.body;
+    }
+}
diff --git a/src/domain/session/room/timeline/tilesCreator.js b/src/domain/session/room/timeline/tilesCreator.js
index 74375f1c..e70088b9 100644
--- a/src/domain/session/room/timeline/tilesCreator.js
+++ b/src/domain/session/room/timeline/tilesCreator.js
@@ -1,13 +1,15 @@
 import GapTile from "./tiles/GapTile.js";
 import TextTile from "./tiles/TextTile.js";
 import ImageTile from "./tiles/ImageTile.js";
+import LocationTile from "./tiles/LocationTile.js";
 import RoomNameTile from "./tiles/RoomNameTile.js";
 import RoomMemberTile from "./tiles/RoomMemberTile.js";
 
-export default function ({timeline}) {
+export default function ({timeline, emitUpdate}) {
     return function tilesCreator(entry) {
+        const options = {entry, emitUpdate};
         if (entry.gap) {
-            return new GapTile(entry, timeline);
+            return new GapTile(options, timeline);
         } else if (entry.event) {
             const event = entry.event;
             switch (event.type) {
@@ -16,18 +18,23 @@ export default function ({timeline}) {
                     const msgtype = content && content.msgtype;
                     switch (msgtype) {
                         case "m.text":
-                            return new TextTile(entry);
+                        case "m.notice":
+                            return new TextTile(options);
                         case "m.image":
-                            return new ImageTile(entry);
+                            return new ImageTile(options);
+                        case "m.location":
+                            return new LocationTile(options);
                         default:
-                            return null;    // unknown tile types are not rendered?
+                            // unknown msgtype not rendered
+                            return null;
                     }
                 }
                 case "m.room.name":
-                    return new RoomNameTile(entry);
+                    return new RoomNameTile(options);
                 case "m.room.member":
-                    return new RoomMemberTile(entry);
+                    return new RoomMemberTile(options);
                 default:
+                    // unknown type not rendered
                     return null;
             }
         }

From c8749a1a06028856c4f93640f371663085d0c156 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 9 Mar 2019 00:40:17 +0100
Subject: [PATCH 13/83] rought impl of tiles collection

---
 .../session/room/timeline/TilesCollection.js  | 86 ++++++++++++++++---
 1 file changed, 75 insertions(+), 11 deletions(-)

diff --git a/src/domain/session/room/timeline/TilesCollection.js b/src/domain/session/room/timeline/TilesCollection.js
index f4ef2236..2b15bb97 100644
--- a/src/domain/session/room/timeline/TilesCollection.js
+++ b/src/domain/session/room/timeline/TilesCollection.js
@@ -1,4 +1,5 @@
 import BaseObservableList from "../../../../observable/list/BaseObservableList.js";
+import sortedIndex from "../../../../utils/sortedIndex.js";
 
 // maps 1..n entries to 0..1 tile. Entries are what is stored in the timeline, either an event or gap
 export default class TilesCollection extends BaseObservableList {
@@ -37,12 +38,27 @@ export default class TilesCollection extends BaseObservableList {
         }
     }
 
-    _findTileIndex(sortKey) {
+    _findTileIdx(sortKey) {
         return sortedIndex(this._tiles, sortKey, (key, tile) => {
-            return tile.compareSortKey(key);
+            // negate result because we're switching the order of the params
+            return -tile.compareSortKey(key);
         });
     }
 
+    _findTileAtIdx(sortKey, idx) {
+        const tile = this._getTileAtIdx(idx);
+        if (tile && tile.compareSortKey(sortKey) === 0) {
+            return tile;
+        }
+    }
+
+    _getTileAtIdx(tileIdx) {
+        if (tileIdx >= 0 && tileIdx < this._tiles.length) {
+            return this._tiles[tileIdx];
+        }
+        return null;
+    }
+
     onUnsubscribeLast() {
         this._entrySubscription = this._entrySubscription();
         this._tiles = null;
@@ -54,25 +70,73 @@ export default class TilesCollection extends BaseObservableList {
         this.emitReset();
     }
 
-    onAdd(index, value) {
+    onAdd(index, entry) {
+        const {sortKey} = entry;
+        const tileIdx = this._findTileIdx(sortKey);
+        const prevTile = this._getTileAtIdx(tileIdx - 1);
+        if (prevTile && prevTile.tryIncludeEntry(entry)) {
+            this.emitUpdate(tileIdx - 1, prevTile);
+            return;
+        }
+        // not + 1 because this entry hasn't been added yet
+        const nextTile = this._getTileAtIdx(tileIdx);
+        if (nextTile && nextTile.tryIncludeEntry(entry)) {
+            this.emitUpdate(tileIdx, nextTile);
+            return;
+        }
+
+        const newTile = this._tileCreator(entry);
+        if (newTile) {
+            prevTile.updateNextSibling(newTile);
+            nextTile.updatePreviousSibling(newTile);
+            this._tiles.splice(tileIdx, 0, newTile);
+            this.emitAdd(tileIdx, newTile);
+        }
         // find position by sort key
         // ask siblings to be included? both? yes, twice: a (insert c here) b, ask a(c), if yes ask b(a), else ask b(c)? if yes then b(a)?
     }
 
-    onUpdate(index, value, params) {
+    onUpdate(index, entry, params) {
+        const {sortKey} = entry;
+        const tileIdx = this._findTileIdx(sortKey);
+        const tile = this._findTileAtIdx(sortKey, tileIdx);
+        if (tile) {
+            const newParams = tile.updateEntry(entry, params);
+            if (newParams) {
+                this.emitUpdate(tileIdx, tile, newParams);
+            }
+        }
+        // technically we should handle adding a tile here as well
+        // in case before we didn't have a tile for it but now we do
+        // but in reality we don't have this use case as the type and msgtype
+        // doesn't change. Decryption maybe is the exception?
+
+
         // outcomes here can be
         //   tiles should be removed (got redacted and we don't want it in the timeline)
         //   tile should be added where there was none before ... ?
         //   entry should get it's own tile now
-        //   merge with neighbours? ... hard to imagine for this use case ...
-
-        // also emit update for tile
+        //   merge with neighbours? ... hard to imagine use case for this  ...
     }
 
-    onRemove(index, value) {
-        // find tile, if any
-        // remove entry from tile
-        // emit update or remove (if empty now) on tile
+    // would also be called when unloading a part of the timeline
+    onRemove(index, entry) {
+        const {sortKey} = entry;
+        const tileIdx = this._findTileIdx(sortKey);
+        const tile = this._findTileAtIdx(sortKey, tileIdx);
+        if (tile) {
+            const removeTile = tile.removeEntry(entry);
+            if (removeTile) {
+                const prevTile = this._getTileAtIdx(tileIdx - 1);
+                const nextTile = this._getTileAtIdx(tileIdx + 1);
+                this._tiles.splice(tileIdx, 1);
+                prevTile && prevTile.updateNextSibling(nextTile);
+                nextTile && nextTile.updatePreviousSibling(prevTile);
+                this.emitRemove(tileIdx, tile);
+            } else {
+                this.emitUpdate(tileIdx, tile);
+            }
+        }
     }
 
     onMove(fromIdx, toIdx, value) {

From 8e8e22fe16454f765bc1f9caf236df68d539525f Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 9 Mar 2019 00:41:06 +0100
Subject: [PATCH 14/83] work on filling gaps

---
 src/matrix/hs-api.js                          |   5 +
 src/matrix/room/persister.js                  | 106 +++++++++++++++++-
 src/matrix/room/room.js                       |   2 +-
 src/matrix/room/timeline.js                   |  22 +++-
 .../storage/idb/stores/RoomTimelineStore.js   |  17 ++-
 src/observable/list/ObservableArray.js        |  22 ++++
 6 files changed, 164 insertions(+), 10 deletions(-)

diff --git a/src/matrix/hs-api.js b/src/matrix/hs-api.js
index 07e9babc..66eed71b 100644
--- a/src/matrix/hs-api.js
+++ b/src/matrix/hs-api.js
@@ -93,6 +93,11 @@ export default class HomeServerApi {
 		return this._get("/sync", {since, timeout, filter});
 	}
 
+    // params is from, dir and optionally to, limit, filter.
+    messages(roomId, params) {
+        return this._get(`/rooms/${roomId}/messages`, params);
+    }
+
 	passwordLogin(username, password) {
         return this._post("/login", undefined, {
           "type": "m.login.password",
diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js
index 6ed72ece..a2b1683b 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/persister.js
@@ -1,8 +1,17 @@
 import SortKey from "../storage/sortkey.js";
 
+function gapEntriesAreEqual(a, b) {
+    if (!a || !b || !a.gap || !b.gap) {
+        return false;
+    }
+    const gapA = a.gap, gapB = b.gap;
+    return gapA.prev_batch === gapB.prev_batch && gapA.next_batch === gapB.next_batch;
+}
+
 export default class RoomPersister {
-	constructor(roomId) {
+	constructor({roomId, storage}) {
 		this._roomId = roomId;
+        this._storage = storage;
 		this._lastSortKey = new SortKey();
 	}
 
@@ -18,7 +27,85 @@ export default class RoomPersister {
 	}
 
 	async persistGapFill(gapEntry, response) {
-        throw new Error("not yet implemented");
+        const {chunk, start, end} = response;
+        if (!Array.isArray(chunk)) {
+            throw new Error("Invalid chunk in response");
+        }
+        if (typeof start !== "string" || typeof end !== "string") {
+            throw new Error("Invalid start or end token in response");
+        }
+        const gapKey = gapEntry.sortKey;
+        const txn = await this._storage.readWriteTxn([this._storage.storeNames.roomTimeline]);
+        try {
+            const roomTimeline = txn.roomTimeline;
+            // make sure what we've been given is actually persisted
+            // in the timeline, otherwise we're replacing something
+            // that doesn't exist (maybe it has been replaced already, or ...)
+            const persistedEntry = await roomTimeline.findEntry(this._roomId, gapKey);
+            if (!gapEntriesAreEqual(gapEntry, persistedEntry)) {
+                throw new Error("Gap is not present in the timeline");
+            }
+            // find the previous event before the gap we could blend with
+            const backwards = !!gapEntry.prev_batch;
+            let neighbourEventEntry;
+            if (backwards) {
+                neighbourEventEntry = await roomTimeline.previousEventFromGap(this._roomId, gapKey);
+            } else {
+                neighbourEventEntry = await roomTimeline.nextEventFromGap(this._roomId, gapKey);
+            }
+            const neighbourEvent = neighbourEventEntry && neighbourEventEntry.event;
+
+            const newEntries = [];
+            let sortKey = gapKey;
+            let eventFound = false;
+            if (backwards) {
+                for (let i = chunk.length - 1; i >= 0; i--) {
+                    const event = chunk[i];
+                    if (event.id === neighbourEvent.id) {
+                        eventFound = true;
+                        break;
+                    }
+                    newEntries.splice(0, 0, this._createEventEntry(sortKey, event));
+                    sortKey = sortKey.previousKey();
+                }
+                if (!eventFound) {
+                    newEntries.splice(0, 0, this._createBackwardGapEntry(sortKey, end));
+                }
+            } else {
+                for (let i = 0; i < chunk.length; i++) {
+                    const event = chunk[i];
+                    if (event.id === neighbourEvent.id) {
+                        eventFound = true;
+                        break;
+                    }
+                    newEntries.push(this._createEventEntry(sortKey, event));
+                    sortKey = sortKey.nextKey();
+                }
+                if (!eventFound) {
+                    // need to check start is correct here
+                    newEntries.push(this._createForwardGapEntry(sortKey, start));
+                }
+            }
+
+            if (eventFound) {
+                // remove gap on the other side as well,
+                // or while we're at it, remove all gaps between gapKey and neighbourEventEntry.sortKey
+            } else {
+                roomTimeline.deleteEntry(this._roomId, gapKey);
+            }
+
+            for (let entry of newEntries) {
+                roomTimeline.add(entry);
+            }
+        } catch (err) {
+            txn.abort();
+            throw err;
+        }
+
+        await txn.complete();
+
+        // somehow also return all the gaps we removed so the timeline can do the same
+        return {newEntries};
 	}
 
 	persistSync(roomResponse, txn) {
@@ -29,7 +116,7 @@ export default class RoomPersister {
 		// I suppose it will, yes
 		if (timeline.limited) {
 			nextKey = nextKey.nextKeyWithGap();
-            entries.push(this._createGapEntry(nextKey, timeline.prev_batch));
+            entries.push(this._createBackwardGapEntry(nextKey, timeline.prev_batch));
         }
         // const startOfChunkSortKey = nextKey;
         if (timeline.events) {
@@ -40,7 +127,7 @@ export default class RoomPersister {
 		}
         // write to store
         for(const entry of entries) {
-            txn.roomTimeline.append(entry);
+            txn.roomTimeline.add(entry);
         }
 		// right thing to do? if the txn fails, not sure we'll continue anyways ...
 		// only advance the key once the transaction has
@@ -68,7 +155,7 @@ export default class RoomPersister {
         return entries;
 	}
 
-    _createGapEntry(sortKey, prevBatch) {
+    _createBackwardGapEntry(sortKey, prevBatch) {
         return {
             roomId: this._roomId,
             sortKey: sortKey.buffer,
@@ -77,6 +164,15 @@ export default class RoomPersister {
         };
     }
 
+    _createForwardGapEntry(sortKey, nextBatch) {
+        return {
+            roomId: this._roomId,
+            sortKey: sortKey.buffer,
+            event: null,
+            gap: {next_batch: nextBatch}
+        };
+    }
+
     _createEventEntry(sortKey, event) {
         return {
             roomId: this._roomId,
diff --git a/src/matrix/room/room.js b/src/matrix/room/room.js
index 014ea529..05560cb5 100644
--- a/src/matrix/room/room.js
+++ b/src/matrix/room/room.js
@@ -10,7 +10,7 @@ export default class Room extends EventEmitter {
         this._storage = storage;
         this._hsApi = hsApi;
 		this._summary = new RoomSummary(roomId);
-		this._persister = new RoomPersister(roomId);
+		this._persister = new RoomPersister({roomId, storage});
         this._emitCollectionChange = emitCollectionChange;
         this._timeline = null;
 	}
diff --git a/src/matrix/room/timeline.js b/src/matrix/room/timeline.js
index 27abee37..b4b61fe4 100644
--- a/src/matrix/room/timeline.js
+++ b/src/matrix/room/timeline.js
@@ -1,4 +1,5 @@
 import { ObservableArray } from "../../observable/index.js";
+import sortedIndex from "../../utils/sortedIndex.js";
 
 export default class Timeline {
     constructor({roomId, storage, closeCallback}) {
@@ -37,15 +38,32 @@ export default class Timeline {
         }
         const token = gap.prev_batch || gap.next_batch;
 
-        const response = await this._hsApi.messages({
-            roomId: this._roomId,
+        const response = await this._hsApi.messages(this._roomId, {
             from: token,
             dir: direction,
             limit: amount
         });
+
         const newEntries = await this._persister.persistGapFill(gapEntry, response);
         // find where to replace existing gap with newEntries by doing binary search
+        const gapIdx = sortedIndex(this._entriesList.array, gapEntry.sortKey, (key, entry) => {
+            return key.compare(entry.sortKey);
+        });
+        // only replace the gap if it's currently in the timeline
+        if (this._entriesList.at(gapIdx) === gapEntry) {
+            this._entriesList.removeAt(gapIdx);
+            this._entriesList.insertMany(gapIdx, newEntries);
+        }
+    }
 
+    async loadAtTop(amount) {
+        const firstEntry = this._entriesList.at(0);
+        if (firstEntry) {
+            const txn = await this._storage.readTxn([this._storage.storeNames.roomTimeline]);
+            const topEntries = await txn.roomTimeline.eventsBefore(this._roomId, firstEntry.sortKey, amount);
+            this._entriesList.insertMany(0, topEntries);
+            return topEntries.length;
+        }
     }
 
     /** @public */
diff --git a/src/matrix/storage/idb/stores/RoomTimelineStore.js b/src/matrix/storage/idb/stores/RoomTimelineStore.js
index cd5ad4e5..f02d2a86 100644
--- a/src/matrix/storage/idb/stores/RoomTimelineStore.js
+++ b/src/matrix/storage/idb/stores/RoomTimelineStore.js
@@ -19,14 +19,27 @@ export default class RoomTimelineStore {
 	}
 
     async eventsBefore(roomId, sortKey, amount) {
-        const range = IDBKeyRange.bound([roomId, SortKey.minKey.buffer], [roomId, sortKey.buffer], false, true);
+        const range = IDBKeyRange.only([roomId, sortKey.buffer]);
         const events = await this._timelineStore.selectLimitReverse(range, amount);
         events.reverse(); // because we fetched them backwards
         return events;
     }
 
+    nextEventFromGap(roomId, sortKey) {
+        
+    }
+
+    previousEventFromGap(roomId, sortKey) {
+        
+    }
+
+    findEntry(roomId, sortKey) {
+        const range = IDBKeyRange.bound([roomId, SortKey.minKey.buffer], [roomId, sortKey.buffer], false, true);
+        return this._timelineStore.selectFirst(range);
+    }
+
     // entry should have roomId, sortKey, event & gap keys
-    append(entry) {
+    add(entry) {
         this._timelineStore.add(entry);
     }
     // should this happen as part of a transaction that stores all synced in changes?
diff --git a/src/observable/list/ObservableArray.js b/src/observable/list/ObservableArray.js
index e33dfe1a..bb5c7758 100644
--- a/src/observable/list/ObservableArray.js
+++ b/src/observable/list/ObservableArray.js
@@ -11,6 +11,28 @@ export default class ObservableArray extends BaseObservableList {
         this.emitAdd(this._items.length - 1, item);
     }
 
+    insertMany(idx, items) {
+        for(let item of items) {
+            this.insert(idx, item);
+            idx += 1;
+        }
+    }
+
+    insert(idx, item) {
+        this._items.splice(idx, 0, item);
+        this.emitAdd(idx, item);
+    }
+
+    get array() {
+        return this._items;
+    }
+
+    at(idx) {
+        if (this._items && idx >= 0 && idx < this._items.length) {
+            return this._items[idx];
+        }
+    }
+
     get length() {
         return this._items.length;
     }

From 2cd9c2344e2e238645bc1242a50d59c6aa489d54 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 9 Mar 2019 00:43:43 +0100
Subject: [PATCH 15/83] expose timeline loading error in viewmodel

---
 src/domain/session/room/RoomViewModel.js | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/src/domain/session/room/RoomViewModel.js b/src/domain/session/room/RoomViewModel.js
index cec084b7..2e9c4053 100644
--- a/src/domain/session/room/RoomViewModel.js
+++ b/src/domain/session/room/RoomViewModel.js
@@ -6,12 +6,18 @@ export default class RoomViewModel extends EventEmitter {
         this._room = room;
         this._timeline = null;
         this._onRoomChange = this._onRoomChange.bind(this);
+        this._timelineError = null;
     }
 
     async enable() {
         this._room.on("change", this._onRoomChange);
-        this._timeline = await this._room.openTimeline();
-        this.emit("change", "timelineEntries");
+        try {
+            this._timeline = await this._room.openTimeline();
+            this.emit("change", "timelineEntries");
+        } catch (err) {
+            this._timelineError = err;
+            this.emit("change", "error");
+        }
     }
 
     disable() {
@@ -33,4 +39,11 @@ export default class RoomViewModel extends EventEmitter {
     get timelineEntries() {
         return this._timeline && this._timeline.entries;
     }
+
+    get error() {
+        if (this._timelineError) {
+            return `Something went wrong loading the timeline: ${this._timelineError.message}`;
+        }
+        return null;
+    }
 }

From 61804073e248c0706a812518eef8ecb2e19ba56a Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 15 Mar 2019 20:29:17 +0100
Subject: [PATCH 16/83] comment how updates should work

---
 src/observable/map/MappedMap.js | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/src/observable/map/MappedMap.js b/src/observable/map/MappedMap.js
index 6d10325f..4b16373b 100644
--- a/src/observable/map/MappedMap.js
+++ b/src/observable/map/MappedMap.js
@@ -1,5 +1,8 @@
 import BaseObservableMap from "./BaseObservableMap.js";
-
+/*
+so a mapped value can emit updates on it's own with this._updater that is passed in the mapping function
+how should the mapped value be notified of an update though? and can it then decide to not propagate the update?
+*/
 export default class MappedMap extends BaseObservableMap {
     constructor(source, mapper) {
         super();

From cc3a1811283ffb1205ed36e1328d0efb6e43e10a Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 15 Mar 2019 20:29:44 +0100
Subject: [PATCH 17/83] prettier color

---
 prototypes/responsive-layout-flex.html | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/prototypes/responsive-layout-flex.html b/prototypes/responsive-layout-flex.html
index 97995aac..c822ac08 100644
--- a/prototypes/responsive-layout-flex.html
+++ b/prototypes/responsive-layout-flex.html
@@ -85,7 +85,6 @@
 
         .description .last-message {
             font-size: 0.8em;
-
         }
 
         .room-panel-placeholder, .room-panel {
@@ -195,7 +194,7 @@
         }
 
         .timeline-panel li {
-            background: darkblue;
+            background: blue;
             padding: 10px;
             margin: 10px;
         }

From 8f7e5a799ca7abc1cf554901bcea24fdb58df52c Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Thu, 21 Mar 2019 21:35:33 +0100
Subject: [PATCH 18/83] work on filling gaps + test (draft only)

---
 src/matrix/room/persister.js | 209 +++++++++++++++++++++++++----------
 1 file changed, 150 insertions(+), 59 deletions(-)

diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js
index a2b1683b..f0c4456b 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/persister.js
@@ -8,6 +8,24 @@ function gapEntriesAreEqual(a, b) {
     return gapA.prev_batch === gapB.prev_batch && gapA.next_batch === gapB.next_batch;
 }
 
+function replaceGapEntries(roomTimeline, newEntries, gapKey, neighbourEventKey, backwards) {
+    let replacedRange;
+    if (neighbourEventKey) {
+        replacedRange = backwards ?
+            roomTimeline.boundRange(neighbourEventKey, gapKey, false, true) :
+            roomTimeline.boundRange(gapKey, neighbourEventKey, true, false);
+    } else {
+        replacedRange = roomTimeline.onlyRange(gapKey);
+    }
+
+    const removedEntries = roomTimeline.getAndRemoveRange(this._roomId, replacedRange);
+    for (let entry of newEntries) {
+        roomTimeline.add(entry);
+    }
+
+    return removedEntries;
+}
+
 export default class RoomPersister {
 	constructor({roomId, storage}) {
 		this._roomId = roomId;
@@ -26,77 +44,38 @@ export default class RoomPersister {
 		}
 	}
 
-	async persistGapFill(gapEntry, response) {
+    async persistGapFill(gapEntry, response) {
+        const backwards = !!gapEntry.prev_batch;
         const {chunk, start, end} = response;
         if (!Array.isArray(chunk)) {
             throw new Error("Invalid chunk in response");
         }
-        if (typeof start !== "string" || typeof end !== "string") {
-            throw new Error("Invalid start or end token in response");
+        if (typeof end !== "string") {
+            throw new Error("Invalid end token in response");
         }
+        if ((backwards && start !== gapEntry.prev_batch) || (!backwards && start !== gapEntry.next_batch)) {
+            throw new Error("start is not equal to prev_batch or next_batch");
+        }
+        
         const gapKey = gapEntry.sortKey;
         const txn = await this._storage.readWriteTxn([this._storage.storeNames.roomTimeline]);
+        let result;
         try {
             const roomTimeline = txn.roomTimeline;
             // make sure what we've been given is actually persisted
             // in the timeline, otherwise we're replacing something
             // that doesn't exist (maybe it has been replaced already, or ...)
-            const persistedEntry = await roomTimeline.findEntry(this._roomId, gapKey);
+            const persistedEntry = await roomTimeline.get(this._roomId, gapKey);
             if (!gapEntriesAreEqual(gapEntry, persistedEntry)) {
                 throw new Error("Gap is not present in the timeline");
             }
-            // find the previous event before the gap we could blend with
-            const backwards = !!gapEntry.prev_batch;
-            let neighbourEventEntry;
-            if (backwards) {
-                neighbourEventEntry = await roomTimeline.previousEventFromGap(this._roomId, gapKey);
-            } else {
-                neighbourEventEntry = await roomTimeline.nextEventFromGap(this._roomId, gapKey);
-            }
-            const neighbourEvent = neighbourEventEntry && neighbourEventEntry.event;
-
-            const newEntries = [];
-            let sortKey = gapKey;
-            let eventFound = false;
-            if (backwards) {
-                for (let i = chunk.length - 1; i >= 0; i--) {
-                    const event = chunk[i];
-                    if (event.id === neighbourEvent.id) {
-                        eventFound = true;
-                        break;
-                    }
-                    newEntries.splice(0, 0, this._createEventEntry(sortKey, event));
-                    sortKey = sortKey.previousKey();
-                }
-                if (!eventFound) {
-                    newEntries.splice(0, 0, this._createBackwardGapEntry(sortKey, end));
-                }
-            } else {
-                for (let i = 0; i < chunk.length; i++) {
-                    const event = chunk[i];
-                    if (event.id === neighbourEvent.id) {
-                        eventFound = true;
-                        break;
-                    }
-                    newEntries.push(this._createEventEntry(sortKey, event));
-                    sortKey = sortKey.nextKey();
-                }
-                if (!eventFound) {
-                    // need to check start is correct here
-                    newEntries.push(this._createForwardGapEntry(sortKey, start));
-                }
-            }
-
-            if (eventFound) {
-                // remove gap on the other side as well,
-                // or while we're at it, remove all gaps between gapKey and neighbourEventEntry.sortKey
-            } else {
-                roomTimeline.deleteEntry(this._roomId, gapKey);
-            }
-
-            for (let entry of newEntries) {
-                roomTimeline.add(entry);
-            }
+            // find the previous event before the gap we could merge with
+            const neighbourEventEntry = await roomTimeline.findSubsequentEvent(this._roomId, gapKey, backwards);
+            const neighbourEventId = neighbourEventEntry ? neighbourEventEntry.event.event_id : undefined;
+            const {newEntries, eventFound} = this._createNewGapEntries(chunk, end, gapKey, neighbourEventId, backwards);
+            const neighbourEventKey = eventFound ? neighbourEventEntry.sortKey : undefined;
+            const replacedEntries = replaceGapEntries(roomTimeline, newEntries, gapKey, neighbourEventKey, backwards);
+            result = {newEntries, replacedEntries};
         } catch (err) {
             txn.abort();
             throw err;
@@ -104,9 +83,35 @@ export default class RoomPersister {
 
         await txn.complete();
 
-        // somehow also return all the gaps we removed so the timeline can do the same
-        return {newEntries};
-	}
+        return result;
+    }
+
+    _createNewGapEntries(chunk, nextPaginationToken, gapKey, neighbourEventId, backwards) {
+        if (backwards) {
+            // if backwards, the last events are the ones closest to the gap,
+            // and need to be assigned a key derived from the gap first,
+            // so swap order to only need one loop for both directions
+            chunk.reverse();
+        }
+        let sortKey = gapKey;
+        const {newEntries, eventFound} = chunk.reduce((acc, event) => {
+            acc.eventFound = acc.eventFound || event.event_id === neighbourEventId;
+            if (!acc.eventFound) {
+                acc.newEntries.push(this._createEventEntry(sortKey, event));
+                sortKey = backwards ? sortKey.previousKey() : sortKey.nextKey();
+            }
+        }, {newEntries: [], eventFound: false});
+
+        if (!eventFound) {
+            // as we're replacing an existing gap, no need to increment the gap index
+            newEntries.push(this._createGapEntry(sortKey, nextPaginationToken, backwards));
+        }
+        if (backwards) {
+            // swap resulting array order again if needed
+            newEntries.reverse();
+        }
+        return {newEntries, eventFound};
+    }
 
 	persistSync(roomResponse, txn) {
 		let nextKey = this._lastSortKey;
@@ -182,3 +187,89 @@ export default class RoomPersister {
         };
     }
 }
+
+//#ifdef TESTS
+import {StorageMock, RoomTimelineMock} from "../../src/mocks/storage.js";
+
+export async function tests() {
+    const roomId = "!abc:hs.tld";
+
+    function areSorted(entries) {
+        for (var i = 1; i < entries.length; i++) {
+            const isSorted = entries[i - 1].sortKey.compare(entries[i].sortKey) < 0;
+            if(!isSorted) {
+                return false
+            }
+        }
+        return true;
+    }
+
+    return {
+        "test backwards gap fill with overlapping neighbouring event": async function(assert) {
+            const currentPaginationToken = "abc";
+            const gap = {gap: {prev_batch: currentPaginationToken}};
+            // assigns roomId and sortKey
+            const roomTimeline = RoomTimelineMock.forRoom(roomId, [
+                {event: {event_id: "b"}},
+                {gap: {next_batch: "ghi"}},
+                gap,
+            ]);
+            const persister = new RoomPersister(roomId, new StorageMock({roomTimeline}));
+            const response = {
+                start: currentPaginationToken,
+                end: "def",
+                chunk: [
+                    {event_id: "a"},
+                    {event_id: "b"},
+                    {event_id: "c"},
+                    {event_id: "d"},
+                ]
+            };
+            const {newEntries, replacedEntries} = await persister.persistGapFill(gap, response);
+            // should only have taken events up till existing event
+            assert.equal(newEntries.length, 2);
+            assert.equal(newEntries[0].event.event_id, "c");
+            assert.equal(newEntries[1].event.event_id, "d");
+            assert.equal(replacedEntries.length, 2);
+            assert.equal(replacedEntries[0].gap.next_batch, "hij");
+            assert.equal(replacedEntries[1].gap.prev_batch, currentPaginationToken);
+            assert(areSorted(newEntries));
+            assert(areSorted(replacedEntries));
+        },
+        "test backwards gap fill with non-overlapping neighbouring event": async function(assert) {
+            const currentPaginationToken = "abc";
+            const newPaginationToken = "def";
+            const gap = {gap: {prev_batch: currentPaginationToken}};
+            // assigns roomId and sortKey
+            const roomTimeline = RoomTimelineMock.forRoom(roomId, [
+                {event: {event_id: "a"}},
+                {gap: {next_batch: "ghi"}},
+                gap,
+            ]);
+            const persister = new RoomPersister(roomId, new StorageMock({roomTimeline}));
+            const response = {
+                start: currentPaginationToken,
+                end: newPaginationToken,
+                chunk: [
+                    {event_id: "c"},
+                    {event_id: "d"},
+                    {event_id: "e"},
+                    {event_id: "f"},
+                ]
+            };
+            const {newEntries, replacedEntries} = await persister.persistGapFill(gap, response);
+            // should only have taken events up till existing event
+            assert.equal(newEntries.length, 5);
+            assert.equal(newEntries[0].gap.prev_batch, newPaginationToken);
+            assert.equal(newEntries[1].event.event_id, "c");
+            assert.equal(newEntries[2].event.event_id, "d");
+            assert.equal(newEntries[3].event.event_id, "e");
+            assert.equal(newEntries[4].event.event_id, "f");
+            assert(areSorted(newEntries));
+
+            assert.equal(replacedEntries.length, 1);
+            assert.equal(replacedEntries[0].gap.prev_batch, currentPaginationToken);
+        },
+    }
+}
+//#endif

From b1e382d7c98367af143cc821c54bb5cf447c960f Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Thu, 21 Mar 2019 21:36:02 +0100
Subject: [PATCH 19/83] thinko with direction

---
 src/matrix/storage/idb/query-target.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/matrix/storage/idb/query-target.js b/src/matrix/storage/idb/query-target.js
index 51b20c7b..0a81f533 100644
--- a/src/matrix/storage/idb/query-target.js
+++ b/src/matrix/storage/idb/query-target.js
@@ -10,7 +10,7 @@ export default class QueryTarget {
 	}
 
 	reduceReverse(range, reducer, initialValue) {
-		return this._reduce(range, reducer, initialValue, "next");
+		return this._reduce(range, reducer, initialValue, "prev");
 	}
 	
 	selectLimit(range, amount) {
@@ -94,4 +94,4 @@ export default class QueryTarget {
 			return result;
 		}
 	}
-}
\ No newline at end of file
+}

From 7d91b2dde3ff6d267f5f3744da34e8de7670c751 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 29 Mar 2019 23:00:22 +0100
Subject: [PATCH 20/83] first go at a timeline memory store

first to use in unit tests for persister
later also to use in production when idb is not available
---
 .../memory/stores/RoomTimelineStore.js        | 220 ++++++++++++++++++
 1 file changed, 220 insertions(+)
 create mode 100644 src/matrix/storage/memory/stores/RoomTimelineStore.js

diff --git a/src/matrix/storage/memory/stores/RoomTimelineStore.js b/src/matrix/storage/memory/stores/RoomTimelineStore.js
new file mode 100644
index 00000000..a5072dba
--- /dev/null
+++ b/src/matrix/storage/memory/stores/RoomTimelineStore.js
@@ -0,0 +1,220 @@
+import SortKey from "../sortkey.js";
+import sortedIndex from "../../../utils/sortedIndex.js";
+
+function compareKeys(key, entry) {
+    if (key.roomId === entry.roomId) {
+        return key.sortKey.compare(entry.sortKey);
+    } else {
+        return key.roomId < entry.roomId ? -1 : 1;
+    }
+}
+
+class Range {
+    constructor(timeline, lower, upper, lowerOpen, upperOpen) {
+        this._timeline = timeline;
+        this._lower = lower;
+        this._upper = upper;
+        this._lowerOpen = lowerOpen;
+        this._upperOpen = upperOpen;
+    }
+
+    /** projects the range onto the timeline array */
+    project(roomId, maxCount = Number.MAX_SAFE_INTEGER) {
+        // determine lowest and highest allowed index.
+        // Important not to bleed into other roomIds here.
+        const lowerKey = {roomId, sortKey: this._lower || SortKey.minKey };
+        // apply lower key being open (excludes given key)
+        let minIndex = sortedIndex(this._timeline, lowerKey, compareKeys);
+        if (this._lowerOpen && minIndex < this._timeline.length && compareKeys(lowerKey, this._timeline[minIndex]) === 0) {
+            minIndex += 1;
+        }
+        const upperKey = {roomId, sortKey: this._upper || SortKey.maxKey };
+        // apply upper key being open (excludes given key)
+        let maxIndex = sortedIndex(this._timeline, upperKey, compareKeys);
+        if (this._upperOpen && maxIndex < this._timeline.length && compareKeys(upperKey, this._timeline[maxIndex]) === 0) {
+            maxIndex -= 1;
+        }
+        // find out from which edge we should grow
+        // if upper or lower bound
+        // again, important not to go below minIndex or above maxIndex
+        // to avoid bleeding into other rooms
+        let startIndex, endIndex;
+        if (!this._lower && this._upper) {
+            startIndex = Math.max(minIndex, maxIndex - maxCount);
+            endIndex = maxIndex;
+        } else if (this._lower && !this._upper) {
+            startIndex = minIndex;
+            endIndex = Math.min(maxIndex, minIndex + maxCount);
+        } else {
+            startIndex = minIndex;
+            endIndex = maxIndex;
+        }
+        
+        // if startIndex is out of range, make range empty
+        if (startIndex === this._timeline.length) {
+            startIndex = endIndex = 0;
+        }
+        const count = endIndex - startIndex;
+        return {startIndex, count};
+    }
+
+    select(roomId, maxCount) {
+        const {startIndex, count} = this.project(roomId, this._timeline, maxCount);
+        return this._timeline.slice(startIndex, startIndex + count);
+    }
+}
+
+export default class RoomTimelineStore {
+    constructor(initialTimeline = []) {
+        this._timeline = initialTimeline;
+    }
+
+    /** Creates a range that only includes the given key
+     *  @param {SortKey} sortKey the key
+     *  @return {Range} the created range
+     */
+    onlyRange(sortKey) {
+        return new Range(this._timeline, sortKey, sortKey);
+    }
+
+    /** Creates a range that includes all keys before sortKey, and optionally also the key itself.
+     *  @param {SortKey} sortKey the key
+     *  @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the upper end.
+     *  @return {Range} the created range
+     */
+    upperBoundRange(sortKey, open=false) {
+        return new Range(this._timeline, undefined, sortKey, undefined, open);
+    }
+
+    /** Creates a range that includes all keys after sortKey, and optionally also the key itself.
+     *  @param {SortKey} sortKey the key
+     *  @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the lower end.
+     *  @return {Range} the created range
+     */
+    lowerBoundRange(sortKey, open=false) {
+        return new Range(this._timeline, sortKey, undefined, open);
+    }
+
+    /** Creates a range that includes all keys between `lower` and `upper`, and optionally the given keys as well.
+     *  @param {SortKey} lower the lower key
+     *  @param {SortKey} upper the upper key
+     *  @param {boolean} [lowerOpen=false] whether the lower key is included (false) or excluded (true) from the range.
+     *  @param {boolean} [upperOpen=false] whether the upper key is included (false) or excluded (true) from the range.
+     *  @return {Range} the created range
+     */
+    boundRange(lower, upper, lowerOpen=false, upperOpen=false) {
+        return new Range(this._timeline, lower, upper, lowerOpen, upperOpen);
+    }
+
+    /** Looks up the last `amount` entries in the timeline for `roomId`.
+     *  @param  {string} roomId
+     *  @param  {number} amount
+     *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
+     */
+    lastEvents(roomId, amount) {
+        return this.eventsBefore(roomId, SortKey.maxKey, amount);
+    }
+
+    /** Looks up the first `amount` entries in the timeline for `roomId`.
+     *  @param  {string} roomId
+     *  @param  {number} amount
+     *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
+     */
+    firstEvents(roomId, amount) {
+        return this.eventsAfter(roomId, SortKey.minKey, amount);
+    }
+
+    /** Looks up `amount` entries after `sortKey` in the timeline for `roomId`.
+     *  The entry for `sortKey` is not included.
+     *  @param  {string} roomId
+     *  @param  {SortKey} sortKey
+     *  @param  {number} amount
+     *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
+     */
+    eventsAfter(roomId, sortKey, amount) {
+        const events = this.lowerBoundRange(sortKey, true).select(roomId, amount);
+        return Promise.resolve(events);
+    }
+
+    /** Looks up `amount` entries before `sortKey` in the timeline for `roomId`.
+     *  The entry for `sortKey` is not included.
+     *  @param  {string} roomId
+     *  @param  {SortKey} sortKey
+     *  @param  {number} amount
+     *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
+     */
+    eventsBefore(roomId, sortKey, amount) {
+        const events = this.upperBoundRange(sortKey, true).select(roomId, amount);
+        return Promise.resolve(events);
+    }
+
+    /** Looks up the first, if any, event entry (so excluding gap entries) after `sortKey`.
+     *  @param  {string} roomId
+     *  @param  {SortKey} sortKey
+     *  @return {Promise<(?Entry)>} a promise resolving to entry, if any.
+     */
+    nextEvent(roomId, sortKey) {
+        const searchSpace = this.lowerBoundRange(sortKey, true).select(roomId);
+        const event = searchSpace.find(entry => !!entry.event);
+        return Promise.resolve(event);
+    }
+
+    /** Looks up the first, if any, event entry (so excluding gap entries) before `sortKey`.
+     *  @param  {string} roomId
+     *  @param  {SortKey} sortKey
+     *  @return {Promise<(?Entry)>} a promise resolving to entry, if any.
+     */
+    previousEvent(roomId, sortKey) {
+        const searchSpace = this.upperBoundRange(sortKey, true).select(roomId);
+        const event = searchSpace.reverse().find(entry => !!entry.event);
+        return Promise.resolve(event);
+    }
+
+    /** Inserts a new entry into the store. The combination of roomId and sortKey should not exist yet, or an error is thrown.
+     *  @param  {Entry} entry the entry to insert
+     *  @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not.
+     *  @throws {StorageError} ...
+     */
+    insert(entry) {
+        const insertIndex = sortedIndex(this._timeline, entry, compareKeys);
+        if (insertIndex < this._timeline.length) {
+            const existingEntry = this._timeline[insertIndex];
+            if (compareKeys(entry, existingEntry) === 0) {
+                return Promise.reject(new Error("entry already exists"));
+            }
+        }
+        this._timeline.splice(insertIndex, 0, entry);
+        return Promise.resolve();
+    }
+
+    /** Updates the entry into the store with the given [roomId, sortKey] combination.
+     *  If not yet present, will insert. Might be slower than add.
+     *  @param  {Entry} entry the entry to update.
+     *  @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not.
+     */
+    update(entry) {
+        let update = false;
+        const updateIndex = sortedIndex(this._timeline, entry, compareKeys);
+        if (updateIndex < this._timeline.length) {
+            const existingEntry = this._timeline[updateIndex];
+            if (compareKeys(entry, existingEntry) === 0) {
+                update = true;
+            }
+        }
+        this._timeline.splice(updateIndex, update ? 1 : 0, entry);
+        return Promise.resolve();
+    }
+
+    get(roomId, sortKey) {
+        const range = this.onlyRange(sortKey);
+        const {startIndex, count} = range.project(roomId);
+        const event = count ? this._timeline[startIndex] : undefined;
+        return Promise.resolve(event);
+    }
+
+    removeRange(roomId, range) {
+        const {startIndex, count} = range.project(roomId);
+        const removedEntries = this._timeline.splice(startIndex, count);
+        return Promise.resolve(removedEntries);
+    }
+}

From aaff9eea6cf19d6dc790a60e7abdef0cda73227d Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 29 Mar 2019 23:01:01 +0100
Subject: [PATCH 21/83] update store api with requirements for gap filling

---
 src/matrix/storage/idb/query-target.js        |   6 +-
 .../storage/idb/stores/RoomTimelineStore.js   | 208 ++++++++++++++----
 2 files changed, 169 insertions(+), 45 deletions(-)

diff --git a/src/matrix/storage/idb/query-target.js b/src/matrix/storage/idb/query-target.js
index 0a81f533..547c2fdd 100644
--- a/src/matrix/storage/idb/query-target.js
+++ b/src/matrix/storage/idb/query-target.js
@@ -1,10 +1,14 @@
-import {iterateCursor} from "./utils.js";
+import {iterateCursor, reqAsPromise} from "./utils.js";
 
 export default class QueryTarget {
 	constructor(target) {
 		this._target = target;
 	}
 
+    get(key) {
+        return reqAsPromise(this._target.get(key));
+    }
+
 	reduce(range, reducer, initialValue) {
 		return this._reduce(range, reducer, initialValue, "next");
 	}
diff --git a/src/matrix/storage/idb/stores/RoomTimelineStore.js b/src/matrix/storage/idb/stores/RoomTimelineStore.js
index f02d2a86..491c1f10 100644
--- a/src/matrix/storage/idb/stores/RoomTimelineStore.js
+++ b/src/matrix/storage/idb/stores/RoomTimelineStore.js
@@ -1,77 +1,197 @@
 import SortKey from "../../sortkey.js";
 
+class Range {
+    constructor(only, lower, upper, lowerOpen, upperOpen) {
+        this._only = only;
+        this._lower = lower;
+        this._upper = upper;
+        this._lowerOpen = lowerOpen;
+        this._upperOpen = upperOpen;
+    }
+
+    asIDBKeyRange(roomId) {
+        // only
+        if (this._only) {
+            return IDBKeyRange.only([roomId, this._only.buffer]);
+        }
+        // lowerBound
+        // also bound as we don't want to move into another roomId
+        if (this._lower && !this._upper) {
+            return IDBKeyRange.bound(
+                [roomId, this._lower.buffer],
+                [roomId, SortKey.maxKey.buffer],
+                this._lowerOpen,
+                false
+            );
+        }
+        // upperBound
+        // also bound as we don't want to move into another roomId
+        if (!this._lower && this._upper) {
+            return IDBKeyRange.bound(
+                [roomId, SortKey.minKey.buffer],
+                [roomId, this._upper.buffer],
+                false,
+                this._upperOpen
+            );
+        }
+        // bound
+        if (this._lower && this._upper) {
+            return IDBKeyRange.bound(
+                [roomId, this._lower.buffer],
+                [roomId, this._upper.buffer],
+                this._lowerOpen,
+                this._upperOpen
+            );
+        }
+    }
+}
+/*
+ * @typedef   {Object} Gap
+ * @property  {?string} prev_batch the pagination token for this backwards facing gap
+ * @property  {?string} next_batch the pagination token for this forwards facing gap
+ *
+ * @typedef   {Object} Event
+ * @property  {string} event_id the id of the event
+ * @property  {string} type the
+ * @property  {?string} state_key the state key of this state event
+ *
+ * @typedef   {Object} Entry
+ * @property  {string} roomId
+ * @property  {SortKey} sortKey
+ * @property  {?Event} event if an event entry, the event
+ * @property  {?Gap} gap if a gap entry, the gap
+*/
 export default class RoomTimelineStore {
 	constructor(timelineStore) {
 		this._timelineStore = timelineStore;
 	}
 
+    /** Creates a range that only includes the given key
+     *  @param {SortKey} sortKey the key
+     *  @return {Range} the created range
+     */
+    onlyRange(sortKey) {
+        return new Range(sortKey);
+    }
+
+    /** Creates a range that includes all keys before sortKey, and optionally also the key itself.
+     *  @param {SortKey} sortKey the key
+     *  @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the upper end.
+     *  @return {Range} the created range
+     */
+    upperBoundRange(sortKey, open=false) {
+        return new Range(undefined, undefined, sortKey, undefined, open);
+    }
+
+    /** Creates a range that includes all keys after sortKey, and optionally also the key itself.
+     *  @param {SortKey} sortKey the key
+     *  @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the lower end.
+     *  @return {Range} the created range
+     */
+    lowerBoundRange(sortKey, open=false) {
+        return new Range(undefined, sortKey, undefined, open);
+    }
+
+    /** Creates a range that includes all keys between `lower` and `upper`, and optionally the given keys as well.
+     *  @param {SortKey} lower the lower key
+     *  @param {SortKey} upper the upper key
+     *  @param {boolean} [lowerOpen=false] whether the lower key is included (false) or excluded (true) from the range.
+     *  @param {boolean} [upperOpen=false] whether the upper key is included (false) or excluded (true) from the range.
+     *  @return {Range} the created range
+     */
+    boundRange(lower, upper, lowerOpen=false, upperOpen=false) {
+        return new Range(undefined, lower, upper, lowerOpen, upperOpen);
+    }
+
+    /** Looks up the last `amount` entries in the timeline for `roomId`.
+     *  @param  {string} roomId
+     *  @param  {number} amount
+     *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
+     */
 	async lastEvents(roomId, amount) {
 		return this.eventsBefore(roomId, SortKey.maxKey, amount);
 	}
 
+    /** Looks up the first `amount` entries in the timeline for `roomId`.
+     *  @param  {string} roomId
+     *  @param  {number} amount
+     *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
+     */
 	async firstEvents(roomId, amount) {
 		return this.eventsAfter(roomId, SortKey.minKey, amount);
 	}
 
+    /** Looks up `amount` entries after `sortKey` in the timeline for `roomId`.
+     *  The entry for `sortKey` is not included.
+     *  @param  {string} roomId
+     *  @param  {SortKey} sortKey
+     *  @param  {number} amount
+     *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
+     */
 	eventsAfter(roomId, sortKey, amount) {
-		const range = IDBKeyRange.bound([roomId, sortKey.buffer], [roomId, SortKey.maxKey.buffer], true, false);
-		return this._timelineStore.selectLimit(range, amount);
+        const idbRange = this.lowerBoundRange(sortKey, true).asIDBKeyRange(roomId);
+		return this._timelineStore.selectLimit(idbRange, amount);
 	}
 
+    /** Looks up `amount` entries before `sortKey` in the timeline for `roomId`.
+     *  The entry for `sortKey` is not included.
+     *  @param  {string} roomId
+     *  @param  {SortKey} sortKey
+     *  @param  {number} amount
+     *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
+     */
     async eventsBefore(roomId, sortKey, amount) {
-        const range = IDBKeyRange.only([roomId, sortKey.buffer]);
+        const range = this.upperBoundRange(sortKey, true).asIDBKeyRange(roomId);
         const events = await this._timelineStore.selectLimitReverse(range, amount);
         events.reverse(); // because we fetched them backwards
         return events;
     }
 
-    nextEventFromGap(roomId, sortKey) {
-        
+    /** Looks up the first, if any, event entry (so excluding gap entries) after `sortKey`.
+     *  @param  {string} roomId
+     *  @param  {SortKey} sortKey
+     *  @return {Promise<(?Entry)>} a promise resolving to entry, if any.
+     */
+    nextEvent(roomId, sortKey) {
+        const range = this.lowerBoundRange(sortKey, true).asIDBKeyRange(roomId);
+        return this._timelineStore.find(range, entry => !!entry.event);
     }
 
-    previousEventFromGap(roomId, sortKey) {
-        
+    /** Looks up the first, if any, event entry (so excluding gap entries) before `sortKey`.
+     *  @param  {string} roomId
+     *  @param  {SortKey} sortKey
+     *  @return {Promise<(?Entry)>} a promise resolving to entry, if any.
+     */
+    previousEvent(roomId, sortKey) {
+        const range = this.upperBoundRange(sortKey, true).asIDBKeyRange(roomId);
+        return this._timelineStore.findReverse(range, entry => !!entry.event);
     }
 
-    findEntry(roomId, sortKey) {
-        const range = IDBKeyRange.bound([roomId, SortKey.minKey.buffer], [roomId, sortKey.buffer], false, true);
-        return this._timelineStore.selectFirst(range);
+    /** Inserts a new entry into the store. The combination of roomId and sortKey should not exist yet, or an error is thrown.
+     *  @param  {Entry} entry the entry to insert
+     *  @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not.
+     *  @throws {StorageError} ...
+     */
+    insert(entry) {
+        // TODO: map error? or in idb/store?
+        return this._timelineStore.add(entry);
     }
 
-    // entry should have roomId, sortKey, event & gap keys
-    add(entry) {
-        this._timelineStore.add(entry);
+    /** Updates the entry into the store with the given [roomId, sortKey] combination.
+     *  If not yet present, will insert. Might be slower than add.
+     *  @param  {Entry} entry the entry to update.
+     *  @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not.
+     */
+    update(entry) {
+        return this._timelineStore.put(entry);
     }
-    // should this happen as part of a transaction that stores all synced in changes?
-    // e.g.:
-    // - timeline events for all rooms
-    // - latest sync token
-    // - new members
-    // - new room state
-    // - updated/new account data
 
-
-
-	appendGap(roomId, sortKey, gap) {
-		this._timelineStore.add({
-			roomId: roomId,
-			sortKey: sortKey.buffer,
-			event: null,
-			gap: gap,
-		});
-	}
-
-	appendEvent(roomId, sortKey, event) {
-		console.info(`appending event for room ${roomId} with key ${sortKey}`);
-		this._timelineStore.add({
-			roomId: roomId,
-			sortKey: sortKey.buffer,
-			event: event,
-			gap: null,
-		});
-	}
-	// could be gap or event
-	async removeEntry(roomId, sortKey) {
-		this._timelineStore.delete([roomId, sortKey.buffer]);
-	}
+    get(roomId, sortKey) {
+        return this._timelineStore.get([roomId, sortKey]);
+    }
+    // returns the entries as well!! (or not always needed? I guess not always needed, so extra method)
+    removeRange(roomId, range) {
+        // TODO: read the entries!
+        return this._timelineStore.delete(range.asIDBKeyRange(roomId));
+    }
 }

From 3f2f656db71dd7bcb3ae7444d011e92301d55dca Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 29 Mar 2019 23:01:27 +0100
Subject: [PATCH 22/83] work on gap filling + tests (doesn't work yet)

---
 src/matrix/room/persister.js | 42 +++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 13 deletions(-)

diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js
index f0c4456b..abcdf1bd 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/persister.js
@@ -70,7 +70,9 @@ export default class RoomPersister {
                 throw new Error("Gap is not present in the timeline");
             }
             // find the previous event before the gap we could merge with
-            const neighbourEventEntry = await roomTimeline.findSubsequentEvent(this._roomId, gapKey, backwards);
+            const neighbourEventEntry = await (backwards ?
+                roomTimeline.previousEvent(this._roomId, gapKey) :
+                roomTimeline.nextEvent(this._roomId, gapKey));
             const neighbourEventId = neighbourEventEntry ? neighbourEventEntry.event.event_id : undefined;
             const {newEntries, eventFound} = this._createNewGapEntries(chunk, end, gapKey, neighbourEventId, backwards);
             const neighbourEventKey = eventFound ? neighbourEventEntry.sortKey : undefined;
@@ -123,7 +125,6 @@ export default class RoomPersister {
 			nextKey = nextKey.nextKeyWithGap();
             entries.push(this._createBackwardGapEntry(nextKey, timeline.prev_batch));
         }
-        // const startOfChunkSortKey = nextKey;
         if (timeline.events) {
             for(const event of timeline.events) {
                 nextKey = nextKey.nextKey();
@@ -132,7 +133,7 @@ export default class RoomPersister {
 		}
         // write to store
         for(const entry of entries) {
-            txn.roomTimeline.add(entry);
+            txn.roomTimeline.insert(entry);
         }
 		// right thing to do? if the txn fails, not sure we'll continue anyways ...
 		// only advance the key once the transaction has
@@ -189,11 +190,28 @@ export default class RoomPersister {
 }
 
 //#ifdef TESTS
-import {StorageMock, RoomTimelineMock} from "../../src/mocks/storage.js";
+import MemoryStorage from "../storage/memory/MemoryStorage.js";
 
-export async function tests() {
+export function tests() {
     const roomId = "!abc:hs.tld";
 
+    // sets sortKey and roomId on an array of entries
+    function createTimeline(roomId, entries) {
+        let key = new SortKey();
+        for (let entry of entries) {
+            if (entry.gap && entry.gap.prev_batch) {
+                key = key.nextKeyWithGap();
+            }
+            entry.sortKey = key;
+            if (entry.gap && entry.gap.next_batch) {
+                key = key.nextKeyWithGap();
+            } else if (!entry.gap) {
+                key = key.nextKey();
+            }
+            entry.roomId = roomId;
+        }
+    }
+
     function areSorted(entries) {
         for (var i = 1; i < entries.length; i++) {
             const isSorted = entries[i - 1].sortKey.compare(entries[i].sortKey) < 0;
@@ -208,13 +226,12 @@ export async function tests() {
         "test backwards gap fill with overlapping neighbouring event": async function(assert) {
             const currentPaginationToken = "abc";
             const gap = {gap: {prev_batch: currentPaginationToken}};
-            // assigns roomId and sortKey
-            const roomTimeline = RoomTimelineMock.forRoom(roomId, [
+            const storage = new MemoryStorage({roomTimeline: createTimeline(roomId, [
                 {event: {event_id: "b"}},
                 {gap: {next_batch: "ghi"}},
                 gap,
-            ]);
-            const persister = new RoomPersister(roomId, new StorageMock({roomTimeline}));
+            ])});
+            const persister = new RoomPersister({roomId, storage});
             const response = {
                 start: currentPaginationToken,
                 end: "def",
@@ -240,13 +257,12 @@ export async function tests() {
             const currentPaginationToken = "abc";
             const newPaginationToken = "def";
             const gap = {gap: {prev_batch: currentPaginationToken}};
-            // assigns roomId and sortKey
-            const roomTimeline = RoomTimelineMock.forRoom(roomId, [
+            const storage = new MemoryStorage({roomTimeline: createTimeline(roomId, [
                 {event: {event_id: "a"}},
                 {gap: {next_batch: "ghi"}},
                 gap,
-            ]);
-            const persister = new RoomPersister(roomId, new StorageMock({roomTimeline}));
+            ])});
+            const persister = new RoomPersister({roomId, storage});
             const response = {
                 start: currentPaginationToken,
                 end: newPaginationToken,

From 1605170f9ed8c0366fffeaa295070566b95fe79a Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Fri, 29 Mar 2019 23:51:25 +0100
Subject: [PATCH 23/83] update test runner to support async tests

---
 package-lock.json | 6 +++---
 package.json      | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/package-lock.json b/package-lock.json
index 6253d97e..56f65bc8 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -95,9 +95,9 @@
       }
     },
     "impunity": {
-      "version": "0.0.5",
-      "resolved": "https://registry.npmjs.org/impunity/-/impunity-0.0.5.tgz",
-      "integrity": "sha512-ro+enrZPFTyY2U1sV9NytsyejE2tS5theAArM95iPYYQHUvO9YN0VjgfXP0KJfxwh4Xb6vBTRBmHIgx9GUx2Xg==",
+      "version": "0.0.7",
+      "resolved": "https://registry.npmjs.org/impunity/-/impunity-0.0.7.tgz",
+      "integrity": "sha512-+DhzXSWrzqI1KNroKt3y1LkLTn/aoJpt4DzxWN+hair+Jfb+iJAbTEsSFkYUG7kASP9TF9GvI0hIBUul6PjpKg==",
       "dev": true,
       "requires": {
         "colors": "^1.3.3",
diff --git a/package.json b/package.json
index 4b5db04a..6ae62e9c 100644
--- a/package.json
+++ b/package.json
@@ -22,7 +22,7 @@
   "homepage": "https://github.com/bwindels/morpheusjs#readme",
   "devDependencies": {
     "finalhandler": "^1.1.1",
-    "impunity": "^0.0.5",
+    "impunity": "^0.0.7",
     "serve-static": "^1.13.2"
   }
 }

From 6ba37e90a32eb52f49383e0a5904aba073ef1ba2 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Thu, 4 Apr 2019 09:27:31 +0200
Subject: [PATCH 24/83] work on memory store

---
 src/matrix/storage/common.js                  |  6 ++
 src/matrix/storage/memory/Storage.js          | 37 ++++++++++++
 src/matrix/storage/memory/Transaction.js      | 57 +++++++++++++++++++
 .../memory/stores/RoomTimelineStore.js        | 14 ++++-
 src/matrix/storage/memory/stores/Store.js     | 27 +++++++++
 5 files changed, 138 insertions(+), 3 deletions(-)
 create mode 100644 src/matrix/storage/common.js
 create mode 100644 src/matrix/storage/memory/Storage.js
 create mode 100644 src/matrix/storage/memory/Transaction.js
 create mode 100644 src/matrix/storage/memory/stores/Store.js

diff --git a/src/matrix/storage/common.js b/src/matrix/storage/common.js
new file mode 100644
index 00000000..0280549b
--- /dev/null
+++ b/src/matrix/storage/common.js
@@ -0,0 +1,6 @@
+export const STORE_NAMES = Object.freeze(["session", "roomState", "roomSummary", "roomTimeline"]);
+
+export const STORE_MAP = Object.freeze(STORE_NAMES.reduce((nameMap, name) => {
+    nameMap[name] = name;
+    return nameMap;
+}, {}));
diff --git a/src/matrix/storage/memory/Storage.js b/src/matrix/storage/memory/Storage.js
new file mode 100644
index 00000000..fb178b01
--- /dev/null
+++ b/src/matrix/storage/memory/Storage.js
@@ -0,0 +1,37 @@
+import Transaction from "./transaction.js";
+import { STORE_MAP, STORE_NAMES } from "../common.js";
+
+export default class Storage {
+	constructor(initialStoreValues = {}) {
+        this._validateStoreNames(Object.keys(initialStoreValues));
+		this.storeNames = STORE_MAP;
+        this._storeValues = STORE_NAMES.reduce((values, name) => {
+            values[name] = initialStoreValues[name] || null;
+        }, {});
+	}
+
+	_validateStoreNames(storeNames) {
+		const idx = storeNames.findIndex(name => !STORE_MAP.hasOwnProperty(name));
+		if (idx !== -1) {
+			throw new Error(`Invalid store name ${storeNames[idx]}`);
+		}
+	}
+
+    _createTxn(storeNames, writable) {
+        this._validateStoreNames(storeNames);
+        const storeValues = storeNames.reduce((values, name) => {
+            return values[name] = this._storeValues[name];
+        }, {});
+        return Promise.resolve(new Transaction(storeValues, writable));
+    }
+
+	readTxn(storeNames) {
+        // TODO: avoid concurrency
+        return this._createTxn(storeNames, false);
+	}
+
+	readWriteTxn(storeNames) {
+        // TODO: avoid concurrency
+		return this._createTxn(storeNames, true);
+	}
+}
diff --git a/src/matrix/storage/memory/Transaction.js b/src/matrix/storage/memory/Transaction.js
new file mode 100644
index 00000000..437962da
--- /dev/null
+++ b/src/matrix/storage/memory/Transaction.js
@@ -0,0 +1,57 @@
+import RoomTimelineStore from "./stores/RoomTimelineStore.js";
+
+export default class Transaction {
+    constructor(storeValues, writable) {
+        this._storeValues = storeValues;
+        this._txnStoreValues = {};
+        this._writable = writable;
+    }
+
+    _store(name, mapper) {
+        if (!this._txnStoreValues.hasOwnProperty(name)) {
+            if (!this._storeValues.hasOwnProperty(name)) {
+                throw new Error(`Transaction wasn't opened for store ${name}`);
+            }
+            const store = mapper(this._storeValues[name]);
+            const clone = store.cloneStoreValue();
+            // extra prevention for writing
+            if (!this._writable) {
+                Object.freeze(clone);
+            }
+            this._txnStoreValues[name] = clone;
+        }
+        return mapper(this._txnStoreValues[name]);
+    }
+
+    get session() {
+        throw new Error("not yet implemented");
+        // return this._store("session", storeValue => new SessionStore(storeValue));
+    }
+
+    get roomSummary() {
+        throw new Error("not yet implemented");
+        // return this._store("roomSummary", storeValue => new RoomSummaryStore(storeValue));
+    }
+
+    get roomTimeline() {
+        return this._store("roomTimeline", storeValue => new RoomTimelineStore(storeValue));
+    }
+
+    get roomState() {
+        throw new Error("not yet implemented");
+        // return this._store("roomState", storeValue => new RoomStateStore(storeValue));
+    }
+
+    complete() {
+        for(let name of Object.keys(this._txnStoreValues)) {
+            this._storeValues[name] = this._txnStoreValues[name];
+        }
+        this._txnStoreValues = null;
+        return Promise.resolve();
+    }
+
+    abort() {
+        this._txnStoreValues = null;
+        return Promise.resolve();
+    }
+}
diff --git a/src/matrix/storage/memory/stores/RoomTimelineStore.js b/src/matrix/storage/memory/stores/RoomTimelineStore.js
index a5072dba..40c521cb 100644
--- a/src/matrix/storage/memory/stores/RoomTimelineStore.js
+++ b/src/matrix/storage/memory/stores/RoomTimelineStore.js
@@ -1,5 +1,6 @@
 import SortKey from "../sortkey.js";
 import sortedIndex from "../../../utils/sortedIndex.js";
+import Store from "./Store";
 
 function compareKeys(key, entry) {
     if (key.roomId === entry.roomId) {
@@ -64,9 +65,13 @@ class Range {
     }
 }
 
-export default class RoomTimelineStore {
-    constructor(initialTimeline = []) {
-        this._timeline = initialTimeline;
+export default class RoomTimelineStore extends Store {
+    constructor(timeline, writable) {
+        super(timeline || [], writable);
+    }
+
+    get _timeline() {
+        return this._storeValue;
     }
 
     /** Creates a range that only includes the given key
@@ -176,6 +181,7 @@ export default class RoomTimelineStore {
      *  @throws {StorageError} ...
      */
     insert(entry) {
+        this.assertWritable();
         const insertIndex = sortedIndex(this._timeline, entry, compareKeys);
         if (insertIndex < this._timeline.length) {
             const existingEntry = this._timeline[insertIndex];
@@ -193,6 +199,7 @@ export default class RoomTimelineStore {
      *  @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not.
      */
     update(entry) {
+        this.assertWritable();
         let update = false;
         const updateIndex = sortedIndex(this._timeline, entry, compareKeys);
         if (updateIndex < this._timeline.length) {
@@ -213,6 +220,7 @@ export default class RoomTimelineStore {
     }
 
     removeRange(roomId, range) {
+        this.assertWritable();
         const {startIndex, count} = range.project(roomId);
         const removedEntries = this._timeline.splice(startIndex, count);
         return Promise.resolve(removedEntries);
diff --git a/src/matrix/storage/memory/stores/Store.js b/src/matrix/storage/memory/stores/Store.js
new file mode 100644
index 00000000..8028a783
--- /dev/null
+++ b/src/matrix/storage/memory/stores/Store.js
@@ -0,0 +1,27 @@
+export default class Store {
+    constructor(storeValue, writable) {
+        this._storeValue = storeValue;
+        this._writable = writable;
+    }
+
+    // makes a copy deep enough that any modifications in the store
+    // won't affect the original
+    // used for transactions
+    cloneStoreValue() {
+        // assumes 1 level deep is enough, and that values will be replaced
+        // rather than updated.
+        if (Array.isArray(this._storeValue)) {
+            return this._storeValue.slice();
+        } else if (typeof this._storeValue === "object") {
+            return Object.assign({}, this._storeValue);
+        } else {
+            return this._storeValue;
+        }
+    }
+
+    assertWritable() {
+        if (!this._writable) {
+            throw new Error("Tried to write in read-only transaction");
+        }
+    }
+}

From 245370c765c51d9cbae60c82e2ddcac9e0800d42 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Thu, 4 Apr 2019 09:27:53 +0200
Subject: [PATCH 25/83] list of questions we've got no good solution for yet

---
 doc/QUESTIONS.md | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 doc/QUESTIONS.md

diff --git a/doc/QUESTIONS.md b/doc/QUESTIONS.md
new file mode 100644
index 00000000..224c7d02
--- /dev/null
+++ b/doc/QUESTIONS.md
@@ -0,0 +1,4 @@
+remaining problems to resolve:
+how to store timelime fragments that we don't yet know how they should be sorted wrt the other events and gaps. the case with event permalinks and showing the replied to event when rendering a reply.
+
+room state... 
\ No newline at end of file

From a8aa97fdf3eacce8e9c568a53fb95423ade4ca19 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Thu, 4 Apr 2019 21:48:13 +0200
Subject: [PATCH 26/83] plan to refactor to support storing /context responses

---
 doc/QUESTIONS.md    | 19 +++++++++++++++++--
 doc/architecture.md | 18 ++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/doc/QUESTIONS.md b/doc/QUESTIONS.md
index 224c7d02..2bad1bae 100644
--- a/doc/QUESTIONS.md
+++ b/doc/QUESTIONS.md
@@ -1,4 +1,19 @@
 remaining problems to resolve:
-how to store timelime fragments that we don't yet know how they should be sorted wrt the other events and gaps. the case with event permalinks and showing the replied to event when rendering a reply.
 
-room state... 
\ No newline at end of file
+how to store timelime fragments that we don't yet know how they should be sorted wrt the other events and gaps. the case with event permalinks and showing the replied to event when rendering a reply (anything from /context).
+    
+    either we could put timeline pieces that were the result of /context in something that is not the timeline. Gaps also don't really make sense there ... You can just paginate backwards and forwards. Or maybe still in the timeline but in a different scope not part of the sortKey, scope: live, or scope: piece-1204. While paginating, we could keep the start and end event_id of all the scopes in memory, and set a marker on them to stitch them together?
+
+    Hmmm, I can see the usefullness of the concept of timeline set with multiple timelines in it for this. for the live timeline it's less convenient as you're not bothered so much by the stitching up, but for /context pieces that run into the live timeline while paginating it seems more useful... we could have a marker entry that refers to the next or previous scope ... this way we could also use gap entries for /context timelines, just one on either end.
+
+    the start and end event_id of a scope, keeping that in memory, how do we make sure this is safe taking transactions into account? our preferred strategy so far has been to read everything from store inside a txn to make sure we don't have any stale caches or races. Would be nice to keep this.
+
+    so while paginating, you'd check the event_id of the event against the start/end event_id of every scope to see if stitching is in order, and add marker entries if so. Perhaps marker entries could also be used to stitch up rooms that have changed versioning?
+
+    What does all of this mean for using sortKey as an identifier? Will we need to take scope into account as well everywhere?
+
+    we'll need to at least contemplate how room state will be handled with all of the above.
+    
+how do we deal with the fact that an event can be rendered (and updated) multiple times in the timeline as part of replies. 
+
+room state... 
diff --git a/doc/architecture.md b/doc/architecture.md
index 1a2f4c3a..2be87d37 100644
--- a/doc/architecture.md
+++ b/doc/architecture.md
@@ -7,6 +7,24 @@ The matrix layer assumes a transaction-based storage layer, modelled much to how
 
 For this reason a `Room` processes a sync response in two phases: `persistSync` & `emitSync`, with the return value of the former being passed into the latter to avoid double processing.
 
+## Timeline, fragments & event indices.
+
+A room in matrix is a DAG (directed, acyclic graph) of events, also known as the timeline. Morpheus is only aware of fragments of this graph, and can be unaware how these fragments relate to each other until a common event is found while paginating a fragment. After doing an initial sync, you start with one fragment. When looking up an event with the `/context` endpoint (for fetching a replied to message, or navigating to a given event id, e.g. through a permalink), a new, unconnected, fragment is created. Also, when receiving a limited sync response during incremental sync, a new fragment is created. Here, the relationship is clear, so they are immediately linked up at creation. Events in morpheus are identified within a room by `[fragment_id, event_index]`. The `event_index` is an unique number within a fragment to sort events in chronological order in the timeline. `fragment_id` cannot be directly compared for sorting (as the relationship may be unknown), but with help of the `FragmentIndex`, one can attempt to sort events by their `FragmentIndex([fragment_id, event_index])`.
+
+A fragment is the following data structure:
+```
+let fragment := {
+    roomId: string
+    id: number
+    previousId: number?
+    nextId: number?
+    prevToken: string?
+    nextToken: string?
+}
+```
+
+## Observing the session
+
 `Room`s on the `Session` are exposed as an `ObservableMap` collection, which is like an ordinary `Map` but emits events when it is modified (here when a room is added, removed, or the properties of a room change). `ObservableMap` can have different operators applied to it like `mapValues()`, `filterValues()` each returning a new `ObservableMap`-like object, and also `sortValues()` returning an `ObservableList` (emitting events when a room at an index is added, removed, moved or changes properties).
 
 So example, for the room list, `Room` objects from `Session.rooms` are mapped to a `RoomTileViewModel` and then sorted. This gives us fine-grained events at the end of the collection chain that can be easily and efficiently rendered by the `ListView` component.

From 0ca0230bc083c57b2304b1351dd7c4ed6bad6c39 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Thu, 4 Apr 2019 21:48:26 +0200
Subject: [PATCH 27/83] specify what is missing

---
 doc/architecture.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/architecture.md b/doc/architecture.md
index 2be87d37..a9a2a2f0 100644
--- a/doc/architecture.md
+++ b/doc/architecture.md
@@ -44,4 +44,4 @@ Updates from view models can come in two ways. View models emit a change event,
 
 For view models part of an observable collection (and to be rendered by a ListView), updates can also propagate through the collection and delivered by the ListView to the view in question. This avoids every child component in a ListView having to attach a listener to it's viewModel. This is the mechanism to update the room name in a RoomTile in the room list for example.
 
-TODO: specify how the collection based updates work.
+TODO: specify how the collection based updates work. (not specified yet, we'd need a way to derive a key from a value to emit an update from within a collection, but haven't found a nice way of specifying that in an api)

From 749bdadf74cb13ca1c5118938a8b01dc7a0ccff3 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Thu, 18 Apr 2019 20:19:09 +0200
Subject: [PATCH 28/83] disable failing tests for now and include fragment
 index so it's tests are found

---
 src/matrix/room/persister.js              |   5 +-
 src/matrix/room/timeline/FragmentIndex.js | 230 ++++++++++++++++++++++
 2 files changed, 233 insertions(+), 2 deletions(-)
 create mode 100644 src/matrix/room/timeline/FragmentIndex.js

diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js
index abcdf1bd..be065b34 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/persister.js
@@ -1,4 +1,5 @@
 import SortKey from "../storage/sortkey.js";
+import FragmentIndex from "./timeline/FragmentIndex.js";
 
 function gapEntriesAreEqual(a, b) {
     if (!a || !b || !a.gap || !b.gap) {
@@ -190,9 +191,9 @@ export default class RoomPersister {
 }
 
 //#ifdef TESTS
-import MemoryStorage from "../storage/memory/MemoryStorage.js";
+//import MemoryStorage from "../storage/memory/MemoryStorage.js";
 
-export function tests() {
+export function xtests() {
     const roomId = "!abc:hs.tld";
 
     // sets sortKey and roomId on an array of entries
diff --git a/src/matrix/room/timeline/FragmentIndex.js b/src/matrix/room/timeline/FragmentIndex.js
new file mode 100644
index 00000000..4be2b315
--- /dev/null
+++ b/src/matrix/room/timeline/FragmentIndex.js
@@ -0,0 +1,230 @@
+class Fragment {
+    constructor(previousId, nextId) {
+        this.previousId = previousId;
+        this.nextId = nextId;
+    }
+}
+
+/*
+lookups will be far more frequent than changing fragment order,
+so data structure should be optimized for fast lookup
+
+we can have a Map: fragmentId to sortIndex
+
+changing the order, we would need to rebuild the index
+lets do this the stupid way for now, changing any fragment rebuilds all islands
+
+to build this:
+first load all fragments
+put them in a map by id
+now iterate through them
+
+until no more fragments
+    get the first
+    create an island array, and add to list with islands
+    going backwards and forwards
+        get and remove sibling and prepend/append it to island array
+        stop when no more previous/next
+    return list with islands
+
+*/
+
+
+function findBackwardSiblingFragments(current, byId) {
+    const sortedSiblings = [];
+    while (current.previousId) {
+        const previous = byId.get(current.previousId);
+        if (!previous) {
+            throw new Error(`Unknown previousId ${current.previousId} on ${current.id}`);
+        }
+        if (previous.nextId !== current.id) {
+            throw new Error(`Previous fragment ${previous.id} doesn't point back to ${current.id}`);
+        }
+        byId.delete(current.previousId);
+        sortedSiblings.push(previous);
+        current = previous;
+    }
+    sortedSiblings.reverse();
+    return sortedSiblings;
+}
+
+function findForwardSiblingFragments(current, byId) {
+    const sortedSiblings = [];
+    while (current.nextId) {
+        const next = byId.get(current.nextId);
+        if (!next) {
+            throw new Error(`Unknown nextId ${current.nextId} on ${current.id}`);
+        }
+        if (next.previousId !== current.id) {
+            throw new Error(`Next fragment ${next.id} doesn't point back to ${current.id}`);
+        }
+        byId.delete(current.nextId);
+        sortedSiblings.push(next);
+        current = next;
+    }
+    return sortedSiblings;
+}
+
+
+function createIslands(fragments) {
+    const byId = new Map();
+    for(let f of fragments) {
+        byId.set(f.id, f);
+    }
+
+    const islands = [];
+    while(byId.size) {
+        const current = byId.values().next().value;
+        byId.delete(current.id);
+        // new island
+        const previousSiblings = findBackwardSiblingFragments(current, byId);
+        const nextSiblings = findForwardSiblingFragments(current, byId);
+        const island = previousSiblings.concat(current, nextSiblings);
+        islands.push(island);
+    }
+    return islands.map(a => new Island(a));
+}
+
+class Island {
+    constructor(sortedFragments) {
+        this._idToSortIndex = new Map();
+        sortedFragments.forEach((f, i) => {
+            this._idToSortIndex.set(f.id, i);
+        });
+    }
+
+    compareIds(idA, idB) {
+        const sortIndexA = this._idToSortIndex.get(idA);
+        if (sortIndexA === undefined) {
+            throw new Error(`first id ${idA} isn't part of this island`);
+        }
+        const sortIndexB = this._idToSortIndex.get(idB);
+        if (sortIndexB === undefined) {
+            throw new Error(`second id ${idB} isn't part of this island`);
+        }
+        return sortIndexA - sortIndexB;
+    }
+
+    get fragmentIds() {
+        return this._idToSortIndex.keys();
+    }
+}
+
+/*
+index for fast lookup of how two fragments can be sorted
+*/
+export default class FragmentIndex {
+    constructor(fragments) {
+        this.rebuild(fragments);
+    }
+
+    _getIsland(id) {
+        const island = this._idToIsland.get(id);
+        if (island === undefined) {
+            throw new Error(`Unknown fragment id ${id}`);
+        }
+        return island;
+    }
+
+    compareIds(idA, idB) {
+        if (idA === idB) {
+            return 0;
+        }
+        const islandA = this._getIsland(idA);
+        const islandB = this._getIsland(idB);
+        if (islandA !== islandB) {
+            throw new Error(`${idA} and ${idB} are on different islands, can't tell order`);
+        }
+        return islandA.compareIds(idA, idB);
+    }
+
+    rebuild(fragments) {
+        const islands = createIslands(fragments);
+        this._idToIsland = new Map();
+        for(let island of islands) {
+            for(let id of island.fragmentIds) {
+                this._idToIsland.set(id, island);
+            }
+        }
+    }
+    // maybe actual persistence shouldn't be done here, just allocate fragment ids and sorting
+
+    // we need to check here that the fragment we think we are appending to doesn't already have a nextId.
+    // otherwise we could create a corrupt state (two fragments not pointing at each other).
+
+    // allocates a fragment id within the live range, that can be compared to each other without a mapping as they are allocated in chronological order
+    // appendLiveFragment(txn, previousToken) {
+
+    // }
+
+    // newFragment(txn, previousToken, nextToken) {
+
+    // }
+
+    // linkFragments(txn, firstFragmentId, secondFragmentId) {
+
+    // }
+}
+
+export function tests() {
+    return {
+        test_1_island_3_fragments(assert) {
+            const index = new FragmentIndex([
+                {id: 3, previousId: 2},
+                {id: 1, nextId: 2},
+                {id: 2, nextId: 3, previousId: 1},
+            ]);
+            assert(index.compareIds(1, 2) < 0);
+            assert(index.compareIds(2, 1) > 0);
+
+            assert(index.compareIds(1, 3) < 0);
+            assert(index.compareIds(3, 1) > 0);
+            
+            assert(index.compareIds(2, 3) < 0);
+            assert(index.compareIds(3, 2) > 0);
+            
+            assert.equal(index.compareIds(1, 1), 0);
+        },
+        test_2_island_dont_compare(assert) {
+            const index = new FragmentIndex([
+                {id: 1},
+                {id: 2},
+            ]);
+            assert.throws(() => index.compareIds(1, 2));
+            assert.throws(() => index.compareIds(2, 1));
+        },
+        test_2_island_compare_internally(assert) {
+            const index = new FragmentIndex([
+                {id: 1, nextId: 2},
+                {id: 2, previousId: 1},
+                {id: 11, nextId: 12},
+                {id: 12, previousId: 11},
+                
+            ]);
+
+            assert(index.compareIds(1, 2) < 0);
+            assert(index.compareIds(11, 12) < 0);
+            
+            assert.throws(() => index.compareIds(1, 11));
+            assert.throws(() => index.compareIds(12, 2));
+        },
+        test_unknown_id(assert) {
+            const index = new FragmentIndex([{id: 1}]);
+            assert.throws(() => index.compareIds(1, 2));
+            assert.throws(() => index.compareIds(2, 1));
+        },
+        test_rebuild_flushes_old_state(assert) {
+            const index = new FragmentIndex([
+                {id: 1, nextId: 2},
+                {id: 2, previousId: 1},
+            ]);
+            index.rebuild([
+                {id: 11, nextId: 12},
+                {id: 12, previousId: 11},
+            ]);
+            
+            assert.throws(() => index.compareIds(1, 2));
+            assert(index.compareIds(11, 12) < 0);
+        },
+    }
+}

From 06d2d2e198275750dce8c04ec11c63c1b1650014 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Thu, 18 Apr 2019 20:19:43 +0200
Subject: [PATCH 29/83] draft of idb store for fragments

---
 src/matrix/storage/idb/create.js | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/src/matrix/storage/idb/create.js b/src/matrix/storage/idb/create.js
index 42f66e04..185f4992 100644
--- a/src/matrix/storage/idb/create.js
+++ b/src/matrix/storage/idb/create.js
@@ -10,10 +10,12 @@ function createStores(db) {
 	db.createObjectStore("session", {keyPath: "key"});
 	// any way to make keys unique here? (just use put?)
 	db.createObjectStore("roomSummary", {keyPath: "roomId"});
+
+    db.createObjectStore("timelineFragments", {keyPath: ["roomId", "id"]});
 	// needs roomId separate because it might hold a gap and no event
-	const timeline = db.createObjectStore("roomTimeline", {keyPath: ["roomId", "sortKey"]});
-	timeline.createIndex("byEventId", [
-		"roomId",
+	const timelineEvents = db.createObjectStore("timelineEvents", {keyPath: ["event.room_id", "fragmentId", "sortKey"]});
+	timelineEvents.createIndex("byEventId", [
+		"event.room_id",
 		"event.event_id"
 	], {unique: true});
 
@@ -29,4 +31,4 @@ function createStores(db) {
 	// 	"event.state_key"
 	// ]});
 	// roomMembers.createIndex("byName", ["room_id", "content.name"]);
-}
\ No newline at end of file
+}

From 8670ab6331b9fe6d38709111ae13c09bc1ce8012 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Thu, 18 Apr 2019 20:20:23 +0200
Subject: [PATCH 30/83] add todo for fragments

---
 doc/FRAGMENTS.md | 10 ++++++++++
 1 file changed, 10 insertions(+)
 create mode 100644 doc/FRAGMENTS.md

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
new file mode 100644
index 00000000..fd082898
--- /dev/null
+++ b/doc/FRAGMENTS.md
@@ -0,0 +1,10 @@
+ - DONE: write FragmentIndex
+ - adapt SortKey
+ - write fragmentStore
+ - adapt timelineStore
+ - adapt persister
+    - persist fragments in /sync
+    - load n items before and after key
+    - fill gaps / fragment filling
+ - add live fragment id optimization if we haven't done so already
+ - lets try to not have to have the fragmentindex in memory if the timeline isn't loaded

From d90411a6dd3f796bf8c44cb3d30d6ddedc0fbc8b Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Wed, 1 May 2019 14:47:39 +0200
Subject: [PATCH 31/83] adjust SortKey to have fragmentKey instead of gapKey

with FragmentIndex to compare fragment keys
---
 doc/FRAGMENTS.md                              |   7 +-
 src/matrix/room/persister.js                  |   4 +-
 src/matrix/room/timeline/FragmentIndex.js     |  49 ++---
 src/matrix/room/timeline/SortKey.js           | 197 ++++++++++++++++++
 .../storage/idb/stores/RoomTimelineStore.js   |   2 +-
 .../memory/stores/RoomTimelineStore.js        |   4 +-
 src/matrix/storage/sortkey.js                 | 181 ----------------
 7 files changed, 231 insertions(+), 213 deletions(-)
 create mode 100644 src/matrix/room/timeline/SortKey.js
 delete mode 100644 src/matrix/storage/sortkey.js

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
index fd082898..a12d1264 100644
--- a/doc/FRAGMENTS.md
+++ b/doc/FRAGMENTS.md
@@ -1,5 +1,10 @@
  - DONE: write FragmentIndex
- - adapt SortKey
+ - adapt SortKey ... naming! :
+    - FragmentIndex (index as in db index)
+        - compare(fragmentKeyA, fragmentKeyB)
+    - SortKey
+        - FragmentKey
+        - EventKey (we don't use id here because we already have event_id in the event)
  - write fragmentStore
  - adapt timelineStore
  - adapt persister
diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js
index be065b34..11b94145 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/persister.js
@@ -1,4 +1,4 @@
-import SortKey from "../storage/sortkey.js";
+import SortKey from "./timeline/SortKey.js";
 import FragmentIndex from "./timeline/FragmentIndex.js";
 
 function gapEntriesAreEqual(a, b) {
@@ -31,6 +31,7 @@ export default class RoomPersister {
 	constructor({roomId, storage}) {
 		this._roomId = roomId;
         this._storage = storage;
+        // TODO: load fragmentIndex?
 		this._lastSortKey = new SortKey();
 	}
 
@@ -38,6 +39,7 @@ export default class RoomPersister {
 		//fetch key here instead?
 		const [lastEvent] = await txn.roomTimeline.lastEvents(this._roomId, 1);
 		if (lastEvent) {
+            // TODO: load fragmentIndex?
 			this._lastSortKey = new SortKey(lastEvent.sortKey);
 			console.log("room persister load", this._roomId, this._lastSortKey.toString());
 		} else {
diff --git a/src/matrix/room/timeline/FragmentIndex.js b/src/matrix/room/timeline/FragmentIndex.js
index 4be2b315..6c31e2d2 100644
--- a/src/matrix/room/timeline/FragmentIndex.js
+++ b/src/matrix/room/timeline/FragmentIndex.js
@@ -1,10 +1,3 @@
-class Fragment {
-    constructor(previousId, nextId) {
-        this.previousId = previousId;
-        this.nextId = nextId;
-    }
-}
-
 /*
 lookups will be far more frequent than changing fragment order,
 so data structure should be optimized for fast lookup
@@ -93,7 +86,7 @@ class Island {
         });
     }
 
-    compareIds(idA, idB) {
+    compare(idA, idB) {
         const sortIndexA = this._idToSortIndex.get(idA);
         if (sortIndexA === undefined) {
             throw new Error(`first id ${idA} isn't part of this island`);
@@ -126,7 +119,7 @@ export default class FragmentIndex {
         return island;
     }
 
-    compareIds(idA, idB) {
+    compare(idA, idB) {
         if (idA === idB) {
             return 0;
         }
@@ -135,7 +128,7 @@ export default class FragmentIndex {
         if (islandA !== islandB) {
             throw new Error(`${idA} and ${idB} are on different islands, can't tell order`);
         }
-        return islandA.compareIds(idA, idB);
+        return islandA.compare(idA, idB);
     }
 
     rebuild(fragments) {
@@ -166,6 +159,7 @@ export default class FragmentIndex {
     // }
 }
 
+//#ifdef TESTS
 export function tests() {
     return {
         test_1_island_3_fragments(assert) {
@@ -174,24 +168,24 @@ export function tests() {
                 {id: 1, nextId: 2},
                 {id: 2, nextId: 3, previousId: 1},
             ]);
-            assert(index.compareIds(1, 2) < 0);
-            assert(index.compareIds(2, 1) > 0);
+            assert(index.compare(1, 2) < 0);
+            assert(index.compare(2, 1) > 0);
 
-            assert(index.compareIds(1, 3) < 0);
-            assert(index.compareIds(3, 1) > 0);
+            assert(index.compare(1, 3) < 0);
+            assert(index.compare(3, 1) > 0);
             
-            assert(index.compareIds(2, 3) < 0);
-            assert(index.compareIds(3, 2) > 0);
+            assert(index.compare(2, 3) < 0);
+            assert(index.compare(3, 2) > 0);
             
-            assert.equal(index.compareIds(1, 1), 0);
+            assert.equal(index.compare(1, 1), 0);
         },
         test_2_island_dont_compare(assert) {
             const index = new FragmentIndex([
                 {id: 1},
                 {id: 2},
             ]);
-            assert.throws(() => index.compareIds(1, 2));
-            assert.throws(() => index.compareIds(2, 1));
+            assert.throws(() => index.compare(1, 2));
+            assert.throws(() => index.compare(2, 1));
         },
         test_2_island_compare_internally(assert) {
             const index = new FragmentIndex([
@@ -202,16 +196,16 @@ export function tests() {
                 
             ]);
 
-            assert(index.compareIds(1, 2) < 0);
-            assert(index.compareIds(11, 12) < 0);
+            assert(index.compare(1, 2) < 0);
+            assert(index.compare(11, 12) < 0);
             
-            assert.throws(() => index.compareIds(1, 11));
-            assert.throws(() => index.compareIds(12, 2));
+            assert.throws(() => index.compare(1, 11));
+            assert.throws(() => index.compare(12, 2));
         },
         test_unknown_id(assert) {
             const index = new FragmentIndex([{id: 1}]);
-            assert.throws(() => index.compareIds(1, 2));
-            assert.throws(() => index.compareIds(2, 1));
+            assert.throws(() => index.compare(1, 2));
+            assert.throws(() => index.compare(2, 1));
         },
         test_rebuild_flushes_old_state(assert) {
             const index = new FragmentIndex([
@@ -223,8 +217,9 @@ export function tests() {
                 {id: 12, previousId: 11},
             ]);
             
-            assert.throws(() => index.compareIds(1, 2));
-            assert(index.compareIds(11, 12) < 0);
+            assert.throws(() => index.compare(1, 2));
+            assert(index.compare(11, 12) < 0);
         },
     }
 }
+//#endif
diff --git a/src/matrix/room/timeline/SortKey.js b/src/matrix/room/timeline/SortKey.js
new file mode 100644
index 00000000..6ca6b549
--- /dev/null
+++ b/src/matrix/room/timeline/SortKey.js
@@ -0,0 +1,197 @@
+const MIN_INT32 = -2147483648;
+const MID_INT32 = 0;
+const MAX_INT32 = 2147483647;
+
+const MIN_UINT32 = 0;
+const MID_UINT32 = 2147483647;
+const MAX_UINT32 = 4294967295;
+
+const MIN = MIN_UINT32;
+const MID = MID_UINT32;
+const MAX = MAX_UINT32;
+
+export default class SortKey {
+	constructor(fragmentIndex, buffer) {
+		if (buffer) {
+			this._keys = new DataView(buffer);
+		} else {
+			this._keys = new DataView(new ArrayBuffer(8));
+			// start default key right at the middle fragment key, min event key
+			// so we have the same amount of key address space either way
+			this.fragmentKey = MID;
+			this.eventKey = MIN;
+		}
+        this._fragmentIndex = fragmentIndex;
+	}
+
+	get fragmentKey() {
+		return this._keys.getUint32(0, false);
+	}
+
+	set fragmentKey(value) {
+		return this._keys.setUint32(0, value, false);
+	}
+
+	get eventKey() {
+		return this._keys.getUint32(4, false);
+	}
+
+	set eventKey(value) {
+		return this._keys.setUint32(4, value, false);
+	}
+
+	get buffer() {
+		return this._keys.buffer;
+	}
+
+	nextFragmentKey() {
+		const k = new SortKey(this._fragmentIndex);
+		k.fragmentKey = this.fragmentKey + 1;
+		k.eventKey = MIN;
+		return k;
+	}
+
+	nextKey() {
+		const k = new SortKey(this._fragmentIndex);
+		k.fragmentKey = this.fragmentKey;
+		k.eventKey = this.eventKey + 1;
+		return k;
+	}
+
+	previousKey() {
+		const k = new SortKey(this._fragmentIndex);
+		k.fragmentKey = this.fragmentKey;
+		k.eventKey = this.eventKey - 1;
+		return k;
+	}
+
+	clone() {
+		const k = new SortKey();
+		k.fragmentKey = this.fragmentKey;
+		k.eventKey = this.eventKey;	
+		return k;
+	}
+
+	static get maxKey() {
+		const maxKey = new SortKey(null);
+		maxKey.fragmentKey = MAX;
+		maxKey.eventKey = MAX;
+		return maxKey;
+	}
+
+	static get minKey() {
+		const minKey = new SortKey(null);
+		minKey.fragmentKey = MIN;
+		minKey.eventKey = MIN;
+		return minKey;
+	}
+
+    compare(otherKey) {
+        const fragmentDiff = this.fragmentKey - otherKey.fragmentKey;
+        if (fragmentDiff === 0) {
+            return this.eventKey - otherKey.eventKey;
+        } else {
+            // minKey and maxKey might not have fragmentIndex, so short-circuit this first ...
+            if (this.fragmentKey === MIN || otherKey.fragmentKey === MAX) {
+                return -1;
+            }
+            if (this.fragmentKey === MAX || otherKey.fragmentKey === MIN) {
+                return 1;
+            }
+            // ... then delegate to fragmentIndex.
+            // This might throw if the relation of two fragments is unknown.
+            return this._fragmentIndex.compare(this.fragmentKey, otherKey.fragmentKey);
+        }
+    }
+
+	toString() {
+		return `[${this.fragmentKey}/${this.eventKey}]`;
+	}
+}
+
+//#ifdef TESTS
+export function tests() {
+    const fragmentIndex = {compare: (a, b) => a - b};
+
+	return {
+		test_default_key(assert) {
+			const k = new SortKey(fragmentIndex);
+			assert.equal(k.fragmentKey, MID);
+			assert.equal(k.eventKey, MIN);
+		},
+
+		test_inc(assert) {
+			const a = new SortKey(fragmentIndex);
+			const b = a.nextKey();
+			assert.equal(a.fragmentKey, b.fragmentKey);
+			assert.equal(a.eventKey + 1, b.eventKey);
+			const c = b.previousKey();
+			assert.equal(b.fragmentKey, c.fragmentKey);
+			assert.equal(c.eventKey + 1, b.eventKey);
+			assert.equal(a.eventKey, c.eventKey);
+		},
+
+		test_min_key(assert) {
+			const minKey = SortKey.minKey;
+			const k = new SortKey(fragmentIndex);
+			assert(minKey.fragmentKey <= k.fragmentKey);
+			assert(minKey.eventKey <= k.eventKey);
+            assert(k.compare(minKey) > 0);
+            assert(minKey.compare(k) < 0);
+		},
+
+		test_max_key(assert) {
+			const maxKey = SortKey.maxKey;
+			const k = new SortKey(fragmentIndex);
+			assert(maxKey.fragmentKey >= k.fragmentKey);
+			assert(maxKey.eventKey >= k.eventKey);
+            assert(k.compare(maxKey) < 0);
+            assert(maxKey.compare(k) > 0);
+		},
+
+		test_immutable(assert) {
+			const a = new SortKey(fragmentIndex);
+			const fragmentKey = a.fragmentKey;
+			const eventKey = a.eventKey;
+			a.nextFragmentKey();
+			assert.equal(a.fragmentKey, fragmentKey);
+			assert.equal(a.eventKey, eventKey);
+		},
+
+		test_cmp_fragmentkey_first(assert) {
+			const a = new SortKey(fragmentIndex);
+			const b = new SortKey(fragmentIndex);
+			a.fragmentKey = 2;
+			a.eventKey = 1;
+			b.fragmentKey = 1;
+			b.eventKey = 100000;
+			assert(a.compare(b) > 0);
+		},
+
+		test_cmp_eventkey_second(assert) {
+			const a = new SortKey(fragmentIndex);
+			const b = new SortKey(fragmentIndex);
+			a.fragmentKey = 1;
+			a.eventKey = 100000;
+			b.fragmentKey = 1;
+			b.eventKey = 2;
+			assert(a.compare(b) > 0);
+		},
+
+		test_cmp_max_larger_than_min(assert) {
+			assert(SortKey.minKey.compare(SortKey.maxKey) < 0);
+		},
+
+		test_cmp_fragmentkey_first_large(assert) {
+			const a = new SortKey(fragmentIndex);
+			const b = new SortKey(fragmentIndex);
+			a.fragmentKey = MAX;
+			a.eventKey = MIN;
+			b.fragmentKey = MIN;
+			b.eventKey = MAX;
+			assert(b < a);
+			assert(a > b);
+		}
+	};
+}
+//#endif
diff --git a/src/matrix/storage/idb/stores/RoomTimelineStore.js b/src/matrix/storage/idb/stores/RoomTimelineStore.js
index 491c1f10..8b47a501 100644
--- a/src/matrix/storage/idb/stores/RoomTimelineStore.js
+++ b/src/matrix/storage/idb/stores/RoomTimelineStore.js
@@ -1,4 +1,4 @@
-import SortKey from "../../sortkey.js";
+import SortKey from "../../../room/timeline/SortKey.js";
 
 class Range {
     constructor(only, lower, upper, lowerOpen, upperOpen) {
diff --git a/src/matrix/storage/memory/stores/RoomTimelineStore.js b/src/matrix/storage/memory/stores/RoomTimelineStore.js
index 40c521cb..3c17c045 100644
--- a/src/matrix/storage/memory/stores/RoomTimelineStore.js
+++ b/src/matrix/storage/memory/stores/RoomTimelineStore.js
@@ -1,6 +1,6 @@
-import SortKey from "../sortkey.js";
+import SortKey from "../../room/timeline/SortKey.js";
 import sortedIndex from "../../../utils/sortedIndex.js";
-import Store from "./Store";
+import Store from "./Store.js";
 
 function compareKeys(key, entry) {
     if (key.roomId === entry.roomId) {
diff --git a/src/matrix/storage/sortkey.js b/src/matrix/storage/sortkey.js
deleted file mode 100644
index bfc15a5f..00000000
--- a/src/matrix/storage/sortkey.js
+++ /dev/null
@@ -1,181 +0,0 @@
-const MIN_INT32 = -2147483648;
-const MID_INT32 = 0;
-const MAX_INT32 = 2147483647;
-
-const MIN_UINT32 = 0;
-const MID_UINT32 = 2147483647;
-const MAX_UINT32 = 4294967295;
-
-const MIN = MIN_UINT32;
-const MID = MID_UINT32;
-const MAX = MAX_UINT32;
-
-export default class SortKey {
-	constructor(buffer) {
-		if (buffer) {
-			this._keys = new DataView(buffer);
-		} else {
-			this._keys = new DataView(new ArrayBuffer(8));
-			// start default key right at the middle gap key, min event key
-			// so we have the same amount of key address space either way
-			this.gapKey = MID;
-			this.eventKey = MIN;
-		}
-	}
-
-	get gapKey() {
-		return this._keys.getUint32(0, false);
-	}
-
-	set gapKey(value) {
-		return this._keys.setUint32(0, value, false);
-	}
-
-	get eventKey() {
-		return this._keys.getUint32(4, false);
-	}
-
-	set eventKey(value) {
-		return this._keys.setUint32(4, value, false);
-	}
-
-	get buffer() {
-		return this._keys.buffer;
-	}
-
-	nextKeyWithGap() {
-		const k = new SortKey();
-		k.gapKey = this.gapKey + 1;
-		k.eventKey = MIN;
-		return k;
-	}
-
-	nextKey() {
-		const k = new SortKey();
-		k.gapKey = this.gapKey;
-		k.eventKey = this.eventKey + 1;
-		return k;
-	}
-
-	previousKey() {
-		const k = new SortKey();
-		k.gapKey = this.gapKey;
-		k.eventKey = this.eventKey - 1;
-		return k;
-	}
-
-	clone() {
-		const k = new SortKey();
-		k.gapKey = this.gapKey;
-		k.eventKey = this.eventKey;	
-		return k;
-	}
-
-	static get maxKey() {
-		const maxKey = new SortKey();
-		maxKey.gapKey = MAX;
-		maxKey.eventKey = MAX;
-		return maxKey;
-	}
-
-	static get minKey() {
-		const minKey = new SortKey();
-		minKey.gapKey = MIN;
-		minKey.eventKey = MIN;
-		return minKey;
-	}
-
-    compare(otherKey) {
-        const gapDiff = this.gapKey - otherKey.gapKey;
-        if (gapDiff === 0) {
-            return this.eventKey - otherKey.eventKey;
-        } else {
-            return gapDiff;
-        }
-    }
-
-	toString() {
-		return `[${this.gapKey}/${this.eventKey}]`;
-	}
-}
-
-//#ifdef TESTS
-export function tests() {
-	return {
-		test_default_key(assert) {
-			const k = new SortKey();
-			assert.equal(k.gapKey, MID);
-			assert.equal(k.eventKey, MIN);
-		},
-
-		test_inc(assert) {
-			const a = new SortKey();
-			const b = a.nextKey();
-			assert.equal(a.gapKey, b.gapKey);
-			assert.equal(a.eventKey + 1, b.eventKey);
-			const c = b.previousKey();
-			assert.equal(b.gapKey, c.gapKey);
-			assert.equal(c.eventKey + 1, b.eventKey);
-			assert.equal(a.eventKey, c.eventKey);
-		},
-
-		test_min_key(assert) {
-			const minKey = SortKey.minKey;
-			const k = new SortKey();
-			assert(minKey.gapKey <= k.gapKey);
-			assert(minKey.eventKey <= k.eventKey);
-		},
-
-		test_max_key(assert) {
-			const maxKey = SortKey.maxKey;
-			const k = new SortKey();
-			assert(maxKey.gapKey >= k.gapKey);
-			assert(maxKey.eventKey >= k.eventKey);
-		},
-
-		test_immutable(assert) {
-			const a = new SortKey();
-			const gapKey = a.gapKey;
-			const eventKey = a.eventKey;
-			a.nextKeyWithGap();
-			assert.equal(a.gapKey, gapKey);
-			assert.equal(a.eventKey, eventKey);
-		},
-
-		test_cmp_gapkey_first(assert) {
-			const a = new SortKey();
-			const b = new SortKey();
-			a.gapKey = 2;
-			a.eventKey = 1;
-			b.gapKey = 1;
-			b.eventKey = 100000;
-			assert(a.compare(b) > 0);
-		},
-
-		test_cmp_eventkey_second(assert) {
-			const a = new SortKey();
-			const b = new SortKey();
-			a.gapKey = 1;
-			a.eventKey = 100000;
-			b.gapKey = 1;
-			b.eventKey = 2;
-			assert(a.compare(b) > 0);
-		},
-
-		test_cmp_max_larger_than_min(assert) {
-			assert(SortKey.minKey.compare(SortKey.maxKey) < 0);
-		},
-
-		test_cmp_gapkey_first_large(assert) {
-			const a = new SortKey();
-			const b = new SortKey();
-			a.gapKey = MAX;
-			a.eventKey = MIN;
-			b.gapKey = MIN;
-			b.eventKey = MAX;
-			assert(b < a);
-			assert(a > b);
-		}
-	};
-}
-//#endif

From 99c8816bf9d8f94239df0baec86536ffb81d24e4 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Wed, 1 May 2019 15:36:32 +0200
Subject: [PATCH 32/83] better naming FragmentIndex -> FragmentIdIndex (index
 of fragment ids,    not an index number in a fragment collection) EventKey ->
 EventIndex (implies being ordered) FragmentKey -> FragmentId   (implies not
 being ordered, hence FragmentIdIndex)

---
 doc/FRAGMENTS.md                              |  10 +-
 .../{FragmentIndex.js => FragmentIdIndex.js}  |  12 +-
 src/matrix/room/timeline/SortKey.js           | 146 +++++++++---------
 3 files changed, 84 insertions(+), 84 deletions(-)
 rename src/matrix/room/timeline/{FragmentIndex.js => FragmentIdIndex.js} (95%)

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
index a12d1264..83f48a8d 100644
--- a/doc/FRAGMENTS.md
+++ b/doc/FRAGMENTS.md
@@ -1,10 +1,10 @@
  - DONE: write FragmentIndex
- - adapt SortKey ... naming! :
-    - FragmentIndex (index as in db index)
-        - compare(fragmentKeyA, fragmentKeyB)
+ - DONE: adapt SortKey ... naming! :
+    - FragmentIdIndex (index as in db index)
+        - compare(idA, idB)
     - SortKey
-        - FragmentKey
-        - EventKey (we don't use id here because we already have event_id in the event)
+        - FragmentId
+        - EventIndex
  - write fragmentStore
  - adapt timelineStore
  - adapt persister
diff --git a/src/matrix/room/timeline/FragmentIndex.js b/src/matrix/room/timeline/FragmentIdIndex.js
similarity index 95%
rename from src/matrix/room/timeline/FragmentIndex.js
rename to src/matrix/room/timeline/FragmentIdIndex.js
index 6c31e2d2..e67be9c2 100644
--- a/src/matrix/room/timeline/FragmentIndex.js
+++ b/src/matrix/room/timeline/FragmentIdIndex.js
@@ -106,7 +106,7 @@ class Island {
 /*
 index for fast lookup of how two fragments can be sorted
 */
-export default class FragmentIndex {
+export default class FragmentIdIndex {
     constructor(fragments) {
         this.rebuild(fragments);
     }
@@ -163,7 +163,7 @@ export default class FragmentIndex {
 export function tests() {
     return {
         test_1_island_3_fragments(assert) {
-            const index = new FragmentIndex([
+            const index = new FragmentIdIndex([
                 {id: 3, previousId: 2},
                 {id: 1, nextId: 2},
                 {id: 2, nextId: 3, previousId: 1},
@@ -180,7 +180,7 @@ export function tests() {
             assert.equal(index.compare(1, 1), 0);
         },
         test_2_island_dont_compare(assert) {
-            const index = new FragmentIndex([
+            const index = new FragmentIdIndex([
                 {id: 1},
                 {id: 2},
             ]);
@@ -188,7 +188,7 @@ export function tests() {
             assert.throws(() => index.compare(2, 1));
         },
         test_2_island_compare_internally(assert) {
-            const index = new FragmentIndex([
+            const index = new FragmentIdIndex([
                 {id: 1, nextId: 2},
                 {id: 2, previousId: 1},
                 {id: 11, nextId: 12},
@@ -203,12 +203,12 @@ export function tests() {
             assert.throws(() => index.compare(12, 2));
         },
         test_unknown_id(assert) {
-            const index = new FragmentIndex([{id: 1}]);
+            const index = new FragmentIdIndex([{id: 1}]);
             assert.throws(() => index.compare(1, 2));
             assert.throws(() => index.compare(2, 1));
         },
         test_rebuild_flushes_old_state(assert) {
-            const index = new FragmentIndex([
+            const index = new FragmentIdIndex([
                 {id: 1, nextId: 2},
                 {id: 2, previousId: 1},
             ]);
diff --git a/src/matrix/room/timeline/SortKey.js b/src/matrix/room/timeline/SortKey.js
index 6ca6b549..54f82304 100644
--- a/src/matrix/room/timeline/SortKey.js
+++ b/src/matrix/room/timeline/SortKey.js
@@ -11,32 +11,32 @@ const MID = MID_UINT32;
 const MAX = MAX_UINT32;
 
 export default class SortKey {
-	constructor(fragmentIndex, buffer) {
+	constructor(fragmentIdComparer, buffer) {
 		if (buffer) {
 			this._keys = new DataView(buffer);
 		} else {
 			this._keys = new DataView(new ArrayBuffer(8));
 			// start default key right at the middle fragment key, min event key
 			// so we have the same amount of key address space either way
-			this.fragmentKey = MID;
-			this.eventKey = MIN;
+			this.fragmentId = MID;
+			this.eventIndex = MIN;
 		}
-        this._fragmentIndex = fragmentIndex;
+        this._fragmentIdComparer = fragmentIdComparer;
 	}
 
-	get fragmentKey() {
+	get fragmentId() {
 		return this._keys.getUint32(0, false);
 	}
 
-	set fragmentKey(value) {
+	set fragmentId(value) {
 		return this._keys.setUint32(0, value, false);
 	}
 
-	get eventKey() {
+	get eventIndex() {
 		return this._keys.getUint32(4, false);
 	}
 
-	set eventKey(value) {
+	set eventIndex(value) {
 		return this._keys.setUint32(4, value, false);
 	}
 
@@ -45,136 +45,136 @@ export default class SortKey {
 	}
 
 	nextFragmentKey() {
-		const k = new SortKey(this._fragmentIndex);
-		k.fragmentKey = this.fragmentKey + 1;
-		k.eventKey = MIN;
+		const k = new SortKey(this._fragmentIdComparer);
+		k.fragmentId = this.fragmentId + 1;
+		k.eventIndex = MIN;
 		return k;
 	}
 
 	nextKey() {
-		const k = new SortKey(this._fragmentIndex);
-		k.fragmentKey = this.fragmentKey;
-		k.eventKey = this.eventKey + 1;
+		const k = new SortKey(this._fragmentIdComparer);
+		k.fragmentId = this.fragmentId;
+		k.eventIndex = this.eventIndex + 1;
 		return k;
 	}
 
 	previousKey() {
-		const k = new SortKey(this._fragmentIndex);
-		k.fragmentKey = this.fragmentKey;
-		k.eventKey = this.eventKey - 1;
+		const k = new SortKey(this._fragmentIdComparer);
+		k.fragmentId = this.fragmentId;
+		k.eventIndex = this.eventIndex - 1;
 		return k;
 	}
 
 	clone() {
 		const k = new SortKey();
-		k.fragmentKey = this.fragmentKey;
-		k.eventKey = this.eventKey;	
+		k.fragmentId = this.fragmentId;
+		k.eventIndex = this.eventIndex;	
 		return k;
 	}
 
 	static get maxKey() {
 		const maxKey = new SortKey(null);
-		maxKey.fragmentKey = MAX;
-		maxKey.eventKey = MAX;
+		maxKey.fragmentId = MAX;
+		maxKey.eventIndex = MAX;
 		return maxKey;
 	}
 
 	static get minKey() {
 		const minKey = new SortKey(null);
-		minKey.fragmentKey = MIN;
-		minKey.eventKey = MIN;
+		minKey.fragmentId = MIN;
+		minKey.eventIndex = MIN;
 		return minKey;
 	}
 
     compare(otherKey) {
-        const fragmentDiff = this.fragmentKey - otherKey.fragmentKey;
+        const fragmentDiff = this.fragmentId - otherKey.fragmentId;
         if (fragmentDiff === 0) {
-            return this.eventKey - otherKey.eventKey;
+            return this.eventIndex - otherKey.eventIndex;
         } else {
-            // minKey and maxKey might not have fragmentIndex, so short-circuit this first ...
-            if (this.fragmentKey === MIN || otherKey.fragmentKey === MAX) {
+            // minKey and maxKey might not have fragmentIdComparer, so short-circuit this first ...
+            if (this.fragmentId === MIN || otherKey.fragmentId === MAX) {
                 return -1;
             }
-            if (this.fragmentKey === MAX || otherKey.fragmentKey === MIN) {
+            if (this.fragmentId === MAX || otherKey.fragmentId === MIN) {
                 return 1;
             }
-            // ... then delegate to fragmentIndex.
+            // ... then delegate to fragmentIdComparer.
             // This might throw if the relation of two fragments is unknown.
-            return this._fragmentIndex.compare(this.fragmentKey, otherKey.fragmentKey);
+            return this._fragmentIdComparer.compare(this.fragmentId, otherKey.fragmentId);
         }
     }
 
 	toString() {
-		return `[${this.fragmentKey}/${this.eventKey}]`;
+		return `[${this.fragmentId}/${this.eventIndex}]`;
 	}
 }
 
 //#ifdef TESTS
 export function tests() {
-    const fragmentIndex = {compare: (a, b) => a - b};
+    const fragmentIdComparer = {compare: (a, b) => a - b};
 
 	return {
 		test_default_key(assert) {
-			const k = new SortKey(fragmentIndex);
-			assert.equal(k.fragmentKey, MID);
-			assert.equal(k.eventKey, MIN);
+			const k = new SortKey(fragmentIdComparer);
+			assert.equal(k.fragmentId, MID);
+			assert.equal(k.eventIndex, MIN);
 		},
 
 		test_inc(assert) {
-			const a = new SortKey(fragmentIndex);
+			const a = new SortKey(fragmentIdComparer);
 			const b = a.nextKey();
-			assert.equal(a.fragmentKey, b.fragmentKey);
-			assert.equal(a.eventKey + 1, b.eventKey);
+			assert.equal(a.fragmentId, b.fragmentId);
+			assert.equal(a.eventIndex + 1, b.eventIndex);
 			const c = b.previousKey();
-			assert.equal(b.fragmentKey, c.fragmentKey);
-			assert.equal(c.eventKey + 1, b.eventKey);
-			assert.equal(a.eventKey, c.eventKey);
+			assert.equal(b.fragmentId, c.fragmentId);
+			assert.equal(c.eventIndex + 1, b.eventIndex);
+			assert.equal(a.eventIndex, c.eventIndex);
 		},
 
 		test_min_key(assert) {
 			const minKey = SortKey.minKey;
-			const k = new SortKey(fragmentIndex);
-			assert(minKey.fragmentKey <= k.fragmentKey);
-			assert(minKey.eventKey <= k.eventKey);
+			const k = new SortKey(fragmentIdComparer);
+			assert(minKey.fragmentId <= k.fragmentId);
+			assert(minKey.eventIndex <= k.eventIndex);
             assert(k.compare(minKey) > 0);
             assert(minKey.compare(k) < 0);
 		},
 
 		test_max_key(assert) {
 			const maxKey = SortKey.maxKey;
-			const k = new SortKey(fragmentIndex);
-			assert(maxKey.fragmentKey >= k.fragmentKey);
-			assert(maxKey.eventKey >= k.eventKey);
+			const k = new SortKey(fragmentIdComparer);
+			assert(maxKey.fragmentId >= k.fragmentId);
+			assert(maxKey.eventIndex >= k.eventIndex);
             assert(k.compare(maxKey) < 0);
             assert(maxKey.compare(k) > 0);
 		},
 
 		test_immutable(assert) {
-			const a = new SortKey(fragmentIndex);
-			const fragmentKey = a.fragmentKey;
-			const eventKey = a.eventKey;
+			const a = new SortKey(fragmentIdComparer);
+			const fragmentId = a.fragmentId;
+			const eventIndex = a.eventIndex;
 			a.nextFragmentKey();
-			assert.equal(a.fragmentKey, fragmentKey);
-			assert.equal(a.eventKey, eventKey);
+			assert.equal(a.fragmentId, fragmentId);
+			assert.equal(a.eventIndex, eventIndex);
 		},
 
-		test_cmp_fragmentkey_first(assert) {
-			const a = new SortKey(fragmentIndex);
-			const b = new SortKey(fragmentIndex);
-			a.fragmentKey = 2;
-			a.eventKey = 1;
-			b.fragmentKey = 1;
-			b.eventKey = 100000;
+		test_cmp_fragmentid_first(assert) {
+			const a = new SortKey(fragmentIdComparer);
+			const b = new SortKey(fragmentIdComparer);
+			a.fragmentId = 2;
+			a.eventIndex = 1;
+			b.fragmentId = 1;
+			b.eventIndex = 100000;
 			assert(a.compare(b) > 0);
 		},
 
-		test_cmp_eventkey_second(assert) {
-			const a = new SortKey(fragmentIndex);
-			const b = new SortKey(fragmentIndex);
-			a.fragmentKey = 1;
-			a.eventKey = 100000;
-			b.fragmentKey = 1;
-			b.eventKey = 2;
+		test_cmp_eventindex_second(assert) {
+			const a = new SortKey(fragmentIdComparer);
+			const b = new SortKey(fragmentIdComparer);
+			a.fragmentId = 1;
+			a.eventIndex = 100000;
+			b.fragmentId = 1;
+			b.eventIndex = 2;
 			assert(a.compare(b) > 0);
 		},
 
@@ -182,13 +182,13 @@ export function tests() {
 			assert(SortKey.minKey.compare(SortKey.maxKey) < 0);
 		},
 
-		test_cmp_fragmentkey_first_large(assert) {
-			const a = new SortKey(fragmentIndex);
-			const b = new SortKey(fragmentIndex);
-			a.fragmentKey = MAX;
-			a.eventKey = MIN;
-			b.fragmentKey = MIN;
-			b.eventKey = MAX;
+		test_cmp_fragmentid_first_large(assert) {
+			const a = new SortKey(fragmentIdComparer);
+			const b = new SortKey(fragmentIdComparer);
+			a.fragmentId = MAX;
+			a.eventIndex = MIN;
+			b.fragmentId = MIN;
+			b.eventIndex = MAX;
 			assert(b < a);
 			assert(a > b);
 		}

From d6ae313bbd93871ecd7ed009a917bb0976392da7 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 11 May 2019 09:51:27 +0200
Subject: [PATCH 33/83] make compare rely less on fragment index

also indenting
---
 src/matrix/room/timeline/SortKey.js | 292 +++++++++++++++-------------
 1 file changed, 157 insertions(+), 135 deletions(-)

diff --git a/src/matrix/room/timeline/SortKey.js b/src/matrix/room/timeline/SortKey.js
index 54f82304..efddc67b 100644
--- a/src/matrix/room/timeline/SortKey.js
+++ b/src/matrix/room/timeline/SortKey.js
@@ -11,80 +11,80 @@ const MID = MID_UINT32;
 const MAX = MAX_UINT32;
 
 export default class SortKey {
-	constructor(fragmentIdComparer, buffer) {
-		if (buffer) {
-			this._keys = new DataView(buffer);
-		} else {
-			this._keys = new DataView(new ArrayBuffer(8));
-			// start default key right at the middle fragment key, min event key
-			// so we have the same amount of key address space either way
-			this.fragmentId = MID;
-			this.eventIndex = MIN;
-		}
+    constructor(fragmentIdComparer, buffer) {
+        if (buffer) {
+            this._keys = new DataView(buffer);
+        } else {
+            this._keys = new DataView(new ArrayBuffer(8));
+            // start default key right at the middle fragment key, min event key
+            // so we have the same amount of key address space either way
+            this.fragmentId = MID;
+            this.eventIndex = MID;
+        }
         this._fragmentIdComparer = fragmentIdComparer;
-	}
+    }
 
-	get fragmentId() {
-		return this._keys.getUint32(0, false);
-	}
+    get fragmentId() {
+        return this._keys.getUint32(0, false);
+    }
 
-	set fragmentId(value) {
-		return this._keys.setUint32(0, value, false);
-	}
+    set fragmentId(value) {
+        return this._keys.setUint32(0, value, false);
+    }
 
-	get eventIndex() {
-		return this._keys.getUint32(4, false);
-	}
+    get eventIndex() {
+        return this._keys.getUint32(4, false);
+    }
 
-	set eventIndex(value) {
-		return this._keys.setUint32(4, value, false);
-	}
+    set eventIndex(value) {
+        return this._keys.setUint32(4, value, false);
+    }
 
-	get buffer() {
-		return this._keys.buffer;
-	}
+    get buffer() {
+        return this._keys.buffer;
+    }
 
-	nextFragmentKey() {
-		const k = new SortKey(this._fragmentIdComparer);
-		k.fragmentId = this.fragmentId + 1;
-		k.eventIndex = MIN;
-		return k;
-	}
+    nextFragmentKey() {
+        const k = new SortKey(this._fragmentIdComparer);
+        k.fragmentId = this.fragmentId + 1;
+        k.eventIndex = MIN;
+        return k;
+    }
 
-	nextKey() {
-		const k = new SortKey(this._fragmentIdComparer);
-		k.fragmentId = this.fragmentId;
-		k.eventIndex = this.eventIndex + 1;
-		return k;
-	}
+    nextKey() {
+        const k = new SortKey(this._fragmentIdComparer);
+        k.fragmentId = this.fragmentId;
+        k.eventIndex = this.eventIndex + 1;
+        return k;
+    }
 
-	previousKey() {
-		const k = new SortKey(this._fragmentIdComparer);
-		k.fragmentId = this.fragmentId;
-		k.eventIndex = this.eventIndex - 1;
-		return k;
-	}
+    previousKey() {
+        const k = new SortKey(this._fragmentIdComparer);
+        k.fragmentId = this.fragmentId;
+        k.eventIndex = this.eventIndex - 1;
+        return k;
+    }
 
-	clone() {
-		const k = new SortKey();
-		k.fragmentId = this.fragmentId;
-		k.eventIndex = this.eventIndex;	
-		return k;
-	}
+    clone() {
+        const k = new SortKey();
+        k.fragmentId = this.fragmentId;
+        k.eventIndex = this.eventIndex; 
+        return k;
+    }
 
-	static get maxKey() {
-		const maxKey = new SortKey(null);
-		maxKey.fragmentId = MAX;
-		maxKey.eventIndex = MAX;
-		return maxKey;
-	}
+    static get maxKey() {
+        const maxKey = new SortKey(null);
+        maxKey.fragmentId = MAX;
+        maxKey.eventIndex = MAX;
+        return maxKey;
+    }
 
-	static get minKey() {
-		const minKey = new SortKey(null);
-		minKey.fragmentId = MIN;
-		minKey.eventIndex = MIN;
-		return minKey;
-	}
+    static get minKey() {
+        const minKey = new SortKey(null);
+        minKey.fragmentId = MIN;
+        minKey.eventIndex = MIN;
+        return minKey;
+    }
 
     compare(otherKey) {
         const fragmentDiff = this.fragmentId - otherKey.fragmentId;
@@ -92,10 +92,10 @@ export default class SortKey {
             return this.eventIndex - otherKey.eventIndex;
         } else {
             // minKey and maxKey might not have fragmentIdComparer, so short-circuit this first ...
-            if (this.fragmentId === MIN || otherKey.fragmentId === MAX) {
+            if ((this.fragmentId === MIN && otherKey.fragmentId !== MIN) || (this.fragmentId !== MAX && otherKey.fragmentId === MAX)) {
                 return -1;
             }
-            if (this.fragmentId === MAX || otherKey.fragmentId === MIN) {
+            if ((this.fragmentId === MAX && otherKey.fragmentId !== MAX) || (this.fragmentId !== MIN && otherKey.fragmentId === MIN)) {
                 return 1;
             }
             // ... then delegate to fragmentIdComparer.
@@ -104,94 +104,116 @@ export default class SortKey {
         }
     }
 
-	toString() {
-		return `[${this.fragmentId}/${this.eventIndex}]`;
-	}
+    toString() {
+        return `[${this.fragmentId}/${this.eventIndex}]`;
+    }
 }
 
 //#ifdef TESTS
 export function tests() {
     const fragmentIdComparer = {compare: (a, b) => a - b};
 
-	return {
-		test_default_key(assert) {
-			const k = new SortKey(fragmentIdComparer);
-			assert.equal(k.fragmentId, MID);
-			assert.equal(k.eventIndex, MIN);
-		},
+    return {
+        test_no_fragment_index(assert) {
+            const min = SortKey.minKey;
+            const max = SortKey.maxKey;
+            const a = new SortKey();
+            a.eventIndex = 1;
+            a.fragmentId = 1;
 
-		test_inc(assert) {
-			const a = new SortKey(fragmentIdComparer);
-			const b = a.nextKey();
-			assert.equal(a.fragmentId, b.fragmentId);
-			assert.equal(a.eventIndex + 1, b.eventIndex);
-			const c = b.previousKey();
-			assert.equal(b.fragmentId, c.fragmentId);
-			assert.equal(c.eventIndex + 1, b.eventIndex);
-			assert.equal(a.eventIndex, c.eventIndex);
-		},
+            assert(min.compare(min) === 0);
+            assert(max.compare(max) === 0);
+            assert(a.compare(a) === 0);
 
-		test_min_key(assert) {
-			const minKey = SortKey.minKey;
-			const k = new SortKey(fragmentIdComparer);
-			assert(minKey.fragmentId <= k.fragmentId);
-			assert(minKey.eventIndex <= k.eventIndex);
+            assert(min.compare(max) < 0);
+            assert(max.compare(min) > 0);
+
+            assert(min.compare(a) < 0);
+            assert(a.compare(min) > 0);
+            
+            assert(max.compare(a) > 0);
+            assert(a.compare(max) < 0);
+        },
+
+        test_default_key(assert) {
+            const k = new SortKey(fragmentIdComparer);
+            assert.equal(k.fragmentId, MID);
+            assert.equal(k.eventIndex, MID);
+        },
+
+        test_inc(assert) {
+            const a = new SortKey(fragmentIdComparer);
+            const b = a.nextKey();
+            assert.equal(a.fragmentId, b.fragmentId);
+            assert.equal(a.eventIndex + 1, b.eventIndex);
+            const c = b.previousKey();
+            assert.equal(b.fragmentId, c.fragmentId);
+            assert.equal(c.eventIndex + 1, b.eventIndex);
+            assert.equal(a.eventIndex, c.eventIndex);
+        },
+
+        test_min_key(assert) {
+            const minKey = SortKey.minKey;
+            const k = new SortKey(fragmentIdComparer);
+            assert(minKey.fragmentId <= k.fragmentId);
+            assert(minKey.eventIndex <= k.eventIndex);
             assert(k.compare(minKey) > 0);
             assert(minKey.compare(k) < 0);
-		},
+        },
 
-		test_max_key(assert) {
-			const maxKey = SortKey.maxKey;
-			const k = new SortKey(fragmentIdComparer);
-			assert(maxKey.fragmentId >= k.fragmentId);
-			assert(maxKey.eventIndex >= k.eventIndex);
+        test_max_key(assert) {
+            const maxKey = SortKey.maxKey;
+            const k = new SortKey(fragmentIdComparer);
+            assert(maxKey.fragmentId >= k.fragmentId);
+            assert(maxKey.eventIndex >= k.eventIndex);
             assert(k.compare(maxKey) < 0);
             assert(maxKey.compare(k) > 0);
-		},
+        },
 
-		test_immutable(assert) {
-			const a = new SortKey(fragmentIdComparer);
-			const fragmentId = a.fragmentId;
-			const eventIndex = a.eventIndex;
-			a.nextFragmentKey();
-			assert.equal(a.fragmentId, fragmentId);
-			assert.equal(a.eventIndex, eventIndex);
-		},
+        test_immutable(assert) {
+            const a = new SortKey(fragmentIdComparer);
+            const fragmentId = a.fragmentId;
+            const eventIndex = a.eventIndex;
+            a.nextFragmentKey();
+            assert.equal(a.fragmentId, fragmentId);
+            assert.equal(a.eventIndex, eventIndex);
+        },
 
-		test_cmp_fragmentid_first(assert) {
-			const a = new SortKey(fragmentIdComparer);
-			const b = new SortKey(fragmentIdComparer);
-			a.fragmentId = 2;
-			a.eventIndex = 1;
-			b.fragmentId = 1;
-			b.eventIndex = 100000;
-			assert(a.compare(b) > 0);
-		},
+        test_cmp_fragmentid_first(assert) {
+            const a = new SortKey(fragmentIdComparer);
+            const b = new SortKey(fragmentIdComparer);
+            a.fragmentId = 2;
+            a.eventIndex = 1;
+            b.fragmentId = 1;
+            b.eventIndex = 100000;
+            assert(a.compare(b) > 0);
+        },
 
-		test_cmp_eventindex_second(assert) {
-			const a = new SortKey(fragmentIdComparer);
-			const b = new SortKey(fragmentIdComparer);
-			a.fragmentId = 1;
-			a.eventIndex = 100000;
-			b.fragmentId = 1;
-			b.eventIndex = 2;
-			assert(a.compare(b) > 0);
-		},
+        test_cmp_eventindex_second(assert) {
+            const a = new SortKey(fragmentIdComparer);
+            const b = new SortKey(fragmentIdComparer);
+            a.fragmentId = 1;
+            a.eventIndex = 100000;
+            b.fragmentId = 1;
+            b.eventIndex = 2;
+            assert(a.compare(b) > 0);
+            assert(b.compare(a) < 0);
+        },
 
-		test_cmp_max_larger_than_min(assert) {
-			assert(SortKey.minKey.compare(SortKey.maxKey) < 0);
-		},
+        test_cmp_max_larger_than_min(assert) {
+            assert(SortKey.minKey.compare(SortKey.maxKey) < 0);
+        },
 
-		test_cmp_fragmentid_first_large(assert) {
-			const a = new SortKey(fragmentIdComparer);
-			const b = new SortKey(fragmentIdComparer);
-			a.fragmentId = MAX;
-			a.eventIndex = MIN;
-			b.fragmentId = MIN;
-			b.eventIndex = MAX;
-			assert(b < a);
-			assert(a > b);
-		}
-	};
+        test_cmp_fragmentid_first_large(assert) {
+            const a = new SortKey(fragmentIdComparer);
+            const b = new SortKey(fragmentIdComparer);
+            a.fragmentId = MAX;
+            a.eventIndex = MIN;
+            b.fragmentId = MIN;
+            b.eventIndex = MAX;
+            assert(b < a);
+            assert(a > b);
+        }
+    };
 }
 //#endif

From 35a5e3f21a0c1456e6ed8698491fcba9936ababc Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 11 May 2019 09:51:57 +0200
Subject: [PATCH 34/83] docs update

---
 doc/FRAGMENTS.md    | 30 ++++++++++++++++++++++++++++++
 doc/architecture.md |  4 ++--
 2 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
index 83f48a8d..cc28cf98 100644
--- a/doc/FRAGMENTS.md
+++ b/doc/FRAGMENTS.md
@@ -6,10 +6,40 @@
         - FragmentId
         - EventIndex
  - write fragmentStore
+    - load all fragments
+    - add a fragment (live on limited sync, or /context)
+    - connect two fragments
+    - update token on fragment (when filling gap or connecting two fragments)
+
+    fragments can need connecting when filling a gap or creating a new /context fragment
  - adapt timelineStore
+
+    how will fragments be exposed in timeline store?
+        - all read operations are passed a fragment id
  - adapt persister
     - persist fragments in /sync
     - load n items before and after key
     - fill gaps / fragment filling
  - add live fragment id optimization if we haven't done so already
  - lets try to not have to have the fragmentindex in memory if the timeline isn't loaded
+    - could do this by only loading all fragments into index when filling gaps, backpaginating, ... and on persister load only load the last fragment. This wouldn't even need a FragmentIndex?
+
+
+so a gap is two connected fragments where either the first fragment has a nextToken and/or the second fragment has a previousToken. It can be both, so we can have a gap where you can fill in from the top, from the bottom (like when limited sync) or both.
+
+
+
+
+also, filling gaps and storing /context, how do we find the fragment we could potentially merge with to look for overlapping events?
+
+with /sync this is all fine and dandy, but with /context is there a way where we don't need to look up every event_id in the store to see if it's already there?
+    we can do a anyOf(event_id) on timelineStore.index("by_index") by sorting the event ids according to IndexedDb.cmp and passing the next value to cursor.continue(nextId).
+
+so we'll need to remove previous/nextEvent on the timeline store and come up with a method to find the first matched event in a list of eventIds.
+    so we'll need to map all event ids to an event and return the first one that is not null. If we haven't read all events but we know that all the previous ones are null, then we can already return the result. 
+
+    we can call this findFirstEventIn(roomId, [event ids])
+
+thoughts:
+    - ranges in timeline store with fragmentId might not make sense anymore as doing queries over multiple fragment ids doesn't make sense anymore ... still makes sense to have them part of SortKey though ...
+    - we need a test for querytarget::lookup, or make sure it works well ...
diff --git a/doc/architecture.md b/doc/architecture.md
index a9a2a2f0..286113d5 100644
--- a/doc/architecture.md
+++ b/doc/architecture.md
@@ -18,7 +18,7 @@ let fragment := {
     id: number
     previousId: number?
     nextId: number?
-    prevToken: string?
+    previousToken: string?
     nextToken: string?
 }
 ```
@@ -27,7 +27,7 @@ let fragment := {
 
 `Room`s on the `Session` are exposed as an `ObservableMap` collection, which is like an ordinary `Map` but emits events when it is modified (here when a room is added, removed, or the properties of a room change). `ObservableMap` can have different operators applied to it like `mapValues()`, `filterValues()` each returning a new `ObservableMap`-like object, and also `sortValues()` returning an `ObservableList` (emitting events when a room at an index is added, removed, moved or changes properties).
 
-So example, for the room list, `Room` objects from `Session.rooms` are mapped to a `RoomTileViewModel` and then sorted. This gives us fine-grained events at the end of the collection chain that can be easily and efficiently rendered by the `ListView` component.
+So for example, the room list, `Room` objects from `Session.rooms` are mapped to a `RoomTileViewModel` and then sorted. This gives us fine-grained events at the end of the collection chain that can be easily and efficiently rendered by the `ListView` component.
 
 On that note, view components are just a simple convention, having these methods:
 

From 53cdabb459d0da32059afb38396b10a8e349bba1 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 11 May 2019 13:10:31 +0200
Subject: [PATCH 35/83] store method to find events to connect with when
 filling gaps

as fragments can be unaware of their chronological relationship,
we need to check whether the events received from /messages or /context
already exists, so we can later hook up the fragments.
---
 prototypes/idb-continue-key.html              | 165 ++++++++++++++++++
 src/matrix/storage/idb/query-target.js        |  44 ++++-
 .../storage/idb/stores/RoomTimelineStore.js   |  53 ++++--
 src/matrix/storage/idb/utils.js               |  10 +-
 4 files changed, 248 insertions(+), 24 deletions(-)
 create mode 100644 prototypes/idb-continue-key.html

diff --git a/prototypes/idb-continue-key.html b/prototypes/idb-continue-key.html
new file mode 100644
index 00000000..dc488deb
--- /dev/null
+++ b/prototypes/idb-continue-key.html
@@ -0,0 +1,165 @@
+<html>
+<head><meta charset="utf-8"></head>
+<body>
+	<script type="text/javascript">
+        function reqAsPromise(req) {
+            return new Promise((resolve, reject) => {
+                req.onsuccess = () => resolve(req);
+                req.onerror = (err) => reject(err);
+            });
+        }
+
+        function txnAsPromise(txn) {
+            return new Promise((resolve, reject) => {
+                txn.addEventListener("complete", resolve);
+                txn.addEventListener("abort", reject);
+            });
+        }
+
+        function iterateCursor(cursor, processValue) {
+            // TODO: does cursor already have a value here??
+            return new Promise((resolve, reject) => {
+                cursor.onerror = (event) => {
+                    reject(new Error("Query failed: " + event.target.errorCode));
+                };
+                // collect results
+                cursor.onsuccess = (event) => {
+                    const cursor = event.target.result;
+                    if (!cursor) {
+                        resolve(false);
+                        return; // end of results
+                    }
+                    const {done, jumpTo} = processValue(cursor.value, cursor.key);
+                    if (done) {
+                        resolve(true);
+                    } else {
+                        cursor.continue(jumpTo);
+                    }
+                };
+            });
+        }
+
+        /**
+         * Checks if a given set of keys exist.
+         * Calls `callback(key, found)` for each key in `keys`, in an unspecified order.
+         * If the callback returns true, the search is halted and callback won't be called again.
+        */
+        async function findKeys(target, keys, backwards, callback) {
+            const direction = backwards ? "prev" : "next";
+            const compareKeys = (a, b) => backwards ? -indexedDB.cmp(a, b) : indexedDB.cmp(a, b);
+            const sortedKeys = keys.slice().sort(compareKeys);
+            console.log(sortedKeys);
+            const firstKey = backwards ? sortedKeys[sortedKeys.length - 1] : sortedKeys[0];
+            const lastKey = backwards ? sortedKeys[0] : sortedKeys[sortedKeys.length - 1];
+            const cursor = target.openKeyCursor(IDBKeyRange.bound(firstKey, lastKey), direction);
+            let i = 0;
+            let consumerDone = false;
+            await iterateCursor(cursor, (value, key) => {
+                // while key is larger than next key, advance and report false
+                while(i < sortedKeys.length && compareKeys(sortedKeys[i], key) < 0 && !consumerDone) {
+                    console.log("before match", sortedKeys[i]);
+                    consumerDone = callback(sortedKeys[i], false);
+                    ++i;
+                }
+                if (i < sortedKeys.length && compareKeys(sortedKeys[i], key) === 0 && !consumerDone) {
+                    console.log("match", sortedKeys[i]);
+                    consumerDone = callback(sortedKeys[i], true);
+                    ++i;
+                }
+                const done = consumerDone || i >= sortedKeys.length;
+                const jumpTo = !done && sortedKeys[i];
+                return {done, jumpTo};
+            });
+            // report null for keys we didn't to at the end
+            while (!consumerDone && i < sortedKeys.length) {
+                console.log("afterwards", sortedKeys[i]);
+                consumerDone = callback(sortedKeys[i], false);
+                ++i;
+            }
+        }
+
+        async function findFirstOrLastOccurringEventId(target, roomId, eventIds, findLast = false) {
+            const keys = eventIds.map(eventId => [roomId, eventId]);
+            const results = new Array(keys.length);
+            let firstFoundEventId;
+
+            // find first result that is found and has no undefined results before it
+            function firstFoundAndPrecedingResolved() {
+                let inc = findLast ? -1 : 1;
+                let start = findLast ? results.length - 1 : 0;
+                for(let i = start; i >= 0 && i < results.length; i += inc) {
+                    if (results[i] === undefined) {
+                        return;
+                    } else if(results[i] === true) {
+                        return keys[i];
+                    }
+                }
+            }
+
+            await findKeys(target, keys, findLast, (key, found) => {
+                const index = keys.indexOf(key);
+                results[index] = found;
+                firstFoundEventId = firstFoundAndPrecedingResolved();
+                return !!firstFoundEventId;
+            });
+
+            return firstFoundEventId;
+        }
+
+        (async () => {
+            let db;
+            let isNew = false;
+            // open db
+            {
+                const req = window.indexedDB.open("prototype-idb-continue-key");
+                req.onupgradeneeded = (ev) => {
+                    const db = ev.target.result;
+                    db.createObjectStore("timeline", {keyPath: ["roomId", "eventId"]});
+                    isNew = true;
+                };
+                db = (await reqAsPromise(req)).result;
+            }
+            const roomId = "!abcdef:localhost";
+            if (isNew) {
+                const txn = db.transaction(["timeline"], "readwrite");
+                const store = txn.objectStore("timeline");
+                for (var i = 1; i <= 100; ++i) {
+                    store.add({roomId, eventId: `$${i * 10}`});
+                }
+                await txnAsPromise(txn);
+            }
+            console.log("show all in order we get them");
+            {
+                const txn = db.transaction(["timeline"], "readonly");
+                const store = txn.objectStore("timeline");
+                const cursor = store.openKeyCursor();
+                await iterateCursor(cursor, (value, key) => {
+                    console.log(key);
+                    return {done: false};
+                });
+            }
+            console.log("run findKeys");
+            {
+                const txn = db.transaction(["timeline"], "readonly");
+                const store = txn.objectStore("timeline");
+                const eventIds = ["$992", "$1000", "$1010", "$991", "$500", "$990"];
+                // const eventIds = ["$992", "$1010"];
+                const keys = eventIds.map(eventId => [roomId, eventId]);
+                await findKeys(store, keys, false, (key, found) => {
+                    console.log(key, found);
+                });
+            }
+            console.log("run findFirstOrLastOccurringEventId");
+            {
+                const txn = db.transaction(["timeline"], "readonly");
+                const store = txn.objectStore("timeline");
+                const eventIds = ["$992", "$1000", "$1010", "$991", "$500", "$990", "$123"];
+                const firstMatch = await findFirstOrLastOccurringEventId(store, roomId, eventIds, false);
+                console.log("firstMatch", firstMatch);
+                const lastMatch = await findFirstOrLastOccurringEventId(store, roomId, eventIds, true);
+                console.log("lastMatch", lastMatch);
+            }
+        })();
+	</script>
+</body>
+</html>
diff --git a/src/matrix/storage/idb/query-target.js b/src/matrix/storage/idb/query-target.js
index 547c2fdd..ec5c4530 100644
--- a/src/matrix/storage/idb/query-target.js
+++ b/src/matrix/storage/idb/query-target.js
@@ -38,7 +38,7 @@ export default class QueryTarget {
 		const results = [];
 		await iterateCursor(cursor, (value) => {
 			results.push(value);
-			return false;
+			return {done: false};
 		});
 		return results;
 	}
@@ -59,12 +59,48 @@ export default class QueryTarget {
 		return this._find(range, predicate, "prev");
 	}
 
+    /**
+     * Checks if a given set of keys exist.
+     * Calls `callback(key, found)` for each key in `keys`, in key sorting order (or reversed if backwards=true).
+     * If the callback returns true, the search is halted and callback won't be called again.
+     * `callback` is called with the same instances of the key as given in `keys`, so direct comparison can be used.
+     */
+    async findExistingKeys(keys, backwards, callback) {
+        const direction = backwards ? "prev" : "next";
+        const compareKeys = (a, b) => backwards ? -indexedDB.cmp(a, b) : indexedDB.cmp(a, b);
+        const sortedKeys = keys.slice().sort(compareKeys);
+        const firstKey = backwards ? sortedKeys[sortedKeys.length - 1] : sortedKeys[0];
+        const lastKey = backwards ? sortedKeys[0] : sortedKeys[sortedKeys.length - 1];
+        const cursor = this._target.openKeyCursor(IDBKeyRange.bound(firstKey, lastKey), direction);
+        let i = 0;
+        let consumerDone = false;
+        await iterateCursor(cursor, (value, key) => {
+            // while key is larger than next key, advance and report false
+            while(i < sortedKeys.length && compareKeys(sortedKeys[i], key) < 0 && !consumerDone) {
+                consumerDone = callback(sortedKeys[i], false);
+                ++i;
+            }
+            if (i < sortedKeys.length && compareKeys(sortedKeys[i], key) === 0 && !consumerDone) {
+                consumerDone = callback(sortedKeys[i], true);
+                ++i;
+            }
+            const done = consumerDone || i >= sortedKeys.length;
+            const jumpTo = !done && sortedKeys[i];
+            return {done, jumpTo};
+        });
+        // report null for keys we didn't to at the end
+        while (!consumerDone && i < sortedKeys.length) {
+            consumerDone = callback(sortedKeys[i], false);
+            ++i;
+        }
+    }
+
 	_reduce(range, reducer, initialValue, direction) {
 		let reducedValue = initialValue;
 		const cursor = this._target.openCursor(range, direction);
 		return iterateCursor(cursor, (value) => {
 			reducedValue = reducer(reducedValue, value);
-			return true;
+			return {done: false};
 		});
 	}
 
@@ -79,7 +115,7 @@ export default class QueryTarget {
 		const results = [];
 		await iterateCursor(cursor, (value) => {
 			results.push(value);
-			return predicate(results);
+			return {done: predicate(results)};
 		});
 		return results;
 	}
@@ -92,7 +128,7 @@ export default class QueryTarget {
 			if (found) {
 				result = value;
 			}
-			return found;
+			return {done: found};
 		});
 		if (found) {
 			return result;
diff --git a/src/matrix/storage/idb/stores/RoomTimelineStore.js b/src/matrix/storage/idb/stores/RoomTimelineStore.js
index 8b47a501..950685d7 100644
--- a/src/matrix/storage/idb/stores/RoomTimelineStore.js
+++ b/src/matrix/storage/idb/stores/RoomTimelineStore.js
@@ -147,24 +147,47 @@ export default class RoomTimelineStore {
         return events;
     }
 
-    /** Looks up the first, if any, event entry (so excluding gap entries) after `sortKey`.
+    /** Finds the first (or last if `findLast=true`) eventId that occurs in the store, if any.
+     *  For optimal performance, `eventIds` should be in chronological order.
+     *
+     *  The order in which results are returned might be different than `eventIds`.
+     *  Call the return value to obtain the next {id, event} pair.
      *  @param  {string} roomId
-     *  @param  {SortKey} sortKey
-     *  @return {Promise<(?Entry)>} a promise resolving to entry, if any.
+     *  @param  {string[]} eventIds
+     *  @return {Function<Promise>}
      */
-    nextEvent(roomId, sortKey) {
-        const range = this.lowerBoundRange(sortKey, true).asIDBKeyRange(roomId);
-        return this._timelineStore.find(range, entry => !!entry.event);
-    }
+    // performance comment from above refers to the fact that their *might*
+    // be a correlation between event_id sorting order and chronology.
+    // In that case we could avoid running over all eventIds, as the reported order by findExistingKeys
+    // would match the order of eventIds. That's why findLast is also passed as backwards to keysExist.
+    // also passing them in chronological order makes sense as that's how we'll receive them almost always.
+    async findFirstOrLastOccurringEventId(roomId, eventIds, findLast = false) {
+        const byEventId = this._timelineStore.index("byEventId");
+        const keys = eventIds.map(eventId => [roomId, eventId]);
+        const results = new Array(keys.length);
+        let firstFoundEventId;
 
-    /** Looks up the first, if any, event entry (so excluding gap entries) before `sortKey`.
-     *  @param  {string} roomId
-     *  @param  {SortKey} sortKey
-     *  @return {Promise<(?Entry)>} a promise resolving to entry, if any.
-     */
-    previousEvent(roomId, sortKey) {
-        const range = this.upperBoundRange(sortKey, true).asIDBKeyRange(roomId);
-        return this._timelineStore.findReverse(range, entry => !!entry.event);
+        // find first result that is found and has no undefined results before it
+        function firstFoundAndPrecedingResolved() {
+            let inc = findLast ? -1 : 1;
+            let start = findLast ? results.length - 1 : 0;
+            for(let i = start; i >= 0 && i < results.length; i += inc) {
+                if (results[i] === undefined) {
+                    return;
+                } else if(results[i] === true) {
+                    return keys[i];
+                }
+            }
+        }
+
+        await byEventId.findExistingKeys(keys, findLast, (key, found) => {
+            const index = keys.indexOf(key);
+            results[index] = found;
+            firstFoundEventId = firstFoundAndPrecedingResolved();
+            return !!firstFoundEventId;
+        });
+
+        return firstFoundEventId;
     }
 
     /** Inserts a new entry into the store. The combination of roomId and sortKey should not exist yet, or an error is thrown.
diff --git a/src/matrix/storage/idb/utils.js b/src/matrix/storage/idb/utils.js
index 1d522f24..83ef3411 100644
--- a/src/matrix/storage/idb/utils.js
+++ b/src/matrix/storage/idb/utils.js
@@ -35,11 +35,11 @@ export function iterateCursor(cursor, processValue) {
                 resolve(false);
                 return; // end of results
             }
-            const isDone = processValue(cursor.value);
-            if (isDone) {
+            const {done, jumpTo} = processValue(cursor.value, cursor.key);
+            if (done) {
                 resolve(true);
             } else {
-                cursor.continue();
+                cursor.continue(jumpTo);
             }
         };
     });
@@ -49,7 +49,7 @@ export async function fetchResults(cursor, isDone) {
     const results = [];
     await iterateCursor(cursor, (value) => {
         results.push(value);
-        return isDone(results);
+        return {done: isDone(results)};
     });
     return results;
 }
@@ -100,4 +100,4 @@ export async function findStoreValue(db, storeName, toCursor, matchesValue) {
         throw new Error("Value not found");
     }
     return match;
-}
\ No newline at end of file
+}

From 783f39c378d30aac3fb3046375881bcad91ff64c Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 11 May 2019 13:21:21 +0200
Subject: [PATCH 36/83] add fragmentId to methods that need it in timeline
 store

---
 .../storage/idb/stores/RoomTimelineStore.js    | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/src/matrix/storage/idb/stores/RoomTimelineStore.js b/src/matrix/storage/idb/stores/RoomTimelineStore.js
index 950685d7..0a540dd5 100644
--- a/src/matrix/storage/idb/stores/RoomTimelineStore.js
+++ b/src/matrix/storage/idb/stores/RoomTimelineStore.js
@@ -105,23 +105,29 @@ export default class RoomTimelineStore {
 
     /** Looks up the last `amount` entries in the timeline for `roomId`.
      *  @param  {string} roomId
+     *  @param  {number} fragmentId
      *  @param  {number} amount
      *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
      */
-	async lastEvents(roomId, amount) {
-		return this.eventsBefore(roomId, SortKey.maxKey, amount);
+	async lastEvents(roomId, fragmentId, amount) {
+        const sortKey = SortKey.maxKey;
+        sortKey.fragmentId = fragmentId;
+		return this.eventsBefore(roomId, sortKey, amount);
 	}
 
     /** Looks up the first `amount` entries in the timeline for `roomId`.
      *  @param  {string} roomId
+     *  @param  {number} fragmentId
      *  @param  {number} amount
      *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
      */
-	async firstEvents(roomId, amount) {
-		return this.eventsAfter(roomId, SortKey.minKey, amount);
+	async firstEvents(roomId, fragmentId, amount) {
+        const sortKey = SortKey.minKey;
+        sortKey.fragmentId = fragmentId;
+		return this.eventsAfter(roomId, sortKey, amount);
 	}
 
-    /** Looks up `amount` entries after `sortKey` in the timeline for `roomId`.
+    /** Looks up `amount` entries after `sortKey` in the timeline for `roomId` within the same fragment.
      *  The entry for `sortKey` is not included.
      *  @param  {string} roomId
      *  @param  {SortKey} sortKey
@@ -133,7 +139,7 @@ export default class RoomTimelineStore {
 		return this._timelineStore.selectLimit(idbRange, amount);
 	}
 
-    /** Looks up `amount` entries before `sortKey` in the timeline for `roomId`.
+    /** Looks up `amount` entries before `sortKey` in the timeline for `roomId` within the same fragment.
      *  The entry for `sortKey` is not included.
      *  @param  {string} roomId
      *  @param  {SortKey} sortKey

From 41f2224454b2232ab6abe4b9c954bafa555ae75a Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 11 May 2019 13:21:58 +0200
Subject: [PATCH 37/83] get live fragment in persister

---
 src/matrix/room/persister.js | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js
index 11b94145..f25615c5 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/persister.js
@@ -33,18 +33,21 @@ export default class RoomPersister {
         this._storage = storage;
         // TODO: load fragmentIndex?
 		this._lastSortKey = new SortKey();
+		this._lastSortKey = null;
+        this._fragmentIdIndex = new FragmentIdIndex([]);   //only used when timeline is loaded ... e.g. "certain" methods on this class... split up?
 	}
 
 	async load(txn) {
-		//fetch key here instead?
-		const [lastEvent] = await txn.roomTimeline.lastEvents(this._roomId, 1);
-		if (lastEvent) {
-            // TODO: load fragmentIndex?
-			this._lastSortKey = new SortKey(lastEvent.sortKey);
-			console.log("room persister load", this._roomId, this._lastSortKey.toString());
-		} else {
-			console.warn("could not recover last sort key for ", this._roomId);
-		}
+        const liveFragment = await txn.roomFragments.liveFragment(this._roomId);
+        if (liveFragment) {
+            const [lastEvent] = await txn.roomTimeline.lastEvents(this._roomId, liveFragment.id, 1);
+            // last event needs to come from the fragment (e.g. passing the last fragment id)
+            const lastSortKey = new SortKey(this._fragmentIdIndex);
+            lastSortKey.fragmentId = liveFragment.id;
+            lastSortKey.eventIndex = lastEvent.eventIndex;
+            this._lastSortKey = lastSortKey;
+        }
+        console.log("room persister load", this._roomId, this._lastSortKey && this._lastSortKey.toString());
 	}
 
     async persistGapFill(gapEntry, response) {

From 152397a292e2c09b33a41ea0c60f3f78cf2997f3 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 11 May 2019 15:41:09 +0200
Subject: [PATCH 38/83] first impl of idb fragment store

---
 .../storage/idb/stores/RoomFragmentStore.js   | 46 +++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100644 src/matrix/storage/idb/stores/RoomFragmentStore.js

diff --git a/src/matrix/storage/idb/stores/RoomFragmentStore.js b/src/matrix/storage/idb/stores/RoomFragmentStore.js
new file mode 100644
index 00000000..86bd243f
--- /dev/null
+++ b/src/matrix/storage/idb/stores/RoomFragmentStore.js
@@ -0,0 +1,46 @@
+export default class RoomFragmentStore {
+    constructor(store) {
+        this._store = store;
+    }
+
+    _allRange(roomId) {
+        return IDBKeyRange.bound(
+            [roomId, Number.MIN_SAFE_INTEGER],
+            [roomId, Number.MAX_SAFE_INTEGER]
+        );
+    }
+
+    all(roomId) {
+        return this._store.selectAll(this._allRange(roomId));
+    }
+
+    /** Returns the fragment without a nextToken and without nextId,
+    if any, with the largest id if there are multiple (which should not happen) */
+    liveFragment(roomId) {
+        // why do we need this?
+        // Ok, take the case where you've got a /context fragment and a /sync fragment
+        // They are not connected. So, upon loading the persister, which one do we take? We can't sort them ...
+        // we assume that the one without a nextToken and without a nextId is a live one
+        // there should really be only one like this
+
+        // reverse because assuming live fragment has bigger id than non-live ones
+        return this._store.findReverse(this._allRange(roomId), fragment => {
+            return typeof fragment.nextId !== "number" && typeof fragment.nextToken !== "string";
+        });
+    }
+
+    // should generate an id an return it?
+    // depends if we want to do anything smart with fragment ids,
+    // like give them meaning depending on range. not for now probably ...
+    add(fragment) {
+        return this._store.add(fragment);
+    }
+
+    update(fragment) {
+        return this._store.put(fragment);
+    }
+
+    get(roomId, fragmentId) {
+        return this._store.get([roomId, fragmentId]);
+    }
+}

From 2b510b24d90b80de235df7d99c2bdd665e60a477 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 11 May 2019 15:41:46 +0200
Subject: [PATCH 39/83] adjust persister to fragments (untested)

---
 doc/FRAGMENTS.md                    |  16 ++++
 src/matrix/room/persister.js        | 116 ++++++++++++++++++----------
 src/matrix/room/timeline/SortKey.js |   8 ++
 src/matrix/storage/idb/create.js    |  47 +++++------
 4 files changed, 122 insertions(+), 65 deletions(-)

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
index cc28cf98..63a20ee5 100644
--- a/doc/FRAGMENTS.md
+++ b/doc/FRAGMENTS.md
@@ -43,3 +43,19 @@ so we'll need to remove previous/nextEvent on the timeline store and come up wit
 thoughts:
     - ranges in timeline store with fragmentId might not make sense anymore as doing queries over multiple fragment ids doesn't make sense anymore ... still makes sense to have them part of SortKey though ...
     - we need a test for querytarget::lookup, or make sure it works well ...
+
+
+# Reading the timeline with fragments
+
+- what format does the persister return newEntries after persisting sync or a gap fill?
+    - a new fragment can be created during a limited sync
+    - when doing a /context or /messages call, we could have joined with another fragment
+    - don't think we need to describe a result spanning multiple fragments here
+    so:
+
+    in case of limited sync, we just say there was a limited sync, this is the fragment that was created for it so we can show a gap in the timeline
+
+    in case of a gap fill, we need to return what was changed to the fragment (was it joined with another fragment, what's the new token), and which events were actually added.
+
+- where do we translate from fragments to gap entries? and back? in the timeline object?
+    that would make sense, that seems to be the only place we need that translation
diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js
index f25615c5..b837f469 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/persister.js
@@ -1,5 +1,5 @@
 import SortKey from "./timeline/SortKey.js";
-import FragmentIndex from "./timeline/FragmentIndex.js";
+import FragmentIdIndex from "./timeline/FragmentIdIndex.js";
 
 function gapEntriesAreEqual(a, b) {
     if (!a || !b || !a.gap || !b.gap) {
@@ -31,9 +31,7 @@ export default class RoomPersister {
 	constructor({roomId, storage}) {
 		this._roomId = roomId;
         this._storage = storage;
-        // TODO: load fragmentIndex?
-		this._lastSortKey = new SortKey();
-		this._lastSortKey = null;
+		this._lastLiveKey = null;
         this._fragmentIdIndex = new FragmentIdIndex([]);   //only used when timeline is loaded ... e.g. "certain" methods on this class... split up?
 	}
 
@@ -41,13 +39,18 @@ export default class RoomPersister {
         const liveFragment = await txn.roomFragments.liveFragment(this._roomId);
         if (liveFragment) {
             const [lastEvent] = await txn.roomTimeline.lastEvents(this._roomId, liveFragment.id, 1);
-            // last event needs to come from the fragment (e.g. passing the last fragment id)
-            const lastSortKey = new SortKey(this._fragmentIdIndex);
-            lastSortKey.fragmentId = liveFragment.id;
-            lastSortKey.eventIndex = lastEvent.eventIndex;
-            this._lastSortKey = lastSortKey;
+            // sorting and identifying (e.g. sort key and pk to insert) are a bit intertwined here
+            // we could split it up into a SortKey (only with compare) and
+            // a EventKey (no compare or fragment index) with nextkey methods and getters/setters for eventIndex/fragmentId
+            // we probably need to convert from one to the other though, so bother?
+            const lastLiveKey = new SortKey(this._fragmentIdIndex);
+            lastLiveKey.fragmentId = liveFragment.id;
+            lastLiveKey.eventIndex = lastEvent.eventIndex;
+            this._lastLiveKey = lastLiveKey;
         }
-        console.log("room persister load", this._roomId, this._lastSortKey && this._lastSortKey.toString());
+        // if there is no live fragment, we don't create it here because load gets a readonly txn.
+        // this is on purpose, load shouldn't modify the store
+        console.log("room persister load", this._roomId, this._lastLiveKey && this._lastLiveKey.toString());
 	}
 
     async persistGapFill(gapEntry, response) {
@@ -121,16 +124,63 @@ export default class RoomPersister {
         return {newEntries, eventFound};
     }
 
-	persistSync(roomResponse, txn) {
-		let nextKey = this._lastSortKey;
+    async _getLiveFragment(txn, previousToken) {
+        const liveFragment = await txn.roomFragments.liveFragment(this._roomId);
+        if (!liveFragment) {
+            if (!previousToken) {
+                previousToken = null;
+            }
+            let defaultId = SortKey.firstLiveFragmentId;
+            txn.roomFragments.add({
+                roomId: this._roomId,
+                id: defaultId,
+                previousId: null,
+                nextId: null,
+                previousToken: previousToken,
+                nextToken: null
+            });
+            return defaultId;
+        } else {
+            return liveFragment.id;
+        }
+    }
+
+    async _replaceLiveFragment(oldFragmentId, newFragmentId, previousToken, txn) {
+        const oldFragment = await txn.roomFragments.get(oldFragmentId);
+        if (!oldFragment) {
+            throw new Error(`old live fragment doesn't exist: ${oldFragmentId}`);
+        }
+        oldFragment.nextId = newFragmentId;
+        txn.roomFragments.update(oldFragment);
+        txn.roomFragments.add({
+            roomId: this._roomId,
+            id: newFragmentId,
+            previousId: oldFragmentId,
+            nextId: null,
+            previousToken: previousToken,
+            nextToken: null
+        });
+    }
+
+	async persistSync(roomResponse, txn) {
+        // means we haven't synced this room yet (just joined or did initial sync)
+        if (!this._lastLiveKey) {
+            // as this is probably a limited sync, prev_batch should be there
+            // (but don't fail if it isn't, we won't be able to back-paginate though)
+            const fragmentId = await this._getLiveFragment(txn, timeline.prev_batch);
+            this._lastLiveKey = new SortKey(this._fragmentIdIndex);
+            this._lastLiveKey.fragmentId = fragmentId;
+            this._lastLiveKey.eventIndex = SortKey.firstLiveEventIndex;
+        }
+        // replace live fragment for limited sync, *only* if we had a live fragment already
+        else if (timeline.limited) {
+            const oldFragmentId = this._lastLiveKey.fragmentId;
+			this._lastLiveKey = this._lastLiveKey.nextFragmentKey();
+            this._replaceLiveFragment(oldFragmentId, this._lastLiveKey.fragmentId, timeline.prev_batch, txn);
+        }
+		let nextKey = this._lastLiveKey;
 		const timeline = roomResponse.timeline;
         const entries = [];
-		// is limited true for initial sync???? or do we need to handle that as a special case?
-		// I suppose it will, yes
-		if (timeline.limited) {
-			nextKey = nextKey.nextKeyWithGap();
-            entries.push(this._createBackwardGapEntry(nextKey, timeline.prev_batch));
-        }
         if (timeline.events) {
             for(const event of timeline.events) {
                 nextKey = nextKey.nextKey();
@@ -146,7 +196,7 @@ export default class RoomPersister {
 		// succeeded 
 		txn.complete().then(() => {
 			console.log("txn complete, setting key");
-			this._lastSortKey = nextKey;
+			this._lastLiveKey = nextKey;
 		});
 
 		// persist state
@@ -156,7 +206,7 @@ export default class RoomPersister {
 				txn.roomState.setStateEvent(this._roomId, event)
 			}
 		}
-
+        // persist live state events in timeline
 		if (timeline.events) {
 			for (const event of timeline.events) {
 				if (typeof event.state_key === "string") {
@@ -164,33 +214,15 @@ export default class RoomPersister {
 				}
 			}
 		}
+
         return entries;
 	}
 
-    _createBackwardGapEntry(sortKey, prevBatch) {
+    _createEventEntry(key, event) {
         return {
-            roomId: this._roomId,
-            sortKey: sortKey.buffer,
-            event: null,
-            gap: {prev_batch: prevBatch}
-        };
-    }
-
-    _createForwardGapEntry(sortKey, nextBatch) {
-        return {
-            roomId: this._roomId,
-            sortKey: sortKey.buffer,
-            event: null,
-            gap: {next_batch: nextBatch}
-        };
-    }
-
-    _createEventEntry(sortKey, event) {
-        return {
-            roomId: this._roomId,
-            sortKey: sortKey.buffer,
+            fragmentId: key.fragmentId,
+            eventIndex: key.eventIndex,
             event: event,
-            gap: null
         };
     }
 }
diff --git a/src/matrix/room/timeline/SortKey.js b/src/matrix/room/timeline/SortKey.js
index efddc67b..b045ce1c 100644
--- a/src/matrix/room/timeline/SortKey.js
+++ b/src/matrix/room/timeline/SortKey.js
@@ -86,6 +86,14 @@ export default class SortKey {
         return minKey;
     }
 
+    static get firstLiveFragmentId() {
+        return MID;
+    }
+
+    static get firstLiveEventIndex() {
+        return MID;
+    }
+
     compare(otherKey) {
         const fragmentDiff = this.fragmentId - otherKey.fragmentId;
         if (fragmentDiff === 0) {
diff --git a/src/matrix/storage/idb/create.js b/src/matrix/storage/idb/create.js
index 185f4992..a118347b 100644
--- a/src/matrix/storage/idb/create.js
+++ b/src/matrix/storage/idb/create.js
@@ -2,33 +2,34 @@ import Storage from "./storage.js";
 import { openDatabase } from "./utils.js";
 
 export default async function createIdbStorage(databaseName) {
-	const db = await openDatabase(databaseName, createStores, 1);
-	return new Storage(db);
+    const db = await openDatabase(databaseName, createStores, 1);
+    return new Storage(db);
 }
 
 function createStores(db) {
-	db.createObjectStore("session", {keyPath: "key"});
-	// any way to make keys unique here? (just use put?)
-	db.createObjectStore("roomSummary", {keyPath: "roomId"});
+    db.createObjectStore("session", {keyPath: "key"});
+    // any way to make keys unique here? (just use put?)
+    db.createObjectStore("roomSummary", {keyPath: "roomId"});
 
+    // need index to find live fragment? prooobably ok without for now
     db.createObjectStore("timelineFragments", {keyPath: ["roomId", "id"]});
-	// needs roomId separate because it might hold a gap and no event
-	const timelineEvents = db.createObjectStore("timelineEvents", {keyPath: ["event.room_id", "fragmentId", "sortKey"]});
-	timelineEvents.createIndex("byEventId", [
-		"event.room_id",
-		"event.event_id"
-	], {unique: true});
+    // needs roomId separate because it might hold a gap and no event
+    const timelineEvents = db.createObjectStore("timelineEvents", {keyPath: ["event.room_id", "fragmentId", "eventIndex"]});
+    timelineEvents.createIndex("byEventId", [
+        "event.room_id",
+        "event.event_id"
+    ], {unique: true});
 
-	db.createObjectStore("roomState", {keyPath: [
-		"roomId",
-		"event.type",
-		"event.state_key"
-	]});
-	
-	// const roomMembers = db.createObjectStore("roomMembers", {keyPath: [
-	// 	"event.room_id",
-	// 	"event.content.membership",
-	// 	"event.state_key"
-	// ]});
-	// roomMembers.createIndex("byName", ["room_id", "content.name"]);
+    db.createObjectStore("roomState", {keyPath: [
+        "roomId",
+        "event.type",
+        "event.state_key"
+    ]});
+    
+    // const roomMembers = db.createObjectStore("roomMembers", {keyPath: [
+    //  "event.room_id",
+    //  "event.content.membership",
+    //  "event.state_key"
+    // ]});
+    // roomMembers.createIndex("byName", ["room_id", "content.name"]);
 }

From bf835ac01d1272eee94f01c734fb121a63ad92cb Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 11 May 2019 18:19:53 +0200
Subject: [PATCH 40/83] create Entry classes and return fragment boundaries as
 entries as well

they can then be used for gap tiles.
---
 doc/FRAGMENTS.md                              |  10 +
 src/matrix/room/persister.js                  | 126 +++++-----
 src/matrix/room/timeline/EventKey.js          | 145 +++++++++++
 src/matrix/room/timeline/SortKey.js           | 227 ------------------
 src/matrix/room/timeline/entries/BaseEntry.js |  20 ++
 .../room/timeline/entries/EventEntry.js       |  28 +++
 .../timeline/entries/FragmentBoundaryEntry.js |  49 ++++
 7 files changed, 315 insertions(+), 290 deletions(-)
 create mode 100644 src/matrix/room/timeline/EventKey.js
 delete mode 100644 src/matrix/room/timeline/SortKey.js
 create mode 100644 src/matrix/room/timeline/entries/BaseEntry.js
 create mode 100644 src/matrix/room/timeline/entries/EventEntry.js
 create mode 100644 src/matrix/room/timeline/entries/FragmentBoundaryEntry.js

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
index 63a20ee5..e68b5051 100644
--- a/doc/FRAGMENTS.md
+++ b/doc/FRAGMENTS.md
@@ -57,5 +57,15 @@ thoughts:
 
     in case of a gap fill, we need to return what was changed to the fragment (was it joined with another fragment, what's the new token), and which events were actually added.
 
+we return entries! fragmentboundaryentry(start or end) or evententry. so looks much like the gaps we had before, but now they are not stored in the timeline store, but based on fragments.
+
 - where do we translate from fragments to gap entries? and back? in the timeline object?
     that would make sense, that seems to be the only place we need that translation
+
+# SortKey
+
+so, it feels simpler to store fragmentId and eventIndex as fields on the entry instead of an array/arraybuffer in the field sortKey. Currently, the tiles code somewhat relies on having sortKeys but nothing too hard to change.
+
+so, what we could do:
+    - we create EventKey(fragmentId, eventIndex) that has the nextKey methods.
+    - we create a class EventEntry that wraps what is stored in the timeline store. This has a reference to the fragmentindex and has an opaque compare method. Tiles delegate to this method. EventEntry could later on also contain methods like MatrixEvent has in the riot js-sdk, e.g. something to safely dig into the event object.
diff --git a/src/matrix/room/persister.js b/src/matrix/room/persister.js
index b837f469..94d811dc 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/persister.js
@@ -1,5 +1,7 @@
-import SortKey from "./timeline/SortKey.js";
+import EventKey from "./timeline/EventKey.js";
 import FragmentIdIndex from "./timeline/FragmentIdIndex.js";
+import EventEntry from "./timeline/entries/EventEntry.js";
+import FragmentBoundaryEntry from "./timeline/entries/FragmentBoundaryEntry.js";
 
 function gapEntriesAreEqual(a, b) {
     if (!a || !b || !a.gap || !b.gap) {
@@ -28,14 +30,14 @@ function replaceGapEntries(roomTimeline, newEntries, gapKey, neighbourEventKey,
 }
 
 export default class RoomPersister {
-	constructor({roomId, storage}) {
-		this._roomId = roomId;
+    constructor({roomId, storage}) {
+        this._roomId = roomId;
         this._storage = storage;
-		this._lastLiveKey = null;
+        this._lastLiveKey = null;
         this._fragmentIdIndex = new FragmentIdIndex([]);   //only used when timeline is loaded ... e.g. "certain" methods on this class... split up?
-	}
+    }
 
-	async load(txn) {
+    async load(txn) {
         const liveFragment = await txn.roomFragments.liveFragment(this._roomId);
         if (liveFragment) {
             const [lastEvent] = await txn.roomTimeline.lastEvents(this._roomId, liveFragment.id, 1);
@@ -43,15 +45,12 @@ export default class RoomPersister {
             // we could split it up into a SortKey (only with compare) and
             // a EventKey (no compare or fragment index) with nextkey methods and getters/setters for eventIndex/fragmentId
             // we probably need to convert from one to the other though, so bother?
-            const lastLiveKey = new SortKey(this._fragmentIdIndex);
-            lastLiveKey.fragmentId = liveFragment.id;
-            lastLiveKey.eventIndex = lastEvent.eventIndex;
-            this._lastLiveKey = lastLiveKey;
+            this._lastLiveKey = new EventKey(liveFragment.id, lastEvent.eventIndex);
         }
         // if there is no live fragment, we don't create it here because load gets a readonly txn.
         // this is on purpose, load shouldn't modify the store
         console.log("room persister load", this._roomId, this._lastLiveKey && this._lastLiveKey.toString());
-	}
+    }
 
     async persistGapFill(gapEntry, response) {
         const backwards = !!gapEntry.prev_batch;
@@ -124,24 +123,24 @@ export default class RoomPersister {
         return {newEntries, eventFound};
     }
 
-    async _getLiveFragment(txn, previousToken) {
+    async _createLiveFragment(txn, previousToken) {
         const liveFragment = await txn.roomFragments.liveFragment(this._roomId);
         if (!liveFragment) {
             if (!previousToken) {
                 previousToken = null;
             }
-            let defaultId = SortKey.firstLiveFragmentId;
-            txn.roomFragments.add({
+            const fragment = {
                 roomId: this._roomId,
-                id: defaultId,
+                id: EventKey.defaultLiveKey.fragmentId,
                 previousId: null,
                 nextId: null,
                 previousToken: previousToken,
                 nextToken: null
-            });
-            return defaultId;
+            };
+            txn.roomFragments.add(fragment);
+            return fragment;
         } else {
-            return liveFragment.id;
+            return liveFragment;
         }
     }
 
@@ -152,71 +151,72 @@ export default class RoomPersister {
         }
         oldFragment.nextId = newFragmentId;
         txn.roomFragments.update(oldFragment);
-        txn.roomFragments.add({
+        const newFragment = {
             roomId: this._roomId,
             id: newFragmentId,
             previousId: oldFragmentId,
             nextId: null,
             previousToken: previousToken,
             nextToken: null
-        });
+        };
+        txn.roomFragments.add(newFragment);
+        return newFragment;
     }
 
-	async persistSync(roomResponse, txn) {
-        // means we haven't synced this room yet (just joined or did initial sync)
+    async persistSync(roomResponse, txn) {
+        const entries = [];
         if (!this._lastLiveKey) {
+            // means we haven't synced this room yet (just joined or did initial sync)
+            
             // as this is probably a limited sync, prev_batch should be there
             // (but don't fail if it isn't, we won't be able to back-paginate though)
-            const fragmentId = await this._getLiveFragment(txn, timeline.prev_batch);
-            this._lastLiveKey = new SortKey(this._fragmentIdIndex);
-            this._lastLiveKey.fragmentId = fragmentId;
-            this._lastLiveKey.eventIndex = SortKey.firstLiveEventIndex;
-        }
-        // replace live fragment for limited sync, *only* if we had a live fragment already
-        else if (timeline.limited) {
+            let liveFragment = await this._createLiveFragment(txn, timeline.prev_batch);
+            this._lastLiveKey = new EventKey(liveFragment.id, EventKey.defaultLiveKey.eventIndex);
+            entries.push(FragmentBoundaryEntry.start(liveFragment, this._fragmentIdIndex));
+        } else if (timeline.limited) {
+            // replace live fragment for limited sync, *only* if we had a live fragment already
             const oldFragmentId = this._lastLiveKey.fragmentId;
-			this._lastLiveKey = this._lastLiveKey.nextFragmentKey();
-            this._replaceLiveFragment(oldFragmentId, this._lastLiveKey.fragmentId, timeline.prev_batch, txn);
+            this._lastLiveKey = this._lastLiveKey.nextFragmentKey();
+            const [oldFragment, newFragment] = this._replaceLiveFragment(oldFragmentId, this._lastLiveKey.fragmentId, timeline.prev_batch, txn);
+            entries.push(FragmentBoundaryEntry.end(oldFragment, this._fragmentIdIndex));
+            entries.push(FragmentBoundaryEntry.start(newFragment, this._fragmentIdIndex));
         }
-		let nextKey = this._lastLiveKey;
-		const timeline = roomResponse.timeline;
-        const entries = [];
+        let currentKey = this._lastLiveKey;
+        const timeline = roomResponse.timeline;
         if (timeline.events) {
             for(const event of timeline.events) {
-                nextKey = nextKey.nextKey();
-                entries.push(this._createEventEntry(nextKey, event));
-			}
-		}
-        // write to store
-        for(const entry of entries) {
-            txn.roomTimeline.insert(entry);
+                currentKey = currentKey.nextKey();
+                const entry = this._createEventEntry(currentKey, event);
+                txn.roomTimeline.insert(entry);
+                entries.push(new EventEntry(entry, this._fragmentIdIndex));
+            }
         }
-		// right thing to do? if the txn fails, not sure we'll continue anyways ...
-		// only advance the key once the transaction has
-		// succeeded 
-		txn.complete().then(() => {
-			console.log("txn complete, setting key");
-			this._lastLiveKey = nextKey;
-		});
+        // right thing to do? if the txn fails, not sure we'll continue anyways ...
+        // only advance the key once the transaction has
+        // succeeded 
+        txn.complete().then(() => {
+            console.log("txn complete, setting key");
+            this._lastLiveKey = currentKey;
+        });
 
-		// persist state
-		const state = roomResponse.state;
-		if (state.events) {
-			for (const event of state.events) {
-				txn.roomState.setStateEvent(this._roomId, event)
-			}
-		}
+        // persist state
+        const state = roomResponse.state;
+        if (state.events) {
+            for (const event of state.events) {
+                txn.roomState.setStateEvent(this._roomId, event)
+            }
+        }
         // persist live state events in timeline
-		if (timeline.events) {
-			for (const event of timeline.events) {
-				if (typeof event.state_key === "string") {
-					txn.roomState.setStateEvent(this._roomId, event);
-				}
-			}
-		}
+        if (timeline.events) {
+            for (const event of timeline.events) {
+                if (typeof event.state_key === "string") {
+                    txn.roomState.setStateEvent(this._roomId, event);
+                }
+            }
+        }
 
         return entries;
-	}
+    }
 
     _createEventEntry(key, event) {
         return {
diff --git a/src/matrix/room/timeline/EventKey.js b/src/matrix/room/timeline/EventKey.js
new file mode 100644
index 00000000..83621060
--- /dev/null
+++ b/src/matrix/room/timeline/EventKey.js
@@ -0,0 +1,145 @@
+const DEFAULT_LIVE_FRAGMENT_ID = 0;
+const MIN_EVENT_INDEX = Number.MIN_SAFE_INTEGER + 1;
+const MAX_EVENT_INDEX = Number.MAX_SAFE_INTEGER - 1;
+const MID_EVENT_INDEX = 0;
+
+export default class EventKey {
+    constructor(fragmentId, eventIndex) {
+        this.fragmentId = fragmentId;
+        this.eventIndex = eventIndex;
+    }
+
+    nextFragmentKey() {
+        // could take MIN_EVENT_INDEX here if it can't be paged back
+        return new EventKey(this.fragmentId + 1, MID_EVENT_INDEX);
+    }
+
+    nextKey() {
+        return new EventKey(this.fragmentId, this.eventIndex + 1);
+    }
+
+    static get maxKey() {
+        return new EventKey(Number.MAX_SAFE_INTEGER, Number.MAX_SAFE_INTEGER);
+    }
+
+    static get minKey() {
+        return new EventKey(Number.MIN_SAFE_INTEGER, Number.MIN_SAFE_INTEGER);
+    }
+
+    static get defaultLiveKey() {
+        return new EventKey(DEFAULT_LIVE_FRAGMENT_ID, MID_EVENT_INDEX);
+    }
+
+    toString() {
+        return `[${this.fragmentId}/${this.eventIndex}]`;
+    }
+}
+
+//#ifdef TESTS
+export function xtests() {
+    const fragmentIdComparer = {compare: (a, b) => a - b};
+
+    return {
+        test_no_fragment_index(assert) {
+            const min = EventKey.minKey;
+            const max = EventKey.maxKey;
+            const a = new EventKey();
+            a.eventIndex = 1;
+            a.fragmentId = 1;
+
+            assert(min.compare(min) === 0);
+            assert(max.compare(max) === 0);
+            assert(a.compare(a) === 0);
+
+            assert(min.compare(max) < 0);
+            assert(max.compare(min) > 0);
+
+            assert(min.compare(a) < 0);
+            assert(a.compare(min) > 0);
+            
+            assert(max.compare(a) > 0);
+            assert(a.compare(max) < 0);
+        },
+
+        test_default_key(assert) {
+            const k = new EventKey(fragmentIdComparer);
+            assert.equal(k.fragmentId, MID);
+            assert.equal(k.eventIndex, MID);
+        },
+
+        test_inc(assert) {
+            const a = new EventKey(fragmentIdComparer);
+            const b = a.nextKey();
+            assert.equal(a.fragmentId, b.fragmentId);
+            assert.equal(a.eventIndex + 1, b.eventIndex);
+            const c = b.previousKey();
+            assert.equal(b.fragmentId, c.fragmentId);
+            assert.equal(c.eventIndex + 1, b.eventIndex);
+            assert.equal(a.eventIndex, c.eventIndex);
+        },
+
+        test_min_key(assert) {
+            const minKey = EventKey.minKey;
+            const k = new EventKey(fragmentIdComparer);
+            assert(minKey.fragmentId <= k.fragmentId);
+            assert(minKey.eventIndex <= k.eventIndex);
+            assert(k.compare(minKey) > 0);
+            assert(minKey.compare(k) < 0);
+        },
+
+        test_max_key(assert) {
+            const maxKey = EventKey.maxKey;
+            const k = new EventKey(fragmentIdComparer);
+            assert(maxKey.fragmentId >= k.fragmentId);
+            assert(maxKey.eventIndex >= k.eventIndex);
+            assert(k.compare(maxKey) < 0);
+            assert(maxKey.compare(k) > 0);
+        },
+
+        test_immutable(assert) {
+            const a = new EventKey(fragmentIdComparer);
+            const fragmentId = a.fragmentId;
+            const eventIndex = a.eventIndex;
+            a.nextFragmentKey();
+            assert.equal(a.fragmentId, fragmentId);
+            assert.equal(a.eventIndex, eventIndex);
+        },
+
+        test_cmp_fragmentid_first(assert) {
+            const a = new EventKey(fragmentIdComparer);
+            const b = new EventKey(fragmentIdComparer);
+            a.fragmentId = 2;
+            a.eventIndex = 1;
+            b.fragmentId = 1;
+            b.eventIndex = 100000;
+            assert(a.compare(b) > 0);
+        },
+
+        test_cmp_eventindex_second(assert) {
+            const a = new EventKey(fragmentIdComparer);
+            const b = new EventKey(fragmentIdComparer);
+            a.fragmentId = 1;
+            a.eventIndex = 100000;
+            b.fragmentId = 1;
+            b.eventIndex = 2;
+            assert(a.compare(b) > 0);
+            assert(b.compare(a) < 0);
+        },
+
+        test_cmp_max_larger_than_min(assert) {
+            assert(EventKey.minKey.compare(EventKey.maxKey) < 0);
+        },
+
+        test_cmp_fragmentid_first_large(assert) {
+            const a = new EventKey(fragmentIdComparer);
+            const b = new EventKey(fragmentIdComparer);
+            a.fragmentId = MAX;
+            a.eventIndex = MIN;
+            b.fragmentId = MIN;
+            b.eventIndex = MAX;
+            assert(b < a);
+            assert(a > b);
+        }
+    };
+}
+//#endif
diff --git a/src/matrix/room/timeline/SortKey.js b/src/matrix/room/timeline/SortKey.js
deleted file mode 100644
index b045ce1c..00000000
--- a/src/matrix/room/timeline/SortKey.js
+++ /dev/null
@@ -1,227 +0,0 @@
-const MIN_INT32 = -2147483648;
-const MID_INT32 = 0;
-const MAX_INT32 = 2147483647;
-
-const MIN_UINT32 = 0;
-const MID_UINT32 = 2147483647;
-const MAX_UINT32 = 4294967295;
-
-const MIN = MIN_UINT32;
-const MID = MID_UINT32;
-const MAX = MAX_UINT32;
-
-export default class SortKey {
-    constructor(fragmentIdComparer, buffer) {
-        if (buffer) {
-            this._keys = new DataView(buffer);
-        } else {
-            this._keys = new DataView(new ArrayBuffer(8));
-            // start default key right at the middle fragment key, min event key
-            // so we have the same amount of key address space either way
-            this.fragmentId = MID;
-            this.eventIndex = MID;
-        }
-        this._fragmentIdComparer = fragmentIdComparer;
-    }
-
-    get fragmentId() {
-        return this._keys.getUint32(0, false);
-    }
-
-    set fragmentId(value) {
-        return this._keys.setUint32(0, value, false);
-    }
-
-    get eventIndex() {
-        return this._keys.getUint32(4, false);
-    }
-
-    set eventIndex(value) {
-        return this._keys.setUint32(4, value, false);
-    }
-
-    get buffer() {
-        return this._keys.buffer;
-    }
-
-    nextFragmentKey() {
-        const k = new SortKey(this._fragmentIdComparer);
-        k.fragmentId = this.fragmentId + 1;
-        k.eventIndex = MIN;
-        return k;
-    }
-
-    nextKey() {
-        const k = new SortKey(this._fragmentIdComparer);
-        k.fragmentId = this.fragmentId;
-        k.eventIndex = this.eventIndex + 1;
-        return k;
-    }
-
-    previousKey() {
-        const k = new SortKey(this._fragmentIdComparer);
-        k.fragmentId = this.fragmentId;
-        k.eventIndex = this.eventIndex - 1;
-        return k;
-    }
-
-    clone() {
-        const k = new SortKey();
-        k.fragmentId = this.fragmentId;
-        k.eventIndex = this.eventIndex; 
-        return k;
-    }
-
-    static get maxKey() {
-        const maxKey = new SortKey(null);
-        maxKey.fragmentId = MAX;
-        maxKey.eventIndex = MAX;
-        return maxKey;
-    }
-
-    static get minKey() {
-        const minKey = new SortKey(null);
-        minKey.fragmentId = MIN;
-        minKey.eventIndex = MIN;
-        return minKey;
-    }
-
-    static get firstLiveFragmentId() {
-        return MID;
-    }
-
-    static get firstLiveEventIndex() {
-        return MID;
-    }
-
-    compare(otherKey) {
-        const fragmentDiff = this.fragmentId - otherKey.fragmentId;
-        if (fragmentDiff === 0) {
-            return this.eventIndex - otherKey.eventIndex;
-        } else {
-            // minKey and maxKey might not have fragmentIdComparer, so short-circuit this first ...
-            if ((this.fragmentId === MIN && otherKey.fragmentId !== MIN) || (this.fragmentId !== MAX && otherKey.fragmentId === MAX)) {
-                return -1;
-            }
-            if ((this.fragmentId === MAX && otherKey.fragmentId !== MAX) || (this.fragmentId !== MIN && otherKey.fragmentId === MIN)) {
-                return 1;
-            }
-            // ... then delegate to fragmentIdComparer.
-            // This might throw if the relation of two fragments is unknown.
-            return this._fragmentIdComparer.compare(this.fragmentId, otherKey.fragmentId);
-        }
-    }
-
-    toString() {
-        return `[${this.fragmentId}/${this.eventIndex}]`;
-    }
-}
-
-//#ifdef TESTS
-export function tests() {
-    const fragmentIdComparer = {compare: (a, b) => a - b};
-
-    return {
-        test_no_fragment_index(assert) {
-            const min = SortKey.minKey;
-            const max = SortKey.maxKey;
-            const a = new SortKey();
-            a.eventIndex = 1;
-            a.fragmentId = 1;
-
-            assert(min.compare(min) === 0);
-            assert(max.compare(max) === 0);
-            assert(a.compare(a) === 0);
-
-            assert(min.compare(max) < 0);
-            assert(max.compare(min) > 0);
-
-            assert(min.compare(a) < 0);
-            assert(a.compare(min) > 0);
-            
-            assert(max.compare(a) > 0);
-            assert(a.compare(max) < 0);
-        },
-
-        test_default_key(assert) {
-            const k = new SortKey(fragmentIdComparer);
-            assert.equal(k.fragmentId, MID);
-            assert.equal(k.eventIndex, MID);
-        },
-
-        test_inc(assert) {
-            const a = new SortKey(fragmentIdComparer);
-            const b = a.nextKey();
-            assert.equal(a.fragmentId, b.fragmentId);
-            assert.equal(a.eventIndex + 1, b.eventIndex);
-            const c = b.previousKey();
-            assert.equal(b.fragmentId, c.fragmentId);
-            assert.equal(c.eventIndex + 1, b.eventIndex);
-            assert.equal(a.eventIndex, c.eventIndex);
-        },
-
-        test_min_key(assert) {
-            const minKey = SortKey.minKey;
-            const k = new SortKey(fragmentIdComparer);
-            assert(minKey.fragmentId <= k.fragmentId);
-            assert(minKey.eventIndex <= k.eventIndex);
-            assert(k.compare(minKey) > 0);
-            assert(minKey.compare(k) < 0);
-        },
-
-        test_max_key(assert) {
-            const maxKey = SortKey.maxKey;
-            const k = new SortKey(fragmentIdComparer);
-            assert(maxKey.fragmentId >= k.fragmentId);
-            assert(maxKey.eventIndex >= k.eventIndex);
-            assert(k.compare(maxKey) < 0);
-            assert(maxKey.compare(k) > 0);
-        },
-
-        test_immutable(assert) {
-            const a = new SortKey(fragmentIdComparer);
-            const fragmentId = a.fragmentId;
-            const eventIndex = a.eventIndex;
-            a.nextFragmentKey();
-            assert.equal(a.fragmentId, fragmentId);
-            assert.equal(a.eventIndex, eventIndex);
-        },
-
-        test_cmp_fragmentid_first(assert) {
-            const a = new SortKey(fragmentIdComparer);
-            const b = new SortKey(fragmentIdComparer);
-            a.fragmentId = 2;
-            a.eventIndex = 1;
-            b.fragmentId = 1;
-            b.eventIndex = 100000;
-            assert(a.compare(b) > 0);
-        },
-
-        test_cmp_eventindex_second(assert) {
-            const a = new SortKey(fragmentIdComparer);
-            const b = new SortKey(fragmentIdComparer);
-            a.fragmentId = 1;
-            a.eventIndex = 100000;
-            b.fragmentId = 1;
-            b.eventIndex = 2;
-            assert(a.compare(b) > 0);
-            assert(b.compare(a) < 0);
-        },
-
-        test_cmp_max_larger_than_min(assert) {
-            assert(SortKey.minKey.compare(SortKey.maxKey) < 0);
-        },
-
-        test_cmp_fragmentid_first_large(assert) {
-            const a = new SortKey(fragmentIdComparer);
-            const b = new SortKey(fragmentIdComparer);
-            a.fragmentId = MAX;
-            a.eventIndex = MIN;
-            b.fragmentId = MIN;
-            b.eventIndex = MAX;
-            assert(b < a);
-            assert(a > b);
-        }
-    };
-}
-//#endif
diff --git a/src/matrix/room/timeline/entries/BaseEntry.js b/src/matrix/room/timeline/entries/BaseEntry.js
new file mode 100644
index 00000000..56566efc
--- /dev/null
+++ b/src/matrix/room/timeline/entries/BaseEntry.js
@@ -0,0 +1,20 @@
+//entries can be sorted, first by fragment, then by entry index.
+
+export default class BaseEntry {
+    get fragmentId() {
+        throw new Error("unimplemented");
+    }
+
+    get entryIndex() {
+        throw new Error("unimplemented");
+    }
+
+    compare(otherEntry) {
+        if (this.fragmentId === otherEntry.fragmentId) {
+            return this.entryIndex - otherEntry.entryIndex;
+        } else {
+            // This might throw if the relation of two fragments is unknown.
+            return this._fragmentIdComparer.compare(this.fragmentId, otherEntry.fragmentId);
+        }
+    }
+}
diff --git a/src/matrix/room/timeline/entries/EventEntry.js b/src/matrix/room/timeline/entries/EventEntry.js
new file mode 100644
index 00000000..07d00a76
--- /dev/null
+++ b/src/matrix/room/timeline/entries/EventEntry.js
@@ -0,0 +1,28 @@
+import BaseEntry from "./BaseEntry.js";
+
+export default class EventEntry extends BaseEntry {
+    constructor(eventEntry, fragmentIdComparator) {
+        super(fragmentIdComparator);
+        this._eventEntry = eventEntry;
+    }
+
+    get fragmentId() {
+        return this._eventEntry.fragmentId;
+    }
+
+    get entryIndex() {
+        return this._eventEntry.eventIndex;
+    }
+
+    get content() {
+        return this._eventEntry.event.content;
+    }
+
+    get type() {
+        return this._eventEntry.event.type;
+    }
+
+    get id() {
+        return this._eventEntry.event.event_id;
+    }
+}
diff --git a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
new file mode 100644
index 00000000..610d32c6
--- /dev/null
+++ b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
@@ -0,0 +1,49 @@
+import BaseEntry from "./BaseEntry.js";
+
+export default class FragmentBoundaryEntry extends BaseEntry {
+    constructor(fragment, isFragmentStart, fragmentIdComparator) {
+        super(fragmentIdComparator);
+        this._fragment = fragment;
+        this._isFragmentStart = isFragmentStart;
+    }
+
+    static start(fragment, fragmentIdComparator) {
+        return new FragmentBoundaryEntry(fragment, true, fragmentIdComparator);
+    }
+
+    static end(fragment, fragmentIdComparator) {
+        return new FragmentBoundaryEntry(fragment, false, fragmentIdComparator);
+    }
+    
+    get hasStarted() {
+        return this._isFragmentStart;
+    }
+
+    get hasEnded() {
+        return !this.hasStarted;
+    }
+
+    get fragment() {
+        return this._fragment;
+    }
+
+    get fragmentId() {
+        return this._fragment.id;
+    }
+
+    get entryIndex() {
+        if (this.hasStarted) {
+            return Number.MIN_SAFE_INTEGER;
+        } else {
+            return Number.MAX_SAFE_INTEGER;
+        }
+    }
+
+    get isGap() {
+        if (this.hasStarted) {
+            return !!this.fragment.nextToken;
+        } else {
+            return !!this.fragment.previousToken;
+        }
+    }
+}

From 89bc0e1696de8a54efc4a99b734914fc30324d92 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 12 May 2019 20:24:06 +0200
Subject: [PATCH 41/83] split up RoomPersister to SyncPersister

also rename stores to timelineEvents and timelineFragments
---
 doc/FRAGMENTS.md                              |   2 +-
 src/matrix/room/room.js                       |  11 +-
 ...agmentIdIndex.js => FragmentIdComparer.js} |   2 +-
 .../persistence/SyncPersister.js}             | 134 ++----------------
 .../room/timeline/persistence/common.js       |   7 +
 src/matrix/storage/idb/storage.js             |   5 +-
 ...TimelineStore.js => TimelineEventStore.js} |  90 ++++++------
 src/matrix/storage/idb/transaction.js         |  13 +-
 8 files changed, 87 insertions(+), 177 deletions(-)
 rename src/matrix/room/timeline/{FragmentIdIndex.js => FragmentIdComparer.js} (99%)
 rename src/matrix/room/{persister.js => timeline/persistence/SyncPersister.js} (63%)
 create mode 100644 src/matrix/room/timeline/persistence/common.js
 rename src/matrix/storage/idb/stores/{RoomTimelineStore.js => TimelineEventStore.js} (73%)

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
index e68b5051..a63a96ea 100644
--- a/doc/FRAGMENTS.md
+++ b/doc/FRAGMENTS.md
@@ -17,7 +17,7 @@
     how will fragments be exposed in timeline store?
         - all read operations are passed a fragment id
  - adapt persister
-    - persist fragments in /sync
+    - DONE: persist fragments in /sync
     - load n items before and after key
     - fill gaps / fragment filling
  - add live fragment id optimization if we haven't done so already
diff --git a/src/matrix/room/room.js b/src/matrix/room/room.js
index 05560cb5..b2ad1312 100644
--- a/src/matrix/room/room.js
+++ b/src/matrix/room/room.js
@@ -1,7 +1,8 @@
 import EventEmitter from "../../EventEmitter.js";
 import RoomSummary from "./summary.js";
-import RoomPersister from "./persister.js";
 import Timeline from "./timeline.js";
+import SyncPersister from "./timeline/persistence/SyncPersister.js";
+import FragmentIdComparer from "./timeline/FragmentIdComparer.js";
 
 export default class Room extends EventEmitter {
 	constructor({roomId, storage, hsApi, emitCollectionChange}) {
@@ -10,14 +11,15 @@ export default class Room extends EventEmitter {
         this._storage = storage;
         this._hsApi = hsApi;
 		this._summary = new RoomSummary(roomId);
-		this._persister = new RoomPersister({roomId, storage});
+        this._fragmentIdComparer = new FragmentIdComparer([]);
+		this._syncPersister = new SyncPersister({roomId, storage, fragmentIdComparer: this._fragmentIdComparer});
         this._emitCollectionChange = emitCollectionChange;
         this._timeline = null;
 	}
 
     persistSync(roomResponse, membership, txn) {
 		const summaryChanged = this._summary.applySync(roomResponse, membership, txn);
-		const newTimelineEntries = this._persister.persistSync(roomResponse, txn);
+		const newTimelineEntries = this._syncPersister.persistSync(roomResponse, txn);
         return {summaryChanged, newTimelineEntries};
     }
 
@@ -33,7 +35,7 @@ export default class Room extends EventEmitter {
 
 	load(summary, txn) {
 		this._summary.load(summary);
-		return this._persister.load(txn);
+		return this._syncPersister.load(txn);
 	}
 
     get name() {
@@ -51,7 +53,6 @@ export default class Room extends EventEmitter {
         this._timeline = new Timeline({
             roomId: this.id,
             storage: this._storage,
-            persister: this._persister,
             hsApi: this._hsApi,
             closeCallback: () => this._timeline = null,
         });
diff --git a/src/matrix/room/timeline/FragmentIdIndex.js b/src/matrix/room/timeline/FragmentIdComparer.js
similarity index 99%
rename from src/matrix/room/timeline/FragmentIdIndex.js
rename to src/matrix/room/timeline/FragmentIdComparer.js
index e67be9c2..7ffb9422 100644
--- a/src/matrix/room/timeline/FragmentIdIndex.js
+++ b/src/matrix/room/timeline/FragmentIdComparer.js
@@ -106,7 +106,7 @@ class Island {
 /*
 index for fast lookup of how two fragments can be sorted
 */
-export default class FragmentIdIndex {
+export default class FragmentIdComparer {
     constructor(fragments) {
         this.rebuild(fragments);
     }
diff --git a/src/matrix/room/persister.js b/src/matrix/room/timeline/persistence/SyncPersister.js
similarity index 63%
rename from src/matrix/room/persister.js
rename to src/matrix/room/timeline/persistence/SyncPersister.js
index 94d811dc..eaa3bfdd 100644
--- a/src/matrix/room/persister.js
+++ b/src/matrix/room/timeline/persistence/SyncPersister.js
@@ -1,40 +1,13 @@
-import EventKey from "./timeline/EventKey.js";
-import FragmentIdIndex from "./timeline/FragmentIdIndex.js";
-import EventEntry from "./timeline/entries/EventEntry.js";
-import FragmentBoundaryEntry from "./timeline/entries/FragmentBoundaryEntry.js";
+import EventKey from "../EventKey.js";
+import EventEntry from "../entries/EventEntry.js";
+import FragmentBoundaryEntry from "../entries/FragmentBoundaryEntry.js";
+import {createEventEntry} from "./common.js";
 
-function gapEntriesAreEqual(a, b) {
-    if (!a || !b || !a.gap || !b.gap) {
-        return false;
-    }
-    const gapA = a.gap, gapB = b.gap;
-    return gapA.prev_batch === gapB.prev_batch && gapA.next_batch === gapB.next_batch;
-}
-
-function replaceGapEntries(roomTimeline, newEntries, gapKey, neighbourEventKey, backwards) {
-    let replacedRange;
-    if (neighbourEventKey) {
-        replacedRange = backwards ?
-            roomTimeline.boundRange(neighbourEventKey, gapKey, false, true) :
-            roomTimeline.boundRange(gapKey, neighbourEventKey, true, false);
-    } else {
-        replacedRange = roomTimeline.onlyRange(gapKey);
-    }
-
-    const removedEntries = roomTimeline.getAndRemoveRange(this._roomId, replacedRange);
-    for (let entry of newEntries) {
-        roomTimeline.add(entry);
-    }
-
-    return removedEntries;
-}
-
-export default class RoomPersister {
-    constructor({roomId, storage}) {
+export default class SyncPersister {
+    constructor({roomId, storage, fragmentIdComparer}) {
         this._roomId = roomId;
         this._storage = storage;
-        this._lastLiveKey = null;
-        this._fragmentIdIndex = new FragmentIdIndex([]);   //only used when timeline is loaded ... e.g. "certain" methods on this class... split up?
+        this._fragmentIdComparer = fragmentIdComparer;
     }
 
     async load(txn) {
@@ -52,77 +25,6 @@ export default class RoomPersister {
         console.log("room persister load", this._roomId, this._lastLiveKey && this._lastLiveKey.toString());
     }
 
-    async persistGapFill(gapEntry, response) {
-        const backwards = !!gapEntry.prev_batch;
-        const {chunk, start, end} = response;
-        if (!Array.isArray(chunk)) {
-            throw new Error("Invalid chunk in response");
-        }
-        if (typeof end !== "string") {
-            throw new Error("Invalid end token in response");
-        }
-        if ((backwards && start !== gapEntry.prev_batch) || (!backwards && start !== gapEntry.next_batch)) {
-            throw new Error("start is not equal to prev_batch or next_batch");
-        }
-        
-        const gapKey = gapEntry.sortKey;
-        const txn = await this._storage.readWriteTxn([this._storage.storeNames.roomTimeline]);
-        let result;
-        try {
-            const roomTimeline = txn.roomTimeline;
-            // make sure what we've been given is actually persisted
-            // in the timeline, otherwise we're replacing something
-            // that doesn't exist (maybe it has been replaced already, or ...)
-            const persistedEntry = await roomTimeline.get(this._roomId, gapKey);
-            if (!gapEntriesAreEqual(gapEntry, persistedEntry)) {
-                throw new Error("Gap is not present in the timeline");
-            }
-            // find the previous event before the gap we could merge with
-            const neighbourEventEntry = await (backwards ?
-                roomTimeline.previousEvent(this._roomId, gapKey) :
-                roomTimeline.nextEvent(this._roomId, gapKey));
-            const neighbourEventId = neighbourEventEntry ? neighbourEventEntry.event.event_id : undefined;
-            const {newEntries, eventFound} = this._createNewGapEntries(chunk, end, gapKey, neighbourEventId, backwards);
-            const neighbourEventKey = eventFound ? neighbourEventEntry.sortKey : undefined;
-            const replacedEntries = replaceGapEntries(roomTimeline, newEntries, gapKey, neighbourEventKey, backwards);
-            result = {newEntries, replacedEntries};
-        } catch (err) {
-            txn.abort();
-            throw err;
-        }
-
-        await txn.complete();
-
-        return result;
-    }
-
-    _createNewGapEntries(chunk, nextPaginationToken, gapKey, neighbourEventId, backwards) {
-        if (backwards) {
-            // if backwards, the last events are the ones closest to the gap,
-            // and need to be assigned a key derived from the gap first,
-            // so swap order to only need one loop for both directions
-            chunk.reverse();
-        }
-        let sortKey = gapKey;
-        const {newEntries, eventFound} = chunk.reduce((acc, event) => {
-            acc.eventFound = acc.eventFound || event.event_id === neighbourEventId;
-            if (!acc.eventFound) {
-                acc.newEntries.push(this._createEventEntry(sortKey, event));
-                sortKey = backwards ? sortKey.previousKey() : sortKey.nextKey();
-            }
-        }, {newEntries: [], eventFound: false});
-
-        if (!eventFound) {
-            // as we're replacing an existing gap, no need to increment the gap index
-            newEntries.push(this._createGapEntry(sortKey, nextPaginationToken, backwards));
-        }
-        if (backwards) {
-            // swap resulting array order again if needed
-            newEntries.reverse();
-        }
-        return {newEntries, eventFound};
-    }
-
     async _createLiveFragment(txn, previousToken) {
         const liveFragment = await txn.roomFragments.liveFragment(this._roomId);
         if (!liveFragment) {
@@ -160,7 +62,7 @@ export default class RoomPersister {
             nextToken: null
         };
         txn.roomFragments.add(newFragment);
-        return newFragment;
+        return {oldFragment, newFragment};
     }
 
     async persistSync(roomResponse, txn) {
@@ -172,23 +74,23 @@ export default class RoomPersister {
             // (but don't fail if it isn't, we won't be able to back-paginate though)
             let liveFragment = await this._createLiveFragment(txn, timeline.prev_batch);
             this._lastLiveKey = new EventKey(liveFragment.id, EventKey.defaultLiveKey.eventIndex);
-            entries.push(FragmentBoundaryEntry.start(liveFragment, this._fragmentIdIndex));
+            entries.push(FragmentBoundaryEntry.start(liveFragment, this._fragmentIdComparer));
         } else if (timeline.limited) {
             // replace live fragment for limited sync, *only* if we had a live fragment already
             const oldFragmentId = this._lastLiveKey.fragmentId;
             this._lastLiveKey = this._lastLiveKey.nextFragmentKey();
-            const [oldFragment, newFragment] = this._replaceLiveFragment(oldFragmentId, this._lastLiveKey.fragmentId, timeline.prev_batch, txn);
-            entries.push(FragmentBoundaryEntry.end(oldFragment, this._fragmentIdIndex));
-            entries.push(FragmentBoundaryEntry.start(newFragment, this._fragmentIdIndex));
+            const {oldFragment, newFragment} = this._replaceLiveFragment(oldFragmentId, this._lastLiveKey.fragmentId, timeline.prev_batch, txn);
+            entries.push(FragmentBoundaryEntry.end(oldFragment, this._fragmentIdComparer));
+            entries.push(FragmentBoundaryEntry.start(newFragment, this._fragmentIdComparer));
         }
         let currentKey = this._lastLiveKey;
         const timeline = roomResponse.timeline;
         if (timeline.events) {
             for(const event of timeline.events) {
                 currentKey = currentKey.nextKey();
-                const entry = this._createEventEntry(currentKey, event);
+                const entry = createEventEntry(currentKey, event);
                 txn.roomTimeline.insert(entry);
-                entries.push(new EventEntry(entry, this._fragmentIdIndex));
+                entries.push(new EventEntry(entry, this._fragmentIdComparer));
             }
         }
         // right thing to do? if the txn fails, not sure we'll continue anyways ...
@@ -217,14 +119,6 @@ export default class RoomPersister {
 
         return entries;
     }
-
-    _createEventEntry(key, event) {
-        return {
-            fragmentId: key.fragmentId,
-            eventIndex: key.eventIndex,
-            event: event,
-        };
-    }
 }
 
 //#ifdef TESTS
diff --git a/src/matrix/room/timeline/persistence/common.js b/src/matrix/room/timeline/persistence/common.js
new file mode 100644
index 00000000..ac5a6c9d
--- /dev/null
+++ b/src/matrix/room/timeline/persistence/common.js
@@ -0,0 +1,7 @@
+export function createEventEntry(key, event) {
+    return {
+        fragmentId: key.fragmentId,
+        eventIndex: key.eventIndex,
+        event: event,
+    };
+}
\ No newline at end of file
diff --git a/src/matrix/storage/idb/storage.js b/src/matrix/storage/idb/storage.js
index b4a6659c..ca7e2256 100644
--- a/src/matrix/storage/idb/storage.js
+++ b/src/matrix/storage/idb/storage.js
@@ -1,6 +1,5 @@
 import Transaction from "./transaction.js";
-
-export const STORE_NAMES = ["session", "roomState", "roomSummary", "roomTimeline"];
+import { STORE_NAMES } from "../common.js";
 
 export default class Storage {
 	constructor(idbDatabase) {
@@ -30,4 +29,4 @@ export default class Storage {
 		const txn = this._db.transaction(storeNames, "readwrite");
 		return new Transaction(txn, storeNames);
 	}
-}
\ No newline at end of file
+}
diff --git a/src/matrix/storage/idb/stores/RoomTimelineStore.js b/src/matrix/storage/idb/stores/TimelineEventStore.js
similarity index 73%
rename from src/matrix/storage/idb/stores/RoomTimelineStore.js
rename to src/matrix/storage/idb/stores/TimelineEventStore.js
index 0a540dd5..a6ce5617 100644
--- a/src/matrix/storage/idb/stores/RoomTimelineStore.js
+++ b/src/matrix/storage/idb/stores/TimelineEventStore.js
@@ -1,4 +1,4 @@
-import SortKey from "../../../room/timeline/SortKey.js";
+import EventKey from "../../../room/timeline/EventKey.js";
 
 class Range {
     constructor(only, lower, upper, lowerOpen, upperOpen) {
@@ -12,14 +12,14 @@ class Range {
     asIDBKeyRange(roomId) {
         // only
         if (this._only) {
-            return IDBKeyRange.only([roomId, this._only.buffer]);
+            return IDBKeyRange.only([roomId, this._only.fragmentId, this._only.eventIndex]);
         }
         // lowerBound
         // also bound as we don't want to move into another roomId
         if (this._lower && !this._upper) {
             return IDBKeyRange.bound(
-                [roomId, this._lower.buffer],
-                [roomId, SortKey.maxKey.buffer],
+                [roomId, this._lower.fragmentId, this._lower.eventIndex],
+                [roomId, EventKey.maxKey.fragmentId, EventKey.maxKey.eventIndex],
                 this._lowerOpen,
                 false
             );
@@ -28,8 +28,8 @@ class Range {
         // also bound as we don't want to move into another roomId
         if (!this._lower && this._upper) {
             return IDBKeyRange.bound(
-                [roomId, SortKey.minKey.buffer],
-                [roomId, this._upper.buffer],
+                [roomId, EventKey.minKey.fragmentId, EventKey.minKey.eventIndex],
+                [roomId, this._upper.fragmentId, this._upper.eventIndex],
                 false,
                 this._upperOpen
             );
@@ -37,8 +37,8 @@ class Range {
         // bound
         if (this._lower && this._upper) {
             return IDBKeyRange.bound(
-                [roomId, this._lower.buffer],
-                [roomId, this._upper.buffer],
+                [roomId, this._lower.fragmentId, this._lower.eventIndex],
+                [roomId, this._upper.fragmentId, this._upper.eventIndex],
                 this._lowerOpen,
                 this._upperOpen
             );
@@ -57,44 +57,44 @@ class Range {
  *
  * @typedef   {Object} Entry
  * @property  {string} roomId
- * @property  {SortKey} sortKey
+ * @property  {EventKey} eventKey
  * @property  {?Event} event if an event entry, the event
  * @property  {?Gap} gap if a gap entry, the gap
 */
-export default class RoomTimelineStore {
+export default class TimelineEventStore {
 	constructor(timelineStore) {
 		this._timelineStore = timelineStore;
 	}
 
     /** Creates a range that only includes the given key
-     *  @param {SortKey} sortKey the key
+     *  @param {EventKey} eventKey the key
      *  @return {Range} the created range
      */
-    onlyRange(sortKey) {
-        return new Range(sortKey);
+    onlyRange(eventKey) {
+        return new Range(eventKey);
     }
 
-    /** Creates a range that includes all keys before sortKey, and optionally also the key itself.
-     *  @param {SortKey} sortKey the key
+    /** Creates a range that includes all keys before eventKey, and optionally also the key itself.
+     *  @param {EventKey} eventKey the key
      *  @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the upper end.
      *  @return {Range} the created range
      */
-    upperBoundRange(sortKey, open=false) {
-        return new Range(undefined, undefined, sortKey, undefined, open);
+    upperBoundRange(eventKey, open=false) {
+        return new Range(undefined, undefined, eventKey, undefined, open);
     }
 
-    /** Creates a range that includes all keys after sortKey, and optionally also the key itself.
-     *  @param {SortKey} sortKey the key
+    /** Creates a range that includes all keys after eventKey, and optionally also the key itself.
+     *  @param {EventKey} eventKey the key
      *  @param {boolean} [open=false] whether the key is included (false) or excluded (true) from the range at the lower end.
      *  @return {Range} the created range
      */
-    lowerBoundRange(sortKey, open=false) {
-        return new Range(undefined, sortKey, undefined, open);
+    lowerBoundRange(eventKey, open=false) {
+        return new Range(undefined, eventKey, undefined, open);
     }
 
     /** Creates a range that includes all keys between `lower` and `upper`, and optionally the given keys as well.
-     *  @param {SortKey} lower the lower key
-     *  @param {SortKey} upper the upper key
+     *  @param {EventKey} lower the lower key
+     *  @param {EventKey} upper the upper key
      *  @param {boolean} [lowerOpen=false] whether the lower key is included (false) or excluded (true) from the range.
      *  @param {boolean} [upperOpen=false] whether the upper key is included (false) or excluded (true) from the range.
      *  @return {Range} the created range
@@ -110,9 +110,9 @@ export default class RoomTimelineStore {
      *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
      */
 	async lastEvents(roomId, fragmentId, amount) {
-        const sortKey = SortKey.maxKey;
-        sortKey.fragmentId = fragmentId;
-		return this.eventsBefore(roomId, sortKey, amount);
+        const eventKey = EventKey.maxKey;
+        eventKey.fragmentId = fragmentId;
+		return this.eventsBefore(roomId, eventKey, amount);
 	}
 
     /** Looks up the first `amount` entries in the timeline for `roomId`.
@@ -122,32 +122,32 @@ export default class RoomTimelineStore {
      *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
      */
 	async firstEvents(roomId, fragmentId, amount) {
-        const sortKey = SortKey.minKey;
-        sortKey.fragmentId = fragmentId;
-		return this.eventsAfter(roomId, sortKey, amount);
+        const eventKey = EventKey.minKey;
+        eventKey.fragmentId = fragmentId;
+		return this.eventsAfter(roomId, eventKey, amount);
 	}
 
-    /** Looks up `amount` entries after `sortKey` in the timeline for `roomId` within the same fragment.
-     *  The entry for `sortKey` is not included.
+    /** Looks up `amount` entries after `eventKey` in the timeline for `roomId` within the same fragment.
+     *  The entry for `eventKey` is not included.
      *  @param  {string} roomId
-     *  @param  {SortKey} sortKey
+     *  @param  {EventKey} eventKey
      *  @param  {number} amount
      *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
      */
-	eventsAfter(roomId, sortKey, amount) {
-        const idbRange = this.lowerBoundRange(sortKey, true).asIDBKeyRange(roomId);
+	eventsAfter(roomId, eventKey, amount) {
+        const idbRange = this.lowerBoundRange(eventKey, true).asIDBKeyRange(roomId);
 		return this._timelineStore.selectLimit(idbRange, amount);
 	}
 
-    /** Looks up `amount` entries before `sortKey` in the timeline for `roomId` within the same fragment.
-     *  The entry for `sortKey` is not included.
+    /** Looks up `amount` entries before `eventKey` in the timeline for `roomId` within the same fragment.
+     *  The entry for `eventKey` is not included.
      *  @param  {string} roomId
-     *  @param  {SortKey} sortKey
+     *  @param  {EventKey} eventKey
      *  @param  {number} amount
      *  @return {Promise<Entry[]>} a promise resolving to an array with 0 or more entries, in ascending order.
      */
-    async eventsBefore(roomId, sortKey, amount) {
-        const range = this.upperBoundRange(sortKey, true).asIDBKeyRange(roomId);
+    async eventsBefore(roomId, eventKey, amount) {
+        const range = this.upperBoundRange(eventKey, true).asIDBKeyRange(roomId);
         const events = await this._timelineStore.selectLimitReverse(range, amount);
         events.reverse(); // because we fetched them backwards
         return events;
@@ -196,7 +196,7 @@ export default class RoomTimelineStore {
         return firstFoundEventId;
     }
 
-    /** Inserts a new entry into the store. The combination of roomId and sortKey should not exist yet, or an error is thrown.
+    /** Inserts a new entry into the store. The combination of roomId and eventKey should not exist yet, or an error is thrown.
      *  @param  {Entry} entry the entry to insert
      *  @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not.
      *  @throws {StorageError} ...
@@ -206,7 +206,7 @@ export default class RoomTimelineStore {
         return this._timelineStore.add(entry);
     }
 
-    /** Updates the entry into the store with the given [roomId, sortKey] combination.
+    /** Updates the entry into the store with the given [roomId, eventKey] combination.
      *  If not yet present, will insert. Might be slower than add.
      *  @param  {Entry} entry the entry to update.
      *  @return {Promise<>} a promise resolving to undefined if the operation was successful, or a StorageError if not.
@@ -215,12 +215,16 @@ export default class RoomTimelineStore {
         return this._timelineStore.put(entry);
     }
 
-    get(roomId, sortKey) {
-        return this._timelineStore.get([roomId, sortKey]);
+    get(roomId, eventKey) {
+        return this._timelineStore.get([roomId, eventKey.fragmentId, eventKey.eventIndex]);
     }
     // returns the entries as well!! (or not always needed? I guess not always needed, so extra method)
     removeRange(roomId, range) {
         // TODO: read the entries!
         return this._timelineStore.delete(range.asIDBKeyRange(roomId));
     }
+
+    getByEventId(roomId, eventId) {
+        return this._timelineStore.index("byEventId").get([roomId, eventId]);
+    }
 }
diff --git a/src/matrix/storage/idb/transaction.js b/src/matrix/storage/idb/transaction.js
index 78883964..1a16d08d 100644
--- a/src/matrix/storage/idb/transaction.js
+++ b/src/matrix/storage/idb/transaction.js
@@ -2,8 +2,9 @@ import {txnAsPromise} from "./utils.js";
 import Store from "./store.js";
 import SessionStore from "./stores/SessionStore.js";
 import RoomSummaryStore from "./stores/RoomSummaryStore.js";
-import RoomTimelineStore from "./stores/RoomTimelineStore.js";
+import TimelineEventStore from "./stores/TimelineEventStore.js";
 import RoomStateStore from "./stores/RoomStateStore.js";
+import TimelineFragmentStore from "./stores/TimelineFragmentStore.js";
 
 export default class Transaction {
 	constructor(txn, allowedStoreNames) {
@@ -41,8 +42,12 @@ export default class Transaction {
 		return this._store("roomSummary", idbStore => new RoomSummaryStore(idbStore));
 	}
 
-	get roomTimeline() {
-		return this._store("roomTimeline", idbStore => new RoomTimelineStore(idbStore));
+    get timelineFragments() {
+        return this._store("timelineFragments", idbStore => new TimelineFragmentStore(idbStore));
+    }
+
+	get timelineEvents() {
+		return this._store("timelineEvents", idbStore => new TimelineEventStore(idbStore));
 	}
 
 	get roomState() {
@@ -56,4 +61,4 @@ export default class Transaction {
 	abort() {
 		this._txn.abort();
 	}
-}
\ No newline at end of file
+}

From a6b17cf25a2ae2e47cd5164fd888e39a759a67e4 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 12 May 2019 20:25:41 +0200
Subject: [PATCH 42/83] first draft of persistFragmentFill

---
 src/matrix/room/timeline/Direction.js         |  29 +++
 src/matrix/room/timeline/EventKey.js          |  13 ++
 src/matrix/room/timeline/entries/BaseEntry.js |   4 +
 .../timeline/entries/FragmentBoundaryEntry.js |  57 ++++-
 .../room/timeline/persistence/GapPersister.js | 208 ++++++++++++++++++
 ...gmentStore.js => TimelineFragmentStore.js} |   0
 6 files changed, 305 insertions(+), 6 deletions(-)
 create mode 100644 src/matrix/room/timeline/Direction.js
 create mode 100644 src/matrix/room/timeline/persistence/GapPersister.js
 rename src/matrix/storage/idb/stores/{RoomFragmentStore.js => TimelineFragmentStore.js} (100%)

diff --git a/src/matrix/room/timeline/Direction.js b/src/matrix/room/timeline/Direction.js
new file mode 100644
index 00000000..d885ed63
--- /dev/null
+++ b/src/matrix/room/timeline/Direction.js
@@ -0,0 +1,29 @@
+
+const _forward = Object.freeze(new Direction(true));
+const _backward = Object.freeze(new Direction(false));
+
+export default class Direction {
+    constructor(isForward) {
+        this._isForward = isForward;
+    }
+
+    get isForward() {
+        return this._isForward;
+    }
+
+    get isBackward() {
+        return !this.isForward;
+    }
+
+    asApiString() {
+        return this.isForward ? "f" : "b";
+    }
+
+    static get Forward() {
+        return _forward;
+    }
+
+    static get Backward() {
+        return _backward;
+    }
+}
diff --git a/src/matrix/room/timeline/EventKey.js b/src/matrix/room/timeline/EventKey.js
index 83621060..e05c38f4 100644
--- a/src/matrix/room/timeline/EventKey.js
+++ b/src/matrix/room/timeline/EventKey.js
@@ -3,6 +3,7 @@ const MIN_EVENT_INDEX = Number.MIN_SAFE_INTEGER + 1;
 const MAX_EVENT_INDEX = Number.MAX_SAFE_INTEGER - 1;
 const MID_EVENT_INDEX = 0;
 
+// key for events in the timelineEvents store
 export default class EventKey {
     constructor(fragmentId, eventIndex) {
         this.fragmentId = fragmentId;
@@ -14,6 +15,18 @@ export default class EventKey {
         return new EventKey(this.fragmentId + 1, MID_EVENT_INDEX);
     }
 
+    nextKeyForDirection(direction) {
+        if (direction.isForward) {
+            return this.nextKey();
+        } else {
+            return this.previousKey();
+        }
+    }
+
+    previousKey() {
+        return new EventKey(this.fragmentId, this.eventIndex - 1);
+    }
+
     nextKey() {
         return new EventKey(this.fragmentId, this.eventIndex + 1);
     }
diff --git a/src/matrix/room/timeline/entries/BaseEntry.js b/src/matrix/room/timeline/entries/BaseEntry.js
index 56566efc..37695ddf 100644
--- a/src/matrix/room/timeline/entries/BaseEntry.js
+++ b/src/matrix/room/timeline/entries/BaseEntry.js
@@ -1,6 +1,10 @@
 //entries can be sorted, first by fragment, then by entry index.
 
 export default class BaseEntry {
+    constructor(fragmentIdComparer) {
+        this._fragmentIdComparer = fragmentIdComparer;
+    }
+
     get fragmentId() {
         throw new Error("unimplemented");
     }
diff --git a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
index 610d32c6..0a336975 100644
--- a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
+++ b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
@@ -1,4 +1,5 @@
 import BaseEntry from "./BaseEntry.js";
+import Direction from "../Direction.js";
 
 export default class FragmentBoundaryEntry extends BaseEntry {
     constructor(fragment, isFragmentStart, fragmentIdComparator) {
@@ -15,12 +16,12 @@ export default class FragmentBoundaryEntry extends BaseEntry {
         return new FragmentBoundaryEntry(fragment, false, fragmentIdComparator);
     }
     
-    get hasStarted() {
+    get started() {
         return this._isFragmentStart;
     }
 
     get hasEnded() {
-        return !this.hasStarted;
+        return !this.started;
     }
 
     get fragment() {
@@ -32,7 +33,7 @@ export default class FragmentBoundaryEntry extends BaseEntry {
     }
 
     get entryIndex() {
-        if (this.hasStarted) {
+        if (this.started) {
             return Number.MIN_SAFE_INTEGER;
         } else {
             return Number.MAX_SAFE_INTEGER;
@@ -40,10 +41,54 @@ export default class FragmentBoundaryEntry extends BaseEntry {
     }
 
     get isGap() {
-        if (this.hasStarted) {
-            return !!this.fragment.nextToken;
+        return !!this.token;
+    }
+
+    get token() {
+        if (this.started) {
+            return this.fragment.nextToken;
         } else {
-            return !!this.fragment.previousToken;
+            return this.fragment.previousToken;
         }
     }
+
+    set token(token) {
+        if (this.started) {
+            this.fragment.nextToken = token;
+        } else {
+            this.fragment.previousToken = token;
+        }
+    }
+
+    get linkedFragmentId() {
+        if (this.started) {
+            return this.fragment.nextId;
+        } else {
+            return this.fragment.previousId;
+        }
+    }
+
+    set linkedFragmentId(id) {
+        if (this.started) {
+            this.fragment.nextId = id;
+        } else {
+            this.fragment.previousId = id;
+        }
+    }
+
+    get direction() {
+        if (this.started) {
+            return Direction.Backward;
+        } else {
+            return Direction.Forward;
+        }
+    }
+
+    withUpdatedFragment(fragment) {
+        return new FragmentBoundaryEntry(fragment, this._isFragmentStart, this._fragmentIdComparator);
+    }
+
+    createNeighbourEntry(neighbour) {
+        return new FragmentBoundaryEntry(neighbour, !this._isFragmentStart, this._fragmentIdComparator);
+    }
 }
diff --git a/src/matrix/room/timeline/persistence/GapPersister.js b/src/matrix/room/timeline/persistence/GapPersister.js
new file mode 100644
index 00000000..b412d0aa
--- /dev/null
+++ b/src/matrix/room/timeline/persistence/GapPersister.js
@@ -0,0 +1,208 @@
+import EventKey from "../EventKey.js";
+import EventEntry from "../entries/EventEntry.js";
+import {createEventEntry} from "./common.js";
+
+function directionalAppend(array, value, direction) {
+    if (direction.isForward) {
+        array.push(value);
+    } else {
+        array.splice(0, 0, value);
+    }
+}
+
+export default class GapPersister {
+    constructor({roomId, storage, fragmentIdComparer}) {
+        this._roomId = roomId;
+        this._storage = storage;
+        this._fragmentIdComparer = fragmentIdComparer;
+    }
+    async persistFragmentFill(fragmentEntry, response) {
+        const {fragmentId, direction} = fragmentEntry;
+        // assuming that chunk is in chronological order when backwards too?
+        const {chunk, start, end} = response;
+        let entries;
+
+        if (!Array.isArray(chunk)) {
+            throw new Error("Invalid chunk in response");
+        }
+        if (typeof end !== "string") {
+            throw new Error("Invalid end token in response");
+        }
+
+        const txn = await this._storage.readWriteTxn([
+            this._storage.storeNames.timelineEvents,
+            this._storage.storeNames.timelineFragments,
+        ]);
+
+        try {
+            // make sure we have the latest fragment from the store
+            const fragment = await txn.timelineFragments.get(fragmentId);
+            if (!fragment) {
+                throw new Error(`Unknown fragment: ${fragmentId}`);
+            }
+            fragmentEntry = fragmentEntry.withUpdatedFragment(fragment);
+            // check that the request was done with the token we are aware of (extra care to avoid timeline corruption)
+            if (fragmentEntry.token !== start) {
+                throw new Error("start is not equal to prev_batch or next_batch");
+            }
+            // find last event in fragment so we get the eventIndex to begin creating keys at
+            let currentKey;
+            if (direction.isBackward) {
+                const [firstEvent] = await txn.timelineEvents.firstEvents(this._roomId, fragmentId, 1);
+                currentKey = new EventKey(firstEvent.fragmentId, firstEvent.eventIndex);
+            } else {
+                const [lastEvent] = await txn.timelineEvents.lastEvents(this._roomId, fragmentId, 1);
+                currentKey = new EventKey(lastEvent.fragmentId, lastEvent.eventIndex);
+            }
+            // find out if any event in chunk is already present using findFirstOrLastOccurringEventId
+            const eventIds = chunk.map(e => e.event_id);
+            const findLast = direction.isBackward;
+            let nonOverlappingEvents = chunk;
+            let neighbourFragmentEntry;
+            const neighbourEventId = await txn.timelineEvents.findFirstOrLastOccurringEventId(this._roomId, eventIds, findLast);
+            if (neighbourEventId) {
+                // trim overlapping events
+                const neighbourEventIndex = chunk.findIndex(e => e.event_id === neighbourEventId);
+                const start = direction.isBackward ? neighbourEventIndex + 1 : 0;
+                const end = direction.isBackward ? chunk.length : neighbourEventIndex;
+                nonOverlappingEvents = chunk.slice(start, end);
+                // get neighbour fragment to link it up later on
+                const neighbourEvent = await txn.timelineEvents.getByEventId(this._roomId, neighbourEventId);
+                const neighbourFragment = await txn.timelineFragments.get(neighbourEvent.fragmentId);
+                neighbourFragmentEntry = fragmentEntry.createNeighbourEntry(neighbourFragment);
+            }
+
+            // create entries for all events in chunk, add them to entries
+            entries = new Array(nonOverlappingEvents.length);
+            const reducer = direction.isBackward ? Array.prototype.reduceRight : Array.prototype.reduce;
+            currentKey = reducer.call(nonOverlappingEvents, (key, event, i) => {
+                key = key.nextKeyForDirection(direction);
+                const eventEntry = createEventEntry(currentKey, event);
+                txn.timelineEvents.insert(eventEntry);
+                entries[i] = new EventEntry(eventEntry, this._fragmentIdComparer);
+            }, currentKey);
+
+            directionalAppend(entries, fragmentEntry, direction);
+            // set `end` as token, and if we found an event in the step before, link up the fragments in the fragment entry
+            if (neighbourFragmentEntry) {
+                fragmentEntry.linkedFragmentId = neighbourFragmentEntry.fragmentId;
+                neighbourFragmentEntry.linkedFragmentId = fragmentEntry.fragmentId;
+                txn.timelineFragments.set(neighbourFragmentEntry.fragment);
+                directionalAppend(entries, neighbourFragmentEntry, direction);
+
+                // update fragmentIdComparer here after linking up fragments?
+                this._fragmentIdComparer.rebuild(await txn.timelineFragments.all());
+            }
+            fragmentEntry.token = end;
+            txn.timelineFragments.set(fragmentEntry.fragment);
+        } catch (err) {
+            txn.abort();
+            throw err;
+        }
+
+        await txn.complete();
+
+        return entries;
+    }
+}
+
+//#ifdef TESTS
+//import MemoryStorage from "../storage/memory/MemoryStorage.js";
+
+export function xtests() {
+    const roomId = "!abc:hs.tld";
+
+    // sets sortKey and roomId on an array of entries
+    function createTimeline(roomId, entries) {
+        let key = new SortKey();
+        for (let entry of entries) {
+            if (entry.gap && entry.gap.prev_batch) {
+                key = key.nextKeyWithGap();
+            }
+            entry.sortKey = key;
+            if (entry.gap && entry.gap.next_batch) {
+                key = key.nextKeyWithGap();
+            } else if (!entry.gap) {
+                key = key.nextKey();
+            }
+            entry.roomId = roomId;
+        }
+    }
+
+    function areSorted(entries) {
+        for (var i = 1; i < entries.length; i++) {
+            const isSorted = entries[i - 1].sortKey.compare(entries[i].sortKey) < 0;
+            if(!isSorted) {
+                return false
+            }
+        }
+        return true;
+    }
+
+    return {
+        "test backwards gap fill with overlapping neighbouring event": async function(assert) {
+            const currentPaginationToken = "abc";
+            const gap = {gap: {prev_batch: currentPaginationToken}};
+            const storage = new MemoryStorage({roomTimeline: createTimeline(roomId, [
+                {event: {event_id: "b"}},
+                {gap: {next_batch: "ghi"}},
+                gap,
+            ])});
+            const persister = new RoomPersister({roomId, storage});
+            const response = {
+                start: currentPaginationToken,
+                end: "def",
+                chunk: [
+                    {event_id: "a"},
+                    {event_id: "b"},
+                    {event_id: "c"},
+                    {event_id: "d"},
+                ]
+            };
+            const {newEntries, replacedEntries} = await persister.persistGapFill(gap, response);
+            // should only have taken events up till existing event
+            assert.equal(newEntries.length, 2);
+            assert.equal(newEntries[0].event.event_id, "c");
+            assert.equal(newEntries[1].event.event_id, "d");
+            assert.equal(replacedEntries.length, 2);
+            assert.equal(replacedEntries[0].gap.next_batch, "hij");
+            assert.equal(replacedEntries[1].gap.prev_batch, currentPaginationToken);
+            assert(areSorted(newEntries));
+            assert(areSorted(replacedEntries));
+        },
+        "test backwards gap fill with non-overlapping neighbouring event": async function(assert) {
+            const currentPaginationToken = "abc";
+            const newPaginationToken = "def";
+            const gap = {gap: {prev_batch: currentPaginationToken}};
+            const storage = new MemoryStorage({roomTimeline: createTimeline(roomId, [
+                {event: {event_id: "a"}},
+                {gap: {next_batch: "ghi"}},
+                gap,
+            ])});
+            const persister = new RoomPersister({roomId, storage});
+            const response = {
+                start: currentPaginationToken,
+                end: newPaginationToken,
+                chunk: [
+                    {event_id: "c"},
+                    {event_id: "d"},
+                    {event_id: "e"},
+                    {event_id: "f"},
+                ]
+            };
+            const {newEntries, replacedEntries} = await persister.persistGapFill(gap, response);
+            // should only have taken events up till existing event
+            assert.equal(newEntries.length, 5);
+            assert.equal(newEntries[0].gap.prev_batch, newPaginationToken);
+            assert.equal(newEntries[1].event.event_id, "c");
+            assert.equal(newEntries[2].event.event_id, "d");
+            assert.equal(newEntries[3].event.event_id, "e");
+            assert.equal(newEntries[4].event.event_id, "f");
+            assert(areSorted(newEntries));
+
+            assert.equal(replacedEntries.length, 1);
+            assert.equal(replacedEntries[0].gap.prev_batch, currentPaginationToken);
+        },
+    }
+}
+//#endif
diff --git a/src/matrix/storage/idb/stores/RoomFragmentStore.js b/src/matrix/storage/idb/stores/TimelineFragmentStore.js
similarity index 100%
rename from src/matrix/storage/idb/stores/RoomFragmentStore.js
rename to src/matrix/storage/idb/stores/TimelineFragmentStore.js

From 75100c1c60ed89d52092039020a089388eece19e Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 12 May 2019 20:26:03 +0200
Subject: [PATCH 43/83] adjust Timeline to changes, gap persister

---
 src/matrix/room/room.js                       |  3 +-
 .../{timeline.js => timeline/Timeline.js}     | 42 +++++++++----------
 2 files changed, 21 insertions(+), 24 deletions(-)
 rename src/matrix/room/{timeline.js => timeline/Timeline.js} (61%)

diff --git a/src/matrix/room/room.js b/src/matrix/room/room.js
index b2ad1312..c6e75e95 100644
--- a/src/matrix/room/room.js
+++ b/src/matrix/room/room.js
@@ -1,7 +1,7 @@
 import EventEmitter from "../../EventEmitter.js";
 import RoomSummary from "./summary.js";
-import Timeline from "./timeline.js";
 import SyncPersister from "./timeline/persistence/SyncPersister.js";
+import Timeline from "./timeline/Timeline.js";
 import FragmentIdComparer from "./timeline/FragmentIdComparer.js";
 
 export default class Room extends EventEmitter {
@@ -54,6 +54,7 @@ export default class Room extends EventEmitter {
             roomId: this.id,
             storage: this._storage,
             hsApi: this._hsApi,
+            fragmentIdComparer: this._fragmentIdComparer,
             closeCallback: () => this._timeline = null,
         });
         await this._timeline.load();
diff --git a/src/matrix/room/timeline.js b/src/matrix/room/timeline/Timeline.js
similarity index 61%
rename from src/matrix/room/timeline.js
rename to src/matrix/room/timeline/Timeline.js
index b4b61fe4..fcadeadf 100644
--- a/src/matrix/room/timeline.js
+++ b/src/matrix/room/timeline/Timeline.js
@@ -1,17 +1,19 @@
-import { ObservableArray } from "../../observable/index.js";
-import sortedIndex from "../../utils/sortedIndex.js";
+import { ObservableArray } from "../../../observable/index.js";
+import sortedIndex from "../../../utils/sortedIndex.js";
+import GapPersister from "./persistence/GapPersister.js";
 
 export default class Timeline {
-    constructor({roomId, storage, closeCallback}) {
+    constructor({roomId, storage, closeCallback, fragmentIdComparer}) {
         this._roomId = roomId;
         this._storage = storage;
         this._closeCallback = closeCallback;
         this._entriesList = new ObservableArray();
+        this._fragmentIdComparer = fragmentIdComparer;
     }
 
     /** @package */
     async load() {
-        const txn = await this._storage.readTxn([this._storage.storeNames.roomTimeline]);
+        const txn = await this._storage.readTxn([this._storage.storeNames.timelineEvents]);
         const entries = await txn.roomTimeline.lastEvents(this._roomId, 100);
         for (const entry of entries) {
             this._entriesList.append(entry);
@@ -26,31 +28,25 @@ export default class Timeline {
     }
 
     /** @public */
-    async fillGap(gapEntry, amount) {
-        const gap = gapEntry.gap;
-        let direction;
-        if (gap.prev_batch) {
-            direction = "b";
-        } else if (gap.next_batch) {
-            direction = "f";
-        } else {
-            throw new Error("Invalid gap, no prev_batch or next_batch field: " + JSON.stringify(gapEntry.gap));
-        }
-        const token = gap.prev_batch || gap.next_batch;
-
+    async fillGap(fragmentEntry, amount) {
         const response = await this._hsApi.messages(this._roomId, {
-            from: token,
-            dir: direction,
+            from: fragmentEntry.token,
+            dir: fragmentEntry.direction.asApiString(),
             limit: amount
         });
 
-        const newEntries = await this._persister.persistGapFill(gapEntry, response);
+        const gapPersister = new GapPersister({
+            roomId: this._roomId,
+            storage: this._storage,
+            fragmentIdComparer: this._fragmentIdComparer
+        });
+        const newEntries = await gapPersister.persistFragmentFill(fragmentEntry, response);
         // find where to replace existing gap with newEntries by doing binary search
-        const gapIdx = sortedIndex(this._entriesList.array, gapEntry.sortKey, (key, entry) => {
-            return key.compare(entry.sortKey);
+        const gapIdx = sortedIndex(this._entriesList.array, fragmentEntry, (fragmentEntry, entry) => {
+            return fragmentEntry.compare(entry);
         });
         // only replace the gap if it's currently in the timeline
-        if (this._entriesList.at(gapIdx) === gapEntry) {
+        if (this._entriesList.at(gapIdx) === fragmentEntry) {
             this._entriesList.removeAt(gapIdx);
             this._entriesList.insertMany(gapIdx, newEntries);
         }
@@ -59,7 +55,7 @@ export default class Timeline {
     async loadAtTop(amount) {
         const firstEntry = this._entriesList.at(0);
         if (firstEntry) {
-            const txn = await this._storage.readTxn([this._storage.storeNames.roomTimeline]);
+            const txn = await this._storage.readTxn([this._storage.storeNames.timelineEvents]);
             const topEntries = await txn.roomTimeline.eventsBefore(this._roomId, firstEntry.sortKey, amount);
             this._entriesList.insertMany(0, topEntries);
             return topEntries.length;

From da5e8794abfa7b96c8c43be5595dbb4243befa26 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 12 May 2019 20:26:20 +0200
Subject: [PATCH 44/83] lint

---
 src/matrix/hs-api.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/matrix/hs-api.js b/src/matrix/hs-api.js
index 66eed71b..4c6ebf86 100644
--- a/src/matrix/hs-api.js
+++ b/src/matrix/hs-api.js
@@ -33,7 +33,7 @@ export default class HomeServerApi {
 
 	_request(method, csPath, queryParams = {}, body) {
 		const queryString = Object.entries(queryParams)
-			.filter(([name, value]) => value !== undefined)
+			.filter(([, value]) => value !== undefined)
 			.map(([name, value]) => `${encodeURIComponent(name)}=${encodeURIComponent(value)}`)
 			.join("&");
 		const url = this._url(`${csPath}?${queryString}`);

From e3328f0fefcd308c780b1a18d17ee512040a681a Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 12 May 2019 20:26:32 +0200
Subject: [PATCH 45/83] add fragments store name

---
 src/matrix/storage/common.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/matrix/storage/common.js b/src/matrix/storage/common.js
index 0280549b..d1050e5a 100644
--- a/src/matrix/storage/common.js
+++ b/src/matrix/storage/common.js
@@ -1,4 +1,4 @@
-export const STORE_NAMES = Object.freeze(["session", "roomState", "roomSummary", "roomTimeline"]);
+export const STORE_NAMES = Object.freeze(["session", "roomState", "roomSummary", "timelineEvents", "timelineFragments"]);
 
 export const STORE_MAP = Object.freeze(STORE_NAMES.reduce((nameMap, name) => {
     nameMap[name] = name;

From 10457611f9d35b4659a2b37e4844ec8115451745 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 12 May 2019 20:26:46 +0200
Subject: [PATCH 46/83] whitespace

---
 src/matrix/session.js |  76 ++++++++--------
 src/matrix/sync.js    | 202 +++++++++++++++++++++---------------------
 2 files changed, 139 insertions(+), 139 deletions(-)

diff --git a/src/matrix/session.js b/src/matrix/session.js
index 9dbeb010..d86065dc 100644
--- a/src/matrix/session.js
+++ b/src/matrix/session.js
@@ -3,59 +3,59 @@ import { ObservableMap } from "../observable/index.js";
 
 export default class Session {
     // sessionInfo contains deviceId, userId and homeServer
-	constructor({storage, hsApi, sessionInfo}) {
-		this._storage = storage;
+    constructor({storage, hsApi, sessionInfo}) {
+        this._storage = storage;
         this._hsApi = hsApi;
-		this._session = null;
+        this._session = null;
         this._sessionInfo = sessionInfo;
-		this._rooms = new ObservableMap();
+        this._rooms = new ObservableMap();
         this._roomUpdateCallback = (room, params) => this._rooms.update(room.id, params);
-	}
+    }
 
-	async load() {
-		const txn = await this._storage.readTxn([
-			this._storage.storeNames.session,
-			this._storage.storeNames.roomSummary,
-			this._storage.storeNames.roomState,
-			this._storage.storeNames.roomTimeline,
-		]);
-		// restore session object
-		this._session = await txn.session.get();
-		if (!this._session) {
+    async load() {
+        const txn = await this._storage.readTxn([
+            this._storage.storeNames.session,
+            this._storage.storeNames.roomSummary,
+            this._storage.storeNames.roomState,
+            this._storage.storeNames.timelineEvents,
+        ]);
+        // restore session object
+        this._session = await txn.session.get();
+        if (!this._session) {
             this._session = {};
-			return;
-		}
-		// load rooms
-		const rooms = await txn.roomSummary.getAll();
-		await Promise.all(rooms.map(summary => {
-			const room = this.createRoom(summary.roomId);
-			return room.load(summary, txn);
-		}));
-	}
+            return;
+        }
+        // load rooms
+        const rooms = await txn.roomSummary.getAll();
+        await Promise.all(rooms.map(summary => {
+            const room = this.createRoom(summary.roomId);
+            return room.load(summary, txn);
+        }));
+    }
 
     get rooms() {
         return this._rooms;
     }
 
-	createRoom(roomId) {
-		const room = new Room({
+    createRoom(roomId) {
+        const room = new Room({
             roomId,
             storage: this._storage,
             emitCollectionChange: this._roomUpdateCallback,
             hsApi: this._hsApi,
         });
-		this._rooms.add(roomId, room);
-		return room;
-	}
+        this._rooms.add(roomId, room);
+        return room;
+    }
 
-	persistSync(syncToken, accountData, txn) {
-		if (syncToken !== this._session.syncToken) {
-			this._session.syncToken = syncToken;
-			txn.session.set(this._session);
-		}
-	}
+    persistSync(syncToken, accountData, txn) {
+        if (syncToken !== this._session.syncToken) {
+            this._session.syncToken = syncToken;
+            txn.session.set(this._session);
+        }
+    }
 
-	get syncToken() {
-		return this._session.syncToken;
-	}
+    get syncToken() {
+        return this._session.syncToken;
+    }
 }
diff --git a/src/matrix/sync.js b/src/matrix/sync.js
index 4ad89466..b96f2110 100644
--- a/src/matrix/sync.js
+++ b/src/matrix/sync.js
@@ -1,7 +1,7 @@
 import {
-	RequestAbortError,
-	HomeServerError,
-	StorageError
+    RequestAbortError,
+    HomeServerError,
+    StorageError
 } from "./error.js";
 import EventEmitter from "../EventEmitter.js";
 
@@ -9,119 +9,119 @@ const INCREMENTAL_TIMEOUT = 30000;
 const SYNC_EVENT_LIMIT = 10;
 
 function parseRooms(roomsSection, roomCallback) {
-	if (!roomsSection) {
-		return;
-	}
-	const allMemberships = ["join", "invite", "leave"];
-	for(const membership of allMemberships) {
-		const membershipSection = roomsSection[membership];
-		if (membershipSection) {
-			const rooms = Object.entries(membershipSection)
-			for (const [roomId, roomResponse] of rooms) {
-				roomCallback(roomId, roomResponse, membership);
-			}
-		}
-	}
+    if (!roomsSection) {
+        return;
+    }
+    const allMemberships = ["join", "invite", "leave"];
+    for(const membership of allMemberships) {
+        const membershipSection = roomsSection[membership];
+        if (membershipSection) {
+            const rooms = Object.entries(membershipSection)
+            for (const [roomId, roomResponse] of rooms) {
+                roomCallback(roomId, roomResponse, membership);
+            }
+        }
+    }
 }
 
 export default class Sync extends EventEmitter {
-	constructor(hsApi, session, storage) {
-		super();
-		this._hsApi = hsApi;
-		this._session = session;
-		this._storage = storage;
-		this._isSyncing = false;
-		this._currentRequest = null;
-	}
-	// returns when initial sync is done
-	async start() {
-		if (this._isSyncing) {
-			return;
-		}
-		this._isSyncing = true;
-		let syncToken = this._session.syncToken;
-		// do initial sync if needed
-		if (!syncToken) {
-			// need to create limit filter here
-			syncToken = await this._syncRequest();
-		}
-		this._syncLoop(syncToken);
-	}
+    constructor(hsApi, session, storage) {
+        super();
+        this._hsApi = hsApi;
+        this._session = session;
+        this._storage = storage;
+        this._isSyncing = false;
+        this._currentRequest = null;
+    }
+    // returns when initial sync is done
+    async start() {
+        if (this._isSyncing) {
+            return;
+        }
+        this._isSyncing = true;
+        let syncToken = this._session.syncToken;
+        // do initial sync if needed
+        if (!syncToken) {
+            // need to create limit filter here
+            syncToken = await this._syncRequest();
+        }
+        this._syncLoop(syncToken);
+    }
 
-	async _syncLoop(syncToken) {
-		// if syncToken is falsy, it will first do an initial sync ... 
-		while(this._isSyncing) {
-			try {
-				console.log(`starting sync request with since ${syncToken} ...`);
-				syncToken = await this._syncRequest(syncToken, INCREMENTAL_TIMEOUT);
-			} catch (err) {
-				this._isSyncing = false;
-				if (!(err instanceof RequestAbortError)) {
-					console.warn("stopping sync because of error");
-					this.emit("error", err);
-				}
-			}
-		}
-		this.emit("stopped");
-	}
+    async _syncLoop(syncToken) {
+        // if syncToken is falsy, it will first do an initial sync ... 
+        while(this._isSyncing) {
+            try {
+                console.log(`starting sync request with since ${syncToken} ...`);
+                syncToken = await this._syncRequest(syncToken, INCREMENTAL_TIMEOUT);
+            } catch (err) {
+                this._isSyncing = false;
+                if (!(err instanceof RequestAbortError)) {
+                    console.warn("stopping sync because of error");
+                    this.emit("error", err);
+                }
+            }
+        }
+        this.emit("stopped");
+    }
 
-	async _syncRequest(syncToken, timeout) {
-		this._currentRequest = this._hsApi.sync(syncToken, undefined, timeout);
-		const response = await this._currentRequest.response();
-		syncToken = response.next_batch;
-		const storeNames = this._storage.storeNames;
-		const syncTxn = await this._storage.readWriteTxn([
-			storeNames.session,
-			storeNames.roomSummary,
-			storeNames.roomTimeline,
-			storeNames.roomState,
-		]);
+    async _syncRequest(syncToken, timeout) {
+        this._currentRequest = this._hsApi.sync(syncToken, undefined, timeout);
+        const response = await this._currentRequest.response();
+        syncToken = response.next_batch;
+        const storeNames = this._storage.storeNames;
+        const syncTxn = await this._storage.readWriteTxn([
+            storeNames.session,
+            storeNames.roomSummary,
+            storeNames.timelineEvents,
+            storeNames.roomState,
+        ]);
         const roomChanges = [];
         try {
             this._session.persistSync(syncToken, response.account_data, syncTxn);
             // to_device
             // presence
-			if (response.rooms) {
-				parseRooms(response.rooms, (roomId, roomResponse, membership) => {
-					let room = this._session.rooms.get(roomId);
-					if (!room) {
-						room = this._session.createRoom(roomId);
-					}
-					console.log(` * applying sync response to room ${roomId} ...`);
-					const changes = room.persistSync(roomResponse, membership, syncTxn);
+            if (response.rooms) {
+                parseRooms(response.rooms, (roomId, roomResponse, membership) => {
+                    let room = this._session.rooms.get(roomId);
+                    if (!room) {
+                        room = this._session.createRoom(roomId);
+                    }
+                    console.log(` * applying sync response to room ${roomId} ...`);
+                    const changes = room.persistSync(roomResponse, membership, syncTxn);
                     roomChanges.push({room, changes});
-				});
-			}
-		} catch(err) {
-			console.warn("aborting syncTxn because of error");
-			// avoid corrupting state by only
-			// storing the sync up till the point
-			// the exception occurred
-			syncTxn.abort();
-			throw err;
-		}
-		try {
-			await syncTxn.complete();
-			console.info("syncTxn committed!!");
-		} catch (err) {
-			throw new StorageError("unable to commit sync tranaction", err);
-		}
+                });
+            }
+        } catch(err) {
+            console.warn("aborting syncTxn because of error");
+            // avoid corrupting state by only
+            // storing the sync up till the point
+            // the exception occurred
+            syncTxn.abort();
+            throw err;
+        }
+        try {
+            await syncTxn.complete();
+            console.info("syncTxn committed!!");
+        } catch (err) {
+            throw new StorageError("unable to commit sync tranaction", err);
+        }
         // emit room related events after txn has been closed
         for(let {room, changes} of roomChanges) {
             room.emitSync(changes);
         }
 
-		return syncToken;
-	}
+        return syncToken;
+    }
 
-	stop() {
-		if (!this._isSyncing) {
-			return;
-		}
-		this._isSyncing = false;
-		if (this._currentRequest) {
-			this._currentRequest.abort();
-			this._currentRequest = null;
-		}
-	}
+    stop() {
+        if (!this._isSyncing) {
+            return;
+        }
+        this._isSyncing = false;
+        if (this._currentRequest) {
+            this._currentRequest.abort();
+            this._currentRequest = null;
+        }
+    }
 }

From 3324fd3afd4bc0c3a85c4420118596ad0d4f792f Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 12 May 2019 20:41:14 +0200
Subject: [PATCH 47/83] split up persistFragmentFill method into smaller ones

---
 .../room/timeline/persistence/GapPersister.js | 116 +++++++++++-------
 1 file changed, 70 insertions(+), 46 deletions(-)

diff --git a/src/matrix/room/timeline/persistence/GapPersister.js b/src/matrix/room/timeline/persistence/GapPersister.js
index b412d0aa..8eabb226 100644
--- a/src/matrix/room/timeline/persistence/GapPersister.js
+++ b/src/matrix/room/timeline/persistence/GapPersister.js
@@ -16,6 +16,69 @@ export default class GapPersister {
         this._storage = storage;
         this._fragmentIdComparer = fragmentIdComparer;
     }
+
+    async _findOverlappingEvents(fragmentEntry, events, txn) {
+        const eventIds = events.map(e => e.event_id);
+        const {direction} = fragmentEntry;
+        const findLast = direction.isBackward;
+        let nonOverlappingEvents = events;
+        let neighbourFragmentEntry;
+        const neighbourEventId = await txn.timelineEvents.findFirstOrLastOccurringEventId(this._roomId, eventIds, findLast);
+        if (neighbourEventId) {
+            // trim overlapping events
+            const neighbourEventIndex = events.findIndex(e => e.event_id === neighbourEventId);
+            const start = direction.isBackward ? neighbourEventIndex + 1 : 0;
+            const end = direction.isBackward ? events.length : neighbourEventIndex;
+            nonOverlappingEvents = events.slice(start, end);
+            // get neighbour fragment to link it up later on
+            const neighbourEvent = await txn.timelineEvents.getByEventId(this._roomId, neighbourEventId);
+            const neighbourFragment = await txn.timelineFragments.get(neighbourEvent.fragmentId);
+            neighbourFragmentEntry = fragmentEntry.createNeighbourEntry(neighbourFragment);
+        }
+
+        return {nonOverlappingEvents, neighbourFragmentEntry};
+    }
+
+    async _findLastFragmentEventKey(fragmentEntry, txn) {
+        const {fragmentId, direction} = fragmentEntry;
+        if (direction.isBackward) {
+            const [firstEvent] = await txn.timelineEvents.firstEvents(this._roomId, fragmentId, 1);
+            return new EventKey(firstEvent.fragmentId, firstEvent.eventIndex);
+        } else {
+            const [lastEvent] = await txn.timelineEvents.lastEvents(this._roomId, fragmentId, 1);
+            return new EventKey(lastEvent.fragmentId, lastEvent.eventIndex);
+        }
+    }
+
+    _storeEvents(events, startKey, direction, txn) {
+        const entries = new Array(events.length);
+        const reducer = direction.isBackward ? Array.prototype.reduceRight : Array.prototype.reduce;
+        reducer.call(events, (key, event, i) => {
+            key = key.nextKeyForDirection(direction);
+            const eventEntry = createEventEntry(key, event);
+            txn.timelineEvents.insert(eventEntry);
+            entries[i] = new EventEntry(eventEntry, this._fragmentIdComparer);
+        }, startKey);
+        return entries;
+    }
+
+    async _updateFragments(fragmentEntry, neighbourFragmentEntry, end, entries, txn) {
+        const {direction} = fragmentEntry;
+        directionalAppend(entries, fragmentEntry, direction);
+        // set `end` as token, and if we found an event in the step before, link up the fragments in the fragment entry
+        if (neighbourFragmentEntry) {
+            fragmentEntry.linkedFragmentId = neighbourFragmentEntry.fragmentId;
+            neighbourFragmentEntry.linkedFragmentId = fragmentEntry.fragmentId;
+            txn.timelineFragments.set(neighbourFragmentEntry.fragment);
+            directionalAppend(entries, neighbourFragmentEntry, direction);
+
+            // update fragmentIdComparer here after linking up fragments?
+            this._fragmentIdComparer.rebuild(await txn.timelineFragments.all());
+        }
+        fragmentEntry.token = end;
+        txn.timelineFragments.set(fragmentEntry.fragment);
+    }
+
     async persistFragmentFill(fragmentEntry, response) {
         const {fragmentId, direction} = fragmentEntry;
         // assuming that chunk is in chronological order when backwards too?
@@ -46,55 +109,16 @@ export default class GapPersister {
                 throw new Error("start is not equal to prev_batch or next_batch");
             }
             // find last event in fragment so we get the eventIndex to begin creating keys at
-            let currentKey;
-            if (direction.isBackward) {
-                const [firstEvent] = await txn.timelineEvents.firstEvents(this._roomId, fragmentId, 1);
-                currentKey = new EventKey(firstEvent.fragmentId, firstEvent.eventIndex);
-            } else {
-                const [lastEvent] = await txn.timelineEvents.lastEvents(this._roomId, fragmentId, 1);
-                currentKey = new EventKey(lastEvent.fragmentId, lastEvent.eventIndex);
-            }
+            let lastKey = this._findLastFragmentEventKey(fragmentEntry, txn);
             // find out if any event in chunk is already present using findFirstOrLastOccurringEventId
-            const eventIds = chunk.map(e => e.event_id);
-            const findLast = direction.isBackward;
-            let nonOverlappingEvents = chunk;
-            let neighbourFragmentEntry;
-            const neighbourEventId = await txn.timelineEvents.findFirstOrLastOccurringEventId(this._roomId, eventIds, findLast);
-            if (neighbourEventId) {
-                // trim overlapping events
-                const neighbourEventIndex = chunk.findIndex(e => e.event_id === neighbourEventId);
-                const start = direction.isBackward ? neighbourEventIndex + 1 : 0;
-                const end = direction.isBackward ? chunk.length : neighbourEventIndex;
-                nonOverlappingEvents = chunk.slice(start, end);
-                // get neighbour fragment to link it up later on
-                const neighbourEvent = await txn.timelineEvents.getByEventId(this._roomId, neighbourEventId);
-                const neighbourFragment = await txn.timelineFragments.get(neighbourEvent.fragmentId);
-                neighbourFragmentEntry = fragmentEntry.createNeighbourEntry(neighbourFragment);
-            }
+            const {
+                nonOverlappingEvents,
+                neighbourFragmentEntry
+            } = this._findOverlappingEvents(fragmentEntry, chunk, txn);
 
             // create entries for all events in chunk, add them to entries
-            entries = new Array(nonOverlappingEvents.length);
-            const reducer = direction.isBackward ? Array.prototype.reduceRight : Array.prototype.reduce;
-            currentKey = reducer.call(nonOverlappingEvents, (key, event, i) => {
-                key = key.nextKeyForDirection(direction);
-                const eventEntry = createEventEntry(currentKey, event);
-                txn.timelineEvents.insert(eventEntry);
-                entries[i] = new EventEntry(eventEntry, this._fragmentIdComparer);
-            }, currentKey);
-
-            directionalAppend(entries, fragmentEntry, direction);
-            // set `end` as token, and if we found an event in the step before, link up the fragments in the fragment entry
-            if (neighbourFragmentEntry) {
-                fragmentEntry.linkedFragmentId = neighbourFragmentEntry.fragmentId;
-                neighbourFragmentEntry.linkedFragmentId = fragmentEntry.fragmentId;
-                txn.timelineFragments.set(neighbourFragmentEntry.fragment);
-                directionalAppend(entries, neighbourFragmentEntry, direction);
-
-                // update fragmentIdComparer here after linking up fragments?
-                this._fragmentIdComparer.rebuild(await txn.timelineFragments.all());
-            }
-            fragmentEntry.token = end;
-            txn.timelineFragments.set(fragmentEntry.fragment);
+            entries = this._storeEvents(nonOverlappingEvents, lastKey, direction, txn);
+            await this._updateFragments(fragmentEntry, neighbourFragmentEntry, end, entries, txn);
         } catch (err) {
             txn.abort();
             throw err;

From 784588440c09797b71a3d80fcca04560569c6362 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 19 May 2019 20:49:46 +0200
Subject: [PATCH 48/83] WIP for fragment support

---
 doc/FRAGMENTS.md                              | 18 ++++--
 .../session/room/timeline/TilesCollection.js  |  1 +
 .../room/timeline/FragmentIdComparer.js       | 10 +--
 src/matrix/room/timeline/Timeline.js          |  7 +++
 .../room/timeline/persistence/GapPersister.js | 10 +--
 .../timeline/persistence/SyncPersister.js     | 17 +++--
 .../timeline/persistence/TimelineReader.js    | 63 +++++++++++++++++++
 .../room/timeline/persistence/common.js       | 18 +++++-
 src/matrix/session.js                         |  1 +
 src/matrix/sync.js                            |  1 +
 src/observable/list/SortedArray.js            | 38 +++++++++++
 11 files changed, 159 insertions(+), 25 deletions(-)
 create mode 100644 src/matrix/room/timeline/persistence/TimelineReader.js
 create mode 100644 src/observable/list/SortedArray.js

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
index a63a96ea..af944ed7 100644
--- a/doc/FRAGMENTS.md
+++ b/doc/FRAGMENTS.md
@@ -5,21 +5,31 @@
     - SortKey
         - FragmentId
         - EventIndex
- - write fragmentStore
+ - DONE: write fragmentStore
     - load all fragments
     - add a fragment (live on limited sync, or /context)
     - connect two fragments
     - update token on fragment (when filling gap or connecting two fragments)
 
     fragments can need connecting when filling a gap or creating a new /context fragment
- - adapt timelineStore
+ - DONE: adapt timelineStore
 
     how will fragments be exposed in timeline store?
         - all read operations are passed a fragment id
  - adapt persister
     - DONE: persist fragments in /sync
-    - load n items before and after key
-    - fill gaps / fragment filling
+    - DONE: fill gaps / fragment filling
+    - load n items before and after key,
+        - need to add fragments as we come across boundaries
+        - also cache fragments? not for now ...
+        - not doing any of the above, just reloading and rebuilding for now
+
+ - adapt Timeline
+    - turn ObservableArray into ObservableSortedArray
+        - upsert already sorted sections
+        - upsert single entry
+ - adapt TilesCollection & Tile to entry changes
+ 
  - add live fragment id optimization if we haven't done so already
  - lets try to not have to have the fragmentindex in memory if the timeline isn't loaded
     - could do this by only loading all fragments into index when filling gaps, backpaginating, ... and on persister load only load the last fragment. This wouldn't even need a FragmentIndex?
diff --git a/src/domain/session/room/timeline/TilesCollection.js b/src/domain/session/room/timeline/TilesCollection.js
index 2b15bb97..36678764 100644
--- a/src/domain/session/room/timeline/TilesCollection.js
+++ b/src/domain/session/room/timeline/TilesCollection.js
@@ -22,6 +22,7 @@ export default class TilesCollection extends BaseObservableList {
         for (let entry of this._entries) {
             if (!currentTile || !currentTile.tryIncludeEntry(entry)) {
                 currentTile = this._tileCreator(entry);
+                // if (currentTile) here?
                 this._tiles.push(currentTile);
             }
         }
diff --git a/src/matrix/room/timeline/FragmentIdComparer.js b/src/matrix/room/timeline/FragmentIdComparer.js
index 7ffb9422..df5c3ac7 100644
--- a/src/matrix/room/timeline/FragmentIdComparer.js
+++ b/src/matrix/room/timeline/FragmentIdComparer.js
@@ -163,7 +163,7 @@ export default class FragmentIdComparer {
 export function tests() {
     return {
         test_1_island_3_fragments(assert) {
-            const index = new FragmentIdIndex([
+            const index = new FragmentIdComparer([
                 {id: 3, previousId: 2},
                 {id: 1, nextId: 2},
                 {id: 2, nextId: 3, previousId: 1},
@@ -180,7 +180,7 @@ export function tests() {
             assert.equal(index.compare(1, 1), 0);
         },
         test_2_island_dont_compare(assert) {
-            const index = new FragmentIdIndex([
+            const index = new FragmentIdComparer([
                 {id: 1},
                 {id: 2},
             ]);
@@ -188,7 +188,7 @@ export function tests() {
             assert.throws(() => index.compare(2, 1));
         },
         test_2_island_compare_internally(assert) {
-            const index = new FragmentIdIndex([
+            const index = new FragmentIdComparer([
                 {id: 1, nextId: 2},
                 {id: 2, previousId: 1},
                 {id: 11, nextId: 12},
@@ -203,12 +203,12 @@ export function tests() {
             assert.throws(() => index.compare(12, 2));
         },
         test_unknown_id(assert) {
-            const index = new FragmentIdIndex([{id: 1}]);
+            const index = new FragmentIdComparer([{id: 1}]);
             assert.throws(() => index.compare(1, 2));
             assert.throws(() => index.compare(2, 1));
         },
         test_rebuild_flushes_old_state(assert) {
-            const index = new FragmentIdIndex([
+            const index = new FragmentIdComparer([
                 {id: 1, nextId: 2},
                 {id: 2, previousId: 1},
             ]);
diff --git a/src/matrix/room/timeline/Timeline.js b/src/matrix/room/timeline/Timeline.js
index fcadeadf..2eb8d19b 100644
--- a/src/matrix/room/timeline/Timeline.js
+++ b/src/matrix/room/timeline/Timeline.js
@@ -1,6 +1,7 @@
 import { ObservableArray } from "../../../observable/index.js";
 import sortedIndex from "../../../utils/sortedIndex.js";
 import GapPersister from "./persistence/GapPersister.js";
+import TimelineReader from "./persistence/TimelineReader.js";
 
 export default class Timeline {
     constructor({roomId, storage, closeCallback, fragmentIdComparer}) {
@@ -9,6 +10,11 @@ export default class Timeline {
         this._closeCallback = closeCallback;
         this._entriesList = new ObservableArray();
         this._fragmentIdComparer = fragmentIdComparer;
+        this._timelineReader = new TimelineReader({
+            roomId: this._roomId,
+            storage: this._storage,
+            fragmentIdComparer: this._fragmentIdComparer
+        });
     }
 
     /** @package */
@@ -53,6 +59,7 @@ export default class Timeline {
     }
 
     async loadAtTop(amount) {
+        // TODO: use TimelineReader::readFrom here, and insert returned array at location for first and last entry.
         const firstEntry = this._entriesList.at(0);
         if (firstEntry) {
             const txn = await this._storage.readTxn([this._storage.storeNames.timelineEvents]);
diff --git a/src/matrix/room/timeline/persistence/GapPersister.js b/src/matrix/room/timeline/persistence/GapPersister.js
index 8eabb226..9b92409f 100644
--- a/src/matrix/room/timeline/persistence/GapPersister.js
+++ b/src/matrix/room/timeline/persistence/GapPersister.js
@@ -1,14 +1,6 @@
 import EventKey from "../EventKey.js";
 import EventEntry from "../entries/EventEntry.js";
-import {createEventEntry} from "./common.js";
-
-function directionalAppend(array, value, direction) {
-    if (direction.isForward) {
-        array.push(value);
-    } else {
-        array.splice(0, 0, value);
-    }
-}
+import {createEventEntry, directionalAppend} from "./common.js";
 
 export default class GapPersister {
     constructor({roomId, storage, fragmentIdComparer}) {
diff --git a/src/matrix/room/timeline/persistence/SyncPersister.js b/src/matrix/room/timeline/persistence/SyncPersister.js
index eaa3bfdd..97b21952 100644
--- a/src/matrix/room/timeline/persistence/SyncPersister.js
+++ b/src/matrix/room/timeline/persistence/SyncPersister.js
@@ -11,7 +11,7 @@ export default class SyncPersister {
     }
 
     async load(txn) {
-        const liveFragment = await txn.roomFragments.liveFragment(this._roomId);
+        const liveFragment = await txn.timelineFragments.liveFragment(this._roomId);
         if (liveFragment) {
             const [lastEvent] = await txn.roomTimeline.lastEvents(this._roomId, liveFragment.id, 1);
             // sorting and identifying (e.g. sort key and pk to insert) are a bit intertwined here
@@ -26,7 +26,7 @@ export default class SyncPersister {
     }
 
     async _createLiveFragment(txn, previousToken) {
-        const liveFragment = await txn.roomFragments.liveFragment(this._roomId);
+        const liveFragment = await txn.timelineFragments.liveFragment(this._roomId);
         if (!liveFragment) {
             if (!previousToken) {
                 previousToken = null;
@@ -39,7 +39,7 @@ export default class SyncPersister {
                 previousToken: previousToken,
                 nextToken: null
             };
-            txn.roomFragments.add(fragment);
+            txn.timelineFragments.add(fragment);
             return fragment;
         } else {
             return liveFragment;
@@ -47,12 +47,12 @@ export default class SyncPersister {
     }
 
     async _replaceLiveFragment(oldFragmentId, newFragmentId, previousToken, txn) {
-        const oldFragment = await txn.roomFragments.get(oldFragmentId);
+        const oldFragment = await txn.timelineFragments.get(oldFragmentId);
         if (!oldFragment) {
             throw new Error(`old live fragment doesn't exist: ${oldFragmentId}`);
         }
         oldFragment.nextId = newFragmentId;
-        txn.roomFragments.update(oldFragment);
+        txn.timelineFragments.update(oldFragment);
         const newFragment = {
             roomId: this._roomId,
             id: newFragmentId,
@@ -61,7 +61,7 @@ export default class SyncPersister {
             previousToken: previousToken,
             nextToken: null
         };
-        txn.roomFragments.add(newFragment);
+        txn.timelineFragments.add(newFragment);
         return {oldFragment, newFragment};
     }
 
@@ -117,6 +117,11 @@ export default class SyncPersister {
             }
         }
 
+        if (timeline.limited) {
+            const fragments = await txn.timelineFragments.all(this._roomId);
+            this._fragmentIdComparer.rebuild(fragments);
+        }
+
         return entries;
     }
 }
diff --git a/src/matrix/room/timeline/persistence/TimelineReader.js b/src/matrix/room/timeline/persistence/TimelineReader.js
new file mode 100644
index 00000000..75402aee
--- /dev/null
+++ b/src/matrix/room/timeline/persistence/TimelineReader.js
@@ -0,0 +1,63 @@
+import {directionalConcat, directionalAppend} from "./common.js";
+import EventKey from "../EventKey.js";
+import EventEntry from "../entries/EventEntry.js";
+import FragmentBoundaryEntry from "../entries/FragmentBoundaryEntry.js";
+
+export default class TimelineReader {
+    constructor({roomId, storage, fragmentIdComparer}) {
+        this._roomId = roomId;
+        this._storage = storage;
+        this._fragmentIdComparer = fragmentIdComparer;
+    }
+
+    async readFrom(eventKey, direction, amount) {
+        const txn = this._storage.readTxn([
+            this._storage.storeNames.timelineEvents,
+            this._storage.storeNames.timelineFragments,
+        ]);
+        let entries = [];
+        let loadedFragment = false;
+
+        const timelineStore = txn.timelineEvents;
+        const fragmentStore = txn.timelineFragments;
+        
+        while (entries.length < amount && eventKey) {
+            let eventsWithinFragment;
+            if (direction.isForward) {
+                eventsWithinFragment = timelineStore.eventsAfter(eventKey, amount);
+            } else {
+                eventsWithinFragment = timelineStore.eventsBefore(eventKey, amount);
+            }
+            const eventEntries = eventsWithinFragment.map(e => new EventEntry(e, this._fragmentIdComparer));
+            entries = directionalConcat(entries, eventEntries, direction);
+            // prepend or append eventsWithinFragment to entries, and wrap them in EventEntry
+
+            if (entries.length < amount) {
+                const fragment = await fragmentStore.get(this._roomId, eventKey.fragmentId);
+                // this._fragmentIdComparer.addFragment(fragment);
+                let fragmentEntry = new FragmentBoundaryEntry(fragment, direction.isBackward, this._fragmentIdComparer);
+                // append or prepend fragmentEntry, reuse func from GapPersister?
+                directionalAppend(entries, fragmentEntry, direction);
+                // don't count it in amount perhaps? or do?
+                if (fragmentEntry.linkedFragmentId) {
+                    const nextFragment = await fragmentStore.get(this._roomId, fragmentEntry.linkedFragmentId);
+                    // this._fragmentIdComparer.addFragment(nextFragment);
+                    const nextFragmentEntry = new FragmentBoundaryEntry(nextFragment, direction.isForward, this._fragmentIdComparer);
+                    directionalAppend(entries, nextFragmentEntry, direction);
+                    eventKey = new EventKey(nextFragmentEntry.fragmentId, nextFragmentEntry.eventIndex);
+                    loadedFragment = true;
+                } else {
+                    eventKey = null;
+                }
+            }
+        }
+
+        // reload fragments
+        if (loadedFragment) {
+            const fragments = await fragmentStore.all(this._roomId);
+            this._fragmentIdComparer.rebuild(fragments);
+        }
+
+        return entries;
+    }
+}
diff --git a/src/matrix/room/timeline/persistence/common.js b/src/matrix/room/timeline/persistence/common.js
index ac5a6c9d..5bec1a82 100644
--- a/src/matrix/room/timeline/persistence/common.js
+++ b/src/matrix/room/timeline/persistence/common.js
@@ -4,4 +4,20 @@ export function createEventEntry(key, event) {
         eventIndex: key.eventIndex,
         event: event,
     };
-}
\ No newline at end of file
+}
+
+export function directionalAppend(array, value, direction) {
+    if (direction.isForward) {
+        array.push(value);
+    } else {
+        array.unshift(value);
+    }
+}
+
+export function directionalConcat(array, otherArray, direction) {
+    if (direction.isForward) {
+        return array.concat(otherArray);
+    } else {
+        return otherArray.concat(array);
+    }
+}
diff --git a/src/matrix/session.js b/src/matrix/session.js
index d86065dc..763d0265 100644
--- a/src/matrix/session.js
+++ b/src/matrix/session.js
@@ -18,6 +18,7 @@ export default class Session {
             this._storage.storeNames.roomSummary,
             this._storage.storeNames.roomState,
             this._storage.storeNames.timelineEvents,
+            this._storage.storeNames.timelineFragments,
         ]);
         // restore session object
         this._session = await txn.session.get();
diff --git a/src/matrix/sync.js b/src/matrix/sync.js
index b96f2110..b12dd73e 100644
--- a/src/matrix/sync.js
+++ b/src/matrix/sync.js
@@ -74,6 +74,7 @@ export default class Sync extends EventEmitter {
             storeNames.session,
             storeNames.roomSummary,
             storeNames.timelineEvents,
+            storeNames.timelineFragments,
             storeNames.roomState,
         ]);
         const roomChanges = [];
diff --git a/src/observable/list/SortedArray.js b/src/observable/list/SortedArray.js
new file mode 100644
index 00000000..e01bae4f
--- /dev/null
+++ b/src/observable/list/SortedArray.js
@@ -0,0 +1,38 @@
+import BaseObservableList from "./BaseObservableList.js";
+import sortedIndex from "../../utils/sortedIndex";
+
+export default class SortedArray extends BaseObservableList {
+    constructor(comparator) {
+        super();
+        this._comparator = comparator;
+        this._items = [];
+    }
+
+    setSortedMany(items) {
+
+    }
+
+    set(item) {
+        const idx = sortedIndex(this._items, item, this._comparator);
+        if (idx < this._items.length || this._comparator(this._items[idx], item) !== 0) {
+            this._items.splice(idx, 0, item);
+            //emitAdd
+        } else {
+            this._items[idx] = item;
+            //emitRemove
+            //emitAdd
+        }
+    }
+
+    get array() {
+        return this._items;
+    }
+
+    get length() {
+        return this._items.length;
+    }
+
+    [Symbol.iterator]() {
+        return this._items.values();
+    }
+}

From fa4efe0132150299d1bedbf79267a7cd9c2c1779 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 15:40:21 +0200
Subject: [PATCH 49/83] rename Gap/SyncPersistence to Writer, in line with
 TimelineReader

---
 src/matrix/room/room.js                                   | 8 ++++----
 src/matrix/room/timeline/Timeline.js                      | 6 +++---
 .../persistence/{GapPersister.js => GapWriter.js}         | 4 ++--
 .../persistence/{SyncPersister.js => SyncWriter.js}       | 4 ++--
 4 files changed, 11 insertions(+), 11 deletions(-)
 rename src/matrix/room/timeline/persistence/{GapPersister.js => GapWriter.js} (99%)
 rename src/matrix/room/timeline/persistence/{SyncPersister.js => SyncWriter.js} (99%)

diff --git a/src/matrix/room/room.js b/src/matrix/room/room.js
index c6e75e95..09b3427c 100644
--- a/src/matrix/room/room.js
+++ b/src/matrix/room/room.js
@@ -1,6 +1,6 @@
 import EventEmitter from "../../EventEmitter.js";
 import RoomSummary from "./summary.js";
-import SyncPersister from "./timeline/persistence/SyncPersister.js";
+import SyncWriter from "./timeline/persistence/SyncWriter.js";
 import Timeline from "./timeline/Timeline.js";
 import FragmentIdComparer from "./timeline/FragmentIdComparer.js";
 
@@ -12,14 +12,14 @@ export default class Room extends EventEmitter {
         this._hsApi = hsApi;
 		this._summary = new RoomSummary(roomId);
         this._fragmentIdComparer = new FragmentIdComparer([]);
-		this._syncPersister = new SyncPersister({roomId, storage, fragmentIdComparer: this._fragmentIdComparer});
+		this._syncWriter = new SyncWriter({roomId, storage, fragmentIdComparer: this._fragmentIdComparer});
         this._emitCollectionChange = emitCollectionChange;
         this._timeline = null;
 	}
 
     persistSync(roomResponse, membership, txn) {
 		const summaryChanged = this._summary.applySync(roomResponse, membership, txn);
-		const newTimelineEntries = this._syncPersister.persistSync(roomResponse, txn);
+		const newTimelineEntries = this._syncWriter.writeSync(roomResponse, txn);
         return {summaryChanged, newTimelineEntries};
     }
 
@@ -35,7 +35,7 @@ export default class Room extends EventEmitter {
 
 	load(summary, txn) {
 		this._summary.load(summary);
-		return this._syncPersister.load(txn);
+		return this._syncWriter.load(txn);
 	}
 
     get name() {
diff --git a/src/matrix/room/timeline/Timeline.js b/src/matrix/room/timeline/Timeline.js
index 2eb8d19b..2ac74e64 100644
--- a/src/matrix/room/timeline/Timeline.js
+++ b/src/matrix/room/timeline/Timeline.js
@@ -1,6 +1,6 @@
 import { ObservableArray } from "../../../observable/index.js";
 import sortedIndex from "../../../utils/sortedIndex.js";
-import GapPersister from "./persistence/GapPersister.js";
+import GapWriter from "./persistence/GapWriter.js";
 import TimelineReader from "./persistence/TimelineReader.js";
 
 export default class Timeline {
@@ -41,12 +41,12 @@ export default class Timeline {
             limit: amount
         });
 
-        const gapPersister = new GapPersister({
+        const gapWriter = new GapWriter({
             roomId: this._roomId,
             storage: this._storage,
             fragmentIdComparer: this._fragmentIdComparer
         });
-        const newEntries = await gapPersister.persistFragmentFill(fragmentEntry, response);
+        const newEntries = await gapWriter.writerFragmentFill(fragmentEntry, response);
         // find where to replace existing gap with newEntries by doing binary search
         const gapIdx = sortedIndex(this._entriesList.array, fragmentEntry, (fragmentEntry, entry) => {
             return fragmentEntry.compare(entry);
diff --git a/src/matrix/room/timeline/persistence/GapPersister.js b/src/matrix/room/timeline/persistence/GapWriter.js
similarity index 99%
rename from src/matrix/room/timeline/persistence/GapPersister.js
rename to src/matrix/room/timeline/persistence/GapWriter.js
index 9b92409f..06d450a6 100644
--- a/src/matrix/room/timeline/persistence/GapPersister.js
+++ b/src/matrix/room/timeline/persistence/GapWriter.js
@@ -2,7 +2,7 @@ import EventKey from "../EventKey.js";
 import EventEntry from "../entries/EventEntry.js";
 import {createEventEntry, directionalAppend} from "./common.js";
 
-export default class GapPersister {
+export default class GapWriter {
     constructor({roomId, storage, fragmentIdComparer}) {
         this._roomId = roomId;
         this._storage = storage;
@@ -71,7 +71,7 @@ export default class GapPersister {
         txn.timelineFragments.set(fragmentEntry.fragment);
     }
 
-    async persistFragmentFill(fragmentEntry, response) {
+    async writeFragmentFill(fragmentEntry, response) {
         const {fragmentId, direction} = fragmentEntry;
         // assuming that chunk is in chronological order when backwards too?
         const {chunk, start, end} = response;
diff --git a/src/matrix/room/timeline/persistence/SyncPersister.js b/src/matrix/room/timeline/persistence/SyncWriter.js
similarity index 99%
rename from src/matrix/room/timeline/persistence/SyncPersister.js
rename to src/matrix/room/timeline/persistence/SyncWriter.js
index 97b21952..f27e8a2e 100644
--- a/src/matrix/room/timeline/persistence/SyncPersister.js
+++ b/src/matrix/room/timeline/persistence/SyncWriter.js
@@ -3,7 +3,7 @@ import EventEntry from "../entries/EventEntry.js";
 import FragmentBoundaryEntry from "../entries/FragmentBoundaryEntry.js";
 import {createEventEntry} from "./common.js";
 
-export default class SyncPersister {
+export default class SyncWriter {
     constructor({roomId, storage, fragmentIdComparer}) {
         this._roomId = roomId;
         this._storage = storage;
@@ -65,7 +65,7 @@ export default class SyncPersister {
         return {oldFragment, newFragment};
     }
 
-    async persistSync(roomResponse, txn) {
+    async writeSync(roomResponse, txn) {
         const entries = [];
         if (!this._lastLiveKey) {
             // means we haven't synced this room yet (just joined or did initial sync)

From 447b0aa03cbbf7e5f8723e0bc805486988ff2828 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 16:42:57 +0200
Subject: [PATCH 50/83] allow adding fragments to comparer, instead of
 reloading from db.

This is a suboptimal implementation now,
but is the API we want to end up with. Readers and Writers
in persistence add fragments to the comparer when they become
aware of, create, or link up fragments.
---
 src/matrix/room/timeline/FragmentIdComparer.js       |  5 +++++
 src/matrix/room/timeline/persistence/GapWriter.js    |  5 +++--
 src/matrix/room/timeline/persistence/SyncWriter.js   |  7 ++-----
 .../room/timeline/persistence/TimelineReader.js      | 12 ++----------
 4 files changed, 12 insertions(+), 17 deletions(-)

diff --git a/src/matrix/room/timeline/FragmentIdComparer.js b/src/matrix/room/timeline/FragmentIdComparer.js
index df5c3ac7..029c1333 100644
--- a/src/matrix/room/timeline/FragmentIdComparer.js
+++ b/src/matrix/room/timeline/FragmentIdComparer.js
@@ -108,6 +108,7 @@ index for fast lookup of how two fragments can be sorted
 */
 export default class FragmentIdComparer {
     constructor(fragments) {
+        this._fragmentsById = fragments.reduce((map, f) => {map.set(f.id, f); return map;}, new Map());
         this.rebuild(fragments);
     }
 
@@ -157,6 +158,10 @@ export default class FragmentIdComparer {
     // linkFragments(txn, firstFragmentId, secondFragmentId) {
 
     // }
+    add(fragment) {
+        this._fragmentsById[fragment.id] = fragment;
+        this.rebuild(this._fragmentsById.values());
+    }
 }
 
 //#ifdef TESTS
diff --git a/src/matrix/room/timeline/persistence/GapWriter.js b/src/matrix/room/timeline/persistence/GapWriter.js
index 06d450a6..415f2292 100644
--- a/src/matrix/room/timeline/persistence/GapWriter.js
+++ b/src/matrix/room/timeline/persistence/GapWriter.js
@@ -64,8 +64,9 @@ export default class GapWriter {
             txn.timelineFragments.set(neighbourFragmentEntry.fragment);
             directionalAppend(entries, neighbourFragmentEntry, direction);
 
-            // update fragmentIdComparer here after linking up fragments?
-            this._fragmentIdComparer.rebuild(await txn.timelineFragments.all());
+            // update fragmentIdComparer here after linking up fragments
+            this._fragmentIdComparer.add(fragmentEntry.fragment);
+            this._fragmentIdComparer.add(neighbourFragmentEntry.fragment);
         }
         fragmentEntry.token = end;
         txn.timelineFragments.set(fragmentEntry.fragment);
diff --git a/src/matrix/room/timeline/persistence/SyncWriter.js b/src/matrix/room/timeline/persistence/SyncWriter.js
index f27e8a2e..ebec1bae 100644
--- a/src/matrix/room/timeline/persistence/SyncWriter.js
+++ b/src/matrix/room/timeline/persistence/SyncWriter.js
@@ -40,6 +40,7 @@ export default class SyncWriter {
                 nextToken: null
             };
             txn.timelineFragments.add(fragment);
+            this._fragmentIdComparer.add(fragment);
             return fragment;
         } else {
             return liveFragment;
@@ -62,6 +63,7 @@ export default class SyncWriter {
             nextToken: null
         };
         txn.timelineFragments.add(newFragment);
+        this._fragmentIdComparer.add(newFragment);
         return {oldFragment, newFragment};
     }
 
@@ -117,11 +119,6 @@ export default class SyncWriter {
             }
         }
 
-        if (timeline.limited) {
-            const fragments = await txn.timelineFragments.all(this._roomId);
-            this._fragmentIdComparer.rebuild(fragments);
-        }
-
         return entries;
     }
 }
diff --git a/src/matrix/room/timeline/persistence/TimelineReader.js b/src/matrix/room/timeline/persistence/TimelineReader.js
index 75402aee..ee7c7865 100644
--- a/src/matrix/room/timeline/persistence/TimelineReader.js
+++ b/src/matrix/room/timeline/persistence/TimelineReader.js
@@ -16,7 +16,6 @@ export default class TimelineReader {
             this._storage.storeNames.timelineFragments,
         ]);
         let entries = [];
-        let loadedFragment = false;
 
         const timelineStore = txn.timelineEvents;
         const fragmentStore = txn.timelineFragments;
@@ -36,28 +35,21 @@ export default class TimelineReader {
                 const fragment = await fragmentStore.get(this._roomId, eventKey.fragmentId);
                 // this._fragmentIdComparer.addFragment(fragment);
                 let fragmentEntry = new FragmentBoundaryEntry(fragment, direction.isBackward, this._fragmentIdComparer);
-                // append or prepend fragmentEntry, reuse func from GapPersister?
+                // append or prepend fragmentEntry, reuse func from GapWriter?
                 directionalAppend(entries, fragmentEntry, direction);
                 // don't count it in amount perhaps? or do?
                 if (fragmentEntry.linkedFragmentId) {
                     const nextFragment = await fragmentStore.get(this._roomId, fragmentEntry.linkedFragmentId);
-                    // this._fragmentIdComparer.addFragment(nextFragment);
+                    this._fragmentIdComparer.add(nextFragment);
                     const nextFragmentEntry = new FragmentBoundaryEntry(nextFragment, direction.isForward, this._fragmentIdComparer);
                     directionalAppend(entries, nextFragmentEntry, direction);
                     eventKey = new EventKey(nextFragmentEntry.fragmentId, nextFragmentEntry.eventIndex);
-                    loadedFragment = true;
                 } else {
                     eventKey = null;
                 }
             }
         }
 
-        // reload fragments
-        if (loadedFragment) {
-            const fragments = await fragmentStore.all(this._roomId);
-            this._fragmentIdComparer.rebuild(fragments);
-        }
-
         return entries;
     }
 }

From 3137f025c79749a51d63d8f9a6ee90dc871b061c Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 16:44:58 +0200
Subject: [PATCH 51/83] remove draft commented code

---
 src/matrix/room/timeline/FragmentIdComparer.js | 16 ----------------
 1 file changed, 16 deletions(-)

diff --git a/src/matrix/room/timeline/FragmentIdComparer.js b/src/matrix/room/timeline/FragmentIdComparer.js
index 029c1333..15c2bc08 100644
--- a/src/matrix/room/timeline/FragmentIdComparer.js
+++ b/src/matrix/room/timeline/FragmentIdComparer.js
@@ -141,23 +141,7 @@ export default class FragmentIdComparer {
             }
         }
     }
-    // maybe actual persistence shouldn't be done here, just allocate fragment ids and sorting
 
-    // we need to check here that the fragment we think we are appending to doesn't already have a nextId.
-    // otherwise we could create a corrupt state (two fragments not pointing at each other).
-
-    // allocates a fragment id within the live range, that can be compared to each other without a mapping as they are allocated in chronological order
-    // appendLiveFragment(txn, previousToken) {
-
-    // }
-
-    // newFragment(txn, previousToken, nextToken) {
-
-    // }
-
-    // linkFragments(txn, firstFragmentId, secondFragmentId) {
-
-    // }
     add(fragment) {
         this._fragmentsById[fragment.id] = fragment;
         this.rebuild(this._fragmentsById.values());

From f8fbfbff9aa6599024988f16e0740ca4f2974587 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 17:04:05 +0200
Subject: [PATCH 52/83] implement reading n events from end of live fragment

---
 src/matrix/room/timeline/Timeline.js          |  3 +-
 .../timeline/persistence/TimelineReader.js    | 34 +++++++++++++++++--
 2 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/src/matrix/room/timeline/Timeline.js b/src/matrix/room/timeline/Timeline.js
index 2ac74e64..600fe143 100644
--- a/src/matrix/room/timeline/Timeline.js
+++ b/src/matrix/room/timeline/Timeline.js
@@ -19,8 +19,7 @@ export default class Timeline {
 
     /** @package */
     async load() {
-        const txn = await this._storage.readTxn([this._storage.storeNames.timelineEvents]);
-        const entries = await txn.roomTimeline.lastEvents(this._roomId, 100);
+        const entries = this._timelineReader.readFromEnd(100);
         for (const entry of entries) {
             this._entriesList.append(entry);
         }
diff --git a/src/matrix/room/timeline/persistence/TimelineReader.js b/src/matrix/room/timeline/persistence/TimelineReader.js
index ee7c7865..0fbc59e6 100644
--- a/src/matrix/room/timeline/persistence/TimelineReader.js
+++ b/src/matrix/room/timeline/persistence/TimelineReader.js
@@ -1,5 +1,6 @@
 import {directionalConcat, directionalAppend} from "./common.js";
 import EventKey from "../EventKey.js";
+import Direction from "../Direction.js";
 import EventEntry from "../entries/EventEntry.js";
 import FragmentBoundaryEntry from "../entries/FragmentBoundaryEntry.js";
 
@@ -10,11 +11,19 @@ export default class TimelineReader {
         this._fragmentIdComparer = fragmentIdComparer;
     }
 
-    async readFrom(eventKey, direction, amount) {
-        const txn = this._storage.readTxn([
+    _openTxn() {
+        return this._storage.readTxn([
             this._storage.storeNames.timelineEvents,
             this._storage.storeNames.timelineFragments,
         ]);
+    }
+
+    async readFrom(eventKey, direction, amount) {
+        const txn = await this._openTxn();
+        return this._readFrom(eventKey, direction, amount, txn);
+    }
+
+    async _readFrom(eventKey, direction, amount, txn) {
         let entries = [];
 
         const timelineStore = txn.timelineEvents;
@@ -52,4 +61,25 @@ export default class TimelineReader {
 
         return entries;
     }
+
+    async readFromEnd(amount) {
+        const txn = await this._openTxn();
+        const liveFragment = await txn.timelineFragments.liveFragment(this._roomId);
+        // room hasn't been synced yet
+        if (!liveFragment) {
+            return [];
+        }
+        this._fragmentIdComparer.add(liveFragment);
+        const liveFragmentEntry = new FragmentBoundaryEntry(liveFragment, Direction.Forward, this._fragmentIdComparer);
+        const eventKey = new EventKey(liveFragmentEntry.fragmentId, liveFragmentEntry.eventIndex);
+        const entries = this._readFrom(eventKey, Direction.Backward, amount, txn);
+        entries.unshift(liveFragmentEntry);
+        return entries;
+    }
+
+    // reads distance up and down from eventId
+    // or just expose eventIdToKey?
+    readAtEventId(eventId, distance) {
+        return null;
+    }
 }

From 843c94b75090479b80dcc9ad2f0501ea74a74adb Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 17:38:23 +0200
Subject: [PATCH 53/83] finished observable SortedArray to something useable

although not as performant as it could be
---
 src/observable/index.js            |  3 ++-
 src/observable/list/SortedArray.js | 26 +++++++++++++++++++-------
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/src/observable/index.js b/src/observable/index.js
index 36e037cd..828a6aea 100644
--- a/src/observable/index.js
+++ b/src/observable/index.js
@@ -3,7 +3,8 @@ import FilteredMap from "./map/FilteredMap.js";
 import MappedMap from "./map/MappedMap.js";
 import BaseObservableMap from "./map/BaseObservableMap.js";
 // re-export "root" (of chain) collections
-export { default as ObservableArray} from "./list/ObservableArray.js";
+export { default as ObservableArray } from "./list/ObservableArray.js";
+export { default as SortedArray } from "./list/SortedArray.js";
 export { default as ObservableMap } from "./map/ObservableMap.js";
 
 // avoid circular dependency between these classes
diff --git a/src/observable/list/SortedArray.js b/src/observable/list/SortedArray.js
index e01bae4f..b18b7250 100644
--- a/src/observable/list/SortedArray.js
+++ b/src/observable/list/SortedArray.js
@@ -8,22 +8,34 @@ export default class SortedArray extends BaseObservableList {
         this._items = [];
     }
 
-    setSortedMany(items) {
-
+    setManySorted(items) {
+        // TODO: we can make this way faster by only looking up the first and last key,
+        // and merging whatever is inbetween with items
+        // if items is not sorted, 💩🌀 will follow!
+        // should we check?
+        // Also, once bulk events are supported in collections,
+        // we can do a bulk add event here probably if there are no updates
+        // BAD CODE!
+        for(let item of items) {
+            this.set(item);
+        }
     }
 
-    set(item) {
+    set(item, updateParams = null) {
         const idx = sortedIndex(this._items, item, this._comparator);
-        if (idx < this._items.length || this._comparator(this._items[idx], item) !== 0) {
+        if (idx >= this._items.length || this._comparator(this._items[idx], item) !== 0) {
             this._items.splice(idx, 0, item);
-            //emitAdd
+            this.emitAdd(idx, item)
         } else {
             this._items[idx] = item;
-            //emitRemove
-            //emitAdd
+            this.emitUpdate(idx, item, updateParams);
         }
     }
 
+    remove(item) {
+        throw new Error("unimplemented");
+    }
+
     get array() {
         return this._items;
     }

From 2a128ed32c557b91c2097570508ce9ef1d40d53f Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 17:39:23 +0200
Subject: [PATCH 54/83] use SortedArray in Timeline, adjust loadAtTop to use
 TimelineReader

---
 src/matrix/room/timeline/Timeline.js | 39 ++++++++++------------------
 1 file changed, 14 insertions(+), 25 deletions(-)

diff --git a/src/matrix/room/timeline/Timeline.js b/src/matrix/room/timeline/Timeline.js
index 600fe143..da40eb53 100644
--- a/src/matrix/room/timeline/Timeline.js
+++ b/src/matrix/room/timeline/Timeline.js
@@ -1,5 +1,6 @@
-import { ObservableArray } from "../../../observable/index.js";
-import sortedIndex from "../../../utils/sortedIndex.js";
+import { SortedArray } from "../../../observable/index.js";
+import EventKey from "./EventKey.js";
+import Direction from "./Direction.js";
 import GapWriter from "./persistence/GapWriter.js";
 import TimelineReader from "./persistence/TimelineReader.js";
 
@@ -8,8 +9,8 @@ export default class Timeline {
         this._roomId = roomId;
         this._storage = storage;
         this._closeCallback = closeCallback;
-        this._entriesList = new ObservableArray();
         this._fragmentIdComparer = fragmentIdComparer;
+        this._entriesList = new SortedArray((a, b) => a.compare(b));
         this._timelineReader = new TimelineReader({
             roomId: this._roomId,
             storage: this._storage,
@@ -20,16 +21,12 @@ export default class Timeline {
     /** @package */
     async load() {
         const entries = this._timelineReader.readFromEnd(100);
-        for (const entry of entries) {
-            this._entriesList.append(entry);
-        }
+        this._entriesList.setManySorted(entries);
     }
 
     /** @package */
     appendLiveEntries(newEntries) {
-        for (const entry of newEntries) {
-            this._entriesList.append(entry);
-        }
+        this._entriesList.setManySorted(newEntries);
     }
 
     /** @public */
@@ -46,26 +43,18 @@ export default class Timeline {
             fragmentIdComparer: this._fragmentIdComparer
         });
         const newEntries = await gapWriter.writerFragmentFill(fragmentEntry, response);
-        // find where to replace existing gap with newEntries by doing binary search
-        const gapIdx = sortedIndex(this._entriesList.array, fragmentEntry, (fragmentEntry, entry) => {
-            return fragmentEntry.compare(entry);
-        });
-        // only replace the gap if it's currently in the timeline
-        if (this._entriesList.at(gapIdx) === fragmentEntry) {
-            this._entriesList.removeAt(gapIdx);
-            this._entriesList.insertMany(gapIdx, newEntries);
-        }
+        this._entriesList.setManySorted(newEntries);
     }
 
+    // tries to prepend `amount` entries to the `entries` list.
     async loadAtTop(amount) {
-        // TODO: use TimelineReader::readFrom here, and insert returned array at location for first and last entry.
-        const firstEntry = this._entriesList.at(0);
-        if (firstEntry) {
-            const txn = await this._storage.readTxn([this._storage.storeNames.timelineEvents]);
-            const topEntries = await txn.roomTimeline.eventsBefore(this._roomId, firstEntry.sortKey, amount);
-            this._entriesList.insertMany(0, topEntries);
-            return topEntries.length;
+        if (this._entriesList.length() === 0) {
+            return;
         }
+        const firstEntry = this._entriesList.array()[0];
+        const firstKey = new EventKey(firstEntry.fragmentId, firstEntry.eventIndex);
+        const entries = await this._timelineReader.readFrom(firstKey, Direction.Backward, amount);
+        this._entriesList.setManySorted(entries);
     }
 
     /** @public */

From 35d90a8535667f3bcc7184fad3088048e4071b26 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 18:20:29 +0200
Subject: [PATCH 55/83] ctor takes a bool, not Direction. But use helper fn
 instead.

---
 src/matrix/room/timeline/entries/FragmentBoundaryEntry.js | 1 +
 src/matrix/room/timeline/persistence/TimelineReader.js    | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
index 0a336975..10ae7ec4 100644
--- a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
+++ b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
@@ -5,6 +5,7 @@ export default class FragmentBoundaryEntry extends BaseEntry {
     constructor(fragment, isFragmentStart, fragmentIdComparator) {
         super(fragmentIdComparator);
         this._fragment = fragment;
+        // TODO: should isFragmentStart be Direction instead of bool?
         this._isFragmentStart = isFragmentStart;
     }
 
diff --git a/src/matrix/room/timeline/persistence/TimelineReader.js b/src/matrix/room/timeline/persistence/TimelineReader.js
index 0fbc59e6..8f2e5a4d 100644
--- a/src/matrix/room/timeline/persistence/TimelineReader.js
+++ b/src/matrix/room/timeline/persistence/TimelineReader.js
@@ -70,7 +70,7 @@ export default class TimelineReader {
             return [];
         }
         this._fragmentIdComparer.add(liveFragment);
-        const liveFragmentEntry = new FragmentBoundaryEntry(liveFragment, Direction.Forward, this._fragmentIdComparer);
+        const liveFragmentEntry = FragmentBoundaryEntry.end(liveFragment, this._fragmentIdComparer);
         const eventKey = new EventKey(liveFragmentEntry.fragmentId, liveFragmentEntry.eventIndex);
         const entries = this._readFrom(eventKey, Direction.Backward, amount, txn);
         entries.unshift(liveFragmentEntry);

From 039bbe038ccd08b2ef5170083b6e3883176bedad Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 18:29:02 +0200
Subject: [PATCH 56/83] adjust tiles(collection) to entry changes

---
 .../session/room/timeline/TilesCollection.js  | 25 ++++++++-----------
 .../session/room/timeline/tiles/GapTile.js    |  2 +-
 .../session/room/timeline/tiles/SimpleTile.js | 15 +++++------
 .../session/room/timeline/tilesCreator.js     |  2 +-
 .../room/timeline/entries/EventEntry.js       |  4 +++
 5 files changed, 25 insertions(+), 23 deletions(-)

diff --git a/src/domain/session/room/timeline/TilesCollection.js b/src/domain/session/room/timeline/TilesCollection.js
index 36678764..3c80fc98 100644
--- a/src/domain/session/room/timeline/TilesCollection.js
+++ b/src/domain/session/room/timeline/TilesCollection.js
@@ -1,7 +1,7 @@
 import BaseObservableList from "../../../../observable/list/BaseObservableList.js";
 import sortedIndex from "../../../../utils/sortedIndex.js";
 
-// maps 1..n entries to 0..1 tile. Entries are what is stored in the timeline, either an event or gap
+// maps 1..n entries to 0..1 tile. Entries are what is stored in the timeline, either an event or fragmentboundary
 export default class TilesCollection extends BaseObservableList {
     constructor(entries, tileCreator) {
         super();
@@ -39,16 +39,16 @@ export default class TilesCollection extends BaseObservableList {
         }
     }
 
-    _findTileIdx(sortKey) {
-        return sortedIndex(this._tiles, sortKey, (key, tile) => {
+    _findTileIdx(entry) {
+        return sortedIndex(this._tiles, entry, (entry, tile) => {
             // negate result because we're switching the order of the params
-            return -tile.compareSortKey(key);
+            return -tile.compareEntry(entry);
         });
     }
 
-    _findTileAtIdx(sortKey, idx) {
+    _findTileAtIdx(entry, idx) {
         const tile = this._getTileAtIdx(idx);
-        if (tile && tile.compareSortKey(sortKey) === 0) {
+        if (tile && tile.compareEntry(entry) === 0) {
             return tile;
         }
     }
@@ -72,8 +72,7 @@ export default class TilesCollection extends BaseObservableList {
     }
 
     onAdd(index, entry) {
-        const {sortKey} = entry;
-        const tileIdx = this._findTileIdx(sortKey);
+        const tileIdx = this._findTileIdx(entry);
         const prevTile = this._getTileAtIdx(tileIdx - 1);
         if (prevTile && prevTile.tryIncludeEntry(entry)) {
             this.emitUpdate(tileIdx - 1, prevTile);
@@ -98,9 +97,8 @@ export default class TilesCollection extends BaseObservableList {
     }
 
     onUpdate(index, entry, params) {
-        const {sortKey} = entry;
-        const tileIdx = this._findTileIdx(sortKey);
-        const tile = this._findTileAtIdx(sortKey, tileIdx);
+        const tileIdx = this._findTileIdx(entry);
+        const tile = this._findTileAtIdx(entry, tileIdx);
         if (tile) {
             const newParams = tile.updateEntry(entry, params);
             if (newParams) {
@@ -122,9 +120,8 @@ export default class TilesCollection extends BaseObservableList {
 
     // would also be called when unloading a part of the timeline
     onRemove(index, entry) {
-        const {sortKey} = entry;
-        const tileIdx = this._findTileIdx(sortKey);
-        const tile = this._findTileAtIdx(sortKey, tileIdx);
+        const tileIdx = this._findTileIdx(entry);
+        const tile = this._findTileAtIdx(entry, tileIdx);
         if (tile) {
             const removeTile = tile.removeEntry(entry);
             if (removeTile) {
diff --git a/src/domain/session/room/timeline/tiles/GapTile.js b/src/domain/session/room/timeline/tiles/GapTile.js
index fe0cb6fe..cebffe24 100644
--- a/src/domain/session/room/timeline/tiles/GapTile.js
+++ b/src/domain/session/room/timeline/tiles/GapTile.js
@@ -29,7 +29,7 @@ export default class GapTile extends SimpleTile {
     }
 
     get direction() {
-        return this._entry.prev_batch ? "backward" : "forward";
+        return this._entry.direction;
     }
 
     get error() {
diff --git a/src/domain/session/room/timeline/tiles/SimpleTile.js b/src/domain/session/room/timeline/tiles/SimpleTile.js
index bacb8259..266b09db 100644
--- a/src/domain/session/room/timeline/tiles/SimpleTile.js
+++ b/src/domain/session/room/timeline/tiles/SimpleTile.js
@@ -18,18 +18,19 @@ export default class SimpleTile {
     get hasDateSeparator() {
         return false;
     }
-
-    get upperSortKey() {
-        return this._entry.sortKey;
+    // TilesCollection contract? unused atm
+    get upperEntry() {
+        return this._entry;
     }
 
-    get lowerSortKey() {
-        return this._entry.sortKey;
+    // TilesCollection contract? unused atm
+    get lowerEntry() {
+        return this._entry;
     }
 
     // TilesCollection contract
-    compareSortKey(key) {
-        return this._entry.sortKey.compare(key);
+    compareEntry(entry) {
+        return this._entry.compare(entry);
     }
 
     // update received for already included (falls within sort keys) entry
diff --git a/src/domain/session/room/timeline/tilesCreator.js b/src/domain/session/room/timeline/tilesCreator.js
index e70088b9..732e0ce6 100644
--- a/src/domain/session/room/timeline/tilesCreator.js
+++ b/src/domain/session/room/timeline/tilesCreator.js
@@ -8,7 +8,7 @@ import RoomMemberTile from "./tiles/RoomMemberTile.js";
 export default function ({timeline, emitUpdate}) {
     return function tilesCreator(entry) {
         const options = {entry, emitUpdate};
-        if (entry.gap) {
+        if (entry.isGap) {
             return new GapTile(options, timeline);
         } else if (entry.event) {
             const event = entry.event;
diff --git a/src/matrix/room/timeline/entries/EventEntry.js b/src/matrix/room/timeline/entries/EventEntry.js
index 07d00a76..98e961cf 100644
--- a/src/matrix/room/timeline/entries/EventEntry.js
+++ b/src/matrix/room/timeline/entries/EventEntry.js
@@ -18,6 +18,10 @@ export default class EventEntry extends BaseEntry {
         return this._eventEntry.event.content;
     }
 
+    get event() {
+        return this._eventEntry.event;
+    }
+
     get type() {
         return this._eventEntry.event.type;
     }

From 1b228b0200bd6f9d998c811f8a4df897926dd295 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 18:29:23 +0200
Subject: [PATCH 57/83] export timelineviewmodel from roomviewmodel

---
 src/domain/session/room/RoomViewModel.js | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/src/domain/session/room/RoomViewModel.js b/src/domain/session/room/RoomViewModel.js
index 2e9c4053..17427305 100644
--- a/src/domain/session/room/RoomViewModel.js
+++ b/src/domain/session/room/RoomViewModel.js
@@ -1,10 +1,12 @@
 import EventEmitter from "../../../EventEmitter.js";
+import TimelineViewModel from "./timeline/TimelineViewModel.js";
 
 export default class RoomViewModel extends EventEmitter {
     constructor(room) {
         super();
         this._room = room;
         this._timeline = null;
+        this._timelineVM = null;
         this._onRoomChange = this._onRoomChange.bind(this);
         this._timelineError = null;
     }
@@ -13,6 +15,7 @@ export default class RoomViewModel extends EventEmitter {
         this._room.on("change", this._onRoomChange);
         try {
             this._timeline = await this._room.openTimeline();
+            this._timelineVM = new TimelineViewModel(this._timeline);
             this.emit("change", "timelineEntries");
         } catch (err) {
             this._timelineError = err;
@@ -22,6 +25,7 @@ export default class RoomViewModel extends EventEmitter {
 
     disable() {
         if (this._timeline) {
+            // will stop the timeline from delivering updates on entries
             this._timeline.close();
         }
     }
@@ -36,8 +40,8 @@ export default class RoomViewModel extends EventEmitter {
         return this._room.name;
     }
 
-    get timelineEntries() {
-        return this._timeline && this._timeline.entries;
+    get timelineViewModel() {
+        return this._timelineVM;
     }
 
     get error() {

From 765a68c76650e45d1b8a4d35edc41466d9376fba Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 18:29:37 +0200
Subject: [PATCH 58/83] adjust fragments status, also add future perf
 optimization notes

---
 doc/FRAGMENTS.md | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
index af944ed7..efe0d64a 100644
--- a/doc/FRAGMENTS.md
+++ b/doc/FRAGMENTS.md
@@ -19,15 +19,15 @@
  - adapt persister
     - DONE: persist fragments in /sync
     - DONE: fill gaps / fragment filling
-    - load n items before and after key,
-        - need to add fragments as we come across boundaries
-        - also cache fragments? not for now ...
-        - not doing any of the above, just reloading and rebuilding for now
+    - DONE: load n items before and after key,
+        - DONE: need to add fragments as we come across boundaries
+        - DONE: also cache fragments? not for now ...
+        - DONE: not doing any of the above, just reloading and rebuilding for now
 
- - adapt Timeline
-    - turn ObservableArray into ObservableSortedArray
+ - DONE: adapt Timeline
+    - DONE: turn ObservableArray into ObservableSortedArray
         - upsert already sorted sections
-        - upsert single entry
+        - DONE: upsert single entry
  - adapt TilesCollection & Tile to entry changes
  
  - add live fragment id optimization if we haven't done so already
@@ -35,6 +35,10 @@
     - could do this by only loading all fragments into index when filling gaps, backpaginating, ... and on persister load only load the last fragment. This wouldn't even need a FragmentIndex?
 
 
+implement SortedArray::setManySorted in a performant manner
+implement FragmentIdComparator::add in a performant manner
+there is some duplication (also in memory) between SortedArray and TilesCollection. Both keep a sorted list based on fragmentId/eventIndex... TilesCollection doesn't use the index in the event handlers at all. we could allow timeline to export a structure that just emits "these entries are a thing (now)" and not have to go through sorting twice. Timeline would have to keep track of the earliest key so it can use it in loadAtTop, but that should be easy. Hmmm. also, Timeline might want to be in charge of unloading parts of the loaded timeline, and for that it would need to know the order of entries. So maybe not ... we'll see.
+
 so a gap is two connected fragments where either the first fragment has a nextToken and/or the second fragment has a previousToken. It can be both, so we can have a gap where you can fill in from the top, from the bottom (like when limited sync) or both.
 
 

From b10aa269d2e3365bb0f91adeeb36abd0c56dc21f Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 18:32:17 +0200
Subject: [PATCH 59/83] very quick emote support

---
 src/domain/session/room/timeline/tiles/TextTile.js | 7 ++++++-
 src/domain/session/room/timeline/tilesCreator.js   | 1 +
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/src/domain/session/room/timeline/tiles/TextTile.js b/src/domain/session/room/timeline/tiles/TextTile.js
index 411b1a55..092ff889 100644
--- a/src/domain/session/room/timeline/tiles/TextTile.js
+++ b/src/domain/session/room/timeline/tiles/TextTile.js
@@ -3,6 +3,11 @@ import MessageTile from "./MessageTile.js";
 export default class TextTile extends MessageTile {
     get text() {
         const content = this._getContent();
-        return content && content.body;
+        const body = content && content.body;
+        if (this._entry.type() === "m.emote") {
+            return `* ${this._entry.event.sender} ${body}`;
+        } else {
+            return body;
+        }
     }
 }
diff --git a/src/domain/session/room/timeline/tilesCreator.js b/src/domain/session/room/timeline/tilesCreator.js
index 732e0ce6..96638aa0 100644
--- a/src/domain/session/room/timeline/tilesCreator.js
+++ b/src/domain/session/room/timeline/tilesCreator.js
@@ -19,6 +19,7 @@ export default function ({timeline, emitUpdate}) {
                     switch (msgtype) {
                         case "m.text":
                         case "m.notice":
+                        case "m.emote":
                             return new TextTile(options);
                         case "m.image":
                             return new ImageTile(options);

From 3de86cdf331bc20279212f6dc8201735b44aef0b Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sat, 1 Jun 2019 18:32:32 +0200
Subject: [PATCH 60/83] obsolete comment

---
 src/matrix/storage/idb/create.js | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/matrix/storage/idb/create.js b/src/matrix/storage/idb/create.js
index a118347b..be5618f0 100644
--- a/src/matrix/storage/idb/create.js
+++ b/src/matrix/storage/idb/create.js
@@ -13,7 +13,6 @@ function createStores(db) {
 
     // need index to find live fragment? prooobably ok without for now
     db.createObjectStore("timelineFragments", {keyPath: ["roomId", "id"]});
-    // needs roomId separate because it might hold a gap and no event
     const timelineEvents = db.createObjectStore("timelineEvents", {keyPath: ["event.room_id", "fragmentId", "eventIndex"]});
     timelineEvents.createIndex("byEventId", [
         "event.room_id",

From 210a00d5412fc033941b843c0b62fbb5192e8a40 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 00:03:55 +0200
Subject: [PATCH 61/83] first attempt at making UI work again, with tiles and
 gaps

---
 .../session/room/timeline/tiles/GapTile.js    | 12 +++-
 .../session/room/timeline/tiles/ImageTile.js  |  4 ++
 .../room/timeline/tiles/MessageTile.js        |  4 ++
 .../room/timeline/tiles/RoomMemberTile.js     |  5 ++
 .../room/timeline/tiles/RoomNameTile.js       |  5 ++
 .../session/room/timeline/tiles/TextTile.js   |  2 +-
 src/ui/web/RoomView.js                        |  8 +--
 src/ui/web/TimelineTile.js                    | 61 +++++++------------
 8 files changed, 55 insertions(+), 46 deletions(-)

diff --git a/src/domain/session/room/timeline/tiles/GapTile.js b/src/domain/session/room/timeline/tiles/GapTile.js
index cebffe24..240c071c 100644
--- a/src/domain/session/room/timeline/tiles/GapTile.js
+++ b/src/domain/session/room/timeline/tiles/GapTile.js
@@ -24,12 +24,20 @@ export default class GapTile extends SimpleTile {
         }
     }
 
+    get shape() {
+        return "gap";
+    }
+
     get isLoading() {
         return this._loading;
     }
 
-    get direction() {
-        return this._entry.direction;
+    get isUp() {
+        return this._entry.direction.isBackward;
+    }
+
+    get isDown() {
+        return this._entry.direction.isForward;
     }
 
     get error() {
diff --git a/src/domain/session/room/timeline/tiles/ImageTile.js b/src/domain/session/room/timeline/tiles/ImageTile.js
index 8c18491e..ebe8d022 100644
--- a/src/domain/session/room/timeline/tiles/ImageTile.js
+++ b/src/domain/session/room/timeline/tiles/ImageTile.js
@@ -19,4 +19,8 @@ export default class ImageTile extends MessageTile {
     get height() {
         return 200;
     }
+
+    get label() {
+        return "this is an image";
+    }
 }
diff --git a/src/domain/session/room/timeline/tiles/MessageTile.js b/src/domain/session/room/timeline/tiles/MessageTile.js
index f57f8379..8ada2310 100644
--- a/src/domain/session/room/timeline/tiles/MessageTile.js
+++ b/src/domain/session/room/timeline/tiles/MessageTile.js
@@ -7,6 +7,10 @@ export default class MessageTile extends SimpleTile {
         this._date = new Date(this._entry.event.origin_server_ts);
     }
 
+    get shape() {
+        return "message";
+    }
+
     get sender() {
         return this._entry.event.sender;
     }
diff --git a/src/domain/session/room/timeline/tiles/RoomMemberTile.js b/src/domain/session/room/timeline/tiles/RoomMemberTile.js
index df31945c..cc9796e9 100644
--- a/src/domain/session/room/timeline/tiles/RoomMemberTile.js
+++ b/src/domain/session/room/timeline/tiles/RoomMemberTile.js
@@ -1,6 +1,11 @@
 import SimpleTile from "./SimpleTile.js";
 
 export default class RoomNameTile extends SimpleTile {
+
+    get shape() {
+        return "announcement";
+    }
+
     get label() {
         const event = this._entry.event;
         const content = event.content;
diff --git a/src/domain/session/room/timeline/tiles/RoomNameTile.js b/src/domain/session/room/timeline/tiles/RoomNameTile.js
index e352b168..756d8d97 100644
--- a/src/domain/session/room/timeline/tiles/RoomNameTile.js
+++ b/src/domain/session/room/timeline/tiles/RoomNameTile.js
@@ -1,6 +1,11 @@
 import SimpleTile from "./SimpleTile.js";
 
 export default class RoomNameTile extends SimpleTile {
+    
+    get shape() {
+        return "annoucement";
+    }
+
     get label() {
         const event = this._entry.event;
         const content = event.content;
diff --git a/src/domain/session/room/timeline/tiles/TextTile.js b/src/domain/session/room/timeline/tiles/TextTile.js
index 092ff889..08c330f5 100644
--- a/src/domain/session/room/timeline/tiles/TextTile.js
+++ b/src/domain/session/room/timeline/tiles/TextTile.js
@@ -1,7 +1,7 @@
 import MessageTile from "./MessageTile.js";
 
 export default class TextTile extends MessageTile {
-    get text() {
+    get label() {
         const content = this._getContent();
         const body = content && content.body;
         if (this._entry.type() === "m.emote") {
diff --git a/src/ui/web/RoomView.js b/src/ui/web/RoomView.js
index 987bdeeb..fd18ba6f 100644
--- a/src/ui/web/RoomView.js
+++ b/src/ui/web/RoomView.js
@@ -14,9 +14,7 @@ export default class RoomView {
     mount() {
         this._viewModel.on("change", this._onViewModelUpdate);
         this._nameLabel = html.h2(null, this._viewModel.name);
-        this._timelineList = new ListView({
-            list: this._viewModel.timelineEntries
-        }, entry => new TimelineTile(entry));
+        this._timelineList = new ListView({}, entry => new TimelineTile(entry));
         this._timelineList.mount();
 
         this._root = html.div({className: "RoomView"}, [
@@ -40,8 +38,8 @@ export default class RoomView {
         if (prop === "name") {
             this._nameLabel.innerText = this._viewModel.name;
         }
-        else if (prop === "timelineEntries") {
-            this._timelineList.update({list: this._viewModel.timelineEntries});
+        else if (prop === "timelineViewModel") {
+            this._timelineList.update({list: this._viewModel.timelineViewModel.tiles});
         }
     }
 
diff --git a/src/ui/web/TimelineTile.js b/src/ui/web/TimelineTile.js
index 7c362d78..a8d7e481 100644
--- a/src/ui/web/TimelineTile.js
+++ b/src/ui/web/TimelineTile.js
@@ -1,29 +1,8 @@
 import * as html from "./html.js";
 
-function tileText(event) {
-    const content = event.content;
-    switch (event.type) {
-        case "m.room.message": {
-            const msgtype = content.msgtype;
-            switch (msgtype) {
-                case "m.text":
-                    return content.body;
-                default:
-                    return `unsupported msgtype: ${msgtype}`;
-            }
-        }
-        case "m.room.name":
-            return `changed the room name to "${content.name}"`;
-        case "m.room.member":
-            return `changed membership to ${content.membership}`;
-        default:
-            return `unsupported event type: ${event.type}`;
-    }
-}
-
 export default class TimelineTile {
-    constructor(entry) {
-        this._entry = entry;
+    constructor(tileVM) {
+        this._tileVM = tileVM;
         this._root = null;
     }
 
@@ -32,21 +11,7 @@ export default class TimelineTile {
     }
 
     mount() {
-        let children;
-        if (this._entry.gap) {
-            children = [
-                html.strong(null, "Gap"),
-                " with prev_batch ",
-                html.strong(null, this._entry.gap.prev_batch)
-            ];
-        } else if (this._entry.event) {
-            const event = this._entry.event;
-            children = [
-                html.strong(null, event.sender),
-                `: ${tileText(event)}`,
-            ];
-        }
-        this._root = html.li(null, children);
+        this._root = renderTile(this._tileVM);
         return this._root;
     }
 
@@ -54,3 +19,23 @@ export default class TimelineTile {
 
     update() {}
 }
+
+function renderTile(tile) {
+    switch (tile.shape) {
+        case "message":
+            return html.li(null, [html.strong(tile.sender), `: ${tile.label}`]);
+        case "gap": {
+            const button = html.button(null, (tile.isUp ? "🠝" : "🠟") + " fill gap");
+            const handler = () => {
+                tile.fill();
+                button.removeEventListener("click", handler);
+            };
+            button.addEventListener("click", handler);
+            return html.li(null, button);
+        }
+        case "announcement":
+            return html.li(null, tile.label);
+        default:
+            return html.li(null, "unknown tile shape: " + tile.shape);
+    }
+}

From a1e527ccbc0aa24a179a17451136c07cc675fd22 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 00:49:47 +0200
Subject: [PATCH 62/83] first round of fixes after running the app again in the
 browser!

---
 .../session/room/timeline/tiles/GapTile.js    |  2 +-
 .../room/timeline/tiles/RoomNameTile.js       |  2 +-
 src/matrix/room/room.js                       |  4 +--
 src/matrix/room/timeline/Direction.js         |  5 ++--
 .../room/timeline/persistence/SyncWriter.js   | 13 ++++-----
 .../room/timeline/persistence/common.js       |  3 +-
 src/matrix/storage/idb/create.js              |  2 +-
 src/matrix/sync.js                            | 28 +++++++++----------
 src/observable/list/SortedArray.js            |  2 +-
 src/observable/list/SortedMapList.js          |  2 +-
 10 files changed, 32 insertions(+), 31 deletions(-)

diff --git a/src/domain/session/room/timeline/tiles/GapTile.js b/src/domain/session/room/timeline/tiles/GapTile.js
index 240c071c..55bc7429 100644
--- a/src/domain/session/room/timeline/tiles/GapTile.js
+++ b/src/domain/session/room/timeline/tiles/GapTile.js
@@ -1,4 +1,4 @@
-import SimpleTile from "./SimpleTile";
+import SimpleTile from "./SimpleTile.js";
 
 export default class GapTile extends SimpleTile {
     constructor(options, timeline) {
diff --git a/src/domain/session/room/timeline/tiles/RoomNameTile.js b/src/domain/session/room/timeline/tiles/RoomNameTile.js
index 756d8d97..f415efe6 100644
--- a/src/domain/session/room/timeline/tiles/RoomNameTile.js
+++ b/src/domain/session/room/timeline/tiles/RoomNameTile.js
@@ -3,7 +3,7 @@ import SimpleTile from "./SimpleTile.js";
 export default class RoomNameTile extends SimpleTile {
     
     get shape() {
-        return "annoucement";
+        return "announcement";
     }
 
     get label() {
diff --git a/src/matrix/room/room.js b/src/matrix/room/room.js
index 09b3427c..7c17b3b1 100644
--- a/src/matrix/room/room.js
+++ b/src/matrix/room/room.js
@@ -17,9 +17,9 @@ export default class Room extends EventEmitter {
         this._timeline = null;
 	}
 
-    persistSync(roomResponse, membership, txn) {
+    async persistSync(roomResponse, membership, txn) {
 		const summaryChanged = this._summary.applySync(roomResponse, membership, txn);
-		const newTimelineEntries = this._syncWriter.writeSync(roomResponse, txn);
+		const newTimelineEntries = await this._syncWriter.writeSync(roomResponse, txn);
         return {summaryChanged, newTimelineEntries};
     }
 
diff --git a/src/matrix/room/timeline/Direction.js b/src/matrix/room/timeline/Direction.js
index d885ed63..6e527e88 100644
--- a/src/matrix/room/timeline/Direction.js
+++ b/src/matrix/room/timeline/Direction.js
@@ -1,6 +1,4 @@
 
-const _forward = Object.freeze(new Direction(true));
-const _backward = Object.freeze(new Direction(false));
 
 export default class Direction {
     constructor(isForward) {
@@ -27,3 +25,6 @@ export default class Direction {
         return _backward;
     }
 }
+
+const _forward = Object.freeze(new Direction(true));
+const _backward = Object.freeze(new Direction(false));
diff --git a/src/matrix/room/timeline/persistence/SyncWriter.js b/src/matrix/room/timeline/persistence/SyncWriter.js
index ebec1bae..faf0d6e6 100644
--- a/src/matrix/room/timeline/persistence/SyncWriter.js
+++ b/src/matrix/room/timeline/persistence/SyncWriter.js
@@ -13,7 +13,7 @@ export default class SyncWriter {
     async load(txn) {
         const liveFragment = await txn.timelineFragments.liveFragment(this._roomId);
         if (liveFragment) {
-            const [lastEvent] = await txn.roomTimeline.lastEvents(this._roomId, liveFragment.id, 1);
+            const [lastEvent] = await txn.timelineEvents.lastEvents(this._roomId, liveFragment.id, 1);
             // sorting and identifying (e.g. sort key and pk to insert) are a bit intertwined here
             // we could split it up into a SortKey (only with compare) and
             // a EventKey (no compare or fragment index) with nextkey methods and getters/setters for eventIndex/fragmentId
@@ -69,6 +69,7 @@ export default class SyncWriter {
 
     async writeSync(roomResponse, txn) {
         const entries = [];
+        const timeline = roomResponse.timeline;
         if (!this._lastLiveKey) {
             // means we haven't synced this room yet (just joined or did initial sync)
             
@@ -86,22 +87,20 @@ export default class SyncWriter {
             entries.push(FragmentBoundaryEntry.start(newFragment, this._fragmentIdComparer));
         }
         let currentKey = this._lastLiveKey;
-        const timeline = roomResponse.timeline;
         if (timeline.events) {
             for(const event of timeline.events) {
                 currentKey = currentKey.nextKey();
-                const entry = createEventEntry(currentKey, event);
-                txn.roomTimeline.insert(entry);
+                const entry = createEventEntry(currentKey, this._roomId, event);
+                txn.timelineEvents.insert(entry);
                 entries.push(new EventEntry(entry, this._fragmentIdComparer));
             }
         }
         // right thing to do? if the txn fails, not sure we'll continue anyways ...
-        // only advance the key once the transaction has
-        // succeeded 
+        // only advance the key once the transaction has succeeded 
         txn.complete().then(() => {
             console.log("txn complete, setting key");
             this._lastLiveKey = currentKey;
-        });
+        })
 
         // persist state
         const state = roomResponse.state;
diff --git a/src/matrix/room/timeline/persistence/common.js b/src/matrix/room/timeline/persistence/common.js
index 5bec1a82..93d96f94 100644
--- a/src/matrix/room/timeline/persistence/common.js
+++ b/src/matrix/room/timeline/persistence/common.js
@@ -1,7 +1,8 @@
-export function createEventEntry(key, event) {
+export function createEventEntry(key, roomId, event) {
     return {
         fragmentId: key.fragmentId,
         eventIndex: key.eventIndex,
+        roomId,
         event: event,
     };
 }
diff --git a/src/matrix/storage/idb/create.js b/src/matrix/storage/idb/create.js
index be5618f0..6ee6af24 100644
--- a/src/matrix/storage/idb/create.js
+++ b/src/matrix/storage/idb/create.js
@@ -13,7 +13,7 @@ function createStores(db) {
 
     // need index to find live fragment? prooobably ok without for now
     db.createObjectStore("timelineFragments", {keyPath: ["roomId", "id"]});
-    const timelineEvents = db.createObjectStore("timelineEvents", {keyPath: ["event.room_id", "fragmentId", "eventIndex"]});
+    const timelineEvents = db.createObjectStore("timelineEvents", {keyPath: ["roomId", "fragmentId", "eventIndex"]});
     timelineEvents.createIndex("byEventId", [
         "event.room_id",
         "event.event_id"
diff --git a/src/matrix/sync.js b/src/matrix/sync.js
index b12dd73e..11c6ad4c 100644
--- a/src/matrix/sync.js
+++ b/src/matrix/sync.js
@@ -9,19 +9,18 @@ const INCREMENTAL_TIMEOUT = 30000;
 const SYNC_EVENT_LIMIT = 10;
 
 function parseRooms(roomsSection, roomCallback) {
-    if (!roomsSection) {
-        return;
-    }
-    const allMemberships = ["join", "invite", "leave"];
-    for(const membership of allMemberships) {
-        const membershipSection = roomsSection[membership];
-        if (membershipSection) {
-            const rooms = Object.entries(membershipSection)
-            for (const [roomId, roomResponse] of rooms) {
-                roomCallback(roomId, roomResponse, membership);
+    if (roomsSection) {
+        const allMemberships = ["join", "invite", "leave"];
+        for(const membership of allMemberships) {
+            const membershipSection = roomsSection[membership];
+            if (membershipSection) {
+                return Object.entries(membershipSection).map(([roomId, roomResponse]) => {
+                    return roomCallback(roomId, roomResponse, membership);
+                });
             }
         }
     }
+    return [];
 }
 
 export default class Sync extends EventEmitter {
@@ -73,9 +72,9 @@ export default class Sync extends EventEmitter {
         const syncTxn = await this._storage.readWriteTxn([
             storeNames.session,
             storeNames.roomSummary,
+            storeNames.roomState,
             storeNames.timelineEvents,
             storeNames.timelineFragments,
-            storeNames.roomState,
         ]);
         const roomChanges = [];
         try {
@@ -83,18 +82,19 @@ export default class Sync extends EventEmitter {
             // to_device
             // presence
             if (response.rooms) {
-                parseRooms(response.rooms, (roomId, roomResponse, membership) => {
+                const promises = parseRooms(response.rooms, async (roomId, roomResponse, membership) => {
                     let room = this._session.rooms.get(roomId);
                     if (!room) {
                         room = this._session.createRoom(roomId);
                     }
                     console.log(` * applying sync response to room ${roomId} ...`);
-                    const changes = room.persistSync(roomResponse, membership, syncTxn);
+                    const changes = await room.persistSync(roomResponse, membership, syncTxn);
                     roomChanges.push({room, changes});
                 });
+                await Promise.all(promises);
             }
         } catch(err) {
-            console.warn("aborting syncTxn because of error");
+            console.warn("aborting syncTxn because of error", err.stack);
             // avoid corrupting state by only
             // storing the sync up till the point
             // the exception occurred
diff --git a/src/observable/list/SortedArray.js b/src/observable/list/SortedArray.js
index b18b7250..8c90ce15 100644
--- a/src/observable/list/SortedArray.js
+++ b/src/observable/list/SortedArray.js
@@ -1,5 +1,5 @@
 import BaseObservableList from "./BaseObservableList.js";
-import sortedIndex from "../../utils/sortedIndex";
+import sortedIndex from "../../utils/sortedIndex.js";
 
 export default class SortedArray extends BaseObservableList {
     constructor(comparator) {
diff --git a/src/observable/list/SortedMapList.js b/src/observable/list/SortedMapList.js
index a3479be0..ebd7e86a 100644
--- a/src/observable/list/SortedMapList.js
+++ b/src/observable/list/SortedMapList.js
@@ -1,5 +1,5 @@
 import BaseObservableList from "./BaseObservableList.js";
-import sortedIndex from "../../utils/sortedIndex";
+import sortedIndex from "../../utils/sortedIndex.js";
 
 /*
 

From e339cb73213b28aed5c5b5cbca38d0cacd5b15ad Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 14:59:30 +0200
Subject: [PATCH 63/83] more fixes, timeline is showing again

---
 index.html                                             |  4 ++++
 src/domain/session/room/RoomViewModel.js               |  3 ++-
 src/domain/session/room/timeline/TilesCollection.js    |  9 +++++++--
 src/domain/session/room/timeline/tiles/TextTile.js     |  7 ++++---
 src/main.js                                            |  2 +-
 src/matrix/room/timeline/Timeline.js                   |  2 +-
 src/matrix/room/timeline/entries/BaseEntry.js          |  5 +++++
 src/matrix/room/timeline/persistence/SyncWriter.js     |  2 +-
 src/matrix/room/timeline/persistence/TimelineReader.js | 10 +++++-----
 src/matrix/storage/idb/create.js                       |  2 +-
 src/matrix/sync.js                                     |  2 +-
 src/observable/list/BaseObservableList.js              |  7 +++++++
 src/ui/web/RoomView.js                                 |  5 +++++
 src/ui/web/TimelineTile.js                             |  2 +-
 src/ui/web/html.js                                     |  1 +
 15 files changed, 46 insertions(+), 17 deletions(-)

diff --git a/index.html b/index.html
index facaa4d3..477d7c05 100644
--- a/index.html
+++ b/index.html
@@ -45,6 +45,10 @@
                 flex: 1;
                 overflow-y: scroll;
             }
+
+            .RoomView_error {
+                color: red;
+            }
         </style>
 	</head>
 	<body>
diff --git a/src/domain/session/room/RoomViewModel.js b/src/domain/session/room/RoomViewModel.js
index 17427305..0e5f9b1c 100644
--- a/src/domain/session/room/RoomViewModel.js
+++ b/src/domain/session/room/RoomViewModel.js
@@ -16,8 +16,9 @@ export default class RoomViewModel extends EventEmitter {
         try {
             this._timeline = await this._room.openTimeline();
             this._timelineVM = new TimelineViewModel(this._timeline);
-            this.emit("change", "timelineEntries");
+            this.emit("change", "timelineViewModel");
         } catch (err) {
+            console.error(`room.openTimeline(): ${err.message}:\n${err.stack}`);
             this._timelineError = err;
             this.emit("change", "error");
         }
diff --git a/src/domain/session/room/timeline/TilesCollection.js b/src/domain/session/room/timeline/TilesCollection.js
index 3c80fc98..0b1dab8e 100644
--- a/src/domain/session/room/timeline/TilesCollection.js
+++ b/src/domain/session/room/timeline/TilesCollection.js
@@ -22,8 +22,9 @@ export default class TilesCollection extends BaseObservableList {
         for (let entry of this._entries) {
             if (!currentTile || !currentTile.tryIncludeEntry(entry)) {
                 currentTile = this._tileCreator(entry);
-                // if (currentTile) here?
-                this._tiles.push(currentTile);
+                if (currentTile) {
+                    this._tiles.push(currentTile);
+                }
             }
         }
         let prevTile = null;
@@ -145,4 +146,8 @@ export default class TilesCollection extends BaseObservableList {
     [Symbol.iterator]() {
         return this._tiles.values();
     }
+
+    get length() {
+        return this._tiles.length;
+    }
 }
diff --git a/src/domain/session/room/timeline/tiles/TextTile.js b/src/domain/session/room/timeline/tiles/TextTile.js
index 08c330f5..03485ea0 100644
--- a/src/domain/session/room/timeline/tiles/TextTile.js
+++ b/src/domain/session/room/timeline/tiles/TextTile.js
@@ -4,10 +4,11 @@ export default class TextTile extends MessageTile {
     get label() {
         const content = this._getContent();
         const body = content && content.body;
-        if (this._entry.type() === "m.emote") {
-            return `* ${this._entry.event.sender} ${body}`;
+        const sender = this._entry.event.sender;
+        if (this._entry.type === "m.emote") {
+            return `* ${sender} ${body}`;
         } else {
-            return body;
+            return `${sender}: ${body}`;
         }
     }
 }
diff --git a/src/main.js b/src/main.js
index 4f840c4b..f0787673 100644
--- a/src/main.js
+++ b/src/main.js
@@ -88,6 +88,6 @@ export default async function main(label, button, container) {
 			label.innerText = "sync stopped";
 		});
 	} catch(err) {
-		console.error(err);
+        console.error(`${err.message}:\n${err.stack}`);
 	}
 }
diff --git a/src/matrix/room/timeline/Timeline.js b/src/matrix/room/timeline/Timeline.js
index da40eb53..505138f9 100644
--- a/src/matrix/room/timeline/Timeline.js
+++ b/src/matrix/room/timeline/Timeline.js
@@ -20,7 +20,7 @@ export default class Timeline {
 
     /** @package */
     async load() {
-        const entries = this._timelineReader.readFromEnd(100);
+        const entries = await this._timelineReader.readFromEnd(100);
         this._entriesList.setManySorted(entries);
     }
 
diff --git a/src/matrix/room/timeline/entries/BaseEntry.js b/src/matrix/room/timeline/entries/BaseEntry.js
index 37695ddf..3ef00862 100644
--- a/src/matrix/room/timeline/entries/BaseEntry.js
+++ b/src/matrix/room/timeline/entries/BaseEntry.js
@@ -1,4 +1,5 @@
 //entries can be sorted, first by fragment, then by entry index.
+import EventKey from "../EventKey.js";
 
 export default class BaseEntry {
     constructor(fragmentIdComparer) {
@@ -21,4 +22,8 @@ export default class BaseEntry {
             return this._fragmentIdComparer.compare(this.fragmentId, otherEntry.fragmentId);
         }
     }
+
+    asEventKey() {
+        return new EventKey(this.fragmentId, this.entryIndex);
+    }
 }
diff --git a/src/matrix/room/timeline/persistence/SyncWriter.js b/src/matrix/room/timeline/persistence/SyncWriter.js
index faf0d6e6..2abbcf13 100644
--- a/src/matrix/room/timeline/persistence/SyncWriter.js
+++ b/src/matrix/room/timeline/persistence/SyncWriter.js
@@ -8,6 +8,7 @@ export default class SyncWriter {
         this._roomId = roomId;
         this._storage = storage;
         this._fragmentIdComparer = fragmentIdComparer;
+        this._lastLiveKey = null;
     }
 
     async load(txn) {
@@ -98,7 +99,6 @@ export default class SyncWriter {
         // right thing to do? if the txn fails, not sure we'll continue anyways ...
         // only advance the key once the transaction has succeeded 
         txn.complete().then(() => {
-            console.log("txn complete, setting key");
             this._lastLiveKey = currentKey;
         })
 
diff --git a/src/matrix/room/timeline/persistence/TimelineReader.js b/src/matrix/room/timeline/persistence/TimelineReader.js
index 8f2e5a4d..dd57ca39 100644
--- a/src/matrix/room/timeline/persistence/TimelineReader.js
+++ b/src/matrix/room/timeline/persistence/TimelineReader.js
@@ -32,9 +32,9 @@ export default class TimelineReader {
         while (entries.length < amount && eventKey) {
             let eventsWithinFragment;
             if (direction.isForward) {
-                eventsWithinFragment = timelineStore.eventsAfter(eventKey, amount);
+                eventsWithinFragment = await timelineStore.eventsAfter(this._roomId, eventKey, amount);
             } else {
-                eventsWithinFragment = timelineStore.eventsBefore(eventKey, amount);
+                eventsWithinFragment = await timelineStore.eventsBefore(this._roomId, eventKey, amount);
             }
             const eventEntries = eventsWithinFragment.map(e => new EventEntry(e, this._fragmentIdComparer));
             entries = directionalConcat(entries, eventEntries, direction);
@@ -52,7 +52,7 @@ export default class TimelineReader {
                     this._fragmentIdComparer.add(nextFragment);
                     const nextFragmentEntry = new FragmentBoundaryEntry(nextFragment, direction.isForward, this._fragmentIdComparer);
                     directionalAppend(entries, nextFragmentEntry, direction);
-                    eventKey = new EventKey(nextFragmentEntry.fragmentId, nextFragmentEntry.eventIndex);
+                    eventKey = nextFragmentEntry.asEventKey();
                 } else {
                     eventKey = null;
                 }
@@ -71,8 +71,8 @@ export default class TimelineReader {
         }
         this._fragmentIdComparer.add(liveFragment);
         const liveFragmentEntry = FragmentBoundaryEntry.end(liveFragment, this._fragmentIdComparer);
-        const eventKey = new EventKey(liveFragmentEntry.fragmentId, liveFragmentEntry.eventIndex);
-        const entries = this._readFrom(eventKey, Direction.Backward, amount, txn);
+        const eventKey = liveFragmentEntry.asEventKey();
+        const entries = await this._readFrom(eventKey, Direction.Backward, amount, txn);
         entries.unshift(liveFragmentEntry);
         return entries;
     }
diff --git a/src/matrix/storage/idb/create.js b/src/matrix/storage/idb/create.js
index 6ee6af24..ce8b6aca 100644
--- a/src/matrix/storage/idb/create.js
+++ b/src/matrix/storage/idb/create.js
@@ -15,7 +15,7 @@ function createStores(db) {
     db.createObjectStore("timelineFragments", {keyPath: ["roomId", "id"]});
     const timelineEvents = db.createObjectStore("timelineEvents", {keyPath: ["roomId", "fragmentId", "eventIndex"]});
     timelineEvents.createIndex("byEventId", [
-        "event.room_id",
+        "roomId",
         "event.event_id"
     ], {unique: true});
 
diff --git a/src/matrix/sync.js b/src/matrix/sync.js
index 11c6ad4c..37425090 100644
--- a/src/matrix/sync.js
+++ b/src/matrix/sync.js
@@ -94,7 +94,7 @@ export default class Sync extends EventEmitter {
                 await Promise.all(promises);
             }
         } catch(err) {
-            console.warn("aborting syncTxn because of error", err.stack);
+            console.warn("aborting syncTxn because of error");
             // avoid corrupting state by only
             // storing the sync up till the point
             // the exception occurred
diff --git a/src/observable/list/BaseObservableList.js b/src/observable/list/BaseObservableList.js
index a1c38e91..97e50b20 100644
--- a/src/observable/list/BaseObservableList.js
+++ b/src/observable/list/BaseObservableList.js
@@ -34,4 +34,11 @@ export default class BaseObservableList extends BaseObservableCollection {
         }
     }
 
+    [Symbol.iterator]() {
+        throw new Error("unimplemented");
+    }
+
+    get length() {
+        throw new Error("unimplemented");
+    }
 }
diff --git a/src/ui/web/RoomView.js b/src/ui/web/RoomView.js
index fd18ba6f..d26cf5df 100644
--- a/src/ui/web/RoomView.js
+++ b/src/ui/web/RoomView.js
@@ -14,11 +14,14 @@ export default class RoomView {
     mount() {
         this._viewModel.on("change", this._onViewModelUpdate);
         this._nameLabel = html.h2(null, this._viewModel.name);
+        this._errorLabel = html.div({className: "RoomView_error"});
+
         this._timelineList = new ListView({}, entry => new TimelineTile(entry));
         this._timelineList.mount();
 
         this._root = html.div({className: "RoomView"}, [
             this._nameLabel,
+            this._errorLabel,
             this._timelineList.root()
         ]);
 
@@ -40,6 +43,8 @@ export default class RoomView {
         }
         else if (prop === "timelineViewModel") {
             this._timelineList.update({list: this._viewModel.timelineViewModel.tiles});
+        } else if (prop === "error") {
+            this._errorLabel.innerText = this._viewModel.error;
         }
     }
 
diff --git a/src/ui/web/TimelineTile.js b/src/ui/web/TimelineTile.js
index a8d7e481..bf03d5f4 100644
--- a/src/ui/web/TimelineTile.js
+++ b/src/ui/web/TimelineTile.js
@@ -23,7 +23,7 @@ export default class TimelineTile {
 function renderTile(tile) {
     switch (tile.shape) {
         case "message":
-            return html.li(null, [html.strong(tile.sender), `: ${tile.label}`]);
+            return html.li(null, tile.label);
         case "gap": {
             const button = html.button(null, (tile.isUp ? "🠝" : "🠟") + " fill gap");
             const handler = () => {
diff --git a/src/ui/web/html.js b/src/ui/web/html.js
index 754e3ebe..ebf57091 100644
--- a/src/ui/web/html.js
+++ b/src/ui/web/html.js
@@ -51,3 +51,4 @@ export function main(... params)    { return el("main", ... params); }
 export function article(... params) { return el("article", ... params); }
 export function aside(... params)   { return el("aside", ... params); }
 export function pre(... params)   { return el("pre", ... params); }
+export function button(... params)   { return el("button", ... params); }

From 1ed3babfecb74120ef9e84f6b495692e8fd91259 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 15:01:14 +0200
Subject: [PATCH 64/83] fragment boundary is a gap if backwards(started) &
 previousToken

---
 src/matrix/room/timeline/entries/FragmentBoundaryEntry.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
index 10ae7ec4..83993315 100644
--- a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
+++ b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
@@ -47,9 +47,9 @@ export default class FragmentBoundaryEntry extends BaseEntry {
 
     get token() {
         if (this.started) {
-            return this.fragment.nextToken;
-        } else {
             return this.fragment.previousToken;
+        } else {
+            return this.fragment.nextToken;
         }
     }
 

From 674007d892110d813664554ae8875f21245dbd7e Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 15:15:14 +0200
Subject: [PATCH 65/83] don't fail when insert first or appending a tile

---
 src/domain/session/room/timeline/TilesCollection.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/domain/session/room/timeline/TilesCollection.js b/src/domain/session/room/timeline/TilesCollection.js
index 0b1dab8e..4726fdde 100644
--- a/src/domain/session/room/timeline/TilesCollection.js
+++ b/src/domain/session/room/timeline/TilesCollection.js
@@ -88,8 +88,8 @@ export default class TilesCollection extends BaseObservableList {
 
         const newTile = this._tileCreator(entry);
         if (newTile) {
-            prevTile.updateNextSibling(newTile);
-            nextTile.updatePreviousSibling(newTile);
+            prevTile  && prevTile.updateNextSibling(newTile);
+            nextTile  && nextTile.updatePreviousSibling(newTile);
             this._tiles.splice(tileIdx, 0, newTile);
             this.emitAdd(tileIdx, newTile);
         }

From d022608a1a1e723a3aa8ea88c46088a3399088f5 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 15:15:26 +0200
Subject: [PATCH 66/83] it's entry.entryIndex, not eventIndex, but use helper
 method instead.

---
 src/matrix/room/timeline/Timeline.js | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/matrix/room/timeline/Timeline.js b/src/matrix/room/timeline/Timeline.js
index 505138f9..01483017 100644
--- a/src/matrix/room/timeline/Timeline.js
+++ b/src/matrix/room/timeline/Timeline.js
@@ -1,5 +1,4 @@
 import { SortedArray } from "../../../observable/index.js";
-import EventKey from "./EventKey.js";
 import Direction from "./Direction.js";
 import GapWriter from "./persistence/GapWriter.js";
 import TimelineReader from "./persistence/TimelineReader.js";
@@ -52,8 +51,11 @@ export default class Timeline {
             return;
         }
         const firstEntry = this._entriesList.array()[0];
-        const firstKey = new EventKey(firstEntry.fragmentId, firstEntry.eventIndex);
-        const entries = await this._timelineReader.readFrom(firstKey, Direction.Backward, amount);
+        const entries = await this._timelineReader.readFrom(
+            firstEntry.asEventKey(),
+            Direction.Backward,
+            amount
+        );
         this._entriesList.setManySorted(entries);
     }
 

From bdad0ad86b6cb4cf96dcf2aff82ae81e2d1f1c74 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 15:46:24 +0200
Subject: [PATCH 67/83] fix some gap fill errors

---
 src/domain/session/room/timeline/tiles/GapTile.js | 12 +++++++-----
 src/matrix/room/timeline/Timeline.js              |  8 ++++----
 src/matrix/room/timeline/persistence/GapWriter.js | 13 +++++++------
 3 files changed, 18 insertions(+), 15 deletions(-)

diff --git a/src/domain/session/room/timeline/tiles/GapTile.js b/src/domain/session/room/timeline/tiles/GapTile.js
index 55bc7429..c063db25 100644
--- a/src/domain/session/room/timeline/tiles/GapTile.js
+++ b/src/domain/session/room/timeline/tiles/GapTile.js
@@ -12,14 +12,16 @@ export default class GapTile extends SimpleTile {
         // prevent doing this twice
         if (!this._loading) {
             this._loading = true;
-            this._emitUpdate("isLoading");
+            // this._emitUpdate("isLoading");
             try {
-                return await this._timeline.fillGap(this._entry, 10);
+                await this._timeline.fillGap(this._entry, 10);
             } catch (err) {
-                this._loading = false;
+                console.error(`timeline.fillGap(): ${err.message}:\n${err.stack}`);
                 this._error = err;
-                this._emitUpdate("isLoading");
-                this._emitUpdate("error");
+                // this._emitUpdate("error");
+            } finally {
+                this._loading = false;
+                // this._emitUpdate("isLoading");
             }
         }
     }
diff --git a/src/matrix/room/timeline/Timeline.js b/src/matrix/room/timeline/Timeline.js
index 01483017..52900777 100644
--- a/src/matrix/room/timeline/Timeline.js
+++ b/src/matrix/room/timeline/Timeline.js
@@ -4,11 +4,12 @@ import GapWriter from "./persistence/GapWriter.js";
 import TimelineReader from "./persistence/TimelineReader.js";
 
 export default class Timeline {
-    constructor({roomId, storage, closeCallback, fragmentIdComparer}) {
+    constructor({roomId, storage, closeCallback, fragmentIdComparer, hsApi}) {
         this._roomId = roomId;
         this._storage = storage;
         this._closeCallback = closeCallback;
         this._fragmentIdComparer = fragmentIdComparer;
+        this._hsApi = hsApi;
         this._entriesList = new SortedArray((a, b) => a.compare(b));
         this._timelineReader = new TimelineReader({
             roomId: this._roomId,
@@ -34,14 +35,13 @@ export default class Timeline {
             from: fragmentEntry.token,
             dir: fragmentEntry.direction.asApiString(),
             limit: amount
-        });
-
+        }).response();
         const gapWriter = new GapWriter({
             roomId: this._roomId,
             storage: this._storage,
             fragmentIdComparer: this._fragmentIdComparer
         });
-        const newEntries = await gapWriter.writerFragmentFill(fragmentEntry, response);
+        const newEntries = await gapWriter.writeFragmentFill(fragmentEntry, response);
         this._entriesList.setManySorted(newEntries);
     }
 
diff --git a/src/matrix/room/timeline/persistence/GapWriter.js b/src/matrix/room/timeline/persistence/GapWriter.js
index 415f2292..ad721586 100644
--- a/src/matrix/room/timeline/persistence/GapWriter.js
+++ b/src/matrix/room/timeline/persistence/GapWriter.js
@@ -47,9 +47,10 @@ export default class GapWriter {
         const reducer = direction.isBackward ? Array.prototype.reduceRight : Array.prototype.reduce;
         reducer.call(events, (key, event, i) => {
             key = key.nextKeyForDirection(direction);
-            const eventEntry = createEventEntry(key, event);
+            const eventEntry = createEventEntry(key, this._roomId, event);
             txn.timelineEvents.insert(eventEntry);
             entries[i] = new EventEntry(eventEntry, this._fragmentIdComparer);
+            return key;
         }, startKey);
         return entries;
     }
@@ -61,7 +62,7 @@ export default class GapWriter {
         if (neighbourFragmentEntry) {
             fragmentEntry.linkedFragmentId = neighbourFragmentEntry.fragmentId;
             neighbourFragmentEntry.linkedFragmentId = fragmentEntry.fragmentId;
-            txn.timelineFragments.set(neighbourFragmentEntry.fragment);
+            txn.timelineFragments.update(neighbourFragmentEntry.fragment);
             directionalAppend(entries, neighbourFragmentEntry, direction);
 
             // update fragmentIdComparer here after linking up fragments
@@ -69,7 +70,7 @@ export default class GapWriter {
             this._fragmentIdComparer.add(neighbourFragmentEntry.fragment);
         }
         fragmentEntry.token = end;
-        txn.timelineFragments.set(fragmentEntry.fragment);
+        txn.timelineFragments.update(fragmentEntry.fragment);
     }
 
     async writeFragmentFill(fragmentEntry, response) {
@@ -92,7 +93,7 @@ export default class GapWriter {
 
         try {
             // make sure we have the latest fragment from the store
-            const fragment = await txn.timelineFragments.get(fragmentId);
+            const fragment = await txn.timelineFragments.get(this._roomId, fragmentId);
             if (!fragment) {
                 throw new Error(`Unknown fragment: ${fragmentId}`);
             }
@@ -102,12 +103,12 @@ export default class GapWriter {
                 throw new Error("start is not equal to prev_batch or next_batch");
             }
             // find last event in fragment so we get the eventIndex to begin creating keys at
-            let lastKey = this._findLastFragmentEventKey(fragmentEntry, txn);
+            let lastKey = await this._findLastFragmentEventKey(fragmentEntry, txn);
             // find out if any event in chunk is already present using findFirstOrLastOccurringEventId
             const {
                 nonOverlappingEvents,
                 neighbourFragmentEntry
-            } = this._findOverlappingEvents(fragmentEntry, chunk, txn);
+            } = await this._findOverlappingEvents(fragmentEntry, chunk, txn);
 
             // create entries for all events in chunk, add them to entries
             entries = this._storeEvents(nonOverlappingEvents, lastKey, direction, txn);

From a269f612b6c292ccb837b0431f073be2473de65b Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 15:46:44 +0200
Subject: [PATCH 68/83] space

---
 src/domain/session/room/timeline/TilesCollection.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/domain/session/room/timeline/TilesCollection.js b/src/domain/session/room/timeline/TilesCollection.js
index 4726fdde..8af2879d 100644
--- a/src/domain/session/room/timeline/TilesCollection.js
+++ b/src/domain/session/room/timeline/TilesCollection.js
@@ -88,8 +88,8 @@ export default class TilesCollection extends BaseObservableList {
 
         const newTile = this._tileCreator(entry);
         if (newTile) {
-            prevTile  && prevTile.updateNextSibling(newTile);
-            nextTile  && nextTile.updatePreviousSibling(newTile);
+            prevTile && prevTile.updateNextSibling(newTile);
+            nextTile && nextTile.updatePreviousSibling(newTile);
             this._tiles.splice(tileIdx, 0, newTile);
             this.emitAdd(tileIdx, newTile);
         }

From a59014475b0c2dfba156a781f3f17d30a8b0c434 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 18:15:08 +0200
Subject: [PATCH 69/83] also swap logic of setter

---
 src/matrix/room/timeline/entries/FragmentBoundaryEntry.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
index 83993315..7af089f9 100644
--- a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
+++ b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
@@ -55,9 +55,9 @@ export default class FragmentBoundaryEntry extends BaseEntry {
 
     set token(token) {
         if (this.started) {
-            this.fragment.nextToken = token;
-        } else {
             this.fragment.previousToken = token;
+        } else {
+            this.fragment.nextToken = token;
         }
     }
 

From e3b4f898d0312f972d62efd55bf17cd550d7ced6 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 18:15:24 +0200
Subject: [PATCH 70/83] show eventkey in ui for debugging purposes

---
 src/domain/session/room/timeline/tiles/SimpleTile.js | 4 ++++
 src/ui/web/TimelineTile.js                           | 8 ++++----
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/src/domain/session/room/timeline/tiles/SimpleTile.js b/src/domain/session/room/timeline/tiles/SimpleTile.js
index 266b09db..605fc1c1 100644
--- a/src/domain/session/room/timeline/tiles/SimpleTile.js
+++ b/src/domain/session/room/timeline/tiles/SimpleTile.js
@@ -58,4 +58,8 @@ export default class SimpleTile {
     updateNextSibling(next) {
     
     }
+
+    get internalId() {
+        return this._entry.asEventKey().toString();
+    }
 }
diff --git a/src/ui/web/TimelineTile.js b/src/ui/web/TimelineTile.js
index bf03d5f4..4c0ff45d 100644
--- a/src/ui/web/TimelineTile.js
+++ b/src/ui/web/TimelineTile.js
@@ -23,7 +23,7 @@ export default class TimelineTile {
 function renderTile(tile) {
     switch (tile.shape) {
         case "message":
-            return html.li(null, tile.label);
+            return html.li(null, [html.strong(null, tile.internalId+" "), tile.label]);
         case "gap": {
             const button = html.button(null, (tile.isUp ? "🠝" : "🠟") + " fill gap");
             const handler = () => {
@@ -31,11 +31,11 @@ function renderTile(tile) {
                 button.removeEventListener("click", handler);
             };
             button.addEventListener("click", handler);
-            return html.li(null, button);
+            return html.li(null, [html.strong(null, tile.internalId+" "), button]);
         }
         case "announcement":
-            return html.li(null, tile.label);
+            return html.li(null, [html.strong(null, tile.internalId+" "), tile.label]);
         default:
-            return html.li(null, "unknown tile shape: " + tile.shape);
+            return html.li(null, [html.strong(null, tile.internalId+" "), "unknown tile shape: " + tile.shape]);
     }
 }

From ef5d2cfa0834725ff4688031d802579955ee68a2 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 18:15:40 +0200
Subject: [PATCH 71/83] chunk is in reverse-chronological order for backward
 pagination

---
 .../room/timeline/persistence/GapWriter.js     | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/src/matrix/room/timeline/persistence/GapWriter.js b/src/matrix/room/timeline/persistence/GapWriter.js
index ad721586..ba25f909 100644
--- a/src/matrix/room/timeline/persistence/GapWriter.js
+++ b/src/matrix/room/timeline/persistence/GapWriter.js
@@ -43,15 +43,17 @@ export default class GapWriter {
     }
 
     _storeEvents(events, startKey, direction, txn) {
-        const entries = new Array(events.length);
-        const reducer = direction.isBackward ? Array.prototype.reduceRight : Array.prototype.reduce;
-        reducer.call(events, (key, event, i) => {
+        const entries = [];
+        // events is in reverse chronological order for backwards pagination,
+        // e.g. order is moving away from the `from` point.
+        let key = startKey;
+        for(let event of events) {
             key = key.nextKeyForDirection(direction);
-            const eventEntry = createEventEntry(key, this._roomId, event);
-            txn.timelineEvents.insert(eventEntry);
-            entries[i] = new EventEntry(eventEntry, this._fragmentIdComparer);
-            return key;
-        }, startKey);
+            const eventStorageEntry = createEventEntry(key, this._roomId, event);
+            txn.timelineEvents.insert(eventStorageEntry);
+            const eventEntry = new EventEntry(eventStorageEntry, this._fragmentIdComparer);
+            directionalAppend(entries, eventEntry, direction);
+        }
         return entries;
     }
 

From c63d94947f08535824c5b67988bc7f2d39767e69 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 18:28:38 +0200
Subject: [PATCH 72/83] fix persisting a gappy sync

---
 src/matrix/room/timeline/persistence/SyncWriter.js | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/matrix/room/timeline/persistence/SyncWriter.js b/src/matrix/room/timeline/persistence/SyncWriter.js
index 2abbcf13..550df584 100644
--- a/src/matrix/room/timeline/persistence/SyncWriter.js
+++ b/src/matrix/room/timeline/persistence/SyncWriter.js
@@ -49,7 +49,7 @@ export default class SyncWriter {
     }
 
     async _replaceLiveFragment(oldFragmentId, newFragmentId, previousToken, txn) {
-        const oldFragment = await txn.timelineFragments.get(oldFragmentId);
+        const oldFragment = await txn.timelineFragments.get(this._roomId, oldFragmentId);
         if (!oldFragment) {
             throw new Error(`old live fragment doesn't exist: ${oldFragmentId}`);
         }
@@ -83,7 +83,7 @@ export default class SyncWriter {
             // replace live fragment for limited sync, *only* if we had a live fragment already
             const oldFragmentId = this._lastLiveKey.fragmentId;
             this._lastLiveKey = this._lastLiveKey.nextFragmentKey();
-            const {oldFragment, newFragment} = this._replaceLiveFragment(oldFragmentId, this._lastLiveKey.fragmentId, timeline.prev_batch, txn);
+            const {oldFragment, newFragment} = await this._replaceLiveFragment(oldFragmentId, this._lastLiveKey.fragmentId, timeline.prev_batch, txn);
             entries.push(FragmentBoundaryEntry.end(oldFragment, this._fragmentIdComparer));
             entries.push(FragmentBoundaryEntry.start(newFragment, this._fragmentIdComparer));
         }
@@ -106,7 +106,7 @@ export default class SyncWriter {
         const state = roomResponse.state;
         if (state.events) {
             for (const event of state.events) {
-                txn.roomState.setStateEvent(this._roomId, event)
+                txn.roomState.setStateEvent(this._roomId, event);
             }
         }
         // persist live state events in timeline

From 4b5b90e19908186ea9746bcfab8ca3f1fc449947 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 19:26:45 +0200
Subject: [PATCH 73/83] fix another direction mismatch

---
 src/matrix/room/timeline/entries/FragmentBoundaryEntry.js | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
index 7af089f9..ce496364 100644
--- a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
+++ b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
@@ -63,17 +63,17 @@ export default class FragmentBoundaryEntry extends BaseEntry {
 
     get linkedFragmentId() {
         if (this.started) {
-            return this.fragment.nextId;
-        } else {
             return this.fragment.previousId;
+        } else {
+            return this.fragment.nextId;
         }
     }
 
     set linkedFragmentId(id) {
         if (this.started) {
-            this.fragment.nextId = id;
-        } else {
             this.fragment.previousId = id;
+        } else {
+            this.fragment.nextId = id;
         }
     }
 

From bb5f1393558a56f448c3eb2ed5f0397468e74f20 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 19:27:23 +0200
Subject: [PATCH 74/83] fix fragmentId:0 being evaluated as falsy

---
 .../room/timeline/FragmentIdComparer.js       | 52 ++++++++++++++++---
 src/matrix/room/timeline/common.js            |  3 ++
 .../timeline/entries/FragmentBoundaryEntry.js |  5 ++
 .../timeline/persistence/TimelineReader.js    |  3 +-
 4 files changed, 54 insertions(+), 9 deletions(-)
 create mode 100644 src/matrix/room/timeline/common.js

diff --git a/src/matrix/room/timeline/FragmentIdComparer.js b/src/matrix/room/timeline/FragmentIdComparer.js
index 15c2bc08..c26aa043 100644
--- a/src/matrix/room/timeline/FragmentIdComparer.js
+++ b/src/matrix/room/timeline/FragmentIdComparer.js
@@ -22,31 +22,31 @@ until no more fragments
 
 */
 
+import {isValidFragmentId} from "./common.js";
 
 function findBackwardSiblingFragments(current, byId) {
     const sortedSiblings = [];
-    while (current.previousId) {
+    while (isValidFragmentId(current.previousId)) {
         const previous = byId.get(current.previousId);
         if (!previous) {
-            throw new Error(`Unknown previousId ${current.previousId} on ${current.id}`);
+            break;
         }
         if (previous.nextId !== current.id) {
             throw new Error(`Previous fragment ${previous.id} doesn't point back to ${current.id}`);
         }
         byId.delete(current.previousId);
-        sortedSiblings.push(previous);
+        sortedSiblings.unshift(previous);
         current = previous;
     }
-    sortedSiblings.reverse();
     return sortedSiblings;
 }
 
 function findForwardSiblingFragments(current, byId) {
     const sortedSiblings = [];
-    while (current.nextId) {
+    while (isValidFragmentId(current.nextId)) {
         const next = byId.get(current.nextId);
         if (!next) {
-            throw new Error(`Unknown nextId ${current.nextId} on ${current.id}`);
+            break;
         }
         if (next.previousId !== current.id) {
             throw new Error(`Next fragment ${next.id} doesn't point back to ${current.id}`);
@@ -143,7 +143,7 @@ export default class FragmentIdComparer {
     }
 
     add(fragment) {
-        this._fragmentsById[fragment.id] = fragment;
+        this._fragmentsById.set(fragment.id, fragment);
         this.rebuild(this._fragmentsById.values());
     }
 }
@@ -168,6 +168,44 @@ export function tests() {
             
             assert.equal(index.compare(1, 1), 0);
         },
+        test_falsy_id(assert) {
+            const index = new FragmentIdComparer([
+                {id: 0, nextId: 1},
+                {id: 1, previousId: 0},
+            ]);
+            assert(index.compare(0, 1) < 0);
+            assert(index.compare(1, 0) > 0);
+        },
+        test_falsy_id_reverse(assert) {
+            const index = new FragmentIdComparer([
+                {id: 1, previousId: 0},
+                {id: 0, nextId: 1},
+            ]);
+            assert(index.compare(0, 1) < 0);
+            assert(index.compare(1, 0) > 0);
+        },
+        test_allow_unknown_id(assert) {
+            // as we tend to load fragments incrementally
+            // as events come into view, we need to allow
+            // unknown previousId/nextId in the fragments that we do load
+            assert.doesNotThrow(() => {
+                new FragmentIdComparer([
+                    {id: 1, previousId: 2},
+                    {id: 0, nextId: 3},
+                ]);
+            });
+        },
+        test_throw_on_link_mismatch(assert) {
+            // as we tend to load fragments incrementally
+            // as events come into view, we need to allow
+            // unknown previousId/nextId in the fragments that we do load
+            assert.throws(() => {
+                new FragmentIdComparer([
+                    {id: 1, previousId: 0},
+                    {id: 0, nextId: 2},
+                ]);
+            });
+        },
         test_2_island_dont_compare(assert) {
             const index = new FragmentIdComparer([
                 {id: 1},
diff --git a/src/matrix/room/timeline/common.js b/src/matrix/room/timeline/common.js
new file mode 100644
index 00000000..7565d8a4
--- /dev/null
+++ b/src/matrix/room/timeline/common.js
@@ -0,0 +1,3 @@
+export function isValidFragmentId(id) {
+    return typeof id === "number";
+}
diff --git a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
index ce496364..3182fd2f 100644
--- a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
+++ b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
@@ -1,5 +1,6 @@
 import BaseEntry from "./BaseEntry.js";
 import Direction from "../Direction.js";
+import {isValidFragmentId} from "../common.js";
 
 export default class FragmentBoundaryEntry extends BaseEntry {
     constructor(fragment, isFragmentStart, fragmentIdComparator) {
@@ -77,6 +78,10 @@ export default class FragmentBoundaryEntry extends BaseEntry {
         }
     }
 
+    get hasLinkedFragment() {
+        return isValidFragmentId(this.linkedFragmentId);
+    }
+
     get direction() {
         if (this.started) {
             return Direction.Backward;
diff --git a/src/matrix/room/timeline/persistence/TimelineReader.js b/src/matrix/room/timeline/persistence/TimelineReader.js
index dd57ca39..7d86115b 100644
--- a/src/matrix/room/timeline/persistence/TimelineReader.js
+++ b/src/matrix/room/timeline/persistence/TimelineReader.js
@@ -25,7 +25,6 @@ export default class TimelineReader {
 
     async _readFrom(eventKey, direction, amount, txn) {
         let entries = [];
-
         const timelineStore = txn.timelineEvents;
         const fragmentStore = txn.timelineFragments;
         
@@ -47,7 +46,7 @@ export default class TimelineReader {
                 // append or prepend fragmentEntry, reuse func from GapWriter?
                 directionalAppend(entries, fragmentEntry, direction);
                 // don't count it in amount perhaps? or do?
-                if (fragmentEntry.linkedFragmentId) {
+                if (fragmentEntry.hasLinkedFragment) {
                     const nextFragment = await fragmentStore.get(this._roomId, fragmentEntry.linkedFragmentId);
                     this._fragmentIdComparer.add(nextFragment);
                     const nextFragmentEntry = new FragmentBoundaryEntry(nextFragment, direction.isForward, this._fragmentIdComparer);

From 0b637f656ac39dc5995f43c45b1b6d71c828f6c3 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 19:27:40 +0200
Subject: [PATCH 75/83] timeline store requests should always be scoped to 1
 fragmentId

as fragmentIds should not be sorted, they are a linked list
and that is what determines their order.
---
 src/matrix/storage/idb/stores/TimelineEventStore.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/matrix/storage/idb/stores/TimelineEventStore.js b/src/matrix/storage/idb/stores/TimelineEventStore.js
index a6ce5617..fc7a07ca 100644
--- a/src/matrix/storage/idb/stores/TimelineEventStore.js
+++ b/src/matrix/storage/idb/stores/TimelineEventStore.js
@@ -19,7 +19,7 @@ class Range {
         if (this._lower && !this._upper) {
             return IDBKeyRange.bound(
                 [roomId, this._lower.fragmentId, this._lower.eventIndex],
-                [roomId, EventKey.maxKey.fragmentId, EventKey.maxKey.eventIndex],
+                [roomId, this._lower.fragmentId, EventKey.maxKey.eventIndex],
                 this._lowerOpen,
                 false
             );
@@ -28,7 +28,7 @@ class Range {
         // also bound as we don't want to move into another roomId
         if (!this._lower && this._upper) {
             return IDBKeyRange.bound(
-                [roomId, EventKey.minKey.fragmentId, EventKey.minKey.eventIndex],
+                [roomId, this._upper.fragmentId, EventKey.minKey.eventIndex],
                 [roomId, this._upper.fragmentId, this._upper.eventIndex],
                 false,
                 this._upperOpen

From 7852f31f7ea1819f9c7f2fe7ec5287663f75d88e Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Sun, 2 Jun 2019 19:28:24 +0200
Subject: [PATCH 76/83] clear token on pagination when events start overlapping

---
 src/matrix/room/timeline/persistence/GapWriter.js | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/src/matrix/room/timeline/persistence/GapWriter.js b/src/matrix/room/timeline/persistence/GapWriter.js
index ba25f909..dfef468e 100644
--- a/src/matrix/room/timeline/persistence/GapWriter.js
+++ b/src/matrix/room/timeline/persistence/GapWriter.js
@@ -64,14 +64,20 @@ export default class GapWriter {
         if (neighbourFragmentEntry) {
             fragmentEntry.linkedFragmentId = neighbourFragmentEntry.fragmentId;
             neighbourFragmentEntry.linkedFragmentId = fragmentEntry.fragmentId;
+            // if neighbourFragmentEntry was found, it means the events were overlapping,
+            // so no pagination should happen anymore.
+            neighbourFragmentEntry.token = null;
+            fragmentEntry.token = null;
+
             txn.timelineFragments.update(neighbourFragmentEntry.fragment);
             directionalAppend(entries, neighbourFragmentEntry, direction);
 
             // update fragmentIdComparer here after linking up fragments
             this._fragmentIdComparer.add(fragmentEntry.fragment);
             this._fragmentIdComparer.add(neighbourFragmentEntry.fragment);
+        } else {
+            fragmentEntry.token = end;
         }
-        fragmentEntry.token = end;
         txn.timelineFragments.update(fragmentEntry.fragment);
     }
 

From c9aaa18151827684d9958358905205e8b33a97e3 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Mon, 3 Jun 2019 00:11:12 +0200
Subject: [PATCH 77/83] return only eventId from
 findFirstOrLastOccurringEventId

---
 src/matrix/storage/idb/stores/TimelineEventStore.js | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/src/matrix/storage/idb/stores/TimelineEventStore.js b/src/matrix/storage/idb/stores/TimelineEventStore.js
index fc7a07ca..ac4ada9a 100644
--- a/src/matrix/storage/idb/stores/TimelineEventStore.js
+++ b/src/matrix/storage/idb/stores/TimelineEventStore.js
@@ -162,7 +162,7 @@ export default class TimelineEventStore {
      *  @param  {string[]} eventIds
      *  @return {Function<Promise>}
      */
-    // performance comment from above refers to the fact that their *might*
+    // performance comment from above refers to the fact that there *might*
     // be a correlation between event_id sorting order and chronology.
     // In that case we could avoid running over all eventIds, as the reported order by findExistingKeys
     // would match the order of eventIds. That's why findLast is also passed as backwards to keysExist.
@@ -171,7 +171,7 @@ export default class TimelineEventStore {
         const byEventId = this._timelineStore.index("byEventId");
         const keys = eventIds.map(eventId => [roomId, eventId]);
         const results = new Array(keys.length);
-        let firstFoundEventId;
+        let firstFoundKey;
 
         // find first result that is found and has no undefined results before it
         function firstFoundAndPrecedingResolved() {
@@ -189,11 +189,11 @@ export default class TimelineEventStore {
         await byEventId.findExistingKeys(keys, findLast, (key, found) => {
             const index = keys.indexOf(key);
             results[index] = found;
-            firstFoundEventId = firstFoundAndPrecedingResolved();
-            return !!firstFoundEventId;
+            firstFoundKey = firstFoundAndPrecedingResolved();
+            return !!firstFoundKey;
         });
-
-        return firstFoundEventId;
+        // key of index is [roomId, eventId], so pick out eventId
+        return firstFoundKey && firstFoundKey[1];
     }
 
     /** Inserts a new entry into the store. The combination of roomId and eventKey should not exist yet, or an error is thrown.

From 0407829b260974ee049ac313067836c577eee0df Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Mon, 3 Jun 2019 00:11:29 +0200
Subject: [PATCH 78/83] fix filling gaps with overlapping events

although event order remains wrong, as events are reversed.
step before removing premature optimization, so it's in the git
commit log
---
 .../room/timeline/persistence/GapWriter.js       | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/src/matrix/room/timeline/persistence/GapWriter.js b/src/matrix/room/timeline/persistence/GapWriter.js
index dfef468e..9d22407a 100644
--- a/src/matrix/room/timeline/persistence/GapWriter.js
+++ b/src/matrix/room/timeline/persistence/GapWriter.js
@@ -8,15 +8,22 @@ export default class GapWriter {
         this._storage = storage;
         this._fragmentIdComparer = fragmentIdComparer;
     }
-
+    // events is in reverse-chronological order (last event comes at index 0) if backwards
     async _findOverlappingEvents(fragmentEntry, events, txn) {
-        const eventIds = events.map(e => e.event_id);
         const {direction} = fragmentEntry;
+        // make it in chronological order, so findFirstOrLastOccurringEventId can
+        // use it's supposed optimization if event ids sorting order correlates with timeline chronology.
+        // premature optimization?
+        if (direction.isBackward) {
+            events = events.slice().reverse();
+        }
+        const eventIds = events.map(e => e.event_id);
         const findLast = direction.isBackward;
         let nonOverlappingEvents = events;
         let neighbourFragmentEntry;
         const neighbourEventId = await txn.timelineEvents.findFirstOrLastOccurringEventId(this._roomId, eventIds, findLast);
         if (neighbourEventId) {
+            console.log("_findOverlappingEvents neighbourEventId", neighbourEventId);
             // trim overlapping events
             const neighbourEventIndex = events.findIndex(e => e.event_id === neighbourEventId);
             const start = direction.isBackward ? neighbourEventIndex + 1 : 0;
@@ -24,10 +31,13 @@ export default class GapWriter {
             nonOverlappingEvents = events.slice(start, end);
             // get neighbour fragment to link it up later on
             const neighbourEvent = await txn.timelineEvents.getByEventId(this._roomId, neighbourEventId);
-            const neighbourFragment = await txn.timelineFragments.get(neighbourEvent.fragmentId);
+            console.log("neighbourEvent", {neighbourEvent, start, end, nonOverlappingEvents, events, neighbourEventIndex});
+            const neighbourFragment = await txn.timelineFragments.get(this._roomId, neighbourEvent.fragmentId);
             neighbourFragmentEntry = fragmentEntry.createNeighbourEntry(neighbourFragment);
         }
 
+        console.log("_findOverlappingEvents events", events, nonOverlappingEvents);
+
         return {nonOverlappingEvents, neighbourFragmentEntry};
     }
 

From 3dbf5e727d2875feebafdf542acd71d7bad5f83f Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Mon, 3 Jun 2019 00:18:52 +0200
Subject: [PATCH 79/83] process in incoming order (reverse-chronological order
 if backward)

makes code simpler, don't need fix to undo reverse ordering of
nonOverlappingEvents. reverse looking is very likely premature
optimization as well.
---
 .../room/timeline/persistence/GapWriter.js       | 16 +++-------------
 .../storage/idb/stores/TimelineEventStore.js     | 10 ++++------
 2 files changed, 7 insertions(+), 19 deletions(-)

diff --git a/src/matrix/room/timeline/persistence/GapWriter.js b/src/matrix/room/timeline/persistence/GapWriter.js
index 9d22407a..bcd3f6ea 100644
--- a/src/matrix/room/timeline/persistence/GapWriter.js
+++ b/src/matrix/room/timeline/persistence/GapWriter.js
@@ -10,28 +10,18 @@ export default class GapWriter {
     }
     // events is in reverse-chronological order (last event comes at index 0) if backwards
     async _findOverlappingEvents(fragmentEntry, events, txn) {
-        const {direction} = fragmentEntry;
-        // make it in chronological order, so findFirstOrLastOccurringEventId can
-        // use it's supposed optimization if event ids sorting order correlates with timeline chronology.
-        // premature optimization?
-        if (direction.isBackward) {
-            events = events.slice().reverse();
-        }
         const eventIds = events.map(e => e.event_id);
-        const findLast = direction.isBackward;
         let nonOverlappingEvents = events;
         let neighbourFragmentEntry;
-        const neighbourEventId = await txn.timelineEvents.findFirstOrLastOccurringEventId(this._roomId, eventIds, findLast);
+        const neighbourEventId = await txn.timelineEvents.findFirstOccurringEventId(this._roomId, eventIds);
         if (neighbourEventId) {
             console.log("_findOverlappingEvents neighbourEventId", neighbourEventId);
             // trim overlapping events
             const neighbourEventIndex = events.findIndex(e => e.event_id === neighbourEventId);
-            const start = direction.isBackward ? neighbourEventIndex + 1 : 0;
-            const end = direction.isBackward ? events.length : neighbourEventIndex;
-            nonOverlappingEvents = events.slice(start, end);
+            nonOverlappingEvents = events.slice(0, neighbourEventIndex);
             // get neighbour fragment to link it up later on
             const neighbourEvent = await txn.timelineEvents.getByEventId(this._roomId, neighbourEventId);
-            console.log("neighbourEvent", {neighbourEvent, start, end, nonOverlappingEvents, events, neighbourEventIndex});
+            console.log("neighbourEvent", {neighbourEvent, nonOverlappingEvents, events, neighbourEventIndex});
             const neighbourFragment = await txn.timelineFragments.get(this._roomId, neighbourEvent.fragmentId);
             neighbourFragmentEntry = fragmentEntry.createNeighbourEntry(neighbourFragment);
         }
diff --git a/src/matrix/storage/idb/stores/TimelineEventStore.js b/src/matrix/storage/idb/stores/TimelineEventStore.js
index ac4ada9a..7c333245 100644
--- a/src/matrix/storage/idb/stores/TimelineEventStore.js
+++ b/src/matrix/storage/idb/stores/TimelineEventStore.js
@@ -153,7 +153,7 @@ export default class TimelineEventStore {
         return events;
     }
 
-    /** Finds the first (or last if `findLast=true`) eventId that occurs in the store, if any.
+    /** Finds the first eventId that occurs in the store, if any.
      *  For optimal performance, `eventIds` should be in chronological order.
      *
      *  The order in which results are returned might be different than `eventIds`.
@@ -167,7 +167,7 @@ export default class TimelineEventStore {
     // In that case we could avoid running over all eventIds, as the reported order by findExistingKeys
     // would match the order of eventIds. That's why findLast is also passed as backwards to keysExist.
     // also passing them in chronological order makes sense as that's how we'll receive them almost always.
-    async findFirstOrLastOccurringEventId(roomId, eventIds, findLast = false) {
+    async findFirstOccurringEventId(roomId, eventIds) {
         const byEventId = this._timelineStore.index("byEventId");
         const keys = eventIds.map(eventId => [roomId, eventId]);
         const results = new Array(keys.length);
@@ -175,9 +175,7 @@ export default class TimelineEventStore {
 
         // find first result that is found and has no undefined results before it
         function firstFoundAndPrecedingResolved() {
-            let inc = findLast ? -1 : 1;
-            let start = findLast ? results.length - 1 : 0;
-            for(let i = start; i >= 0 && i < results.length; i += inc) {
+            for(let i = 0; i < results.length; ++i) {
                 if (results[i] === undefined) {
                     return;
                 } else if(results[i] === true) {
@@ -186,7 +184,7 @@ export default class TimelineEventStore {
             }
         }
 
-        await byEventId.findExistingKeys(keys, findLast, (key, found) => {
+        await byEventId.findExistingKeys(keys, false, (key, found) => {
             const index = keys.indexOf(key);
             results[index] = found;
             firstFoundKey = firstFoundAndPrecedingResolved();

From 45528580ede6e516b6de60d73bd49f9f6627640e Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Mon, 3 Jun 2019 00:30:16 +0200
Subject: [PATCH 80/83] fix comparator/comparer mismatch

---
 src/matrix/room/timeline/entries/EventEntry.js   |  4 ++--
 .../timeline/entries/FragmentBoundaryEntry.js    | 16 ++++++++--------
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/src/matrix/room/timeline/entries/EventEntry.js b/src/matrix/room/timeline/entries/EventEntry.js
index 98e961cf..ce3697fa 100644
--- a/src/matrix/room/timeline/entries/EventEntry.js
+++ b/src/matrix/room/timeline/entries/EventEntry.js
@@ -1,8 +1,8 @@
 import BaseEntry from "./BaseEntry.js";
 
 export default class EventEntry extends BaseEntry {
-    constructor(eventEntry, fragmentIdComparator) {
-        super(fragmentIdComparator);
+    constructor(eventEntry, fragmentIdComparer) {
+        super(fragmentIdComparer);
         this._eventEntry = eventEntry;
     }
 
diff --git a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
index 3182fd2f..5eb772f3 100644
--- a/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
+++ b/src/matrix/room/timeline/entries/FragmentBoundaryEntry.js
@@ -3,19 +3,19 @@ import Direction from "../Direction.js";
 import {isValidFragmentId} from "../common.js";
 
 export default class FragmentBoundaryEntry extends BaseEntry {
-    constructor(fragment, isFragmentStart, fragmentIdComparator) {
-        super(fragmentIdComparator);
+    constructor(fragment, isFragmentStart, fragmentIdComparer) {
+        super(fragmentIdComparer);
         this._fragment = fragment;
         // TODO: should isFragmentStart be Direction instead of bool?
         this._isFragmentStart = isFragmentStart;
     }
 
-    static start(fragment, fragmentIdComparator) {
-        return new FragmentBoundaryEntry(fragment, true, fragmentIdComparator);
+    static start(fragment, fragmentIdComparer) {
+        return new FragmentBoundaryEntry(fragment, true, fragmentIdComparer);
     }
 
-    static end(fragment, fragmentIdComparator) {
-        return new FragmentBoundaryEntry(fragment, false, fragmentIdComparator);
+    static end(fragment, fragmentIdComparer) {
+        return new FragmentBoundaryEntry(fragment, false, fragmentIdComparer);
     }
     
     get started() {
@@ -91,10 +91,10 @@ export default class FragmentBoundaryEntry extends BaseEntry {
     }
 
     withUpdatedFragment(fragment) {
-        return new FragmentBoundaryEntry(fragment, this._isFragmentStart, this._fragmentIdComparator);
+        return new FragmentBoundaryEntry(fragment, this._isFragmentStart, this._fragmentIdComparer);
     }
 
     createNeighbourEntry(neighbour) {
-        return new FragmentBoundaryEntry(neighbour, !this._isFragmentStart, this._fragmentIdComparator);
+        return new FragmentBoundaryEntry(neighbour, !this._isFragmentStart, this._fragmentIdComparer);
     }
 }

From 6bdf44d114e672893f2639cc9439ac4e830f587d Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Mon, 3 Jun 2019 00:30:37 +0200
Subject: [PATCH 81/83] adjust comment

---
 src/matrix/room/timeline/persistence/GapWriter.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/matrix/room/timeline/persistence/GapWriter.js b/src/matrix/room/timeline/persistence/GapWriter.js
index bcd3f6ea..8edeb313 100644
--- a/src/matrix/room/timeline/persistence/GapWriter.js
+++ b/src/matrix/room/timeline/persistence/GapWriter.js
@@ -83,7 +83,7 @@ export default class GapWriter {
 
     async writeFragmentFill(fragmentEntry, response) {
         const {fragmentId, direction} = fragmentEntry;
-        // assuming that chunk is in chronological order when backwards too?
+        // chunk is in reverse-chronological order when backwards
         const {chunk, start, end} = response;
         let entries;
 

From 0524f06722a5c7405cf6e9cd7103ea87c432adb5 Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Mon, 3 Jun 2019 00:31:21 +0200
Subject: [PATCH 82/83] remove logging

---
 src/matrix/room/timeline/persistence/GapWriter.js | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/src/matrix/room/timeline/persistence/GapWriter.js b/src/matrix/room/timeline/persistence/GapWriter.js
index 8edeb313..9941c712 100644
--- a/src/matrix/room/timeline/persistence/GapWriter.js
+++ b/src/matrix/room/timeline/persistence/GapWriter.js
@@ -15,19 +15,14 @@ export default class GapWriter {
         let neighbourFragmentEntry;
         const neighbourEventId = await txn.timelineEvents.findFirstOccurringEventId(this._roomId, eventIds);
         if (neighbourEventId) {
-            console.log("_findOverlappingEvents neighbourEventId", neighbourEventId);
             // trim overlapping events
             const neighbourEventIndex = events.findIndex(e => e.event_id === neighbourEventId);
             nonOverlappingEvents = events.slice(0, neighbourEventIndex);
             // get neighbour fragment to link it up later on
             const neighbourEvent = await txn.timelineEvents.getByEventId(this._roomId, neighbourEventId);
-            console.log("neighbourEvent", {neighbourEvent, nonOverlappingEvents, events, neighbourEventIndex});
             const neighbourFragment = await txn.timelineFragments.get(this._roomId, neighbourEvent.fragmentId);
             neighbourFragmentEntry = fragmentEntry.createNeighbourEntry(neighbourFragment);
         }
-
-        console.log("_findOverlappingEvents events", events, nonOverlappingEvents);
-
         return {nonOverlappingEvents, neighbourFragmentEntry};
     }
 

From c838edb6c47b9a1d6f24567a0570d11d5043b3cd Mon Sep 17 00:00:00 2001
From: Bruno Windels <brunow@matrix.org>
Date: Mon, 3 Jun 2019 00:33:19 +0200
Subject: [PATCH 83/83] update todo lists

---
 doc/FRAGMENTS.md | 3 +++
 doc/TODO.md      | 5 +++++
 2 files changed, 8 insertions(+)

diff --git a/doc/FRAGMENTS.md b/doc/FRAGMENTS.md
index efe0d64a..621e155a 100644
--- a/doc/FRAGMENTS.md
+++ b/doc/FRAGMENTS.md
@@ -34,11 +34,14 @@
  - lets try to not have to have the fragmentindex in memory if the timeline isn't loaded
     - could do this by only loading all fragments into index when filling gaps, backpaginating, ... and on persister load only load the last fragment. This wouldn't even need a FragmentIndex?
 
+# Leftover items
 
 implement SortedArray::setManySorted in a performant manner
 implement FragmentIdComparator::add in a performant manner
 there is some duplication (also in memory) between SortedArray and TilesCollection. Both keep a sorted list based on fragmentId/eventIndex... TilesCollection doesn't use the index in the event handlers at all. we could allow timeline to export a structure that just emits "these entries are a thing (now)" and not have to go through sorting twice. Timeline would have to keep track of the earliest key so it can use it in loadAtTop, but that should be easy. Hmmm. also, Timeline might want to be in charge of unloading parts of the loaded timeline, and for that it would need to know the order of entries. So maybe not ... we'll see.
 
+check: do /sync events not have a room_id and /messages do???
+
 so a gap is two connected fragments where either the first fragment has a nextToken and/or the second fragment has a previousToken. It can be both, so we can have a gap where you can fill in from the top, from the bottom (like when limited sync) or both.
 
 
diff --git a/doc/TODO.md b/doc/TODO.md
index 48be2e5a..f3e41d42 100644
--- a/doc/TODO.md
+++ b/doc/TODO.md
@@ -25,6 +25,11 @@
  - DONE: support timeline
  	- DONE: clicking on a room list, you see messages (userId -> body)
  - DONE: style minimal UI
+ - DONE: implement gap filling and fragments (see FRAGMENTS.md)
+ - allow collection items (especially tiles) to self-update
+ - improve fragmentidcomparer::add
+ - send messages
+ - better UI
  - fix MappedMap update mechanism
  - see if in BaseObservableMap we need to change ...params
  - put sync button and status label inside SessionView