mirrors/vector-im-hydrogen-web
1
0
Fork 0
mirror of https://github.com/vector-im/hydrogen-web.git synced 2024-12-22 19:14:52 +01:00
Code Issues Projects Releases Wiki Activity

Merge pull request #988 from vector-im/bwindels/calls-errorhandling

Browse Source 
Error handling for calls (and room)
This commit is contained in:
Bruno Windels 2023-01-19 10:39:33 +00:00 committed by GitHub
commit 4270e300d6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
23 changed files with 935 additions and 507 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
doc/error-handling.md Normal file
View File

@ -0,0 +1,15 @@
# Error handling
Ideally, every error that is unexpected and can't be automatically recovered from without degrading the experience is shown in the UI. This is the task of the view model, and you can use `ErrorReportViewModel` for this purpose, a dedicated base view model class. It exposes a child view model, `ErrorViewModel`, when `reportError` is called which can be paired with `ErrorView` in the view to present an error message from which debug logs can also be sent.
Methods on classes from the `matrix` layer can often throw errors and those errors should be caught in the view model and reported to the UI. When inheriting from `ErrorReportViewModel`, there is the low-level `reportError` method, but typically you'd use the convenience method `logAndCatch`. The latter makes it easy to get both error handlikng and logging right. You would typically use `logAndCatch` for every public method in the view model (e.g methods called from the view or from the parent view model). It calls a callback within a log item and also a try catch that reports the error.
## Sync errors & ErrorBoundary
There are some errors that are thrown during background processes though, most notably the sync loop. These processes are not triggered by the view model directly, and hence there is not always a method call they can wrap in a try/catch. For this, there is the `ErrorBoundary` utility class. Since almost all aspects of the client can be updated through the sync loop, it is also not too helpful if there is only one try/catch around the whole sync and we stop sync if something goes wrong.
Instead, it's more helpful to split up the error handling into different scopes, where errors are stored and not rethrown when leaving the scope. One example is to have a scope per room. In this way, we can isolate an error occuring during sync to a specific room, and report it in the UI of that room. This is typically where you would use `reportError` from `ErrorReportViewModel` rather than `logAndCatch`. You observe changes from your model in the view model (see docs on updates), and if the `error` property is set (by the `ErrorBoundary`), you call reportError with it. You can do this repeatedly without problems, if the same error is already reported, it's a No-Op.
### `writeSync` and preventing data loss when dealing with errors.
There is an extra complication though. The `writeSync` sync lifecycle step should not swallow any errors, or data loss can occur. This is because the whole `writeSync` lifecycle step writes all changes (for all rooms, the session, ...) for a sync response in one transaction (including the sync token), and aborts the transaction and stops sync if there is an error thrown during this step. So if there is an error in `writeSync` of a given room, it's fair to assume not all changes it was planning to write were passed to the transaction, as it got interrupted by the exception. Therefore, if we would swallow the error, data loss can occur as we'd not get another chance to write these changes to disk as we would have advanced the sync token. Therefore, code in the `writeSync` lifecycle step should be written defensively but always throw.

2
doc/impl-thoughts/RELATIONS.md
View File

@ -237,7 +237,7 @@ room.sendEvent(eventEntry.eventType, replacement);
## Replies
```js
const reply = eventEntry.reply({});
const reply = eventEntry.createReplyContent({});
room.sendEvent("m.room.message", reply);
```

72
src/domain/ErrorReportViewModel.ts Normal file
View File

@ -0,0 +1,72 @@
/*
Copyright 2023 The Matrix.org Foundation C.I.C.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
import { ViewModel } from "./ViewModel";
import type { Options as BaseOptions } from "./ViewModel";
import type { Session } from "../matrix/Session";
import { ErrorViewModel } from "./ErrorViewModel";
import type { LogCallback, LabelOrValues } from "../logging/types";
export type Options = BaseOptions & {
session: Session
};
/** Base class for view models that need to report errors to the UI. */
export class ErrorReportViewModel<O extends Options = Options> extends ViewModel<O> {
private _errorViewModel?: ErrorViewModel;
get errorViewModel(): ErrorViewModel | undefined {
return this._errorViewModel;
}
/** Typically you'd want to use `logAndCatch` when implementing a view model method.
* Use `reportError` when showing errors on your model that were set by
* background processes using `ErrorBoundary` or you have some other
* special low-level need to write your try/catch yourself. */
protected reportError(error: Error) {
if (this._errorViewModel?.error === error) {
return;
}
this.disposeTracked(this._errorViewModel);
this._errorViewModel = this.track(new ErrorViewModel(this.childOptions({
error,
onClose: () => {
this._errorViewModel = this.disposeTracked(this._errorViewModel);
this.emitChange("errorViewModel");
}
})));
this.emitChange("errorViewModel");
}
/** Combines logging and error reporting in one method.
* Wrap the implementation of public view model methods
* with this to ensure errors are logged and reported.*/
protected logAndCatch<T>(labelOrValues: LabelOrValues, callback: LogCallback<T>, errorValue: T = undefined as unknown as T): T {
try {
let result = this.logger.run(labelOrValues, callback);
if (result instanceof Promise) {
result = result.catch(err => {
this.reportError(err);
return errorValue;
}) as unknown as T;
}
return result;
} catch (err) {
this.reportError(err);
return errorValue;
}
}
}

48
src/domain/ErrorViewModel.ts Normal file
View File

@ -0,0 +1,48 @@
/*
Copyright 2023 The Matrix.org Foundation C.I.C.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
import { ViewModel, Options as BaseOptions } from "./ViewModel";
import {submitLogsFromSessionToDefaultServer} from "./rageshake";
import type { Session } from "../matrix/Session";
type Options = {
error: Error
session: Session,
onClose: () => void
} & BaseOptions;
export class ErrorViewModel extends ViewModel<Options> {
get message(): string {
return this.getOption("error")?.message;
}
get error(): Error {
return this.getOption("error");
}
close() {
this.getOption("onClose")();
}
async submitLogs(): Promise<boolean> {
try {
await submitLogsFromSessionToDefaultServer(this.getOption("session"), this.platform);
return true;
} catch (err) {
return false;
}
}
}

33
src/domain/rageshake.ts
View File

@ -16,11 +16,15 @@ limitations under the License.
import type {BlobHandle} from "../platform/web/dom/BlobHandle";
import type {RequestFunction} from "../platform/types/types";
import type {Platform} from "../platform/web/Platform";
import type {ILogger} from "../logging/types";
import type { IDBLogPersister } from "../logging/IDBLogPersister";
import type { Session } from "../matrix/Session";
// see https://github.com/matrix-org/rageshake#readme
type RageshakeData = {
// A textual description of the problem. Included in the details.log.gz file.
text: string | undefined;
text?: string;
// Application user-agent. Included in the details.log.gz file.
userAgent: string;
// Identifier for the application (eg 'riot-web'). Should correspond to a mapping configured in the configuration file for github issue reporting to work.
@ -28,7 +32,7 @@ type RageshakeData = {
// Application version. Included in the details.log.gz file.
version: string;
// Label to attach to the github issue, and include in the details file.
label: string | undefined;
label?: string;
};
export async function submitLogsToRageshakeServer(data: RageshakeData, logsBlob: BlobHandle, submitUrl: string, request: RequestFunction): Promise<void> {
@ -63,3 +67,28 @@ export async function submitLogsToRageshakeServer(data: RageshakeData, logsBlob:
// we don't bother with reading report_url from the body as the rageshake server doesn't always return it
// and would have to have CORS setup properly for us to be able to read it.
}
/** @throws {Error} */
export async function submitLogsFromSessionToDefaultServer(session: Session, platform: Platform): Promise<void> {
const {bugReportEndpointUrl} = platform.config;
if (!bugReportEndpointUrl) {
throw new Error("no server configured to submit logs");
}
const logReporters = (platform.logger as ILogger).reporters;
const exportReporter = logReporters.find(r => !!r["export"]) as IDBLogPersister | undefined;
if (!exportReporter) {
throw new Error("No logger that can export configured");
}
const logExport = await exportReporter.export();
await submitLogsToRageshakeServer(
{
app: "hydrogen",
userAgent: platform.description,
version: platform.version,
text: `Submit logs from settings for user ${session.userId} on device ${session.deviceId}`,
},
logExport.asBlob(),
bugReportEndpointUrl,
platform.request
);
}

100
src/domain/session/room/CallViewModel.ts
View File

@ -15,11 +15,13 @@ limitations under the License.
*/
import {AvatarSource} from "../../AvatarSource";
import {ViewModel, Options as BaseOptions} from "../../ViewModel";
import type {ViewModel} from "../../ViewModel";
import {ErrorReportViewModel, Options as BaseOptions} from "../../ErrorReportViewModel";
import {getStreamVideoTrack, getStreamAudioTrack} from "../../../matrix/calls/common";
import {avatarInitials, getIdentifierColorNumber, getAvatarHttpUrl} from "../../avatar";
import {EventObservableValue} from "../../../observable/value/EventObservableValue";
import {ObservableValueMap} from "../../../observable/map/ObservableValueMap";
import {ErrorViewModel} from "../../ErrorViewModel";
import type {Room} from "../../../matrix/room/Room";
import type {GroupCall} from "../../../matrix/calls/group/GroupCall";
import type {Member} from "../../../matrix/calls/group/Member";
@ -28,22 +30,32 @@ import type {BaseObservableList} from "../../../observable/list/BaseObservableLi
import type {BaseObservableValue} from "../../../observable/value/BaseObservableValue";
import type {Stream} from "../../../platform/types/MediaDevices";
import type {MediaRepository} from "../../../matrix/net/MediaRepository";
import type {Session} from "../../../matrix/Session";
type Options = BaseOptions & {
call: GroupCall,
room: Room,
};
export class CallViewModel extends ViewModel<Options> {
export class CallViewModel extends ErrorReportViewModel<Options> {
public readonly memberViewModels: BaseObservableList<IStreamViewModel>;
constructor(options: Options) {
super(options);
const ownMemberViewModelMap = new ObservableValueMap("self", new EventObservableValue(this.call, "change"))
const callObservable = new EventObservableValue(this.call, "change");
this.track(callObservable.subscribe(() => this.onUpdate()));
const ownMemberViewModelMap = new ObservableValueMap("self", callObservable)
.mapValues((call, emitChange) => new OwnMemberViewModel(this.childOptions({call, emitChange})), () => {});
this.memberViewModels = this.call.members
.filterValues(member => member.isConnected)
.mapValues(member => new CallMemberViewModel(this.childOptions({member, mediaRepository: this.getOption("room").mediaRepository})))
.mapValues(
(member, emitChange) => new CallMemberViewModel(this.childOptions({
member,
emitChange,
mediaRepository: this.getOption("room").mediaRepository
})),
(vm: CallMemberViewModel) => vm.onUpdate(),
)
.join(ownMemberViewModelMap)
.sortValues((a, b) => a.compare(b));
this.track(this.memberViewModels.subscribe({
@ -83,43 +95,54 @@ export class CallViewModel extends ViewModel<Options> {
return this.getOption("call");
}
async hangup() {
if (this.call.hasJoined) {
await this.call.leave();
private onUpdate() {
if (this.call.error) {
this.reportError(this.call.error);
}
}
async hangup() {
this.logAndCatch("CallViewModel.hangup", async log => {
if (this.call.hasJoined) {
await this.call.leave(log);
}
});
}
async toggleCamera() {
const {localMedia, muteSettings} = this.call;
if (muteSettings && localMedia) {
// unmute but no track?
if (muteSettings.camera && !getStreamVideoTrack(localMedia.userMedia)) {
const stream = await this.platform.mediaDevices.getMediaTracks(!muteSettings.microphone, true);
await this.call.setMedia(localMedia.withUserMedia(stream));
} else {
await this.call.setMuted(muteSettings.toggleCamera());
this.logAndCatch("Call.toggleCamera", async log => {
const {localMedia, muteSettings} = this.call;
if (muteSettings && localMedia) {
// unmute but no track?
if (muteSettings.camera && !getStreamVideoTrack(localMedia.userMedia)) {
const stream = await this.platform.mediaDevices.getMediaTracks(!muteSettings.microphone, true);
await this.call.setMedia(localMedia.withUserMedia(stream));
} else {
await this.call.setMuted(muteSettings.toggleCamera());
}
this.emitChange();
}
this.emitChange();
}
});
}
async toggleMicrophone() {
const {localMedia, muteSettings} = this.call;
if (muteSettings && localMedia) {
// unmute but no track?
if (muteSettings.microphone && !getStreamAudioTrack(localMedia.userMedia)) {
const stream = await this.platform.mediaDevices.getMediaTracks(true, !muteSettings.camera);
console.log("got tracks", Array.from(stream.getTracks()).map((t: MediaStreamTrack) => { return {kind: t.kind, id: t.id};}))
await this.call.setMedia(localMedia.withUserMedia(stream));
} else {
await this.call.setMuted(muteSettings.toggleMicrophone());
this.logAndCatch("Call.toggleMicrophone", async log => {
const {localMedia, muteSettings} = this.call;
if (muteSettings && localMedia) {
// unmute but no track?
if (muteSettings.microphone && !getStreamAudioTrack(localMedia.userMedia)) {
const stream = await this.platform.mediaDevices.getMediaTracks(true, !muteSettings.camera);
await this.call.setMedia(localMedia.withUserMedia(stream));
} else {
await this.call.setMuted(muteSettings.toggleMicrophone());
}
this.emitChange();
}
this.emitChange();
}
});
}
}
class OwnMemberViewModel extends ViewModel<Options> implements IStreamViewModel {
class OwnMemberViewModel extends ErrorReportViewModel<Options> implements IStreamViewModel {
private memberObservable: undefined | BaseObservableValue<RoomMember>;
constructor(options: Options) {
@ -135,6 +158,10 @@ class OwnMemberViewModel extends ViewModel<Options> implements IStreamViewModel
}));
}
get errorViewModel(): ErrorViewModel | undefined {
return undefined;
}
get stream(): Stream | undefined {
return this.call.localPreviewMedia?.userMedia;
}
@ -187,10 +214,10 @@ class OwnMemberViewModel extends ViewModel<Options> implements IStreamViewModel
type MemberOptions = BaseOptions & {
member: Member,
mediaRepository: MediaRepository
mediaRepository: MediaRepository,
};
export class CallMemberViewModel extends ViewModel<MemberOptions> implements IStreamViewModel {
export class CallMemberViewModel extends ErrorReportViewModel<MemberOptions> implements IStreamViewModel {
get stream(): Stream | undefined {
return this.member.remoteMedia?.userMedia;
}
@ -225,6 +252,16 @@ export class CallMemberViewModel extends ViewModel<MemberOptions> implements ISt
return this.member.member.name;
}
onUpdate() {
this.mapMemberSyncErrorIfNeeded();
}
private mapMemberSyncErrorIfNeeded() {
if (this.member.error) {
this.reportError(this.member.error);
}
}
compare(other: OwnMemberViewModel | CallMemberViewModel): number {
if (other instanceof OwnMemberViewModel) {
return -other.compare(this);
@ -242,4 +279,5 @@ export interface IStreamViewModel extends AvatarSource, ViewModel {
get stream(): Stream | undefined;
get isCameraMuted(): boolean;
get isMicrophoneMuted(): boolean;
get errorViewModel(): ErrorViewModel | undefined;
}

152
src/domain/session/room/RoomViewModel.js
View File

@ -20,6 +20,7 @@ import {ComposerViewModel} from "./ComposerViewModel.js"
import {CallViewModel} from "./CallViewModel"
import {PickMapObservableValue} from "../../../observable/value/PickMapObservableValue";
import {avatarInitials, getIdentifierColorNumber, getAvatarHttpUrl} from "../../avatar";
import {ErrorReportViewModel} from "../../ErrorReportViewModel";
import {ViewModel} from "../../ViewModel";
import {imageToInfo} from "../common.js";
import {LocalMedia} from "../../../matrix/calls/LocalMedia";
@ -27,7 +28,7 @@ import {LocalMedia} from "../../../matrix/calls/LocalMedia";
// this is a breaking SDK change though to make this option mandatory
import {tileClassForEntry as defaultTileClassForEntry} from "./timeline/tiles/index";
export class RoomViewModel extends ViewModel {
export class RoomViewModel extends ErrorReportViewModel {
constructor(options) {
super(options);
const {room, tileClassForEntry} = options;
@ -36,8 +37,6 @@ export class RoomViewModel extends ViewModel {
this._tileClassForEntry = tileClassForEntry ?? defaultTileClassForEntry;
this._tileOptions = undefined;
this._onRoomChange = this._onRoomChange.bind(this);
this._timelineError = null;
this._sendError = null;
this._composerVM = null;
if (room.isArchived) {
this._composerVM = new ArchivedViewModel(this.childOptions({archivedRoom: room}));
@ -74,9 +73,9 @@ export class RoomViewModel extends ViewModel {
}
async load() {
this._room.on("change", this._onRoomChange);
try {
const timeline = await this._room.openTimeline();
this.logAndCatch("RoomViewModel.load", async log => {
this._room.on("change", this._onRoomChange);
const timeline = await this._room.openTimeline(log);
this._tileOptions = this.childOptions({
session: this.getOption("session"),
roomVM: this,
@ -88,32 +87,32 @@ export class RoomViewModel extends ViewModel {
timeline,
})));
this.emitChange("timelineViewModel");
} catch (err) {
console.error(`room.openTimeline(): ${err.message}:\n${err.stack}`);
this._timelineError = err;
this.emitChange("error");
}
this._clearUnreadAfterDelay();
await this._clearUnreadAfterDelay(log);
});
}
async _clearUnreadAfterDelay() {
async _clearUnreadAfterDelay(log) {
if (this._room.isArchived || this._clearUnreadTimout) {
return;
}
this._clearUnreadTimout = this.clock.createTimeout(2000);
try {
await this._clearUnreadTimout.elapsed();
await this._room.clearUnread();
await this._room.clearUnread(log);
this._clearUnreadTimout = null;
} catch (err) {
if (err.name !== "AbortError") {
if (err.name === "AbortError") {
log.set("clearUnreadCancelled", true);
} else {
throw err;
}
}
}
focus() {
this._clearUnreadAfterDelay();
this.logAndCatch("RoomViewModel.focus", async log => {
this._clearUnreadAfterDelay(log);
});
}
dispose() {
@ -143,16 +142,6 @@ export class RoomViewModel extends ViewModel {
get timelineViewModel() { return this._timelineVM; }
get isEncrypted() { return this._room.isEncrypted; }
get error() {
if (this._timelineError) {
return `Something went wrong loading the timeline: ${this._timelineError.message}`;
}
if (this._sendError) {
return `Something went wrong sending your message: ${this._sendError.message}`;
}
return "";
}
get avatarLetter() {
return avatarInitials(this.name);
}
@ -202,65 +191,62 @@ export class RoomViewModel extends ViewModel {
}
}
async _sendMessage(message, replyingTo) {
if (!this._room.isArchived && message) {
try {
_sendMessage(message, replyingTo) {
return this.logAndCatch("RoomViewModel.sendMessage", async log => {
let success = false;
if (!this._room.isArchived && message) {
let msgtype = "m.text";
if (message.startsWith("/me ")) {
message = message.substr(4).trim();
msgtype = "m.emote";
}
let content;
if (replyingTo) {
await replyingTo.reply(msgtype, message);
log.set("replyingTo", replyingTo.eventId);
content = await replyingTo.createReplyContent(msgtype, message);
} else {
await this._room.sendEvent("m.room.message", {msgtype, body: message});
content = {msgtype, body: message};
}
} catch (err) {
console.error(`room.sendMessage(): ${err.message}:\n${err.stack}`);
this._sendError = err;
this._timelineError = null;
this.emitChange("error");
return false;
await this._room.sendEvent("m.room.message", content, undefined, log);
success = true;
}
return true;
}
return false;
log.set("success", success);
return success;
}, false);
}
async _pickAndSendFile() {
try {
_pickAndSendFile() {
return this.logAndCatch("RoomViewModel.sendFile", async log => {
const file = await this.platform.openFile();
if (!file) {
log.set("cancelled", true);
return;
}
return this._sendFile(file);
} catch (err) {
console.error(err);
}
return this._sendFile(file, log);
});
}
async _sendFile(file) {
async _sendFile(file, log) {
const content = {
body: file.name,
msgtype: "m.file"
};
await this._room.sendEvent("m.room.message", content, {
"url": this._room.createAttachment(file.blob, file.name)
});
}, log);
}
async _pickAndSendVideo() {
try {
_pickAndSendVideo() {
return this.logAndCatch("RoomViewModel.sendVideo", async log => {
if (!this.platform.hasReadPixelPermission()) {
alert("Please allow canvas image data access, so we can scale your images down.");
return;
throw new Error("Please allow canvas image data access, so we can scale your images down.");
}
const file = await this.platform.openFile("video/*");
if (!file) {
return;
}
if (!file.blob.mimeType.startsWith("video/")) {
return this._sendFile(file);
return this._sendFile(file, log);
}
let video;
try {
@ -288,26 +274,23 @@ export class RoomViewModel extends ViewModel {
content.info.thumbnail_info = imageToInfo(thumbnail);
attachments["info.thumbnail_url"] =
this._room.createAttachment(thumbnail.blob, file.name);
await this._room.sendEvent("m.room.message", content, attachments);
} catch (err) {
this._sendError = err;
this.emitChange("error");
console.error(err.stack);
}
await this._room.sendEvent("m.room.message", content, attachments, log);
});
}
async _pickAndSendPicture() {
try {
this.logAndCatch("RoomViewModel.sendPicture", async log => {
if (!this.platform.hasReadPixelPermission()) {
alert("Please allow canvas image data access, so we can scale your images down.");
return;
}
const file = await this.platform.openFile("image/*");
if (!file) {
log.set("cancelled", true);
return;
}
if (!file.blob.mimeType.startsWith("image/")) {
return this._sendFile(file);
return this._sendFile(file, log);
}
let image = await this.platform.loadImage(file.blob);
const limit = await this.platform.settingsStorage.getInt("sentImageSizeLimit");
@ -330,12 +313,8 @@ export class RoomViewModel extends ViewModel {
attachments["info.thumbnail_url"] =
this._room.createAttachment(thumbnail.blob, file.name);
}
await this._room.sendEvent("m.room.message", content, attachments);
} catch (err) {
this._sendError = err;
this.emitChange("error");
console.error(err.stack);
}
await this._room.sendEvent("m.room.message", content, attachments, log);
});
}
get room() {
@ -363,17 +342,36 @@ export class RoomViewModel extends ViewModel {
}
}
async startCall() {
try {
startCall() {
return this.logAndCatch("RoomViewModel.startCall", async log => {
log.set("roomId", this._room.id);
let localMedia;
try {
const stream = await this.platform.mediaDevices.getMediaTracks(false, true);
localMedia = new LocalMedia().withUserMedia(stream);
} catch (err) {
throw new Error(`Could not get local audio and/or video stream: ${err.message}`);
}
const session = this.getOption("session");
const stream = await this.platform.mediaDevices.getMediaTracks(false, true);
const localMedia = new LocalMedia().withUserMedia(stream);
// this will set the callViewModel above as a call will be added to callHandler.calls
const call = await session.callHandler.createCall(this._room.id, "m.video", "A call " + Math.round(this.platform.random() * 100));
await call.join(localMedia);
} catch (err) {
console.error(err.stack);
}
let call;
try {
// this will set the callViewModel above as a call will be added to callHandler.calls
call = await session.callHandler.createCall(
this._room.id,
"m.video",
"A call " + Math.round(this.platform.random() * 100),
undefined,
log
);
} catch (err) {
throw new Error(`Could not create call: ${err.message}`);
}
try {
await call.join(localMedia, log);
} catch (err) {
throw new Error(`Could not join call: ${err.message}`);
}
});
}
}

4
src/domain/session/room/timeline/tiles/BaseMessageTile.js
View File

@ -146,8 +146,8 @@ export class BaseMessageTile extends SimpleTile {
this._roomVM.startReply(this._entry);
}
reply(msgtype, body, log = null) {
return this._room.sendEvent("m.room.message", this._entry.reply(msgtype, body), null, log);
createReplyContent(msgtype, body) {
return this._entry.createReplyContent(msgtype, body);
}
redact(reason, log) {

21
src/domain/session/room/timeline/tiles/CallTile.js
View File

@ -16,7 +16,6 @@ limitations under the License.
import {SimpleTile} from "./SimpleTile.js";
import {LocalMedia} from "../../../../../matrix/calls/LocalMedia";
// TODO: timeline entries for state events with the same state key and type
// should also update previous entries in the timeline, so we can update the name of the call, whether it is terminated, etc ...
@ -73,17 +72,21 @@ export class CallTile extends SimpleTile {
}
async join() {
if (this.canJoin) {
const stream = await this.platform.mediaDevices.getMediaTracks(false, true);
const localMedia = new LocalMedia().withUserMedia(stream);
await this._call.join(localMedia);
}
await this.logAndCatch("CallTile.join", async log => {
if (this.canJoin) {
const stream = await this.platform.mediaDevices.getMediaTracks(false, true);
const localMedia = new LocalMedia().withUserMedia(stream);
await this._call.join(localMedia, log);
}
});
}
async leave() {
if (this.canLeave) {
this._call.leave();
}
await this.logAndCatch("CallTile.leave", async log => {
if (this.canLeave) {
await this._call.leave(log);
}
});
}
dispose() {

4
src/domain/session/room/timeline/tiles/SimpleTile.js
View File

@ -15,10 +15,10 @@ limitations under the License.
*/
import {UpdateAction} from "../UpdateAction.js";
import {ViewModel} from "../../../../ViewModel";
import {ErrorReportViewModel} from "../../../../ErrorReportViewModel";
import {SendStatus} from "../../../../../matrix/room/sending/PendingEvent.js";
export class SimpleTile extends ViewModel {
export class SimpleTile extends ErrorReportViewModel {
constructor(entry, options) {
super(options);
this._entry = entry;

30
src/domain/session/settings/SettingsViewModel.js
View File

@ -16,7 +16,7 @@ limitations under the License.
import {ViewModel} from "../../ViewModel";
import {KeyBackupViewModel} from "./KeyBackupViewModel.js";
import {submitLogsToRageshakeServer} from "../../../domain/rageshake";
import {submitLogsFromSessionToDefaultServer} from "../../../domain/rageshake";
class PushNotificationStatus {
constructor() {
@ -175,29 +175,13 @@ export class SettingsViewModel extends ViewModel {
}
async sendLogsToServer() {
const {bugReportEndpointUrl} = this.platform.config;
if (bugReportEndpointUrl) {
this._logsFeedbackMessage = this.i18n`Sending logs…`;
this._logsFeedbackMessage = this.i18n`Sending logs…`;
try {
await submitLogsFromSessionToDefaultServer(this._session, this.platform);
this._logsFeedbackMessage = this.i18n`Logs sent succesfully!`;
} catch (err) {
this._logsFeedbackMessage = err.message;
this.emitChange();
try {
const logExport = await this.logger.export();
await submitLogsToRageshakeServer(
{
app: "hydrogen",
userAgent: this.platform.description,
version: DEFINE_VERSION,
text: `Submit logs from settings for user ${this._session.userId} on device ${this._session.deviceId}`,
},
logExport.asBlob(),
bugReportEndpointUrl,
this.platform.request
);
this._logsFeedbackMessage = this.i18n`Logs sent succesfully!`;
this.emitChange();
} catch (err) {
this._logsFeedbackMessage = err.message;
this.emitChange();
}
}
}

141
src/matrix/calls/CallHandler.ts
View File

@ -65,16 +65,26 @@ export class CallHandler implements RoomStateHandler {
});
}
async loadCalls(intent: CallIntent = CallIntent.Ring) {
const txn = await this._getLoadTxn();
const callEntries = await txn.calls.getByIntent(intent);
this._loadCallEntries(callEntries, txn);
loadCalls(intent?: CallIntent, log?: ILogItem) {
return this.options.logger.wrapOrRun(log, "CallHandler.loadCalls", async log => {
if (!intent) {
intent = CallIntent.Ring;
}
log.set("intent", intent);
const txn = await this._getLoadTxn();
const callEntries = await txn.calls.getByIntent(intent);
await this._loadCallEntries(callEntries, txn, log);
});
}
async loadCallsForRoom(intent: CallIntent, roomId: string) {
const txn = await this._getLoadTxn();
const callEntries = await txn.calls.getByIntentAndRoom(intent, roomId);
this._loadCallEntries(callEntries, txn);
loadCallsForRoom(intent: CallIntent, roomId: string, log?: ILogItem) {
return this.options.logger.wrapOrRun(log, "CallHandler.loadCallsForRoom", async log => {
log.set("intent", intent);
log.set("roomId", roomId);
const txn = await this._getLoadTxn();
const callEntries = await txn.calls.getByIntentAndRoom(intent, roomId);
await this._loadCallEntries(callEntries, txn, log);
});
}
private async _getLoadTxn(): Promise<Transaction> {
@ -86,68 +96,71 @@ export class CallHandler implements RoomStateHandler {
return txn;
}
private async _loadCallEntries(callEntries: CallEntry[], txn: Transaction): Promise<void> {
return this.options.logger.run({l: "loading calls", t: CALL_LOG_TYPE}, async log => {
log.set("entries", callEntries.length);
await Promise.all(callEntries.map(async callEntry => {
if (this._calls.get(callEntry.callId)) {
return;
private async _loadCallEntries(callEntries: CallEntry[], txn: Transaction, log: ILogItem): Promise<void> {
log.set("entries", callEntries.length);
await Promise.all(callEntries.map(async callEntry => {
if (this._calls.get(callEntry.callId)) {
return;
}
const event = await txn.roomState.get(callEntry.roomId, EventType.GroupCall, callEntry.callId);
if (event) {
const call = new GroupCall(event.event.state_key, false, event.event.content, event.roomId, this.groupCallOptions);
this._calls.set(call.id, call);
}
}));
const roomIds = Array.from(new Set(callEntries.map(e => e.roomId)));
await Promise.all(roomIds.map(async roomId => {
// TODO: don't load all members until we need them
const callsMemberEvents = await txn.roomState.getAllForType(roomId, EventType.GroupCallMember);
await Promise.all(callsMemberEvents.map(async entry => {
const userId = entry.event.sender;
const roomMemberState = await txn.roomState.get(roomId, MEMBER_EVENT_TYPE, userId);
let roomMember;
if (roomMemberState) {
roomMember = RoomMember.fromMemberEvent(roomMemberState.event);
}
const event = await txn.roomState.get(callEntry.roomId, EventType.GroupCall, callEntry.callId);
if (event) {
const call = new GroupCall(event.event.state_key, false, event.event.content, event.roomId, this.groupCallOptions);
this._calls.set(call.id, call);
if (!roomMember) {
// we'll be missing the member here if we received a call and it's members
// as pre-gap state and the members weren't active in the timeline we got.
roomMember = RoomMember.fromUserId(roomId, userId, "join");
}
this.handleCallMemberEvent(entry.event, roomMember, roomId, log);
}));
const roomIds = Array.from(new Set(callEntries.map(e => e.roomId)));
await Promise.all(roomIds.map(async roomId => {
// TODO: don't load all members until we need them
const callsMemberEvents = await txn.roomState.getAllForType(roomId, EventType.GroupCallMember);
await Promise.all(callsMemberEvents.map(async entry => {
const userId = entry.event.sender;
const roomMemberState = await txn.roomState.get(roomId, MEMBER_EVENT_TYPE, userId);
let roomMember;
if (roomMemberState) {
roomMember = RoomMember.fromMemberEvent(roomMemberState.event);
}
if (!roomMember) {
// we'll be missing the member here if we received a call and it's members
// as pre-gap state and the members weren't active in the timeline we got.
roomMember = RoomMember.fromUserId(roomId, userId, "join");
}
this.handleCallMemberEvent(entry.event, roomMember, roomId, log);
}));
}));
log.set("newSize", this._calls.size);
});
}));
log.set("newSize", this._calls.size);
}
async createCall(roomId: string, type: "m.video" | "m.voice", name: string, intent: CallIntent = CallIntent.Ring): Promise<GroupCall> {
const call = new GroupCall(makeId("conf-"), true, {
"m.name": name,
"m.intent": intent
}, roomId, this.groupCallOptions);
this._calls.set(call.id, call);
createCall(roomId: string, type: "m.video" | "m.voice", name: string, intent?: CallIntent, log?: ILogItem): Promise<GroupCall> {
return this.options.logger.wrapOrRun(log, "CallHandler.createCall", async log => {
if (!intent) {
intent = CallIntent.Ring;
}
const call = new GroupCall(makeId("conf-"), true, {
"m.name": name,
"m.intent": intent
}, roomId, this.groupCallOptions);
this._calls.set(call.id, call);
try {
await call.create(type);
// store call info so it will ring again when reopening the app
const txn = await this.options.storage.readWriteTxn([this.options.storage.storeNames.calls]);
txn.calls.add({
intent: call.intent,
callId: call.id,
timestamp: this.options.clock.now(),
roomId: roomId
});
await txn.complete();
} catch (err) {
//if (err.name === "ConnectionError") {
// if we're offline, give up and remove the call again
this._calls.remove(call.id);
//}
throw err;
}
return call;
try {
await call.create(type, log);
// store call info so it will ring again when reopening the app
const txn = await this.options.storage.readWriteTxn([this.options.storage.storeNames.calls]);
txn.calls.add({
intent: call.intent,
callId: call.id,
timestamp: this.options.clock.now(),
roomId: roomId
});
await txn.complete();
} catch (err) {
//if (err.name === "ConnectionError") {
// if we're offline, give up and remove the call again
this._calls.remove(call.id);
//}
throw err;
}
return call;
});
}
get calls(): BaseObservableMap<string, GroupCall> { return this._calls; }

423
src/matrix/calls/group/GroupCall.ts
View File

@ -21,6 +21,7 @@ import {MuteSettings, CALL_LOG_TYPE, CALL_MEMBER_VALIDITY_PERIOD_MS, mute} from
import {MemberChange, RoomMember} from "../../room/members/RoomMember";
import {EventEmitter} from "../../../utils/EventEmitter";
import {EventType, CallIntent} from "../callEventTypes";
import { ErrorBoundary } from "../../../utils/ErrorBoundary";
import type {Options as MemberOptions} from "./Member";
import type {TurnServerSource} from "../TurnServerSource";
@ -92,6 +93,14 @@ export class GroupCall extends EventEmitter<{change: never}> {
private bufferedDeviceMessages = new Map<string, Set<SignallingMessage<MGroupCallBase>>>();
/** Set between calling join and leave. */
private joinedData?: JoinedData;
private errorBoundary = new ErrorBoundary(err => {
this.emitChange();
if (this.joinedData) {
// in case the error happens in code that does not log,
// log it here to make sure it isn't swallowed
this.joinedData.logItem.log("error at boundary").catch(err);
}
});
constructor(
public readonly id: string,
@ -154,45 +163,52 @@ export class GroupCall extends EventEmitter<{change: never}> {
return this.joinedData?.logItem;
}
async join(localMedia: LocalMedia): Promise<void> {
if (this._state !== GroupCallState.Created || this.joinedData) {
return;
}
const logItem = this.options.logger.child({
l: "answer call",
t: CALL_LOG_TYPE,
id: this.id,
ownSessionId: this.options.sessionId
});
const turnServer = await this.options.turnServerSource.getSettings(logItem);
const membersLogItem = logItem.child("member connections");
const localMuteSettings = new MuteSettings();
localMuteSettings.updateTrackInfo(localMedia.userMedia);
const localPreviewMedia = localMedia.asPreview();
const joinedData = new JoinedData(
logItem,
membersLogItem,
localMedia,
localPreviewMedia,
localMuteSettings,
turnServer
);
this.joinedData = joinedData;
await joinedData.logItem.wrap("join", async log => {
this._state = GroupCallState.Joining;
this.emitChange();
await log.wrap("update member state", async log => {
const memberContent = await this._createMemberPayload(true);
log.set("payload", memberContent);
// send m.call.member state event
const request = this.options.hsApi.sendState(this.roomId, EventType.GroupCallMember, this.options.ownUserId, memberContent, {log});
await request.response();
this.emitChange();
});
// send invite to all members that are < my userId
for (const [,member] of this._members) {
this.connectToMember(member, joinedData, log);
get error(): Error | undefined {
return this.errorBoundary.error;
}
join(localMedia: LocalMedia, log?: ILogItem): Promise<void> {
return this.options.logger.wrapOrRun(log, "Call.join", async joinLog => {
if (this._state !== GroupCallState.Created || this.joinedData) {
return;
}
const logItem = this.options.logger.child({
l: "Call.connection",
t: CALL_LOG_TYPE,
id: this.id,
ownSessionId: this.options.sessionId
});
const turnServer = await this.options.turnServerSource.getSettings(logItem);
const membersLogItem = logItem.child("member connections");
const localMuteSettings = new MuteSettings();
localMuteSettings.updateTrackInfo(localMedia.userMedia);
const localPreviewMedia = localMedia.asPreview();
const joinedData = new JoinedData(
logItem,
membersLogItem,
localMedia,
localPreviewMedia,
localMuteSettings,
turnServer
);
this.joinedData = joinedData;
await joinedData.logItem.wrap("join", async log => {
joinLog.refDetached(log);
this._state = GroupCallState.Joining;
this.emitChange();
await log.wrap("update member state", async log => {
const memberContent = await this._createMemberPayload(true);
log.set("payload", memberContent);
// send m.call.member state event
const request = this.options.hsApi.sendState(this.roomId, EventType.GroupCallMember, this.options.ownUserId, memberContent, {log});
await request.response();
this.emitChange();
});
// send invite to all members that are < my userId
for (const [,member] of this._members) {
this.connectToMember(member, joinedData, log);
}
});
});
}
@ -206,6 +222,9 @@ export class GroupCall extends EventEmitter<{change: never}> {
// and update the track info so PeerCall can use it to send up to date metadata,
this.joinedData.localMuteSettings.updateTrackInfo(localMedia.userMedia);
this.emitChange(); //allow listeners to see new media/mute settings
// TODO: if setMedia fails on one of the members, we should revert to the old media
// on the members processed so far, and show an error that we could not set the new media
// for this, we will need to remove the usage of the errorBoundary in member.setMedia.
await Promise.all(Array.from(this._members.values()).map(m => {
return m.setMedia(localMedia, oldMedia);
}));
@ -234,6 +253,9 @@ export class GroupCall extends EventEmitter<{change: never}> {
if (this.localMedia) {
mute(this.localMedia, muteSettings, this.joinedData!.logItem);
}
// TODO: if setMuted fails on one of the members, we should revert to the old media
// on the members processed so far, and show an error that we could not set the new media
// for this, we will need to remove the usage of the errorBoundary in member.setMuted.
await Promise.all(Array.from(this._members.values()).map(m => {
return m.setMuted(joinedData.localMuteSettings);
}));
@ -249,12 +271,12 @@ export class GroupCall extends EventEmitter<{change: never}> {
return this._state === GroupCallState.Joining || this._state === GroupCallState.Joined;
}
async leave(): Promise<void> {
const {joinedData} = this;
if (!joinedData) {
return;
}
await joinedData.logItem.wrap("leave", async log => {
async leave(log?: ILogItem): Promise<void> {
await this.options.logger.wrapOrRun(log, "Call.leave", async log => {
const {joinedData} = this;
if (!joinedData) {
return;
}
try {
joinedData.renewMembershipTimeout?.dispose();
joinedData.renewMembershipTimeout = undefined;
@ -271,12 +293,19 @@ export class GroupCall extends EventEmitter<{change: never}> {
log.set("already_left", true);
}
} finally {
this.disconnect(log);
// disconnect is called both from the sync loop and from methods like this one that
// are called from the view model. We want errors during the sync loop being caught
// by the errorboundary, but since leave is called from the view model, we want
// the error to be thrown. So here we check if disconnect succeeded, and if not
// we rethrow the error put into the errorBoundary.
if(!this.disconnect(log)) {
throw this.errorBoundary.error;
}
}
});
}
terminate(log?: ILogItem): Promise<void> {
private terminate(log?: ILogItem): Promise<void> {
return this.options.logger.wrapOrRun(log, {l: "terminate call", t: CALL_LOG_TYPE}, async log => {
if (this._state === GroupCallState.Fledgling) {
return;
@ -289,8 +318,8 @@ export class GroupCall extends EventEmitter<{change: never}> {
}
/** @internal */
create(type: "m.video" | "m.voice", log?: ILogItem): Promise<void> {
return this.options.logger.wrapOrRun(log, {l: "create call", t: CALL_LOG_TYPE}, async log => {
create(type: "m.video" | "m.voice", log: ILogItem): Promise<void> {
return log.wrap({l: "create call", t: CALL_LOG_TYPE}, async log => {
if (this._state !== GroupCallState.Fledgling) {
return;
}
@ -308,126 +337,134 @@ export class GroupCall extends EventEmitter<{change: never}> {
/** @internal */
updateCallEvent(callContent: Record<string, any>, syncLog: ILogItem) {
syncLog.wrap({l: "update call", t: CALL_LOG_TYPE, id: this.id}, log => {
this.callContent = callContent;
if (this._state === GroupCallState.Creating) {
this._state = GroupCallState.Created;
}
log.set("status", this._state);
this.emitChange();
this.errorBoundary.try(() => {
syncLog.wrap({l: "update call", t: CALL_LOG_TYPE, id: this.id}, log => {
this.callContent = callContent;
if (this._state === GroupCallState.Creating) {
this._state = GroupCallState.Created;
}
log.set("status", this._state);
this.emitChange();
});
});
}
/** @internal */
updateRoomMembers(memberChanges: Map<string, MemberChange>) {
for (const change of memberChanges.values()) {
const {member} = change;
for (const callMember of this._members.values()) {
// find all call members for a room member (can be multiple, for every device)
if (callMember.userId === member.userId) {
callMember.updateRoomMember(member);
this.errorBoundary.try(() => {
for (const change of memberChanges.values()) {
const {member} = change;
for (const callMember of this._members.values()) {
// find all call members for a room member (can be multiple, for every device)
if (callMember.userId === member.userId) {
callMember.updateRoomMember(member);
}
}
}
}
}
/** @internal */
updateMembership(userId: string, roomMember: RoomMember, callMembership: CallMembership, syncLog: ILogItem) {
syncLog.wrap({l: "update call membership", t: CALL_LOG_TYPE, id: this.id, userId}, log => {
const now = this.options.clock.now();
const devices = callMembership["m.devices"];
const previousDeviceIds = this.getDeviceIdsForUserId(userId);
for (const device of devices) {
const deviceId = device.device_id;
const memberKey = getMemberKey(userId, deviceId);
if (userId === this.options.ownUserId && deviceId === this.options.ownDeviceId) {
log.wrap("update own membership", log => {
if (this.hasJoined) {
if (this.joinedData) {
this.joinedData.logItem.refDetached(log);
}
this._setupRenewMembershipTimeout(device, log);
}
if (this._state === GroupCallState.Joining) {
log.set("joined", true);
this._state = GroupCallState.Joined;
this.emitChange();
}
});
} else {
log.wrap({l: "update device membership", id: memberKey, sessionId: device.session_id}, log => {
if (isMemberExpired(device, now)) {
log.set("expired", true);
const member = this._members.get(memberKey);
if (member) {
member.dispose();
this._members.remove(memberKey);
log.set("removed", true);
}
return;
}
let member = this._members.get(memberKey);
const sessionIdChanged = member && member.sessionId !== device.session_id;
if (member && !sessionIdChanged) {
log.set("update", true);
member.updateCallInfo(device, log);
} else {
if (member && sessionIdChanged) {
log.set("removedSessionId", member.sessionId);
const disconnectLogItem = member.disconnect(false);
if (disconnectLogItem) {
log.refDetached(disconnectLogItem);
}
member.dispose();
this._members.remove(memberKey);
member = undefined;
}
log.set("add", true);
member = new Member(
roomMember,
device, this._memberOptions,
log
);
this._members.add(memberKey, member);
if (this.joinedData) {
this.connectToMember(member, this.joinedData, log);
}
}
// flush pending messages, either after having created the member,
// or updated the session id with updateCallInfo
this.flushPendingIncomingDeviceMessages(member, log);
});
}
}
const newDeviceIds = new Set<string>(devices.map(call => call.device_id));
// remove user as member of any calls not present anymore
for (const previousDeviceId of previousDeviceIds) {
if (!newDeviceIds.has(previousDeviceId)) {
this.removeMemberDevice(userId, previousDeviceId, log);
}
}
if (userId === this.options.ownUserId && !newDeviceIds.has(this.options.ownDeviceId)) {
this.removeOwnDevice(log);
}
});
}
/** @internal */
updateMembership(userId: string, roomMember: RoomMember, callMembership: CallMembership, syncLog: ILogItem) {
this.errorBoundary.try(() => {
syncLog.wrap({l: "update call membership", t: CALL_LOG_TYPE, id: this.id, userId}, log => {
const now = this.options.clock.now();
const devices = callMembership["m.devices"];
const previousDeviceIds = this.getDeviceIdsForUserId(userId);
for (const device of devices) {
const deviceId = device.device_id;
const memberKey = getMemberKey(userId, deviceId);
if (userId === this.options.ownUserId && deviceId === this.options.ownDeviceId) {
log.wrap("update own membership", log => {
if (this.hasJoined) {
if (this.joinedData) {
this.joinedData.logItem.refDetached(log);
}
this._setupRenewMembershipTimeout(device, log);
}
if (this._state === GroupCallState.Joining) {
log.set("joined", true);
this._state = GroupCallState.Joined;
this.emitChange();
}
});
} else {
log.wrap({l: "update device membership", id: memberKey, sessionId: device.session_id}, log => {
if (isMemberExpired(device, now)) {
log.set("expired", true);
const member = this._members.get(memberKey);
if (member) {
member.dispose();
this._members.remove(memberKey);
log.set("removed", true);
}
return;
}
let member = this._members.get(memberKey);
const sessionIdChanged = member && member.sessionId !== device.session_id;
if (member && !sessionIdChanged) {
log.set("update", true);
member.updateCallInfo(device, log);
} else {
if (member && sessionIdChanged) {
log.set("removedSessionId", member.sessionId);
const disconnectLogItem = member.disconnect(false);
if (disconnectLogItem) {
log.refDetached(disconnectLogItem);
}
member.dispose();
this._members.remove(memberKey);
member = undefined;
}
log.set("add", true);
member = new Member(
roomMember,
device, this._memberOptions,
log
);
this._members.add(memberKey, member);
if (this.joinedData) {
this.connectToMember(member, this.joinedData, log);
}
}
// flush pending messages, either after having created the member,
// or updated the session id with updateCallInfo
this.flushPendingIncomingDeviceMessages(member, log);
});
}
}
const newDeviceIds = new Set<string>(devices.map(call => call.device_id));
// remove user as member of any calls not present anymore
for (const previousDeviceId of previousDeviceIds) {
if (!newDeviceIds.has(previousDeviceId)) {
this.removeMemberDevice(userId, previousDeviceId, log);
}
}
if (userId === this.options.ownUserId && !newDeviceIds.has(this.options.ownDeviceId)) {
this.removeOwnDevice(log);
}
});
});
}
/** @internal */
removeMembership(userId: string, syncLog: ILogItem) {
const deviceIds = this.getDeviceIdsForUserId(userId);
syncLog.wrap({
l: "remove call member",
t: CALL_LOG_TYPE,
id: this.id,
userId
}, log => {
for (const deviceId of deviceIds) {
this.removeMemberDevice(userId, deviceId, log);
}
if (userId === this.options.ownUserId) {
this.removeOwnDevice(log);
}
this.errorBoundary.try(() => {
const deviceIds = this.getDeviceIdsForUserId(userId);
syncLog.wrap({
l: "remove call member",
t: CALL_LOG_TYPE,
id: this.id,
userId
}, log => {
for (const deviceId of deviceIds) {
this.removeMemberDevice(userId, deviceId, log);
}
if (userId === this.options.ownUserId) {
this.removeOwnDevice(log);
}
});
});
}
@ -465,19 +502,21 @@ export class GroupCall extends EventEmitter<{change: never}> {
}
/** @internal */
disconnect(log: ILogItem) {
if (this.hasJoined) {
for (const [,member] of this._members) {
const disconnectLogItem = member.disconnect(true);
if (disconnectLogItem) {
log.refDetached(disconnectLogItem);
disconnect(log: ILogItem): boolean {
return this.errorBoundary.try(() => {
if (this.hasJoined) {
for (const [,member] of this._members) {
const disconnectLogItem = member.disconnect(true);
if (disconnectLogItem) {
log.refDetached(disconnectLogItem);
}
}
this._state = GroupCallState.Created;
}
this._state = GroupCallState.Created;
}
this.joinedData?.dispose();
this.joinedData = undefined;
this.emitChange();
this.joinedData?.dispose();
this.joinedData = undefined;
this.emitChange();
}, false) || true;
}
/** @internal */
@ -500,31 +539,33 @@ export class GroupCall extends EventEmitter<{change: never}> {
/** @internal */
handleDeviceMessage(message: SignallingMessage<MGroupCallBase>, userId: string, deviceId: string, syncLog: ILogItem) {
// TODO: return if we are not membering to the call
const key = getMemberKey(userId, deviceId);
let member = this._members.get(key);
if (member && message.content.sender_session_id === member.sessionId) {
member.handleDeviceMessage(message, syncLog);
} else {
const item = syncLog.log({
l: "call: buffering to_device message, member not found",
t: CALL_LOG_TYPE,
id: this.id,
userId,
deviceId,
sessionId: message.content.sender_session_id,
type: message.type
});
syncLog.refDetached(item);
// we haven't received the m.call.member yet for this caller (or with this session id).
// buffer the device messages or create the member/call as it should arrive in a moment
let messages = this.bufferedDeviceMessages.get(key);
if (!messages) {
messages = new Set();
this.bufferedDeviceMessages.set(key, messages);
this.errorBoundary.try(() => {
// TODO: return if we are not membering to the call
const key = getMemberKey(userId, deviceId);
let member = this._members.get(key);
if (member && message.content.sender_session_id === member.sessionId) {
member.handleDeviceMessage(message, syncLog);
} else {
const item = syncLog.log({
l: "call: buffering to_device message, member not found",
t: CALL_LOG_TYPE,
id: this.id,
userId,
deviceId,
sessionId: message.content.sender_session_id,
type: message.type
});
syncLog.refDetached(item);
// we haven't received the m.call.member yet for this caller (or with this session id).
// buffer the device messages or create the member/call as it should arrive in a moment
let messages = this.bufferedDeviceMessages.get(key);
if (!messages) {
messages = new Set();
this.bufferedDeviceMessages.set(key, messages);
}
messages.add(message);
}
messages.add(message);
}
});
}
private async _createMemberPayload(includeOwn: boolean): Promise<CallMemberContent> {

190
src/matrix/calls/group/Member.ts
View File

@ -19,6 +19,7 @@ import {makeTxnId, makeId} from "../../common";
import {EventType, CallErrorCode} from "../callEventTypes";
import {formatToDeviceMessagesPayload} from "../../common";
import {sortedIndex} from "../../../utils/sortedIndex";
import { ErrorBoundary } from "../../../utils/ErrorBoundary";
import type {MuteSettings} from "../common";
import type {Options as PeerCallOptions, RemoteMedia} from "../PeerCall";
@ -94,6 +95,14 @@ class MemberConnection {
export class Member {
private connection?: MemberConnection;
private expireTimeout?: Timeout;
private errorBoundary = new ErrorBoundary(err => {
this.options.emitUpdate(this, "error");
if (this.connection) {
// in case the error happens in code that does not log,
// log it here to make sure it isn't swallowed
this.connection.logItem.log("error at boundary").catch(err);
}
});
constructor(
public member: RoomMember,
@ -104,6 +113,10 @@ export class Member {
this._renewExpireTimeout(updateMemberLog);
}
get error(): Error | undefined {
return this.errorBoundary.error;
}
private _renewExpireTimeout(log: ILogItem) {
this.expireTimeout?.dispose();
this.expireTimeout = undefined;
@ -166,23 +179,26 @@ export class Member {
/** @internal */
connect(localMedia: LocalMedia, localMuteSettings: MuteSettings, turnServer: BaseObservableValue<RTCIceServer>, memberLogItem: ILogItem): ILogItem | undefined {
if (this.connection) {
return;
}
// Safari can't send a MediaStream to multiple sources, so clone it
const connection = new MemberConnection(
localMedia.clone(),
localMuteSettings,
turnServer,
memberLogItem
);
this.connection = connection;
let connectLogItem;
connection.logItem.wrap("connect", async log => {
connectLogItem = log;
await this.callIfNeeded(log);
return this.errorBoundary.try(() => {
if (this.connection) {
return;
}
// Safari can't send a MediaStream to multiple sources, so clone it
const connection = new MemberConnection(
localMedia.clone(),
localMuteSettings,
turnServer,
memberLogItem
);
this.connection = connection;
let connectLogItem: ILogItem | undefined;
connection.logItem.wrap("connect", async log => {
connectLogItem = log;
await this.callIfNeeded(log);
});
throw new Error("connect failed!");
return connectLogItem;
});
return connectLogItem;
}
private callIfNeeded(log: ILogItem): Promise<void> {
@ -211,30 +227,34 @@ export class Member {
/** @internal */
disconnect(hangup: boolean): ILogItem | undefined {
const {connection} = this;
if (!connection) {
return;
}
let disconnectLogItem;
// if if not sending the hangup, still log disconnect
connection.logItem.wrap("disconnect", async log => {
disconnectLogItem = log;
if (hangup && connection.peerCall) {
await connection.peerCall.hangup(CallErrorCode.UserHangup, log);
return this.errorBoundary.try(() => {
const {connection} = this;
if (!connection) {
return;
}
let disconnectLogItem;
// if if not sending the hangup, still log disconnect
connection.logItem.wrap("disconnect", async log => {
disconnectLogItem = log;
if (hangup && connection.peerCall) {
await connection.peerCall.hangup(CallErrorCode.UserHangup, log);
}
});
connection.dispose();
this.connection = undefined;
return disconnectLogItem;
});
connection.dispose();
this.connection = undefined;
return disconnectLogItem;
}
/** @internal */
updateCallInfo(callDeviceMembership: CallDeviceMembership, causeItem: ILogItem) {
this.callDeviceMembership = callDeviceMembership;
this._renewExpireTimeout(causeItem);
if (this.connection) {
this.connection.logItem.refDetached(causeItem);
}
this.errorBoundary.try(() => {
this.callDeviceMembership = callDeviceMembership;
this._renewExpireTimeout(causeItem);
if (this.connection) {
this.connection.logItem.refDetached(causeItem);
}
});
}
/** @internal */
@ -308,49 +328,51 @@ export class Member {
/** @internal */
handleDeviceMessage(message: SignallingMessage<MGroupCallBase>, syncLog: ILogItem): void {
const {connection} = this;
if (connection) {
const destSessionId = message.content.dest_session_id;
if (destSessionId !== this.options.sessionId) {
const logItem = connection.logItem.log({l: "ignoring to_device event with wrong session_id", destSessionId, type: message.type});
syncLog.refDetached(logItem);
return;
}
// if there is no peerCall, we either create it with an invite and Handle is implied or we'll ignore it
let action = IncomingMessageAction.Handle;
if (connection.peerCall) {
action = connection.peerCall.getMessageAction(message);
// deal with glare and replacing the call before creating new calls
if (action === IncomingMessageAction.InviteGlare) {
const {shouldReplace, log} = connection.peerCall.handleInviteGlare(message, this.deviceId, connection.logItem);
if (log) {
syncLog.refDetached(log);
}
if (shouldReplace) {
connection.peerCall = undefined;
action = IncomingMessageAction.Handle;
}
this.errorBoundary.try(() => {
const {connection} = this;
if (connection) {
const destSessionId = message.content.dest_session_id;
if (destSessionId !== this.options.sessionId) {
const logItem = connection.logItem.log({l: "ignoring to_device event with wrong session_id", destSessionId, type: message.type});
syncLog.refDetached(logItem);
return;
}
}
if (message.type === EventType.Invite && !connection.peerCall) {
connection.peerCall = this._createPeerCall(message.content.call_id);
}
if (action === IncomingMessageAction.Handle) {
const idx = sortedIndex(connection.queuedSignallingMessages, message, (a, b) => a.content.seq - b.content.seq);
connection.queuedSignallingMessages.splice(idx, 0, message);
// if there is no peerCall, we either create it with an invite and Handle is implied or we'll ignore it
let action = IncomingMessageAction.Handle;
if (connection.peerCall) {
const hasNewMessageBeenDequeued = this.dequeueSignallingMessages(connection, connection.peerCall, message, syncLog);
if (!hasNewMessageBeenDequeued) {
syncLog.refDetached(connection.logItem.log({l: "queued signalling message", type: message.type, seq: message.content.seq}));
action = connection.peerCall.getMessageAction(message);
// deal with glare and replacing the call before creating new calls
if (action === IncomingMessageAction.InviteGlare) {
const {shouldReplace, log} = connection.peerCall.handleInviteGlare(message, this.deviceId, connection.logItem);
if (log) {
syncLog.refDetached(log);
}
if (shouldReplace) {
connection.peerCall = undefined;
action = IncomingMessageAction.Handle;
}
}
}
} else if (action === IncomingMessageAction.Ignore && connection.peerCall) {
const logItem = connection.logItem.log({l: "ignoring to_device event with wrong call_id", callId: message.content.call_id, type: message.type});
syncLog.refDetached(logItem);
if (message.type === EventType.Invite && !connection.peerCall) {
connection.peerCall = this._createPeerCall(message.content.call_id);
}
if (action === IncomingMessageAction.Handle) {
const idx = sortedIndex(connection.queuedSignallingMessages, message, (a, b) => a.content.seq - b.content.seq);
connection.queuedSignallingMessages.splice(idx, 0, message);
if (connection.peerCall) {
const hasNewMessageBeenDequeued = this.dequeueSignallingMessages(connection, connection.peerCall, message, syncLog);
if (!hasNewMessageBeenDequeued) {
syncLog.refDetached(connection.logItem.log({l: "queued signalling message", type: message.type, seq: message.content.seq}));
}
}
} else if (action === IncomingMessageAction.Ignore && connection.peerCall) {
const logItem = connection.logItem.log({l: "ignoring to_device event with wrong call_id", callId: message.content.call_id, type: message.type});
syncLog.refDetached(logItem);
}
} else {
syncLog.log({l: "member not connected", userId: this.userId, deviceId: this.deviceId});
}
} else {
syncLog.log({l: "member not connected", userId: this.userId, deviceId: this.deviceId});
}
});
}
private dequeueSignallingMessages(connection: MemberConnection, peerCall: PeerCall, newMessage: SignallingMessage<MGroupCallBase>, syncLog: ILogItem): boolean {
@ -373,19 +395,23 @@ export class Member {
/** @internal */
async setMedia(localMedia: LocalMedia, previousMedia: LocalMedia): Promise<void> {
const {connection} = this;
if (connection) {
connection.localMedia = localMedia.replaceClone(connection.localMedia, previousMedia);
await connection.peerCall?.setMedia(connection.localMedia, connection.logItem);
}
return this.errorBoundary.try(async () => {
const {connection} = this;
if (connection) {
connection.localMedia = localMedia.replaceClone(connection.localMedia, previousMedia);
await connection.peerCall?.setMedia(connection.localMedia, connection.logItem);
}
});
}
async setMuted(muteSettings: MuteSettings): Promise<void> {
const {connection} = this;
if (connection) {
connection.localMuteSettings = muteSettings;
await connection.peerCall?.setMuted(muteSettings, connection.logItem);
}
return this.errorBoundary.try(async () => {
const {connection} = this;
if (connection) {
connection.localMuteSettings = muteSettings;
await connection.peerCall?.setMuted(muteSettings, connection.logItem);
}
});
}
private _createPeerCall(callId: string): PeerCall {

2
src/matrix/room/timeline/entries/BaseEventEntry.js
View File

@ -181,7 +181,7 @@ export class BaseEventEntry extends BaseEntry {
return createAnnotation(this.id, key);
}
reply(msgtype, body) {
createReplyContent(msgtype, body) {
return createReplyContent(this, msgtype, body);
}

17
src/platform/web/ui/css/themes/element/call.css
View File

@ -24,6 +24,14 @@ limitations under the License.
grid-row: 1;
}
.CallView_error {
color: red;
font-weight: bold;
align-self: start;
justify-self: center;
margin: 16px;
}
.CallView_members {
display: grid;
gap: 12px;
@ -59,6 +67,15 @@ limitations under the License.
justify-self: center;
}
.StreamView_error {
align-self: start;
justify-self: center;
/** Chrome (v100) requires this to make the buttons clickable
* where they overlap with the video element, even though
* the buttons come later in the DOM. */
z-index: 1;
}
.StreamView_muteStatus {
align-self: start;
justify-self: end;

51
src/platform/web/ui/css/themes/element/error.css Normal file
View File

@ -0,0 +1,51 @@
.ErrorView_block {
background: var(--error-color);
color: var(--fixed-white);
margin: 16px;
}
.ErrorView_inline {
color: var(--error-color);
margin: 4px;
}
.ErrorView {
font-weight: bold;
margin: 16px;
border-radius: 8px;
padding: 12px;
display: flex;
gap: 8px;
}
.ErrorView_message {
flex-basis: 0;
flex-grow: 1;
margin: 0px;
word-break: break-all;
word-break: break-word;
align-self: center;
}
.ErrorView_submit {
align-self: end;
}
.ErrorView_close {
align-self: start;
width: 16px;
height: 16px;
border: none;
background: none;
background-repeat: no-repeat;
background-size: contain;
cursor: pointer;
}
.ErrorView_block .ErrorView_close {
background-image: url('icons/clear.svg?primary=fixed-white');
}
.ErrorView_inline .ErrorView_close {
background-image: url('icons/clear.svg?primary=text-color');
}

1
src/platform/web/ui/css/themes/element/theme.css
View File

@ -19,6 +19,7 @@ limitations under the License.
@import url('inter.css');
@import url('timeline.css');
@import url('call.css');
@import url('error.css');
:root {
font-size: 10px;

60
src/platform/web/ui/general/ErrorView.ts Normal file
View File

@ -0,0 +1,60 @@
/*
Copyright 2023 The Matrix.org Foundation C.I.C.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
import {TemplateView, Builder} from "./TemplateView";
import { disableTargetCallback } from "./utils";
import type { ViewNode } from "./types";
import type {ErrorViewModel} from "../../../../domain/ErrorViewModel";
export class ErrorView extends TemplateView<ErrorViewModel> {
constructor(vm: ErrorViewModel, private readonly options: {inline: boolean} = {inline: false}) {
super(vm);
}
override render(t: Builder<ErrorViewModel>, vm: ErrorViewModel): ViewNode {
const submitLogsButton = t.button({
className: "ErrorView_submit",
onClick: disableTargetCallback(async evt => {
evt.stopPropagation();
if (await vm.submitLogs()) {
alert("Logs submitted!");
} else {
alert("Could not submit logs");
}
})
}, "Submit logs");
const closeButton = t.button({
className: "ErrorView_close",
onClick: evt => {
evt.stopPropagation();
vm.close();
},
title: "Dismiss error"
});
return t.div({
className: {
"ErrorView": true,
"ErrorView_inline": this.options.inline,
"ErrorView_block": !this.options.inline
}}, [
t.p({className: "ErrorView_message"}, vm.message),
submitLogsButton,
closeButton
]);
}
}

9
src/platform/web/ui/session/room/CallView.ts
View File

@ -20,6 +20,7 @@ import {ListView} from "../../general/ListView";
import {classNames} from "../../general/html";
import {Stream} from "../../../../types/MediaDevices";
import type {CallViewModel, CallMemberViewModel, IStreamViewModel} from "../../../../../domain/session/room/CallViewModel";
import { ErrorView } from "../../general/ErrorView";
export class CallView extends TemplateView<CallViewModel> {
private resizeObserver?: ResizeObserver;
@ -43,7 +44,10 @@ export class CallView extends TemplateView<CallViewModel> {
"CallView_unmutedCamera": vm => !vm.isCameraMuted,
}, onClick: disableTargetCallback(() => vm.toggleCamera())}),
t.button({className: "CallView_hangup", onClick: disableTargetCallback(() => vm.hangup())}),
])
]),
t.if(vm => !!vm.errorViewModel, t => {
return t.div({className: "CallView_error"}, t.view(new ErrorView(vm.errorViewModel!)));
})
]);
}
@ -112,6 +116,9 @@ class StreamView extends TemplateView<IStreamViewModel> {
microphoneMuted: vm => vm.isMicrophoneMuted && !vm.isCameraMuted,
cameraMuted: vm => vm.isCameraMuted,
}
}),
t.if(vm => !!vm.errorViewModel, t => {
return t.div({className: "StreamView_error"}, t.view(new ErrorView(vm.errorViewModel!)));
})
]);
}

3
src/platform/web/ui/session/room/RoomView.js
View File

@ -24,6 +24,7 @@ import {MessageComposer} from "./MessageComposer.js";
import {RoomArchivedView} from "./RoomArchivedView.js";
import {AvatarView} from "../../AvatarView.js";
import {CallView} from "./CallView";
import { ErrorView } from "../../general/ErrorView";
export class RoomView extends TemplateView {
constructor(vm, viewClassForTile) {
@ -53,7 +54,7 @@ export class RoomView extends TemplateView {
})
]),
t.div({className: "RoomView_body"}, [
t.div({className: "RoomView_error"}, vm => vm.error),
t.if(vm => vm.errorViewModel, t => t.div({className: "RoomView_error"}, t.view(new ErrorView(vm.errorViewModel)))),
t.mapView(vm => vm.callViewModel, callViewModel => callViewModel ? new CallView(callViewModel) : null),
t.mapView(vm => vm.timelineViewModel, timelineViewModel => {
return timelineViewModel ?

21
src/platform/web/ui/session/room/timeline/CallTileView.ts
View File

@ -14,17 +14,24 @@ See the License for the specific language governing permissions and
limitations under the License.
*/
import {TemplateView} from "../../../general/TemplateView";
import {Builder, TemplateView} from "../../../general/TemplateView";
import type {CallTile} from "../../../../../../domain/session/room/timeline/tiles/CallTile";
import {ErrorView} from "../../../general/ErrorView";
export class CallTileView extends TemplateView<CallTile> {
render(t, vm) {
render(t: Builder<CallTile>, vm: CallTile) {
return t.li(
{className: "AnnouncementView"},
t.div([
vm => vm.label,
t.button({className: "CallTileView_join", hidden: vm => !vm.canJoin}, "Join"),
t.button({className: "CallTileView_leave", hidden: vm => !vm.canLeave}, "Leave")
{className: "CallTileView AnnouncementView"},
t.div(
[
t.if(vm => vm.errorViewModel, t => {
return t.div({className: "CallTileView_error"}, t.view(new ErrorView(vm.errorViewModel, {inline: true})));
}),
t.div([
vm => vm.label,
t.button({className: "CallTileView_join", hidden: vm => !vm.canJoin}, "Join"),
t.button({className: "CallTileView_leave", hidden: vm => !vm.canLeave}, "Leave")
])
])
);
}

39
src/utils/ErrorBoundary.ts
View File

@ -19,28 +19,36 @@ export const ErrorValue = Symbol("ErrorBoundary:Error");
export class ErrorBoundary {
private _error?: Error;
constructor(private readonly errorCallback: (Error) => void) {}
constructor(private readonly errorCallback: (err: Error) => void) {}
/**
* Executes callback() and then runs errorCallback() on error.
* This will never throw but instead return `errorValue` if an error occured.
*/
try<T>(callback: () => T): T | typeof ErrorValue;
try<T>(callback: () => Promise<T>): Promise<T | typeof ErrorValue> | typeof ErrorValue {
try<T, E>(callback: () => T, errorValue?: E): T | typeof errorValue;
try<T, E>(callback: () => Promise<T>, errorValue?: E): Promise<T | typeof errorValue> | typeof errorValue {
try {
let result: T | Promise<T | typeof ErrorValue> = callback();
let result: T | Promise<T | typeof errorValue> = callback();
if (result instanceof Promise) {
result = result.catch(err => {
this._error = err;
this.errorCallback(err);
return ErrorValue;
this.reportError(err);
return errorValue;
});
}
return result;
} catch (err) {
this._error = err;
this.reportError(err);
return errorValue;
}
}
private reportError(err: Error) {
try {
this.errorCallback(err);
return ErrorValue;
} catch (err) {
console.error("error in ErrorBoundary callback", err);
}
}
@ -56,9 +64,9 @@ export function tests() {
const boundary = new ErrorBoundary(() => emitted = true);
const result = boundary.try(() => {
throw new Error("fail!");
});
}, 0);
assert(emitted);
assert.strictEqual(result, ErrorValue);
assert.strictEqual(result, 0);
},
"return value of callback is forwarded": assert => {
let emitted = false;
@ -74,9 +82,18 @@ export function tests() {
const boundary = new ErrorBoundary(() => emitted = true);
const result = await boundary.try(async () => {
throw new Error("fail!");
});
}, 0);
assert(emitted);
assert.strictEqual(result, ErrorValue);
assert.strictEqual(result, 0);
},
"exception in error callback is swallowed": async assert => {
let emitted = false;
const boundary = new ErrorBoundary(() => { throw new Error("bug in errorCallback"); });
assert.doesNotThrow(() => {
boundary.try(() => {
throw new Error("fail!");
});
});
}
}
}