Repository navigation
fix(call): resume the mixed call audio after it was suspended - #20113
AbdurRazzaq2004 wants to merge 1 commit into
Conversation
Problem
-------
In Safari and on iOS, the audio of all the participants is mixed with an
AudioContext and played by a single audio element. The browser can
suspend that context, for example on iOS when the audio is interrupted
by a phone call or Siri, or when the screen is locked. Nothing resumed
it afterwards, so no participant could be heard again until the call was
left and joined again.
Solution
--------
- Resume the audio context, and play the element again if needed, when
the state of the context changes, when the page becomes visible again
and on the next user interaction (in case the browser requires a user
gesture for it).
- The listeners are removed when the player is destroyed.
- A user interaction resumes it also with touch input: "pointerup" and
"touchend" are listened to as well, as "pointerdown" is a user gesture
only with a mouse. The element is also played again right after its
source is replaced, which is done in Safari when a participant is
muted.
Testing
-------
- Lint (eslint on the changed files) passes, and the unit tests pass on
main with all five related fixes applied (vitest src/utils: 29 files,
616 tests).
- The change has been running in production since 2026-11-02 on a
Nextcloud 34.0.5 server with Talk 24.0.6 and the external signaling
server (2.1.1, Janus 1.4.1), where all media goes through TURN.
- End-to-end checks were run with headless Chromium against that live
server, each time with the unmodified Talk 24.0.6 JavaScript
("original", served to the browser through request interception) and
with this change ("patched"), on the same server and under the same
conditions.
- The mixed audio path was tested with Chromium identifying itself as
Safari (Talk then mixes the audio as in Safari). The mixing
AudioContext was suspended during a call, original against patched:
- Original: it stayed "suspended", also after a user interaction.
- Patched: it was "running" again within 2 seconds, without any user
interaction.
- Unit tests for the mixed audio were added: the audio is resumed on
"pointerup", "touchend", "mousedown" and "keydown", not after the
player was destroyed, and the element is played again after its source
is replaced.
- A listen-only guest (no microphone, so the browser does not allow
autoplay) joined the call without any user activation, with Chromium
identifying itself as Safari. With the first version of this pull
request (resuming only on "pointerdown") the mixing AudioContext
stayed "suspended" after a full tap; with this version it was
"running" and the element playing right after the tap. A touch-start
alone did not unlock it in either version, as the HTML specification
says.
- This could not be checked on a real iOS device.
Signed-off-by: Abdur Razzaq <153341210+AbdurRazzaq2004@users.noreply.github.com>
f024c94 to
cc6083f
Compare
|
Updated: "pointerdown" is not a user gesture for touch input, so on iOS a tap could not resume the audio. It is now also resumed on "pointerup", "touchend" and "mousedown", and the element is played again after Safari's source swap on mute. Unit tests for the mixed audio were added (eslint and vitest |
Antreesy
left a comment
There was a problem hiding this comment.
Wish for all open PRs:
- please disclose agent model with
Assisted-by: MODEL:VERSIONgit trailer - keep commit body of a reasonable size 🙈 core change, git trailer and DCO are enough, the rest is OK to be documented in the PR description or the code itself
- same for large code comments, if it's self-explanatory
Otherwise nice! Thank you, waiting for you to rebase and looking forward to merge!
The change has been running in production since 2026-11-02
Let us know, if you noticed it's better now?
| // soon as possible; if the browser requires a user gesture for that it | ||
| // will be done on the next interaction with the page. | ||
| this._resumeAudioBound = this._resumeAudio.bind(this) | ||
| this._audioContext.addEventListener?.('statechange', this._resumeAudioBound) |
There was a problem hiding this comment.
Optional chaining should not be needed here
| this._audioContext.addEventListener?.('statechange', this._resumeAudioBound) | |
| this._audioContext.addEventListener('statechange', this._resumeAudioBound) |
| this._audioContext.resume()?.catch(() => { | ||
| // It will be tried again on the next state change or interaction. | ||
| }) |
There was a problem hiding this comment.
Same should be here. Promise should reject to go into catch; if it's the real case to throw an error here because of function missing - then it's an accepted risk to learn it the hard way later from another report
Also more compact writing would be appreciated =)
| this._audioContext.resume()?.catch(() => { | |
| // It will be tried again on the next state change or interaction. | |
| }) | |
| // Rejected Promise will be tried again on the next state change or interaction. | |
| this._audioContext.resume()?.catch(() => {}) |
| vi.restoreAllMocks() | ||
| }) | ||
|
|
||
| test.each(['pointerup', 'touchend', 'mousedown', 'keydown'])('resumes the audio on %s', (eventName) => { |
There was a problem hiding this comment.
also can include:
| test.each(['pointerup', 'touchend', 'mousedown', 'keydown'])('resumes the audio on %s', (eventName) => { | |
| test.each(['pointerup', 'touchend', 'mousedown', 'keydown', 'visibilitychange'])('resumes the audio on %s', (eventName) => { |
And from what I see, statechange is not covered with test, could you add it as well?
☑️ Resolves
Problem
In Safari and on iOS, the audio of all the participants is mixed with an
AudioContext and played by a single audio element. The browser can
suspend that context, for example on iOS when the audio is interrupted
by a phone call or Siri, or when the screen is locked. Nothing resumed
it afterwards, so no participant could be heard again until the call was
left and joined again.
Solution
the state of the context changes, when the page becomes visible again
and on the next user interaction (in case the browser requires a user
gesture for it).
"touchend" are listened to as well, as "pointerdown" is a user gesture
only with a mouse. The element is also played again right after its
source is replaced, which is done in Safari when a participant is
muted.
Testing
main with all five related fixes applied (vitest src/utils: 29 files,
616 tests).
Nextcloud 34.0.5 server with Talk 24.0.6 and the external signaling
server (2.1.1, Janus 1.4.1), where all media goes through TURN.
server, each time with the unmodified Talk 24.0.6 JavaScript
("original", served to the browser through request interception) and
with this change ("patched"), on the same server and under the same
conditions.
Safari (Talk then mixes the audio as in Safari). The mixing
AudioContext was suspended during a call, original against patched:
interaction.
"pointerup", "touchend", "mousedown" and "keydown", not after the
player was destroyed, and the element is played again after its source
is replaced.
autoplay) joined the call without any user activation, with Chromium
identifying itself as Safari. With the first version of this pull
request (resuming only on "pointerdown") the mixing AudioContext
stayed "suspended" after a full tap; with this version it was
"running" and the element playing right after the tap. A touch-start
alone did not unlock it in either version, as the HTML specification
says.
AI (if applicable)
🖌️ UI Checklist
🏁 Checklist
(Chromium identifying itself as Safari, so Talk uses the Safari audio path; not checked on a real iOS device)