[FSSDK-13413] Run browser tests with Vitest 5 by raju-opti · Pull Request #1186 · optimizely/javascript-sdk

Raju Ahmed (raju-opti) · 2026-11-06T14:46:37Z

Summary

Vitest browser mode is not stable in version 3. It fails frequently in CI, is very slow and also requires hacky modification to files in node_modules. This PR runs the tests using vitest@5 in a separate subproject. The root project stays on Vitest 3 so the Node tests keep running on Node 18.

Add tests/browser, a standalone Vitest 5 project that runs the same browser and UMD spec files from lib/ in real browsers, locally or on BrowserStack. It has its own package.json/lockfile, configs and runner (run.js).
Point CI's browser_tests job at it. The job now uses Node 22 (Vitest 5 needs 22.12+), installs root deps, runs npm ci in tests/browser, then npm run test-browser. Matrix and secrets are unchanged.
Run the UMD tests in CI again (npm run test-umd in tests/browser, as a step in the same job). It builds the UMD bundle and tests it in the same browsers, and runs even if the browser tests fail. The old UMD job was commented out.
Make the backoff_retry_runner cancel test deterministic. It depended on how many microtask ticks an await takes, which differs on Safari 15. It now uses ConstantBackoff(100) and stops the last iteration 10ms short of the retry, so the cancel lands while the retry is waiting on a fake timer. The expected count is still 6.

Changes needed for the same specs to pass on Vitest 5 (details in tests/browser/README.md and comments in vitest.shared.mts):

isolate: true. With shared iframes, a module loaded unmocked by one file stays unmocked for later files under Vitest 5.
browser.importMapMocks: false. Always mock by request interception, as Vitest 3 does and as old browsers require.
clearMocks: false + vitest3-compat.setup.ts. Vitest 5 defaults clearMocks to true, and vi.restoreAllMocks() no longer resets vi.fn() mocks. The specs rely on the Vitest 3 behavior of both.
Polyfills for Array.prototype.at, findLast(Index) and Object.hasOwn, injected into both the orchestrator page (browser.orchestratorScripts) and the test iframe. Vitest 5's browser runtime calls them, and Edge 89/90 and Firefox 91 lack them. Without the polyfills, those runs hang until the connect timeout. lib/ doesn't use these functions, so the polyfills can't hide an SDK issue today.
Oxc target set to the oldest tested browsers instead of es2015. Vite 8 replaces esbuild with Oxc, and Vitest 5's mock modules use top-level await and string export names, which can't be compiled down to ES2015. Dependencies are still compiled to ES2015 via Rolldown.
lightningcss is replaced by an empty local stub (tests/browser/stubs/lightningcss, via overrides). It is MPL-2.0, which license policy doesn't allow. Vite 8 depends on it but only loads it lazily for css.transformer: 'lightningcss' or CSS minification in production builds, neither of which happens under Vitest. If anything ever loads it, the stub throws a clear error. This also removes its 11 platform binaries.
No viewport patch. Vitest 5 falls back to classic WebDriver's setWindowSize.
UMD plugin and console capture use transformIndexHtml. Patching the response doesn't reach the tester page under Vitest 5.
Tunnels start without force: true, with a unique localIdentifier. Runs can go in parallel, one per browser.
Small runner fixes.
- Tunnel startup errors now fail the run with exit 1. Previously they were swallowed and the run exited 0 with no tests run.
- USE_LOCAL_BROWSER is passed to Vitest when credentials are missing.
- Extra arguments are passed through to Vitest (e.g. --reporter=verbose <spec>).
- Set WDIO_LOG_LEVEL to see webdriverio's commands.

Type tests (*.test-d.ts) are not run in this project. They run tsc in Node, and the root npm test already covers them.

Test plan

Local Chrome (USE_LOCAL_BROWSER=true npm run test-browser): 65 files, 1026/1026 tests. Same count as the Vitest 3 browser setup.
Local UMD run (npm run test-umd-local, full rolldown build and test): passes (2/2). The UMD step hasn't run on BrowserStack yet; this PR's CI is its first run.
With the lightningcss stub: local Chrome full suite 1026/1026, UMD 2/2, clean npm ci OK, no MPL packages in the lockfile.
BrowserStack, full suite with Vitest 5:
- Chrome 102 and latest: pass
- Edge 89 and latest: 1026/1026
- Firefox 91 and latest: 1026/1026
- Safari Monterey and Sequoia: the only failure was the backoff_retry_runner cancel test. An earlier version of the fix passed on both. The final version shown in this diff passes in Node and local Chrome under both Vitest 3 and 5, but hasn't been rerun on Safari.
Parallel tunnels: ran Edge and Firefox on BrowserStack at the same time, both passed.
npm ci verified against the committed lockfile in a clean directory.

Issues

FSSDK-13413

🤖 Generated with Claude Code

Coveralls (coveralls) · 2026-11-06T14:49:37Z

coverage: 77.513%. remained the same — raju/vitest5-browser-tests into master

The root project stays on Vitest 3 so the Node tests can keep running on Node 18. tests/browser is a standalone Vitest 5 project that runs the same browser and UMD spec files from lib/ in real browsers (locally or on BrowserStack). CI's browser_tests job now uses it, on Node 22. Adjustments needed for the same specs to pass on Vitest 5 across the BrowserStack matrix: - isolate: true, and mocks always by request interception (browser.importMapMocks: false) - clearMocks: false plus a setup file restoring Vitest 3's vi.restoreAllMocks() behavior - polyfills for Array.prototype.at, findLast(Index) and Object.hasOwn in the orchestrator page and test iframe (Vitest 5's browser runtime needs them; Edge 89/90 and Firefox 91 lack them, which made runs hang) - Oxc target set to the oldest tested browsers instead of es2015 - no viewport patch needed - tunnels started without force: true, so runs can go in parallel Also makes the backoff_retry_runner cancel test deterministic: it depended on how many microtask ticks an await takes, which differs on Safari 15. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Adds a UMD step to the browser_tests job: builds the UMD bundle and tests it in the same BrowserStack browsers. Runs even if the browser tests fail, so both results are reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

Shell-based argument forwarding can alter literal test filters, and the declared Node support range conflicts with the locked WebdriverIO dependencies’ requirements.

2 open issues

What changed in this PR

Moves browser testing into a standalone Vitest 5 project while preserving the root Vitest 3 setup for Node 18 compatibility.

Changes:

Adds browser and UMD configurations, compatibility helpers, and a shared runner for local browsers and BrowserStack.
Updates browser CI to Node 22 and restores UMD testing.
Makes retry-cancellation testing independent of browser microtask timing.

File	Description
`tests/browser/vitest3-compat.setup.ts`	Preserves mock-reset behavior expected by existing specs.
`tests/browser/vitest.umd.config.mts`	Configures UMD spec execution.
`tests/browser/vitest.shared.mts`	Shares browser connections, transforms, and compatibility settings.
`tests/browser/vitest.browser.config.mts`	Selects browser specs and optional logging plugins.
`tests/browser/umd-plugin.ts`	Injects the UMD bundle into test pages.
`tests/browser/run.js`	Runs browser matrices, manages tunnels, and builds UMD bundles.
`tests/browser/README.md`	Documents setup, usage, and compatibility differences.
`tests/browser/package.json`	Defines standalone dependencies and test commands.
`tests/browser/package-lock.json`	Locks the browser project's dependency tree.
`tests/browser/legacy-polyfills-plugin.ts`	Supplies runtime polyfills for older browsers.
`tests/browser/console-capture-plugin.ts`	Adapts console capture to HTML transformation hooks.
`lib/utils/executor/backoff_retry_runner.spec.ts`	Uses timed backoff for deterministic cancellation.
`.github/workflows/javascript.yml`	Switches browser CI to the new project and enables UMD tests.

_{Review effort: Balanced}

+  "private": true,
+  "description": "Runs the SDK browser and UMD spec files (../../lib) in real browsers using Vitest 5. Kept separate from the root project, which stays on Vitest 3 for Node 18 support.",
+  "engines": {
+    "node": "^22.12.0 || ^24.0.0 || >=26.0.0"


+      };
+
+      try {
+        execSync(`npx vitest run --config ${mode.config} ${extraVitestArgs}`, {


Vite 8 depends on lightningcss, which is MPL-2.0 and not allowed by our license policy. Vite only loads it lazily for css.transformer 'lightningcss' or CSS minification in production builds; neither happens when Vitest runs the dev server. An npm override points it at a local stub that throws if anything ever loads it, which also drops the 11 platform binaries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- run.js starts Vitest with execFileSync (node + Vitest's CLI script) instead of execSync through a shell, so extra arguments reach Vitest unchanged. JSON-quoting didn't stop the shell from expanding $ and backticks inside double quotes. - engines now match the locked webdriverio packages (^22.23.4 || ^24.22.1 || >=26.12.1); README and the CI comment updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Md Junaed Hossain (junaed-optimizely)

Looks solid overall.
One issue is that the PR description says one reason for this change is to stop patching node_modules. But the root test-browser / test-umd scripts, scripts/patch-vitest-viewport.js, the root browser/UMD vitest configs and the root devDeps (@vitest/browser, webdriverio, browserstack-local) are all still there.

BROWSER_CONFIGS is now duplicated between scripts/run-browser-tests.js and tests/browser/run.js. Is the plan to remove the root setup in a follow-up? If so, could we link a ticket? Otherwise the two matrices will drift.

!cancelled() alone also ran the UMD step after a failed install, adding a second, unrelated failure. Gate it on the browser-deps install step, which is itself skipped if the SDK install fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This comment has been minimized.

Sign in to view