0057 — The e2e suite runs in duration-balanced CI shards
Status: Accepted
Date: 2026-11-09
Amends: ADR-0008 (its CI job: one
e2e job running the whole suite)
Context and Problem Statement
Section titled “Context and Problem Statement”The e2e job ran all 760 tests on one runner with one Playwright worker and took
about 46 minutes (master run 42508737009: the test step alone 2755 s). Setup was
not the cost: the cached debug build took 15 s and the servers were up 30 s into
the step. The tests were: 45.1 minutes of test time, spread over 65 spec files
from 1 s to 7.2 minutes each.
One worker is a constraint of the suite, not of CI: chartgen’s OAuth store is
process-global, so tests sharing one server must not run concurrently
(fullyParallel: false, workers: 1). Each set of servers can serve only one
test at a time, but nothing stops several runners from each having their own.
How do we cut the wall time of a PR’s e2e run without dropping, skipping or weakening any test?
Decision Drivers
Section titled “Decision Drivers”- Coverage stays exactly as it is: every test runs, with the same assertions, timeouts and retries.
- The single worker per set of servers stays.
- A spec file’s tests stay together, in order, on one set of servers: some share state across tests.
- The check name
e2estays. npm run test:e2elocally does not change.
Considered Options
Section titled “Considered Options”- More Playwright workers on one runner. Needs one set of servers per worker and four projects’ worth of servers each; a 4-core runner would be CPU-bound with that many chartgen, Vite and Chromium processes.
- Playwright’s
--shard=i/non n runners. WithfullyParalleloff it keeps files whole, but balances shards by test count. Seconds per test differ tenfold between files (signin-workspace: 32 tests, 435 s; drawing-modifiers: 77 tests, 331 s), so test count is a poor stand-in for time. - n runners, whole files assigned by measured duration. A committed table of seconds per spec file and a greedy longest-first assignment.
- Removing fixed waits in the specs. The heavy files wait on purpose: the 700 ms after each click keeps two clicks out of klinecharts’ 500 ms double-click window, and the sign-in specs wait out sync intervals, back-offs and slow answers to show what happens, or does not, within them. Not a way to make a large cut without changing what they test.
Decision Outcome
Section titled “Decision Outcome”Chosen: option 3, six shards.
e2e-buildbuilds the debug binary once (cargo build --bin chartgen, the Swatinem cache under the shared keye2e, which the single job used) and uploads it.e2e-shardis a matrix of six,fail-fast: false. Each downloads the binary and runsnpx playwright teston the filesscripts/e2e-shard.mjs <i> <n>prints.CHARTGEN_BINmakesplaywright.config.tsstart that binary instead ofcargo run --, andoauth-stale-client.spec.ts, which starts a chartgen of its own, does the same; without it nothing changes locally. Each shard starts all of the config’s servers, so every project works as before.scripts/e2e-shard.mjslists every*.spec.tsunderweb/tests/e2e, takes each file’s seconds fromscripts/e2e-shard-durations.json(a file not in the table counts as 20 s, so a new spec runs before it is measured), and assigns files longest first to the least-loaded shard. The order is deterministic, so every shard computes the same split. A file’s seconds are its wall time in a shard’s log, from the previous result to its last, so they include its hooks (abeforeAllthat builds the app); a test’s own duration leaves those out. With the durations of the first green sharded run the six shards get 470–473 s each; the longest file alone is 448 s, so a seventh shard would barely help.- Shards write blob reports. The
e2ejob needs all shards, fails unless all passed, and on failure merges the blobs into one HTML report (artifactplaywright-report, as before). Test results are uploaded per shard (playwright-test-results-<i>).
Refresh the table when the split drifts, from every shard’s job log of one run (see the Testing guide). Retries are left out, so a flake does not count against its file.
Consequences
Section titled “Consequences”- Good: a PR’s e2e wall time drops from about 46 minutes to the longest shard plus
per-runner setup. On the implementing PR (run 42630402422) the e2e jobs took
10.1 minutes from the first shard’s start to the end of
e2e(shards 7.7–9.6 minutes), and the whole CI run 12.4 minutes instead of 49.4 on master (run 42508737009). - Good: a failing shard does not stop the others, and the merged report still shows every failure in one place.
- Bad: total runner minutes rise, since each of six runners pays checkout,
npm ci, the Chromium install and server start-up: 54 minutes on that run against 46.5 for the single job. The Chromium install’s--with-depsapt step also runs six times, and one slow mirror once cost a shard 3 minutes. - Bad: the duration table can go stale. A stale table only costs balance, never coverage: every spec file is assigned to exactly one shard regardless.
- Neutral: the floor is the longest spec file (signin-workspace, about 7 minutes). Cutting below it means splitting that file.