fix(test, frontend): raise Vitest timeouts for CI stalls - #7623
fix(test, frontend): raise Vitest timeouts for CI stalls#7623aglinxinyuan wants to merge 1 commit into
Conversation
The macOS leg of `build / frontend` goes red on a different unit test every few days -- always a timeout, never the same spec, always green on rerun. Three occurrences in the last four days: | Run | Test | Error | | --- | --- | --- | | 31665399757 | UserDatasetVersionCreatorComponent > onClickCreate ... | Test timed out in 5000ms | | 31630884042 | AdminUserComponent > sortByAffiliation ... | Hook timed out in 10000ms | | 31411656559 | WorkflowRuntimeStatisticsComponent > should create | Test timed out in 5000ms | The tests are not the problem: the runner stalls, and the stall lands on whichever test is executing. In run 31665399757 the offending spec file took 11727ms on macos-latest and 240ms on ubuntu-latest for the same commit; in an earlier run the same file took 219ms on macOS. Suite totals from that run show the same picture -- 252.88s wall on macOS vs 89.85s on ubuntu, with a cumulative test time of 307.69s vs 182.34s. macos-latest gives 3 cores and 7 GB against ubuntu's 4 and 16, so the jsdom + v8-coverage workers run under real memory pressure there. Raise testTimeout and hookTimeout to 30s in both Vitest configs, which absorbs a stall an order of magnitude worse than any observed so far. A spec that legitimately needs 30s is broken, and the job's own timeout still bounds a true hang. Per-test timeouts would be whack-a-mole: the next stall picks a different test. Also opt the frontend matrix out of fail-fast, as every other multi-leg matrix in build.yml already does. Today one flaky OS cancels the other two legs, which destroys exactly the evidence needed to tell a runner flake from a real break. Before: macOS stalls 5s -> that test fails -> ubuntu + windows cancelled After: macOS stalls 5s -> absorbed; a real break still fails all legs
Backport auto-label reportThis
|
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7623 +/- ##
============================================
+ Coverage 88.96% 89.00% +0.03%
- Complexity 4338 4350 +12
============================================
Files 1178 1178
Lines 46835 46836 +1
Branches 5226 5226
============================================
+ Hits 41667 41685 +18
+ Misses 3422 3413 -9
+ Partials 1746 1738 -8
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 434 | 0.265 | 21,648/32,127/32,127 us | 🔴 -5.0% / 🔴 +97.0% |
| 🔴 | bs=100 sw=10 sl=64 | 939 | 0.573 | 104,692/159,560/159,560 us | 🔴 +17.3% / 🔴 +47.5% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,084 | 0.662 | 925,470/1,003,573/1,003,573 us | ⚪ within ±5% / 🟢 -7.1% |
Baseline details
Latest main 310ab88 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 434 tuples/sec | 457 tuples/sec | 775.33 tuples/sec | -5.0% | -44.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.265 MB/s | 0.279 MB/s | 0.473 MB/s | -5.0% | -44.0% |
| bs=10 sw=10 sl=64 | p50 | 21,648 us | 22,627 us | 12,743 us | -4.3% | +69.9% |
| bs=10 sw=10 sl=64 | p95 | 32,127 us | 31,747 us | 16,310 us | +1.2% | +97.0% |
| bs=10 sw=10 sl=64 | p99 | 32,127 us | 31,747 us | 18,926 us | +1.2% | +69.7% |
| bs=100 sw=10 sl=64 | throughput | 939 tuples/sec | 973 tuples/sec | 1,001 tuples/sec | -3.5% | -6.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.573 MB/s | 0.594 MB/s | 0.611 MB/s | -3.5% | -6.2% |
| bs=100 sw=10 sl=64 | p50 | 104,692 us | 101,825 us | 101,399 us | +2.8% | +3.2% |
| bs=100 sw=10 sl=64 | p95 | 159,560 us | 136,023 us | 108,206 us | +17.3% | +47.5% |
| bs=100 sw=10 sl=64 | p99 | 159,560 us | 136,023 us | 118,195 us | +17.3% | +35.0% |
| bs=1000 sw=10 sl=64 | throughput | 1,084 tuples/sec | 1,110 tuples/sec | 1,026 tuples/sec | -2.3% | +5.6% |
| bs=1000 sw=10 sl=64 | MB/s | 0.662 MB/s | 0.677 MB/s | 0.626 MB/s | -2.2% | +5.7% |
| bs=1000 sw=10 sl=64 | p50 | 925,470 us | 901,558 us | 996,304 us | +2.7% | -7.1% |
| bs=1000 sw=10 sl=64 | p95 | 1,003,573 us | 995,220 us | 1,042,531 us | +0.8% | -3.7% |
| bs=1000 sw=10 sl=64 | p99 | 1,003,573 us | 995,220 us | 1,074,934 us | +0.8% | -6.6% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,461.15,200,128000,434,0.265,21647.74,32127.13,32127.13
1,100,10,64,20,2131.01,2000,1280000,939,0.573,104691.64,159559.51,159559.51
2,1000,10,64,20,18447.26,20000,12800000,1084,0.662,925470.16,1003572.54,1003572.54
What changes were proposed in this PR?
The macOS leg of
build / frontendgoes red on a different unit test every few days — always a timeout, never the same spec, always green on rerun. Three occurrences in the last four days:UserDatasetVersionCreatorComponent > onClickCreate creates a dataset with a sanitized name …Test timed out in 5000msAdminUserComponent > sortByAffiliation compares affiliations …Hook timed out in 10000msWorkflowRuntimeStatisticsComponent > should createTest timed out in 5000msRoot cause: the runner stalls, not the test. The stall lands on whichever test happens to be executing. From run
31665399757— one commit, one matrix, two OSes:That file is not systematically slow — it took 219 ms on macOS in an earlier run, and 443 ms locally. The 11.7 s is a stall. jsdom + v8-coverage workers on 3 cores / 7 GB run under real memory pressure, which is where multi-second pauses come from.
testTimeout5s → 30s,hookTimeout10s → 30sfrontend/vitest.config.ts,frontend/vitest.browser.config.tsfail-fast: falseon the frontend matrix.github/workflows/build.ymlfrontend/TESTING.mdBumping the one test's timeout would be whack-a-mole — the next stall picks a different spec. 30 s absorbs a stall an order of magnitude worse than any observed; a spec that legitimately needs 30 s is broken, and the job's own timeout still bounds a true hang.
The
fail-fastopt-out matches every other multi-leg matrix inbuild.yml(platform,platform-integration,agent-service,infra). Today one flaky OS cancels the other two legs, so the run no longer says whether the failure reproduces off that OS — exactly the evidence needed to tell a runner flake from a real break.If macOS keeps flaking after this, the next lever is capping
maxWorkerson that leg to cut memory pressure. Left out here: it trades wall clock for stability and can't be measured from a non-macOS box.Any related issues, documentation, discussions?
Related to #6073 — the same failure mode on the Windows leg, which already proposes both of these remedies. Left open and assigned; the three new macOS occurrences are recorded there.
How was this PR tested?
No production code is touched; the change is to the test harness and CI config.
yarn test:ci(full jsdom suite, new config)beforeEach+ 8 s body passes (14 026 ms); the 8 s body fails on the old 5 s default. Removed before commitbuild.ymlparsesjs-yamlload →jobs.frontend.strategy['fail-fast'] === falseprettier-eslint --list-differentcleanThe flake itself can't be reproduced on demand — that's the nature of a runner stall. What this PR asserts is verifiable: the ceiling that the stalls blow past is now 6× higher, and the surviving matrix legs still report their own results.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)