Skip to content

Coverage audit: wrong numbers and a report phase that scales badly (tracking) #1062

Description

@Chemaclass

Problem

An audit of the coverage feature found two classes of defect: numbers that are wrong, and a report phase that scales badly. This issue tracks the work and fixes the order, because two of the fixes multiply the cost of the others.

All measurements below: Bash 3.2 arm64, macOS, trap engine, running tests/unit/assert/basic_test.sh against --coverage-paths src.

Numbers that are wrong

finding evidence issue
files that never ran are absent from the report 11 of 121 src/*.sh files reported, denominator 2,200 against a real 9,285 executable lines #1053
--coverage-diff passes a new untested file Total: 0/0 (100%) and exit 0 under --coverage-min 90 #1054
the case-pattern rule drops real statements 546 src/ lines excluded by that rule alone, 236 of them containing $( or = #1055
branch data has no execution counts BRDA reports 0 or 1, documented MVP limit #1061

Cost

run wall
no coverage 0.39s
--coverage, nothing tracked (trap fires, early return) 1.63s
--coverage --coverage-paths src, 11 files tracked 6.42s

Split of the 6.42s: 0.39s test work, roughly 1.24s raw trap dispatch, roughly 1.27s report phase, roughly 3.5s capture bookkeeping.

Report phase alone, synthetic hits, every 3rd line:

tracked files precompute report_text report_lcov default total
11 0.33s 0.03s 0.91s 1.27s
40 1.02s 0.16s 2.66s 3.84s
80 2.19s 0.85s 5.60s 8.64s
121 3.34s 3.12s 9.93s 16.39s

report_text per-file cost grew 6.5x while file count grew 3x. Cause: the stats cache is a string scanned with a leading-* glob, measured at 26.0 ms per lookup at 121 entries. The cache costs more than the computation it replaces.

Order

Performance first, because #1053 raises report input by roughly 11x on this repo. Landing the honest denominator on today's report phase means a 16s report.

  1. perf(coverage): the string-scan caches cost 26ms per lookup at 121 files #1056 kill the quadratic string-scan caches. Cheapest change, largest single win, no behaviour change.
  2. perf(coverage): aggregate the hit data once instead of scanning it per file #1057 aggregate the hit data once instead of scanning it per file. Removes 484 forks and 121 full scans at 121 files.
  3. perf(coverage): classify each file once and share it across report sections #1058 classify each file once and share it. Removes the second and third pass over every source line.
  4. perf(coverage): reject untracked files inside the DEBUG trap #1060 reject untracked files inside the DEBUG trap. Targets the capture bookkeeping, ceiling set by the 1.24s of raw trap dispatch.
  5. fix(coverage): the case-pattern rule drops 236 real statements from the denominator #1055 fix the case-pattern rule. Before the awk port, so only one set of rules gets ported.
  6. perf(coverage): move line classification and LCOV emission into awk #1059 move classification and LCOV emission into awk. Largest remaining cost, largest risk, needs the differential harness.
  7. fix(coverage): files that never ran are missing from the report, so 0% code is invisible #1053 seed the tracked files from the coverage paths. The honest denominator, once the report can afford it.
  8. fix(coverage): --coverage-diff passes a new file that no test executed #1054 compute diff coverage over the changed files. Independent of fix(coverage): files that never ran are missing from the report, so 0% code is invisible #1053 by design, but reads best after it.
  9. feat(coverage): report real per-arm execution counts in BRDA #1061 per-arm branch counts. Data quality, not a wrong number.

Expect the coverage badge to drop when #1053 lands. That is the point of it.

Notes for whoever picks these up

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions