Skip to content

v0.68.0 — Test Health, part 2

v0.67.0 made apps/web's 8,635 tests run. It did not make them mean anything, and it said so: findings 4–8 were left untouched, needing the spec the roadmap asked for. This is that work, and the spec is the findings themselves — each one is a measurement with a number attached, and every number below was re-taken against v0.67.0 before a task was written from it.

Finding 4 is the thesis and the other four are its symptoms. 708 of 1,038 test files in this repo read source code off disk and assert a regex against it. That style earns its place — it is how v0.66.0 could assert "nothing renders sourcePath" across the entire tree, which no unit test can do — and it has one failure mode: it fails towards passing. A unit test whose subject moves throws. A regex whose subject moves matches nothing, and matching nothing is indistinguishable from having nothing to complain about.

Everything in this release is a way of removing that asymmetry:

  • Findings 4 + 5 — 133 fixed-window slices (src.slice(idx, idx + 800)) in 57 files, each of which silently stops covering its target the moment anyone adds a comment above the fields at the bottom. v0.66.0 broke three this way in four tasks. Bound each slice by its own closing delimiter and the window can no longer drift off the end.
  • Finding 6 — apps/web's lint is eslint src/ || true, swallowing 8 real errors in 7 files, while repo-wide pnpm lint emits 1,736 warning lines. CLAUDE.md has to tell every contributor which command is the real gate, which is the definition of a broken signal.
  • Finding 7 — e2e/ and playwright-report/ commit a passing report for a suite that does not exist. No Playwright dependency, no config, no specs — just .last-run.json reading "status": "passed" and a screenshot, and TODO.md marking the work [x].
  • Finding 8 — pnpm test:coverage exits 1 today. Two packages fail their own thresholds; the two that pass measure one file out of 163 and two component directories out of ten respectively.

The last task is the one that makes the rest durable: everything above was introduced by someone who ran nothing before pushing, and CI only speaks after the fact.

Bound the guards (findings 4 and 5)

  • [x] 1. Add a declarationSlice() helper for source-text guards, with its own tests. Replaces src.slice(idx, idx + N) with a slice bounded by the construct's own closing delimiter — balanced-brace scan from the opening {/(/[ after the match, with an explicit throw when the construct is unbalanced or absent rather than a silent empty string. A guard whose target has been deleted must fail loudly; that is the entire point of the release.
  • [x] 2. Convert the fixed-window slices in apps/web components (batch 1 of 2). Expect real failures on conversion — a correctly bounded window sees code the fixed one had drifted past. Each is triaged the way v0.67.0 triaged its 44: re-point the assertion at the contract it was written to protect, or fix the code if the contract is still right and the code broke it.
  • [x] 3. Convert the fixed-window slices in apps/web hooks, stores and utils (batch 2).
  • [x] 4. Convert the fixed-window slices in packages/.
  • [x] 5. Add a guard against new fixed-window slices. (Done, and stricter than planned. The task expected a ratchet on the systemLiterals model, reasoning that the pattern is legitimate in places and a hard ban would be a lie. After tasks 2–4 the count in test files is zero, and a ratchet with a baseline of zero is a ban with extra machinery — so it is a flat ban, scoped to test files. That scoping is what makes it honest: the surviving uses are in sandbox/tokenizer.ts, reading \xNN and \uNNNN escapes as slice(i, i + 2) and slice(i, i + 4), where the width is part of the grammar rather than a guess. The guard also asserts it scanned >900 files, because a guard that walks the wrong directory reports zero violations and looks exactly like success.)

Make the signals readable (finding 6)

  • [x] 6. Fix apps/web's 8 lint errors and delete the || true. One react-hooks/rules-of-hooks violation in AttributeGroupSection.tsx (a conditional useState — a real bug, not a style nit), three unused declarations, three no-regex-spaces, one no-useless-escape.
  • [x] 7. Make pnpm lint the real gate, and retire CLAUDE.md's workaround. 1,736 warning lines is why the instructions have to name a different command; the fix is for the default command to be the one worth running.

Stop reporting on a suite that does not exist (finding 7)

  • [x] 8. Delete the fabricated e2e artifacts and correct TODO.md. e2e/test-results/ and playwright-report/ are 888 KB of committed output from a suite with no source. TODO.md's [x] on "Implement integration/end-to-end tests for VTT features" becomes [ ]. This lands before task 9 so the record is honest even if 9 slips.
  • [x] 9. Add a real Playwright smoke suite, unauthenticated paths only. Config, dependency, and specs that assert what can be asserted without Clerk credentials: the app boots, an unauthenticated visit to a campaign route redirects, and the SPA shell renders. Scoped deliberately small — an e2e suite that needs secrets to run locally is one nobody runs.

Make coverage mean something (finding 8)

  • [x] 10. Make the coverage scope honest and ratchet the thresholds to measured reality. (Done. Widening the scope turned two "passing" packages into the numbers they actually earn: server-core 80% on one file of 163 → 52.84% on the package, and ui-components 80% on two component directories of ten → 35.58%. shared (78.78) and client-sdk (66.07) were already honest and already failing. Every threshold is now the measured value floored to an integer, may only be revised upward, and says so. apps/web gets coverage for the first time, and its number is the release's thesis in one figure: 8,640 tests execute 19.69% of statements. 591 of its 803 test files read a .tsx off disk and assert a regex against the text, and readFileSync executes nothing — a guard can pin a contract exactly and contribute zero coverage. The floor is set at 19 so the ratio can only improve. The root vitest.config.ts's 80% is deleted rather than lowered: pnpm test:coverage is pnpm -r, so that block was never the config in play and could not fire.)
  • [x] 11. Wire test:coverage into CI. (Done, replacing the pnpm test step rather than joining it — it runs the same suites plus the ratchets, so running both would only double the wall clock. It turned up a trap on the way in: pnpm -r silently skips a package that lacks the script, and create-purple-cat, apps/help and scripts/setup all had test and no test:coverage. Swapping the commands would have stopped running three suites while the step went greener than before. All three now have one, and a coverageScriptParity guard fails the build if the gap reopens.)

Close the loop (the adjacent TODO)

  • [x] 12. Add a pre-push verification script and hook. (Done. pnpm verify runs CI's gate in CI's order — lint, then build:packages (which is tsc per package, so it is also the typecheck), then test:coverage — in 76 seconds. --quick skips coverage instrumentation, --e2e adds the browser suite. apps/web is deliberately not typechecked: it carries 192 pre-existing tsc errors and CI does not gate them either, since build is vite build. Including it would make pnpm verify a command that always fails, which is a command nobody runs — the same mistake as the 1,736-line lint output. The hook is opt-in (pnpm hooks:install, via core.hooksPath) and nothing installs it for you, because a hook somebody did not choose is one they disable at the first inconvenient moment — and a disabled hook is worse than none, since it still looks installed. SKIP_VERIFY=1, VERIFY_QUICK=1 and --no-verify are the escape hatches. Both paths were exercised: a planted lint error made it exit 1 naming the failed step, and the installer round-trips cleanly.)