Performance Budgets

The tiers

Reference baselines (2026-08-30, rousseau-my-workstation, Ryzen AI 9 HX 370)

All medians in milliseconds. Budget = 2× median (or the tier ceiling, whichever is larger).

INSTANT tier

FAST tier

Currently empty.

dot status and dot diff were placed here at 2000ms against reference medians of 1510 ms and 1420 ms — 1.3× and 1.4× headroom, short of the ≥ 2× this document requires of every budget. They have since moved to MEDIUM; see that section for the measurements that prompted it.

The baselines of 28 ms and 32 ms recorded before those were not real. The perf sandbox never created chezmoi's source directory, so both commands aborted immediately with "no such file or directory" — and _measure discarded exit codes, so the gate timed the failure path and called it excellent. The sandbox now links the repo into XDG_DATA_HOME.

The QA gates and test suites that used to sit in this tier moved to GATES below: they are not interactive operations, so a ceiling defined by human-perceived latency never described them.

MEDIUM tier

The dot status and dot diff baselines are the worst of the two hosted runners, not the reference machine: macOS measured 2290 ms / 2157 ms and Ubuntu 2329 ms / 2136 ms, against 1139 ms / 1174 ms locally. Two unrelated platforms agreeing within 8% is a property of the commands rather than of one slow runner — both shell out to chezmoi, which walks the whole source tree, and that is disk-bound. A budget taken from the faster machine would have been a gate that only ever fired on other people's hardware.

dot doctor and bench.sh --quick are diagnostics that report findings through their exit status — dot doctor exits 1 whenever it finds issues, and bench.sh exits 1 when a shell breaches its own startup threshold. They are gated with _gate_diag, which permits exit 1 but still fails on 2+ (not-found, permission, signal, syntax error). That is far narrower than the blanket || true it replaced.

Exit 1 therefore has to mean a breach and nothing else. bench.sh exits 2 when it measured nothing (hyperfine or jq missing, or a hyperfine run with no reading); before that, a runner without hyperfine exited 1 and the gate reported a 2ms pass. CI installs both tools for the performance job, and a machine without them gets a visible SKIPPED, never a pass.

GATES tier (CI quality gates and test suites)

Budgets are 2× the median measured on the slowest supported platform, not the fastest. Medians below: rousseau-mbp-m1, macOS 26 (Darwin 25.6), 2026-08-30.

† The manifest test hashes an entire sandbox tree before and after every scenario, so it is disk-bound in a way the other gates are not. Its budget is set at 3.2× the macOS median rather than the 2.0× used above, deliberately: dot status and dot diff were first budgeted at 1.3× and 1.4×, and both breached on the first CI run that measured them. The Linux median is left unfilled until CI reports one — guessing it would defeat the point of a table of measurements.

These gates run 3–6× slower on macOS than on Linux — fork/exec is markedly more expensive there and every one of them is fork-heavy shell. CI covers both platforms, so the original Linux-only calibration could not hold, and five of these sat red as a result. Re-capture the Linux column when convenient; it is carried over from the 2026-08-30 Ryzen baseline and is not the binding constraint.

ACCEPTED-SLOW (documented, not gated per-run)

OUT-OF-SCOPE (not gated per-run)

Ratchet vs aspiration

The budgets above are regression gates, not aspirations. If a real optimisation lowers a baseline, edit the doc + the perf test to lower the budget too. If a change pushes something over the budget, the test fails and CI blocks the merge.

The aspirational shell-startup target (<30ms) is tracked separately in benches/bench.sh — that's a bench, not a budget.

Adding a new operation

When you add a new script that runs at a shell prompt:

1. Time it 5 runs on a warm system: for _ in {1..5}; do time bash your-script; done
2. Take the median.
3. Place it in the tier where budget ≥ 2 × median. If a median is 400ms it goes in FAST (500ms is uncomfortably tight); if it's 300ms, INSTANT is fine.
4. Add it to benches/test_perf_budgets.sh in the correct tier section.
5. Add its baseline to this doc.

Where the enforcement lives

• Per-op budget test: benches/test_perf_budgets.sh
• Suite-level wall-clock ratchet: benches/test_help_gates_wall_clock.sh
• CI wiring: .github/workflows/ci.yml, job quality-performance — runs on ubuntu-latest and macos-latest for every PR, with no || true and no budget scaling beyond the runner factor below. On the gates the hosted macOS runner is comparable to or faster than the reference machine — docs-coverage 679 ms vs 969, traceability 1775 vs 2389, help-registry 3318 vs 4449, dot doctor 1746 vs 4423. The exception is anything that drives chezmoi over the whole source tree: dot status and dot diff measured ~2× the local median on both hosted platforms, which is why they sit in MEDIUM rather than FAST. Budget a new operation against the slowest platform it will run on, not against this machine.

How the time is measured

_measure uses the time keyword with TIMEFORMAT='%3R', not date +%s%N.

%N is a GNU extension. BSD date — macOS 14 and earlier — copies the literal N through, so every arithmetic conversion failed, _measure returned nothing, and an empty median compares as 0 against any budget. The gate reported every budget met on those machines, for any command, including one that could not parse. time is a shell builtin, is millisecond-accurate in bash 3.2, and needs no external clock at all.

Two consequences worth keeping:

• _gate_max_rc rejects a non-numeric median outright rather than comparing it. Both fields go through -gt / -le, where bash reads a non-numeric operand as 0 — so a gate that cannot measure would otherwise report success.
• tests/regression/test_gate_integrity.sh puts a date without %N first on PATH and requires the same verdicts. A future timing rewrite that reintroduces the dependency fails there rather than going quiet.

Runner calibration

Every gate is fork/exec-heavy shell, so its wall clock tracks the runner's process-spawn speed as much as the code. On shared macos-latest runners the same code measured 3109–10149 ms (test_dot_subcommand_smoke.sh) and 3433–10293 ms (test_dot_help_registry_symmetry.sh) over 2026-10-22/23, and taking five samples instead of three did not stop them failing main.

So each sample of each gate is paired with a sample of a fixed reference workload, timed just before it: 30 starts of the bash the gates resolve from PATH, each running one command substitution. The median per-spawn cost of those samples, divided by the committed reference (CALIBRATION_REF_US, the same workload on the reference machine), is the gate's runner factor, and its budget is multiplied by max(1, factor):

• Paired per sample, not measured once: the red runs had one gate over budget while its neighbour passed, so the runner slowed down mid-run.
• Never below 1×: a runner faster than the reference gets the budgets in the tables above, no tighter. Linux spawns cost a fraction of the macOS reference, so Linux runs at exactly those budgets.
• Capped at 4×: beyond it the gate fails with runner=…x … exceeds the 4.00x cap — this runner is too slow to gate on, rather than scaling every budget until nothing can fail.
• All wall-clock gates use it: they share the runner and are all spawn-heavy. The factor moves only when the reference workload slows down, so a change that slows one gated command past its headroom still fails.

Every gate line prints the factor it used:

✓ gate_dot_subcommand_smoke: median=4012ms budget=7500ms (runner=1.00x per_spawn=5400us reference=5800us)

The reference (5800 µs per spawn) is the median of 7 runs of PERF_GATE_FILTER=instant_dot_version on an Apple A18 Pro MacBook (8 GB, macOS 26.6, Homebrew bash 5.3.20 first on PATH), 2026-10-23. Refresh it on the machine the budgets are measured on, in the same PR as any budget change:

for i in 1 2 3 4 5 6 7; do
  PERF_GATE_FILTER=instant_dot_version bash benches/test_perf_budgets.sh |
    grep -o 'per_spawn=[0-9]*'
done   # take the median

tests/regression/test_perf_gate_calibration.sh proves the three properties against the real gate: a simulated slow runner (a bash on PATH that pays for an extra bash start every time) breaks the unscaled budget but passes the calibrated one; a gated command made 3× costlier on that runner still fails; and a runner 16× slower than the reference fails on the cap.

Environment knobs

When a budget fires

The error looks like:

✗ instant_dot_help: median=612ms EXCEEDS budget=500ms (runner=1.00x per_spawn=5400us reference=5800us)

Steps:

1. Read the runner= factor. A cap failure means the runner, not the code, was too slow; re-run. Otherwise the budget was already scaled for the runner.
2. Bisect the change that pushed it over.
3. Fix the regression, or
4. If the increase is legitimate (real new work), move the operation to the next tier + update this doc + test_perf_budgets.sh in the same PR.

Never silently bump the budget. The tier a thing lives in is a promise to users.