.dotfiles

Operations

Testing strategy

Overview

The repo uses a multi-layer testing approach: unit tests for individual functions, integration tests for system-wide behavior, and performance benchmarks for resource efficiency.

Quick start

# Run all unit tests
./tests/framework/test_runner.sh

# Run a specific test suite
./tests/framework/test_runner.sh extract

# Run integration tests
RUN_INTEGRATION=1 ./tests/framework/test_runner.sh

# Run performance benchmarks
./benches/benchmark_runner.sh

# Run unit tests in parallel (parsed per-file output, deterministic order)
./tests/framework/test_runner.sh --jobs auto

# Strict mode: promote silent `command not found` / `unbound variable`
# inside cov_exercise_functions_file to test failures. Default is
# tolerant (the helper sources dot command files in a clean shell where
# their lib/dot/ helpers are intentionally unresolved). Use STRICT=1
# locally before pushing to catch the class of bug that escaped
# review pre-v0.2.503 (e.g. agent.sh: _agent_repo_root falling
# through to a missing require_source_dir).
DOT_STRICT=1 ./tests/framework/test_runner.sh

The runner's FINAL SUMMARY lists which test files failed (not just the total count) so locating a regression in a 4000+ assertion run doesn't require grepping back through the full log. Per-file failure counts and crashed-file markers appear inline.

Test structure

tests/
├── framework/           # Test framework
│   ├── assertions.sh    # 16 assertion functions
│   ├── mocks.sh        # Mock utilities
│   └── test_runner.sh  # Test executor
├── unit/               # 425 unit test files
├── integration/        # 11 integration test files
└── performance/        # Benchmarks

Writing tests

Test file template

#!/usr/bin/env bash
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../framework/assertions.sh"

# Source the function under test
source "$HOME/.dotfiles/.chezmoitemplates/functions/myfunction.sh"

# Test cases
test_start "function_basic_usage"
assert_exit_code 0 "myfunction arg1"

test_start "function_no_args"
assert_exit_code 1 "myfunction"

Available assertions

Function Description
assert_equals expected actual [msg] Two values are equal
assert_not_equals unexpected actual [msg] Two values differ
assert_exit_code code command Command exits with given code
assert_output_contains needle command Output contains string
assert_output_not_contains needle command Output lacks string
assert_output_matches pattern command Output matches regex
assert_file_exists path [msg] File exists
assert_file_not_exists path [msg] File doesn't exist
assert_dir_exists path [msg] Directory exists
assert_dir_not_exists path [msg] Directory doesn't exist
assert_true condition [msg] Condition is true
assert_false condition [msg] Condition is false
assert_empty value [msg] String is empty
assert_not_empty value [msg] String isn't empty
assert_file_contains file needle [msg] File contains text
assert_file_not_contains file needle [msg] File lacks text

Mock utilities

Function Description
mock_init Initialize mock environment
mock_command name output [exit_code] Create a mock command
mock_command_spy name [output] [exit_code] Create a mock that records calls
mock_get_calls name Get spy call history
mock_call_count name Get number of spy calls
mock_file content [filename] Create temp file with content
mock_dir [prefix] Create temp directory
mock_archive type [content] Create a mock archive file
mock_env var_name value Set an environment variable
mock_cleanup Clean up all mocks

Test categories

Unit tests (tests/unit/)

Test individual functions in isolation. Each file follows the test_*.sh naming convention and is discoverable with ls tests/unit/.

Integration tests (tests/integration/)

Test complete workflows like the installation script and end-to-end apply behavior.

Performance tests (benches/)

Measure resource efficiency with shell startup benchmarks and load tests.

Coverage goals

Category Target Current
Module coverage >=95% 100%
Unit test files - 425
Integration test files - 11
Total test files - 436
Named tests (test_start) - 2149
Unit test pass rate 100% 100%

CI enforces module coverage via:

MIN_COVERAGE=95 ./tests/framework/module_coverage.sh

For a current local baseline, run:

bash ./scripts/qa/coverage-baseline.sh --with-module-coverage

For core internal behavior traceability, run:

bash ./scripts/qa/traceability-coverage.sh

Mutation score

Line coverage says a line ran. The mutation score says whether the suite would notice if that line were wrong. tools/ci/mutation-test.py plants one small, plausible bug per eligible line (a dropped regex anchor, a flipped exit code, a swapped comparison), runs the behavioural tests that exercise the file, and counts a mutant as killed only if one of them fails.

# What CI runs on a pull request: only the lines the PR changed
python3 tools/ci/mutation-test.py --base origin/main --min-score 80

# Every eligible line of one file
python3 tools/ci/mutation-test.py --full scripts/ops/rollback.sh

# See the mutants without running anything
python3 tools/ci/mutation-test.py --list-mutants --full scripts/ops/rollback.sh

Rules that keep the score honest:

  • Kills only count from behavioural tests. A test that greps or lints the source text would "kill" any edit, so it declares # test-kind: structural in its first 30 lines and its kills are ignored. # test-kind: structural except scripts/x.sh keeps the kills on a source the test does run.
  • A test that fails on the unmutated tree is dropped and reported.
  • A mutant with no related test counts as survived.
  • A line that cannot be meaningfully mutated is opted out in place with # mutation: ignore <reason>; the reason is mandatory.

A survivor means the behaviour on that line is unprotected. Kill it with a test that pins the observable outcome (exit status, message, file written or not), never with a source grep. tests/unit/ci/test_mutation_engine.sh checks the engine itself fails a weak suite.

Tests must run the code

assert_file_contains "$SCRIPT" "main()" proves the text is there, not that the behaviour holds: it passes with a guard inverted and fails when a line is reworded. tools/ci/check-source-grep-tests.py finds assertions whose operand is a repository source file (a text tool or a file-content assertion on a path under scripts/, lib/, bin/, defaults/, install.sh and so on, directly, through a variable, or inside an assignment's command substitution such as first=$(head -n 1 "$SCRIPT")).

python3 tools/ci/check-source-grep-tests.py            # what CI runs
python3 tools/ci/check-source-grep-tests.py --report   # counts only
python3 tools/ci/check-source-grep-tests.py --write-baseline

The suite carries hundreds of these from before the mutation gate, so the lint is a ratchet. tools/ci/source-grep-baseline.txt records the ceiling per file; a new file or a file above its ceiling fails CI. When you rewrite a case to run the code, rerun --write-baseline so the ceiling drops. A test whose purpose is the source text (a lint, a naming rule, a header check) declares # test-kind: structural and is skipped.

Tests run automatically on every push to main, every pull request, and weekly scheduled runs (Monday 6 AM UTC).

GitHub Actions example

- name: Run Tests
  run: |
    chmod +x ./tests/framework/test_runner.sh
    ./tests/framework/test_runner.sh

- name: Run Integration Tests
  run: |
    RUN_INTEGRATION=1 ./tests/framework/test_runner.sh

- name: Run Performance Benchmarks
  run: |
    ./benches/benchmark_runner.sh

Environment variables

Variable Default Description
TEST_FILE_TIMEOUT 900 Seconds a single test file may run before the runner kills it, names it as timed out and counts a failure (0 disables)
RUN_INTEGRATION 0 Set to 1 to include integration tests
VERBOSE 0 Set to 1 for verbose output
REPO_ROOT Auto-detected Repository root directory
TESTS_DIR Auto-detected Tests directory

Best practices

  1. Isolation -- Each test should be independent and self-contained.
  2. Cleanup -- Use mocks that auto-cleanup via traps.
  3. No sleep -- Avoid sleep or hardcoded delays.
  4. Descriptive names -- Test names should describe what's being verified.
  5. Edge cases -- Test error conditions, not just happy paths.
  6. Security -- Include tests for dangerous input rejection.

Troubleshooting

Tests not found

Make sure test files match the test_*.sh pattern and are executable:

chmod +x tests/unit/test_*.sh
chmod +x tests/integration/test_*.sh

Function not available

If a test reports "function not available", verify the source file exists:

ls -la .chezmoitemplates/functions/

Mock cleanup issues

If mocks aren't cleaning up, make sure you aren't running with set -e before mock operations that might intentionally fail.