Verification and Validation Report¶
Document Control¶
| Field | Value |
|---|---|
| Document ID | VV-DOT-001 |
| Version | 0.2.501 |
| Date | 2026-03-20 |
| Author | Repository Maintainer |
| Status | Active |
| Classification | Internal |
| Standard | ISO 13485:2016 Sections 7.3.6 / 7.3.7 (adapted) |
Purpose¶
This document constitutes the formal Verification and Validation (V&V) report for the chezmoi-managed dotfiles repository, adapted from ISO 13485:2016 Sections 7.3.6 (Design and Development Verification) and 7.3.7 (Design and Development Validation) for infrastructure-as-code.
Verification confirms that each design output meets its corresponding design input requirement through inspection, analysis, and testing. Validation confirms that the deployed system satisfies its intended use under representative operating conditions (Linux, macOS, WSL).
All verification activities are executed through automated CI pipelines, pre-commit hooks, and the repository test framework. Evidence is retained in CI logs, test output, and this document.
Verification Matrix¶
| Req ID | Requirement | Verification Method | Test File(s) | Result |
|---|---|---|---|---|
| VR-01 | Shell scripts pass ShellCheck (severity=error) | Static analysis via shellcheck --severity=error -e SC1091 -e SC2030 -e SC2031 |
ci.yml, ci-enforced.yml, pre-commit hook |
Pass |
| VR-02 | All scripts have set -euo pipefail |
Pattern scan across all .sh files; CI enforcement |
ci-enforced.yml, compliance-guard.yml |
Pass |
| VR-03 | No secrets in source control (gitleaks clean) | Gitleaks scan with config/gitleaks.toml ruleset |
Pre-commit hook, security-enhanced.yml |
Pass |
| VR-04 | Cross-platform compatibility (BSD + GNU) | Dual-platform CI matrix (ubuntu-latest, macos-latest) | cross-platform-test.yml |
Pass |
| VR-05 | TLS enforcement (no insecure curl/wget) | Pattern blocking for curl -k, --insecure, wget --no-check-certificate |
Pre-commit hook, compliance-guard.yml |
Pass |
| VR-06 | Permission controls (no chmod 777/666) | CI pattern scan blocks world-writable permissions | Pre-commit hook, compliance-guard.yml |
Pass |
| VR-07 | Signed commits enforced | Pre-push hook verifies git verify-commit for all commits in range |
scripts/git-hooks/pre-push, compliance-guard.yml |
Pass |
| VR-08 | Conventional commit format | Pre-commit hook validates feat:, fix:, docs: prefixes |
Pre-commit hook (conventional-commits) | Pass |
| VR-09 | Template rendering correctness | chezmoi apply --dry-run on Linux and macOS CI runners |
ci.yml (Linux + macOS matrix), tests/unit/test_*.sh |
Pass |
| VR-10 | Alias system functional | Unit tests source alias files and verify command availability | tests/unit/aliases/test_*.sh (104 files) |
Pass |
| VR-11 | Neovim config loads without errors | Headless Neovim launch with error detection | tests/unit/neovim/test_*.sh (15 files) |
Pass |
| VR-12 | Fish/Zsh/Nushell configs valid | Syntax validation and functional tests per shell | tests/unit/fish/ (33), tests/unit/shell/ (12), tests/unit/nushell/ (5) |
Pass |
| VR-13 | CI pipelines execute successfully | GitHub Actions workflow status across all triggers | ci.yml, ci-enforced.yml, compliance-guard.yml, cross-platform-test.yml, security-enhanced.yml, codeql.yml |
Pass |
| VR-14 | Pre-commit hooks functional | Hook execution via pre-commit run --all-files |
config/pre-commit-config.yaml |
Pass |
| VR-15 | Age encryption operational | Chezmoi age-encrypted file decryption during apply | chezmoi apply --dry-run (encrypted targets), integration tests |
Pass |
Validation Summary¶
Coverage Metrics¶
| Metric | Value | Threshold | Status |
|---|---|---|---|
| Total test files | 430 | N/A | Measured |
| Total assertions | 2,217 | N/A | Measured |
| Pass rate | 100% | 100% | Pass |
| Module coverage | 100% | >= 95% | Pass |
Category Breakdown¶
| Category | Test Files | Status |
|---|---|---|
| Alias files | 104 | Tested |
| Function files | 76 | Tested |
| Fish functions + conf.d | 33 | Tested |
| Misc / compliance | 34 | Tested |
| Dot CLI | 30 | Tested |
| Diagnostics | 28 | Tested |
| Shell configs | 20 | Tested |
| Ops | 18 | Tested |
| Install | 17 | Tested |
| Security | 16 | Tested |
| Neovim Lua configs | 15 | Tested |
| Theme | 11 | Tested |
| Tools | 7 | Tested |
| Nushell configs | 6 | Tested |
| Secrets | 5 | Tested |
| Integration tests | 10 | Tested |
| Total | 430 | All pass |
Validation Environments¶
| Environment | OS | Shell | Bash Version | Status |
|---|---|---|---|---|
| CI (Linux) | Ubuntu latest | Bash | 5.x | Validated |
| CI (macOS) | macOS latest | Zsh | 3.2 (system) | Validated |
| Local (Linux) | CachyOS / Arch | Fish, Zsh, Bash | 5.x | Validated |
| Local (WSL) | Ubuntu | Bash | 5.x | Validated |
Test Framework Architecture¶
Directory Structure¶
tests/
├── framework/
│ ├── test_runner.sh # Test executor — discovers and runs all test_*.sh files
│ ├── assertions.sh # 16 assertion functions (assert_equals, assert_contains, etc.)
│ └── mocks.sh # Mock utilities for isolating external dependencies
├── unit/
│ ├── aliases/ # Alias and command-surface tests
│ ├── functions/ # Function behavior tests
│ ├── dot-cli/ # `dot` command workflow tests
│ ├── shell/ # Shell integration and completion tests
│ ├── fish/ # Fish function and completion tests
│ ├── nushell/ # Nushell configuration tests
│ ├── nvim/ # Neovim configuration tests
│ └── test_*.sh # 425 unit test files total
├── integration/
│ └── test_*.sh # 11 integration test files
└── performance/
└── benchmark_runner.sh # Performance regression benchmarks
Execution Model¶
test_runner.shrecursively discovers alltest_*.shfiles undertests/.- Each test file is executed in a subshell with
assertions.shsourced. - Tests source bash files directly from the repository (not Go templates).
mocks.shprovides stubs for external tools (chezmoi,git,brew, etc.).- Exit codes and assertion counts are aggregated into a final report.
Assertion Library¶
| Function | Purpose |
|---|---|
assert_equals |
Exact string equality |
assert_not_equals |
String inequality |
assert_contains |
Substring match |
assert_not_contains |
Substring absence |
assert_starts_with |
Prefix match |
assert_ends_with |
Suffix match |
assert_matches |
Regex match |
assert_file_exists |
File presence |
assert_file_not_exists |
File absence |
assert_directory_exists |
Directory presence |
assert_command_exists |
Command availability |
assert_exit_code |
Process exit code |
assert_empty |
Empty string |
assert_not_empty |
Non-empty string |
assert_true |
Boolean truth |
assert_false |
Boolean false |
Traceability Matrix¶
This matrix maps compliance controls from COMPLIANCE.md to verification requirements defined in this document.
SOC 2 Type II Traceability¶
| SOC 2 Control | Description | Verification Requirement(s) |
|---|---|---|
| CC6.1 | Logical Access Security | VR-06 (Permission controls), VR-15 (Age encryption) |
| CC6.8 | Unauthorized Code Protection | VR-07 (Signed commits), VR-03 (No secrets) |
| CC7.1 | System Operations | VR-05 (TLS enforcement), VR-13 (CI pipelines) |
| CC7.2 | Change Detection | VR-14 (Pre-commit hooks), VR-08 (Conventional commits) |
| CC8.1 | Change Management | VR-07 (Signed commits), VR-08 (Conventional commits), VR-13 (CI pipelines) |
| A1.2 | Audit Logs | VR-13 (CI pipelines — logs retained in GitHub Actions) |
ISO 27001 Traceability¶
| ISO Control | Description | Verification Requirement(s) |
|---|---|---|
| A.9.1.1 | Access Control Policy | VR-06 (Permission controls), VR-15 (Age encryption) |
| A.9.4.1 | Information Access Restriction | VR-06 (No chmod 777/666) |
| A.12.1.2 | Change Management | VR-07 (Signed commits), VR-08 (Conventional commits) |
| A.12.6.1 | Technical Vulnerability Management | VR-03 (Gitleaks), VR-05 (TLS enforcement) |
| A.13.1.1 | Network Controls | VR-05 (TLS enforcement) |
| A.14.2.5 | Secure Engineering | VR-01 (ShellCheck), VR-02 (set -euo pipefail), VR-04 (Cross-platform) |
| A.14.2.8 | System Security Testing | VR-09 (Template rendering), VR-10 (Aliases), VR-11 (Neovim), VR-12 (Shell configs) |
GDPR / Privacy Traceability¶
| Requirement | Description | Verification Requirement(s) |
|---|---|---|
| Data Minimization | Limit data collection | VR-03 (No secrets in source) |
| Integrity | Protect against tampering | VR-07 (Signed commits), VR-14 (Pre-commit hooks) |
| Encryption | Protect data at rest | VR-15 (Age encryption operational) |
Acceptance Criteria¶
All of the following criteria must be satisfied for this V&V report to remain valid.
| Criterion | Verification Method | Required Outcome |
|---|---|---|
| All unit and integration tests pass | ./tests/framework/test_runner.sh |
Exit code 0, 0 failures |
| ShellCheck clean | shellcheck --severity=error on all .sh files |
Zero findings |
| Gitleaks clean | gitleaks detect --config config/gitleaks.toml |
Zero findings |
| Chezmoi dry-run clean | chezmoi apply --dry-run on Linux and macOS |
Exit code 0, no errors |
| Pre-commit hooks pass | pre-commit run --all-files |
Exit code 0 |
| Cross-platform CI green | cross-platform-test.yml on ubuntu-latest and macos-latest |
All jobs pass |
| Module coverage at threshold | Test runner coverage report | >= 95% (currently 100%) |
| No world-writable permissions | compliance-guard.yml pattern scan |
Zero chmod 777 or chmod 666 matches |
| No insecure TLS patterns | compliance-guard.yml pattern scan |
Zero curl -k or wget --no-check-certificate matches |
| All commits signed | git verify-commit on HEAD~50..HEAD range |
All commits verified |
Sign-off¶
| Role | Name | Date | Signature |
|---|---|---|---|
| Repository Owner | |||
| Security Reviewer | |||
| Quality Assurance | |||
| Release Manager |
References¶
- ISO 13485:2016 — Medical devices, Quality management systems
- ISO 27001:2022 — Information security management
- SOC 2 Type II — Trust Services Criteria
- SLSA Framework — Supply chain Levels for Software Artifacts
- ShellCheck — Static analysis for shell scripts
- Gitleaks — Secrets detection
- Chezmoi — Dotfiles manager