# 0062 — End-to-end proof: the `tests-e2e` crate, what is real and what is simulated, demos

- Status: accepted (implements ARCHITECTURE v5 §13.4, §14.3, §14.5, §14.6 and the owner overrides of ADR 0056 §3/§1 for the e2e level; one deviation from §14.6 placement, §1)
- Date: 2026-10-23
- Area: tooling

## Context

§14.6 asks for end-to-end tests "in `custos-daemon/tests/`: in-process relay + one or two daemons + a simulated approver … + fake upstreams", one per row of §13.4. The owner's integration brief (2026-10-23) asks for a separate e2e crate that boots a real relay, two host daemons (`laptop`, `devbox`), a scripted approver built on `custos-core`, a Playwright run of the real approver PWA for use cases 2 and 3, Lowkey Vault (env-gated) and fake Azure, a stub `op` plus an optional live 1Password test against the vault "Custos PoC" (`CUSTOS_LIVE_1P=1`), runnable demos per use case, and every landing claim mapped to a test. The questions: where the code lives so the 100 % line gate (§14.1) is not distorted, which parts are real, how a night passes in a test, how the service-mode host is represented without a second OS user, and how the PWA reaches real daemons.

## Decision

1. **A workspace member `poc/tests-e2e` (`custos-e2e`)** with an empty library and everything in `tests/` and `examples/`. cargo-llvm-cov's default report excludes `tests/`, `examples/` and `benches/` directories (verified: `cargo llvm-cov -p custos-e2e --summary-only` reports 0 lines for this crate), so the gate keeps measuring product code while `make test` and `make cov` still run every e2e test (they also add coverage to the product crates). No new external crate: only workspace dependencies (tokio, axum, reqwest, serde_json, jiff, tempfile) are used as dev-dependencies. Deviation from §14.6: the flows live here instead of `custos-daemon/tests/`; the daemon keeps its in-crate flow tests (ADR 0060 §9).
2. **What is real.** The relay is `custos_relay::serve` on a TCP port with a SQLite **file** (scanned after each test) and its request log captured. Each daemon is `custos_daemon::Daemon` with the production socket server (`server::unix::bind/serve`), sync loop, reqwest relay client, `ProcessRunner` (the stub `op` is a real child process) and `RealSys`; dev-mode hosts run the production `start::preflight`. Agents reach daemons only as agents can: raw JSON-RPC lines on the socket, the real `custos-mcp` server (`CustosMcp` over a byte stream with MCP JSON-RPC, as Claude Code drives it over stdio), and the real `custos` CLI entry (`custos_cli::run` with the production `RealHost`, wrapped only to give the shell its own environment variables, `HOME` and working directory). Enrollment is the real commit-reveal ceremony through the relay mailbox; the phone's grants carry p256 WebAuthn assertions with user verification.
3. **What is simulated, stated in every demo header.** (a) Upstream APIs: a fake `Upstream` that echoes the request (credential reflected raw, base64, hex, in a response header, plus `Set-Cookie`), fake Entra and Key Vault routes asserting the §9.3 URLs/forms; no test needs the network. (b) Time: a `WarpClock` shared by relay, daemons and approver that runs in real time and can jump forward, so "until 07:00" spans a real 9 h of log time in about one second; the host zone is Europe/Zurich from a checked-in TZif file (§14.1). (c) The approver UI in the Rust tests: a scripted approver calling the same `core::approver` builders as the PWA's wasm, offering only grants inside `core::grant::envelope` (§7.5.3). The Playwright run uses the real PWA. (d) The second OS user of service mode: the `devbox` daemon runs in-process as the test's uid with `RunMode::Service`, so it enrolls with isolation 1 and every service-mode rule in the daemon applies; local subcommands run without the service-mode ancestor check. The two-uid layout itself is the subject of `real_perms.rs` (Linux, ADR 0055), not of this crate.
4. **Canary rule (§14.3)** implemented independently of `core::redact`: 12 encodings (raw; base64 std/url × 3 alignments, only fully determined characters; hex lower/upper; percent; JSON; base64∘base64), applied after every flow. *Recorded*: every canary value, `AS`, `ASK`, the `AK` seed, the approver's and every host's signing and KEM seeds, issued session tokens, and (amended after review round 1) every per-secret key recomputed from the approver's verified log at scan time: per policy version `PK` (field 11 under `ASK`) and each `a_i` (field 8); per secret version the transit `a_i` of host proposals, each host's `h_i` (opened with that host's KEM seed), `K = a_i ⊕ h_i` and the DEK. *Scanned*: every agent-visible message (RPC, MCP, CLI stdout/stderr/terminal, local `custosd` subcommand output), the relay database file (+WAL/journal/SHM), the relay log, `custos-relay inspect`, all panic output of the test process (a capturing panic hook), and surfaces a test adds (the `.env` after import). The in-process daemons write no log of their own; the binary's stderr lines are static strings tested in the daemon crate. *Not recorded*: the request ephemeral X-Wing seeds, which exist only in the daemon's memory (`PendingReq.eph`, crate-private); exporting them would add a secret-returning API to the product daemon, so the scan covers what they open (`a_i`) instead. Plaintext names, reasons, URLs and hosts are searched on the relay side (P1) in raw form at any length plus the ≥ 8-byte encodings. Positive controls: `canary_scanner_fires_on_every_encoding` and `tests/canary_rule.rs` (each recorded or derived key planted hex-encoded in the relay log, a key in panic output, a key in the post-import `.env`, a 7-byte plaintext name).
5. **Malicious relay double** (`common/hostile.rs`): holds the relay key (A1 = breach or operator), forwards until told to fork one host (serving it a private branch from a size on, accepting its appends and signing checkpoints for it, also discarding what an in-flight long-poll would have carried across — a deliberate forker) or to tamper (flip a byte under a covering checkpoint).
6. **Playwright against real daemons**: `examples/harness.rs` boots the real relay (CORS for the PWA origin), the two daemons on fixed socket paths and a loopback control endpoint that runs the local `custosd` subcommands a human types on a host; `approver/playwright.real.config.ts` starts it with `cargo run --example harness` and serves the real PWA build with `vite preview`; the spec drives the phone in Chromium with a virtual authenticator (UV on) and the agent over the sockets.
7. **Env-gated external runs** use cargo features plus Makefile targets that check the flag, never a runtime skip (§14.1): `make e2e-azure-lowkey` (`CUSTOS_IT_LOWKEY=1`, feature `lowkey-vault`, the pinned image of ADR 0056) and `make e2e-1password-live` (`CUSTOS_LIVE_1P=1`, feature `live-1p`, token in `CUSTOS_LIVE_1P_TOKEN`, reference asserted to start with `op://Custos PoC/`).
8. **Demos** are the e2e tests with `--nocapture`: each test prints a transcript (`step`/`say`), `demo/run.sh` filters it, `make demo-1 … demo-5` and `make demo-drop` run them.

## Alternatives considered

| Option | Why not |
| --- | --- |
| Tests in `custos-daemon/tests/` (§14.6 text) | One crate would depend on relay, CLI and MCP as dev-dependencies; the owner asked for a separate crate; no coverage benefit |
| Spawn the built binaries (`custos-relay`, `custosd`, `custos`, `custos-mcp`) | `CARGO_BIN_EXE_*` exists only for the same package; the daemon binary cannot take a fake upstream or clock, so the proxy would need the internet and a night would take a night. Binary entry points are covered by each crate's own spawn tests |
| Real network upstreams (GitHub, httpbin) | Non-hermetic, needs real tokens, cannot reflect values in chosen encodings |
| Paused tokio time instead of a warp clock | Relay rate limits, long-polls and reqwest timeouts would all pause; daemons compare relay-signed times with their own wall clock, which must move consistently for every party |
| Runtime `if env … return` skips for Lowkey/1Password | Forbidden by §14.1 ("no test skips at runtime") |
| Driving the PWA against the mock relay only | Already exists (`tests/e2e`); the owner asked for the real PWA on the real stack |

## Evidence

1. **[T]** `cargo llvm-cov -p custos-e2e --test uc1_blind_use --summary-only` on the build Mac (cargo-llvm-cov 0.9.1): TOTAL 0 lines — files under `tests-e2e/tests/` are not in the report — 2026-10-23
2. **[T]** Review round 1 (2026-10-23): `cargo test -p custos-e2e` 18 tests pass, incl. `tests/canary_rule.rs` (3), `shown_number_waits_for_the_terminal` and `e2e_uc4_1password_op_under_group_writable_dir_refused`; the 3 canary-rule controls fail with the round-1 scanner changes reverted.
2a. **[T]** `cargo test -p custos-e2e`: 13 tests pass (uc1 ×2, uc2, uc3, uc4 ×3, uc5 ×4, drop) — 2026-10-23
3. **[T]** `make e2e-azure-lowkey` with `nagyesta/lowkey-vault:7.3.112@sha256:2636ad67…` (Docker, amd64 image under emulation on arm64): `e2e_uc4_bridge_azure_lowkey_vault` passes — 2026-10-23
4. **[T]** `pnpm test:real` (Playwright 1.63.0, Chromium virtual authenticator): `real stack: account, two real hosts, remote approval (UC2) and night shift (UC3)` passes in ≈ 21 s — 2026-10-23
5. [The Cargo Book, Cargo Targets — integration tests, examples and `CARGO_BIN_EXE_<name>` "only available to integration tests and benchmarks of the same package"](https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-crates) — accessed 2026-10-23
6. ARCHITECTURE v5 §13.4, §14.1, §14.3, §14.6; ADR 0055, 0056, 0060.

## Consequences

- `poc/Cargo.toml` lists `tests-e2e`; `make test`/`make cov` run the e2e suite (≈ 25 s added); `make e2e`, `e2e-azure-lowkey`, `e2e-1password-live`, `e2e-pwa`, `demo-*` exist.
- `landing/CLAIMS.md` maps every claim to the e2e test(s) that prove it.
- The live 1Password test is written and compiled but was not run by the builder (no service-account token for "Custos PoC" was provided); it needs the owner. Until it has run, owner override (1) (1Password on macOS in dev mode) is proven end to end only with the stub `op` (a real child process with the real argv and cleared environment); the real `op` binary has not passed through Custos. The live test defaults to the root-owned `/opt/custos/bin/op` and runs the daemon's `op_path_check` before any setup, printing the `sudo -k install -o root -m 0755 …` remedy: the Homebrew cask path (`/opt/homebrew/bin/op` → `/opt/homebrew/Caskroom/…`, `Caskroom` is `drwxrwxr-x … admin`) fails that check, as review round 1 measured. `e2e_uc4_1password_op_under_group_writable_dir_refused` covers that layout: `custos bridge add` accepts it (the CLI only hashes the file), the daemon never runs that `op`, `doctor` reports `op_path` with the remedy, and after `chmod 755` the same policy works.
- What remains unproven end to end: service mode with two real uids (Linux `real_perms.rs`, not in this crate), Windows (compiles only, ADR 0056 §2), real upstream TLS, push notifications (N15).