Debugging

Written for whoever is reading this next, which is usually an agent with no memory of yesterday. It answers three questions in the order they get asked: where is the evidence, how do I tie it together, and what do I hand a model.

There is a page for all of it — #/logs in the console (Debug until 7.0, and every #/debug… address still lands on its section) — and the page is a reader of the endpoints below rather than a second source of truth. If you are in a terminal, skip to One bundle, for a model.

The one command

curl -s localhost:4123/api/debug/bundle | pbcopy

One redacted JSON snapshot of the whole surface: versions, capability flags, health, the watch clock, every metric family, the recent log rows across every source, and the delivery tally. It is sized to be pasted into a context window — around 600 rows, a few hundred kilobytes — and it says what it left out in its own notes array. Start here; open a file only when the bundle points at one.

Add ?slug=<plan> for one plan in full rather than the five most active.

Where the evidence is

Seven places, and the log explorer merges all of them onto one time axis. The names below are the source values on every row and in ?source=.

Two more per-run files are not in the index because neither is a log:

• runs/<instance>/<slug>/run-<id>.json — the checkpoint: one record per phase, its status, its session id, its worktree and branch. This is the file to read when you want state rather than history.
• runs/<instance>/<slug>/run-<id>-p<N>-tasks.ndjson — a phase's published task list (phase-tasks.sh), which the console renders as "What it is doing".

<instance> is derived from the source directory's path — ask for it rather than spelling it: node -e "import('./viewer/shared/instances.mjs').then(m => console.log(m.instanceId('/path/to/repo')))", or just read root off the bundle and let the endpoints resolve it.

Reading the logs

GET /api/debug/index?source=journal&level=error&slug=<plan>&q=<text>&since=<ISO>&limit=500

Every parameter is optional and repeatable (?source=a&source=b). An unknown word is dropped, not honoured — ?source=journals shows everything rather than nothing, because a typo that silently selects an empty set looks exactly like a quiet system. ?slug= and ?run= are the exceptions: both name a path segment, so a value that is not a plan slug / an eight-hex run id is refused with a 400 rather than ignored. (They are a pair for a reason — guarding one of them left the other reading files outside the state directory, and creating a directory out there on the way, because the journal reader makes its parent.)

The answer is { entries, sources, truncated, slugs }:

• entries — newest first. A row with no usable timestamp carries at: "" and sorts to the end, where you can see it is unplaced rather than old.
• sources — one status per source, including the ones that could not be read, each with a note saying why. "There is no supervisor log on this machine" and "the supervisor log is empty" mean opposite things and this is where the difference lives.
• truncated — the limit cut the answer. Narrow, or ask with ?since=.
• slugs — which plans this answer covers. An unscoped read walks the 8 most recently active, never every plan on disk.

GET /api/debug/tail?… is the same query as a Server-Sent Events stream, one entries frame every two seconds. It exists for the page's Follow switch; in a terminal, ?since= is cheaper and it does not hold a socket open.

Each frame carries an id:, so a browser resends it as Last-Event-ID and a reconnect continues instead of skipping — the gap a sleeping laptop opens is bridged on the next tick. ?after=<ISO> is the same thing for a caller that is not a browser, and the handshake reports resumed.

The one loss a tail really has is a gap wider than a single pass's row cap. A frame that filled it carries capped: true, and the page says so. (An earlier build computed a behind flag at the handshake and claimed rows were gone that the next tick delivered — announcing a loss that did not happen is the same defect as absorbing one that did.)

Levels

The journal has no severity column — it was written for a timeline, where every line is equal — so a level is derived from the event kind's own suffix: -failed / -refused / .halted / -broken are error; .stall / .rung / .parked / -timeout / .wall / -red / .retry / .interrupted are warn; everything else is info. An unrecognised kind reads info on purpose — a false warning trains a reader to filter warnings out, which costs more than a missed one.

Tying it together — grep the trace id

Since 5.1.0 there is one id, and it is on everything. A run's trace is runTraceId(instance, slug, runId) — 32 hex, derived rather than minted, so a console that restarts and resumes the run recomputes the same id instead of starting a second one. It is on every journal line of that run, on every console-log line written while its drive was running, on every git.command the drive caused, in the environment of every session it spawned (PE_TRACE_ID, TRACEPARENT), on the lines those sessions write with phase-tasks.sh and phase-outcome.sh, on the lock they take (trace=) and in their presence-hook record.

# the run's own id, from its journal
grep -m1 '"event":"run.trace"' ~/.local/state/phase-console/runs/<instance>/<slug>/run-<id>.jsonl

# everything, everywhere, in one pass
grep -r "$TRACE" ~/.local/state/phase-console/ ~/work/<repo>/docs/handoffs/

The Logs page takes the same id: ?trace=<id> on GET /api/debug/index and in its own address (#/logs?trace=<id>), and any row's record, opened in the panel, filters by its trace with one press.

Two things it does NOT answer, by design. A line written outside any span carries no trace — that is "no trace", not "some trace", and the filter will not sweep it in. And a v1 file (anything written before 5.1.0) has no ids at all; the journal reader derives one on read (withDerivedIds), marking those lines v: 1, but a v1 console log cannot be joined and the walk below is the fallback for it.

The walk, for v1 files and for what the id does not cover

Six identifiers name the same piece of work, and each file knows only some of them. This is the walk:

1. Start with the plan slug. Everything is under runs/<instance>/<slug>/, and every log row carries slug when it has one.
2. slug + phase → the lock. docs/handoffs/<slug>/.locks/phase-NN.lock is a flat key=value file holding owner, scope, session, and — when the work rides its own checkout — branch and worktree. This is the only file that ties a session id to a place on disk.
3. slug → runId. The run files are run-<id>.*; the newest is the live one. GET /api/debug/runs?slug=<plan> lists every run that has a journal, including finished ones.
4. runId + phase → the session. The checkpoint (run-<id>.json) carries sessionId per phase record, and the journal's phase.boarded line names the session an attempt was given.
5. session → the transcript. run-<id>.p<N>.log.jsonl is the SSE replay ring for phase N of that run, one capped file per phase since control-tower phase 94; run-<id>.log.jsonl holds the lines that name no phase, and a run written before the split keeps every line there. A file moved aside (.old, .full-<stamp>) is an archive retention sweeps with its run. A Claude session's own transcript is the CLI's, not the console's.
6. phase → why it is not done. GET /api/run/<slug>/diagnosis/<phase> is the richest single payload this console has: the command output, the session's closing words, the lint summary, the board-vs-handoff disagreement, the lock, the working tree, and what can still be done about it. Logs ▸ Journal (#/logs/journal?slug=&run=&phase=) renders it under the run's Gantt.

A worked example — "phase 7 says stuck and I do not know why":

# what the board thinks, and what the handoff says
scripts/phase-graph.sh <slug>
curl -s 'localhost:4123/api/run/<slug>/diagnosis/7'

# every error the run wrote, in order
curl -s 'localhost:4123/api/debug/index?source=journal&level=error&slug=<slug>&phase=7'

# did the session declare an outcome nobody acted on?
curl -s 'localhost:4123/api/debug/index?source=outcome&slug=<slug>'

# who holds the lock, on which branch, in which tree
cat docs/handoffs/<slug>/.locks/phase-07.lock

Logs ▸ Timeline — the run across all four sources

The Journal section draws the run's PHASES. It cannot draw what the console was doing while a phase was quiet, because three of the four files that hold that answer are not the journal.

Timeline is the join: one axis, six kinds of row.

The join key is the run's trace id, so the same filter that narrows the log explorer narrows this. A row you did not ask for is absent rather than empty — ?lanes=phase,git is in the URL, because the output of this page is evidence and evidence has to be quotable.

Three things it will not smooth over. A truncated journal read says so, and the lanes whose opening it cut begin mid-flight. An entry the projection did not model is counted (unmapped) rather than dropped. And a git span opens at at - ms: the log line is stamped when the command FINISHED, so drawing it at at would have every command start where it ended.

Selecting a bar opens the inspector: the span's attributes, the chain of spans it sits inside, and the redacted source line it was read from.

Open it in Perfetto

The console draws the picture; it is deliberately not a trace viewer. Zoom, pan and measure are what Perfetto is:

curl -sO 'http://127.0.0.1:4123/api/debug/trace?slug=<plan>&run=<id>&format=chrome'

or press Chrome trace on the section. Then open the file in ui.perfetto.dev — drag it onto the page. Perfetto runs entirely in your browser: nothing is uploaded anywhere.

Each lane is a track of one process, so any two can be measured against each other with one drag. Timestamps are microseconds; open spans carry args.open: true; phase.resources and run.usage-window arrive as counter tracks; and the arrows are flows — a boarding to the session it spawned, and a git command to the console line that complained about it. The file names itself (metadata.schema: "phase-console/chrome-trace").

Logs ▸ Git — every command, and what it cost

Every command the shell seam ran for one run, newest last: the argv, how long it took, its exit code, what it was for, and — expanded — the working tree, the tail of stderr it failed with, and how many child processes git started of its own when a Trace2 record was folded.

This table has all of them; the Timeline does not. The axis drops anything under 250 ms, because a sub-pixel bar is a lie drawn precisely. A reader who believed both surfaces would conclude the fast commands did not happen.

Two exit values that must not be read as one: 0 is a command that succeeded, and no exit is one that never started or was killed by a signal.

The delivery ledger

One row per device per announcement. The five outcomes, and what each counts as:

The bundle's delivery block counts an announcement undelivered only when no device took it, which is a different question from how many rows said failed: a fan-out to three devices where one succeeded is delivered. An announcement every device held as quiet is not undelivered either — nothing was attempted, on purpose.

One bundle, for a model

GET /api/debug/bundle — ?slug=<plan> to scope it, ?download=1 to save it as a file.

schema        "phase-console/debug-bundle"
version       1
generatedAt   ISO 8601
console       version, instance, generation, platform, node, uptime, the
              capability flags, whether the terminal went away, whether the
              previous process exited cleanly, and the /api/state facts a
              bundle is worth carrying
root          the source directory, home-masked, or null
plans         [{ slug, runs: [runId] }] — 5 plans, 5 runs each, unless scoped
health        { environment: [{ kind, detail, fix }], watches: { passes, asked, open } }
metrics       every family from /api/metrics, parsed: { name, type, help, samples }
delivery      { outcomes: {sent: n, …}, undelivered, announcements, devices }
entries       up to 600 log rows across every source, newest first
sources       the same per-source status the index returns
notes         what was left out, and how to ask for it

version moves when the SHAPE moves — a key added, removed or retyped — and not when a key's contents change. A reader keyed on version: 1 can trust these keys are there; it cannot assume a fixed set of journal event kinds inside entries, and should not.

What is redacted, and what that costs you

Everything that leaves the server — the index as well as the bundle — goes through two passes:

1. Secret shapes, via the same redact() the webhook payloads use: PEM blocks, sk-/ghp_/xox/AWS-key/JWT prefixes, token= and Authorization: Bearer assignments, credentials in a URL's userinfo, vendor webhook URLs, and any run of 40+ mixed-case-with-digits characters. A git sha, a UUID, a slug and an ISO timestamp all survive — none of them has the case mix. Since 5.1.0 there is a backstop under those named shapes: any run of 32+ token characters whose Shannon entropy exceeds 4.5 bits is replaced with [high-entropy]. The two numbers are each chosen against one false positive. 32, because shorter mixed runs are ordinary — a base64'd id, a slug, a filename. And 4.5 bits, because a 16-symbol alphabet cannot exceed 4.0 however random it is, and hex is what a log line is FULL of: every sha, every trace id, every span id survives by arithmetic rather than by luck, while a base64 or base62 secret draws on 62–64 symbols and lands near 5.5–6.0. The threshold sits in a genuinely empty band.
2. The operator's home path, masked to ~ — the macOS Users form and the Linux /home/<name> one alike, whether or not it is this machine's: a journal written on one box and read on another still names somebody. (.github/scripts/scrub.sh refuses the literal macOS prefix anywhere in a committed file, which is why this paragraph spells it out rather than showing it.)

This applies to the L3 raw record the page shows you too, which is the one surprise worth stating: a value in the file on disk can read [redacted] here. The reason is that the console is reachable from a phone over a tailnet, and a log line is the single most likely place for a token to have been echoed by a command. One rule for every export path, so there is no surface where a secret survives.

One run, for a post-mortem

The bundle above is a snapshot of the CONSOLE, and it is the right artefact for "is this console well". It is the wrong one for "why did phase 7 park at 03:14 on Tuesday" — for that it held the journal of no run, the transcript of no session, no task ledger, no outcome, no ruling, no lock, no git trace, and a log slice chosen by recency rather than by relevance.

So there is a second bundle, of ONE run:

GET /api/debug/bundle?slug=<plan>&run=<id>[&since=30m]
phase-console diagnostics --run <id> [--slug <plan>] [--since 30m] [--out FILE]

Both answer a gzipped tar. since takes a window back from now (30m, 2h, 7d) or an instant, and narrows every time-stamped member; a line whose time cannot be read is always kept, because "I could not date it" and "it is old" are different facts.

MANIFEST.json      schema "phase-console/run-bundle", version 2, every member
                   with its size, everything that was ABSENT, and the notes
record.json        the run record — status, phases, timings
journal.ndjson     the run's journal
transcript.ndjson  the session transcript — the run's own replay (lines that name no phase)
transcripts/pN.ndjson  each phase's replay (control-tower phase 94)
tasks/             this run's per-phase ledgers, and the plan's inbox copies
outcomes/          the same, for declared outcomes
rulings.ndjson     what the sessions DECIDED, as opposed to how they ended
messages.ndjson    the plan's message ledger
git-trace.ndjson   git's own Trace2, folded one line per process
sessions/          each session's raw hook payloads, in arrival order
locks/             the plan's phase locks
worktrees.json     what each live run's checkout looked like
console/           the console log, cut to this run
env.json           an ALLOW-LISTED environment; see below
versions.json      console, node, platform, instance
diagnosis/         one file per phase on record
SUMMARY.json       counts, phases, and the journal events that read as trouble
ANALYSIS.md        the same in prose (Pro)

The CLI verb works with the console down, and that is the point of it existing as a verb rather than only a button: the moment somebody most needs a bundle is the moment the console will not start. With a console answering on the instance's own port it streams from it instead, because a live console holds run state that has not been checkpointed. It checks the console's identity before it does — a port is derived from a root, and the answer on it may be a different project's console.

env.json is an allow-list, not a deny-list: PATH, HOME, SHELL, LANG, TERM, TZ, TMPDIR, NODE_ENV, NODE_OPTIONS, the two XDG dirs, CLAUDE_CONFIG_DIR, and anything under PHASE_CONSOLE_ or PE_. A name matching /TOKEN|SECRET|KEY|PASS|AUTH/i is then dropped WHOLE — the name of a secret is itself a hint. PATH leads the list because the worst environment bug this console has had was a PATH bug (an unaccepted Xcode licence shadowing git under launchd), and it was unanswerable from a bundle that did not carry it.

The manifest says the redaction is best-effort, and means it. It is regexes, a home mask and an entropy heuristic. It catches the shapes we know and the ones random enough to guess at; it does not catch a password that looks like a word. Read a bundle before you share it.

What is kept, and for how long

Every sink this console writes is bounded by one table, in viewer/server/retention-policy.ts, swept at boot and once a day. GET /api/debug/retention reports each sink's live size, its policy row, and what the NEXT sweep would do — which is also what Settings ▸ This instance ▸ Logs and retention shows. The shipped table:

The fleet supervisor's own log is trimmed on the same rule as the supervisor stdio above, by whichever console sweeps first.

Three verbs and one non-verb, and the choice per sink is the design. delete for a file nothing is writing to. rotate (rename to .1) for a file whose writer opens it fresh each time. truncate — copy-truncate on the same inode — for a file whose writer holds a descriptor for the life of the process: the supervisor's stdout is opened ONCE by launchd, and rotating it with rename would leave launchd writing to an unlinked inode, so the console keeps logging, the log file stays empty forever, and nothing anywhere reports an error. And oversized, which acts on nothing and only says the number: the ruling ledger IS the record, and the console log is rotated by the writer that owns it.

Traps

• sent is not "delivered to a human". See the table above. A ledger read as a green tick is the one way this data actively misleads.
• Metrics are a snapshot, not a series. /api/metrics renders the current facts on every scrape and the console stores no history. The Health section charts one sample; the thing on the Logs destination that genuinely is a timeline is the run journal.
• The console log and the supervisor log are different failures. A crash before the logger opened its file exists only in console.err.log, and that file exists only when something supervising the console started it.
• A restarted console has an empty ring and a full file. The index reads both and de-duplicates, so the newest lines survive a rotation — but log.recent() alone is per-process.
• A row beginning … is one line longer than the read window. A crashing process printing an unbroken blob is how the supervisor log fills; the tail is shown, marked, rather than the file being reported as empty.
• A […N more] element or a […] key means breadth was cut, and [deep] means depth was. All three appear inside an L3 record; none of them is data the source had.
• An unscoped read covers 8 plans, and the bundle 5. Both say so. If the plan you care about is quiet, name it with ?slug=.
• ?source= filters, ?level= filters, and an unknown word does neither. Check the sources block if a read comes back empty: it will say whether the source was unreadable or merely had nothing.

See also

• Journal events — every event kind, what emits it, and what it means. Machine-checked in both directions.
• Metrics — the exposition /api/metrics serves and the contract its names are.
• The loop — how a stopped phase is read, and what the ladder tries before it asks a person.
• Safety rails — what an unattended session may and may not do.