Case study: trace

A private incident notebook and receipts ledger for the self-hosted stack — and the real bug it caught in its own deploy story on the night it launched.

Context

/under-the-hood — this site’s own build-and-deploy explainer — already had a proposal for “canonical deployment receipts” as a later phase: build identity embedded in the image, a /api/site-meta endpoint, and evidence shared with something that could correlate incidents against the deployments immediately before and after them. trace is that something, built as its own small service rather than folded into goonk’s API, so every other self-hosted project in this family can report into the same place goonk does.

Problem

Two things that were already true, separately, before trace existed:

1. CI pipelines across this stack build and push images reliably. None of them had anything downstream confirming the pushed image was ever actually running anywhere.
2. Real incidents — bugs, dead ends, deploy mistakes — kept getting reconstructed from memory after the fact for /failures, instead of captured while they were happening.

Both are versions of the same failure mode: a fact that only exists in someone’s head, un-timestamped, un-verified, easy to get subtly wrong in the retelling.

Approach

1) Build receipts and deployment receipts are different event types, on purpose

The obvious simpler design is one “deployment” record that CI writes when it finishes. That’s exactly the shape that lets a green build get displayed as a live production deploy without anyone having actually checked. trace’s schema keeps eventType: build (submitted by CI, right after image push) and eventType: deployment (submitted only by a host that pulled the image, ran docker compose up -d, and passed a real health check) as genuinely separate rows, and the public projection (GET /api/v1/public/projects/:project/latest) only ever returns the latest deployment — never a build standing in for one.

2) Idempotency is the whole reason retries are safe

Receipt submission is explicitly best-effort everywhere it’s documented — CI shouldn’t fail a build because trace was down for a minute, and a deploy host shouldn’t roll back a healthy deploy over a failed POST. That only works if resubmitting is safe. eventId is the idempotency key (UNIQUE in SQLite); resubmitting the same one returns {"ok":true,"duplicate":true} instead of a second row. A deterministic eventId (project + environment + short commit + timestamp) means “retry the curl that timed out” is always correct, never a risk of double-counting.

3) Private by default, published by one explicit action

The incident notebook (/, Basic Auth) is where the messy real version lives — internal hostnames, half-formed hypotheses that turned out wrong, impact notes that are nobody else’s business. None of it is reachable through the public API until an incident is explicitly published, which mints a public slug and switches on the allowlisted projection (publicIncident()/publicDetail() in src/receipts.mjs and src/server.mjs — deliberately narrow functions, not “serialize the row”). Unpublish is instant and reversible. This is the same shape as keep’s server/client trust boundary applied to a different kind of secret: not credentials, but “things not ready to be said publicly yet.”

4) The Git inbox works with or without a checkout

Two ways to discover fix-shaped commits: a locally mounted read-only repo mirror (TRACE_REPOS, for a deploy host that already has one), or Gitea’s commit API with no checkout at all (TRACE_GITEA_URL/TRACE_GITEA_REPOS). The API path exists because requiring a Git checkout on trace’s own host — a service whose entire job is being a small, boring, always-on ledger — would have made “which repos can it see” a much bigger surface than “which repos is it allowed to ask Gitea about.” A failed or unreachable repo is skipped, never fatal to the scan.

5) It’s one file, on purpose

src/server.mjs is routing, the private notebook UI, and the public API in about 70 dense lines, node:sqlite as the only real dependency (built into Node 24, not npm-installed). Not a stylistic flex — this is a service meant to sit quietly on a VPS being trustworthy evidence, and the honest test for “is this too clever” is whether the whole request-handling path fits on one screen during a 2am read. It does.

What actually happened the night this shipped

The most convincing test of a receipts system is whether it catches something real, not a synthetic demo. It did, within hours of trace being wired into goonk’s CI.

goonk’s /under-the-hood page started reporting Built: Jul 21, 3:06 AM and Deployed: Jul 20, 11:00 PM — deployed before built, and Health: healthy. Read literally, that’s nonsense; a deploy can’t precede the build it’s supposedly running.

It wasn’t nonsense once the two-receipt-type design was taken seriously: Built was current, from CI’s build receipt. Deployed was real too — just for a different, older commit. Querying GET /api/v1/public/projects/goonk/latest directly confirmed it: the deployment receipt on file predated the new build entirely. The actual root cause was almost boring by comparison — goonk’s deploy host had only ever run a plain docker compose pull && docker compose up -d, no wrapper, so nothing had ever told trace about this deploy. The app itself was fine; the evidence trail about it was just empty.

The fix landed in the actual deploy mechanism, per trace’s own integration doc (“this last POST belongs in the actual deployment wrapper”): keep’s shared deploy.sh — already used across every project on keep — now polls the freshly-deployed app’s own embedded build identity until it reports healthy, then submits a deployment receipt with the real running commit. Opt-in per project via a .trace-meta marker file, so an unmigrated project is honestly unconfigured rather than silently wrong. Verified end-to-end: submitted the receipt by hand once while the deploy host was still mid-migration to keep, confirmed the public projection updated to the correct commit and a Deployed timestamp that now actually comes after Built.

This incident is itself recorded in trace, published, and visible at /failures — the tool’s first entry is a bug in its own supporting infrastructure, caught the same night it shipped.

What I’d do differently

POST /api/receipts — an older, unauthenticated-by-token, pre-v1 endpoint — is still in src/server.mjs, writing to a receipts table nothing reads from anymore. Dead code from before the v1 events API existed, left in rather than cleaned up during the receipts work. Harmless (still behind Basic Auth), but it shouldn’t ship a second, worse-documented way to do the same thing.

No retention policy on deployment_events or candidates yet — both just grow forever. Fine at this scale (a handful of projects, infrequent deploys), but worth a bound before this becomes a service anyone forgets is quietly accumulating rows.

Stack

Server: Node.js (24+), node:sqlite (built-in, no ORM), zero runtime npm dependencies — the entire app is src/server.mjs, src/receipts.mjs, src/gitea.mjs.

Storage: SQLite, WAL mode, a single file on a named Docker volume.

Infrastructure: Docker, single service, HEALTHCHECK against /api/health, Gitea Actions → private registry. CI runs the real test suite (node --test) and a syntax check before it will push an image — the same build-vs-deploy discipline trace itself enforces, applied to trace’s own pipeline.