trace
Highlights
- One ~70-line HTTP server, node:sqlite, zero runtime dependencies — no framework to hold the shape of something this small
- Structurally separates three claims that usually get conflated: 'CI built this', 'a host actually ran and verified this', and 'a human investigated and understood this' — a green build can never masquerade as a live deployment on the dashboard
- Private incident notebook: timeline, hypotheses (including rejected ones, on purpose), linked commits, confidence tracked separately from status — nothing public until a human explicitly publishes it
- Git fix-candidate inbox scans commit subjects for fix-shaped keywords, via either a mounted read-only mirror or Gitea's API with no checkout at all
- Deployment receipts v1: idempotent, project-scoped bearer tokens, an allowlisted public projection that never leaks raw payloads or internal hostnames
- Found and fixed a real bug in its own operational story the same night it went live: goonk's deploy dashboard was showing a stale commit because nothing had ever wired up the deploy-host half of the receipt chain — see the case study
Every self-hosted project in this family now has a CI pipeline that builds and pushes images. Fewer of them have anything on the deploy host that reports back “and this is what’s actually running” — which means the honest answer to “is production current?” was, for a while, “probably, I think I ran the deploy.”
trace treats that as two separate facts rather than one blurry one. A build receipt, submitted by CI right after an image push, says what was successfully produced. A deployment receipt, submitted only by a host that pulled the image, started it, and ran a real health check, says what’s actually live. Keeping them structurally distinct — different event types, different submitters, a public projection that only ever shows the latest deployment — means a passing CI run can never get displayed as a successful production deploy just because nobody checked.
The other half is a private incident notebook, because “what broke and why” deserves the same discipline as “what’s deployed”: a timeline instead of a reconstructed-after-the-fact summary, hypotheses tracked even when they turn out wrong, and a deliberate publish step before anything crosses from private notebook into the public /failures page. A Git inbox scans commit subjects for fix-shaped language and lets you link the actual fixing commit to the incident that prompted it.
It found its first real bug in production the same night it shipped: goonk’s own deploy dashboard was reporting a commit that wasn’t actually the one running, because nothing on the deploy host had ever been wired up to tell trace about a deploy in the first place. Full story in the case study.