Architecture decision records · Jacob Stephens
The reasoning
behind systems
that run.
What I chose, what I gave up, and when I would choose differently. 52 decisions from production software and the infrastructure around it.
The results were right.
The query shape was wrong.
A production dashboard aggregate, before and after a rule I had to revise. The fix became a lint check.
Read the decisionA reservations platform. Agent sandboxes. A self-hosted fleet. Public tools and private systems.
Where these come from ↗A few ways in
Start with the trade-off.
Eight records that show the range.
Each one names the cost of the choice.
A review finding carries a failing test, and a decline cites the run
Spend reviewer effort on runnable evidence and declared coverage so declines can be checked
A feature's tickets merge into one branch so review happens once
Trade ticket-level review for a whole-feature review, accepting early ticket closure and branch CI limits
The hosted runner asks and the host decides
Use short-lived OIDC requests while the host rechecks identity, the commit, CI, and live state
Release the session lock before slow work, on the request that holds it
Restore concurrent requests for one user, and make accidental mutex behavior explicit
Read the reconciling view by default, inline the union for aggregates, and lint the shape of the restriction
Accept two ways to read one dataset, guarded by a lint test, because MySQL cannot push a non-constant predicate through a UNION - eleven dashboard queries at 493 to 1,441 seconds, the worst to 0.196 once inlined
One secret manifest per consumer, generated from a tracked declaration; and validate every manifest the machine reads, not every manifest a harness launches from
Take on a generator, a startup assertion, and a constants file so a vault rename kills one consumer instead of the box - after a 252-reference shared manifest took down four units for six hours while the validator printed green
Preview containment lives in the application, and isolation is proved by an egress test rather than inferred from a config file
Route outbound effects into inspectable sinks, and refuse to call a slot contained until an egress test passes - after a per-slot override file turned out to be present, well-formed, and never read
Authority over a shared record follows the person it is about, not the account that pays
Separate billing from authority and make handover, departure, and deletion obey the same rule
The collection
All decisions52
Browse by problem, or search the full records.
Numbers stay fixed; gaps include withdrawn records.
52 decisions
Running AI agents against production systems
Docker over bare-metal for per-tenant isolation
Pay container overhead for separate files and processes; shared sandbox data remains a deliberate cost
Data, records, and correctness
External managed database over a containerized one
Give up "one compose up" simplicity for durable, backup-friendly state
Running AI agents against production systems
Periodic snapshot over live replication for Manager Sandboxes
Accept some staleness to gain isolation, reset-ability, and no prod write-path risk
Deploy, release, and provisioning
A guarded shell deploy over a hosted CI runner
Keep guarded execution on the host; automatic requests now arrive through OIDC from hosted CI
Running AI agents against production systems
A scoped system user over a shared service account for an autonomous agent
More host setup for per-identity attribution, with database grants and container boundaries reviewed separately
Data, records, and correctness
Binlog-tailing daemons over database triggers for denormalization
Accept eventual consistency to keep derive-logic in versioned code, off the hot write path
Observability and capacity
Pull-based health probing over push agents for a small fleet
Forgo deep metrics/history to keep the monitored fleet agent-free and the failure domain legible
Data, records, and correctness
Embedded SQLite over a networked database for single-node tooling
Give up cross-host sharing for zero operational surface on state one process owns
Running AI agents against production systems
Default-deny, host-pinned database access over a trusted network
Take on provisioning friction so a leaked credential isn't portable off its host
Running AI agents against production systems
A pixel-equality gate over diff review for changes to generated markup
Pay a render/diff harness to safely change markup you can't audit by eye - it proves visual, not semantic, equality
Observability and capacity
A pull-based metrics stack alongside a bespoke prober, with the data plane kept private
Take on a TSDB to run for real metrics/history/alerting - keep co-located backends on loopback, pin remote exporters to the monitoring host, and publish a static pane while live Grafana stays behind login
Data, records, and correctness
Per-app copy-deployed sync services over a shared multi-tenant backend
Run N near-identical small services to gain physical blast-radius isolation and per-app sync-model freedom, at the cost of hand-applying shared-auth fixes across copies
Deploy, release, and provisioning
Import live DNS into Terraform over recreating it from a desired-state list
Accept verbose generated config and provider quirks to adopt traffic-serving records with zero downtime and a no-op baseline plan, instead of risking duplicate-creates and silent deletion of forgotten records
Deploy, release, and provisioning
Keep Terraform state off the provider it provisions
Take on a second vendor + scoped IAM key so a provider-level outage can't destroy both the infrastructure and the state needed to rebuild it
Secrets, access, and perimeter
Self-host the signing instrument and its audit trail over a SaaS signature service
Take on one container plus its own cert and upkeep so the legally-binding document and its audit trail stay on infra you govern, with a named-human gate on every consequential action - the highest-stakes case of keeping the authoritative copy where you control it
Secrets, access, and perimeter
Runtime-injected secrets over plaintext config, with the store matched to the team
Hold one invariant (encrypted at rest, injected into the environment at runtime) and pay for two backends - a broker where a team needs sharing/revocation/audit, SOPS + age where a solo fleet needs offline zero-vendor recovery - rather than force one tool to fit both
Secrets, access, and perimeter
Reuse an existing passkey session to gate a second app (Caddy forward_auth) over standing up a second auth stack
Gate a second internal app by delegating to one hardened passkey session via Caddy forward_auth - one proxy block and no second credential store, every tool inheriting the gate's future hardening - at the cost of concentrating trust in a single session
Secrets, access, and perimeter
A private mesh for operator shells, public authenticated endpoints for browser consoles, over one VPN for everything
Put SSH behind a WireGuard mesh and drop public :22 on enrolled hosts, but keep browser admin consoles public behind application authentication - size the boundary to who uses it, rather than hide the consoles non-technical staff reach by URL behind a VPN client they can't maintain
Observability and capacity
A shared OpenTelemetry collector seam over direct backend wiring, metrics-only for Claude Code
Add one collector as the ingestion seam so new telemetry sources and backends are configuration, not new architecture - agent usage metrics only, never prompt content
Observability and capacity
Complete the logs pillar with a co-located local-disk Loki behind the collector seam, over a dedicated or hosted log stack
Add single-binary Loki beside Prometheus/Tempo on the one monitoring node, ingested only through the shared collector seam and bound to localhost - one query surface and native trace-to-logs for near-zero new infrastructure, at the cost of a deeper single-box blast radius and a 31-day local-disk retention ceiling with object storage as the written-down escape hatch
Secrets, access, and perimeter
Accept an unsigned DNS provider behind an adapter, with proof keyed to the pin
Pin the artifact and repeat its DNS lifecycle proof when it changes; keep provider quirks inside a checked adapter
Secrets, access, and perimeter
One secret manifest per consumer, generated from a tracked declaration; and validate every manifest the machine reads, not every manifest a harness launches from
Take on a generator, a startup assertion, and a constants file so a vault rename kills one consumer instead of the box - after a 252-reference shared manifest took down four units for six hours while the validator printed green
Deploy, release, and provisioning
Pin the shared serving checkout fast-forward-only, refuse-and-alert on a dirty tree, and edit only in worktrees
Give up "restart and it's live" on one service to stop the inverted-signal failure where the newest commit carries the oldest content - the pin detects dirt, a stale-path hook guards commits, and interactive sessions now land in the serving checkout
Deploy, release, and provisioning
Immutable per-commit releases with atomic promotion and an append-only journal; re-promotion is a named attended operation; root-pinned authority drift is reported, never a failure
Pay a promoter and a journal so no exit leaves production unnamed, hand-editing the symlink is the thing the design refuses, and a merge can never expand what root pinned - drift is shown, not paged
Deploy, release, and provisioning
A CI deploy key that reaches a shared host, restricted to one installed forced command, over having the host pull
Let a CI secret reach a host that serves client work because it cannot open a shell - forced command outside the deployed tree, pinned host key - in exchange for sub-minute deploys without a listener or a timer
Observability and capacity
The first workload with unauthenticated browser ingestion gets its own host; the store is private by construction; admin is gated by a named permission; proof is per property, and the abort is written before launch
Pay a dedicated small host and a volume so browser-driven POSTs never land beside fleet keys or prod sandboxes, and prove recording first, then inspect stored probes against the policy in force
Observability and capacity
Grow the monitoring host's own disk over attaching a block volume, and cap retention by size so the bigger disk is not just a later failure
Take a one-way, power-off resize because a grandfathered allocation made it free - and accept that the bigger disk is not the guard; the size-based retention cap and log hygiene are
Secrets, access, and perimeter
A strict root allowlist returning 404 over a growing blocklist returning 403, for a legacy repository-as-webroot
Fail closed against files nobody anticipated and stop confirming their existence to scanners, without the multi-quarter refactor of moving the document root - which the allowlist now guards
Running AI agents against production systems
Serial rounds run as harness subagents, over detached parallel CLI processes
Give up throughput and process survival - parallelism saved about ten minutes and cost a merge round, a re-gate, and a second correction - to delete a process-management layer that once resumed the wrong session
Running AI agents against production systems
One primary agent instruction file, a byte-identical shared layer, and per-host environment facts beside it
Add a pointer and a checksum check so every harness, not just one vendor's, is told what a host can touch - after five 'generic' instruction files turned out to have drifted into five versions
Data, records, and correctness
Read the reconciling view by default, inline the union for aggregates, and lint the shape of the restriction
Accept two ways to read one dataset, guarded by a lint test, because MySQL cannot push a non-constant predicate through a UNION - eleven dashboard queries at 493 to 1,441 seconds, the worst to 0.196 once inlined
Secrets, access, and perimeter
Audit history never retains secret values, and existing history is migrated to match
Deliberately destroy the recoverability of old credential values, because an audit trail is not a credential store - history keeps who and when, redacts diffs, and omits the field from anything restorable
Secrets, access, and perimeter
Section permissions are additive on top of page permissions and fail closed
Make granting more verbose so a hand-built request cannot mutate a section its author cannot view, and a misconfiguration fails as 'cannot see it' rather than 'can'
Deploy, release, and provisioning
Preview containment lives in the application, and isolation is proved by an egress test rather than inferred from a config file
Route outbound effects into inspectable sinks, and refuse to call a slot contained until an egress test passes - after a per-slot override file turned out to be present, well-formed, and never read
Data, records, and correctness
On a generated status site, scripts hide rows and derive no figures
Cost readers a filter-scoped distribution so every number has exactly one definition - enforced as a property of the rendered output, with the check's three blind spots written down beside it
Deploy, release, and provisioning
Ship static musl release binaries with a CI guard, over glibc builds from whatever the runner has
Give up in-process NSS lookups to remove the glibc floor entirely - after a CI image bump put a non-weak `GLIBC_2.39` reference in a binary that called nothing from 2.39 and broke every stable server distro
Data, records, and correctness
A closed, provider-independent entitlement state, migrated expand-then-contract, with the old endpoint kept indefinitely and reads failing open
Carry a permanent compatibility endpoint because released desktop builds embed old JavaScript, leave ambiguous legacy rows `unclassified` rather than guess, and give away uploads during my own outage rather than lock out a paying customer
Data, records, and correctness
One language-neutral conformance corpus over three hand-written engines, until a shared core can retire them
Keep implementing every rule three times for now, but make drift mechanical to detect - amend the corpus first, watch three suites go red - without depending on a shared-engine rewrite that is no longer planned
Data, records, and correctness
A repeatable LLM judge over a more accurate one when the first verdict is cached as precedent
Pick the model with a constant bias over the one with less bias and more noise, because calibration subtracts an offset once and nothing subtracts variance after the cache has frozen it
Running AI agents against production systems
A review finding carries a failing test, and a decline cites the run
Spend reviewer effort on runnable evidence and declared coverage so declines can be checked
Running AI agents against production systems
Merge the exact head whose CI was watched, over platform auto-merge
Keep merge authority inside the repair loop, and stop when repository review policy refuses
Running AI agents against production systems
Decide CI outcomes from structured state, not the agent's account
Match failures against the base and grant one unchanged-head re-run, without interpreting agent prose
Running AI agents against production systems
A feature's tickets merge into one branch so review happens once
Trade ticket-level review for a whole-feature review, accepting early ticket closure and branch CI limits
Running AI agents against production systems
A sandbox reads the primary, not a replica that can stall silently
Read canonical state under server-enforced query and account limits, accepting production read load
Deploy, release, and provisioning
The hosted runner asks and the host decides
Use short-lived OIDC requests while the host rechecks identity, the commit, CI, and live state
Deploy, release, and provisioning
Two processes, one release; refuse when they disagree
Pin the console and worker together and refuse production work when their revisions or liveness disagree
Deploy, release, and provisioning
A failed deploy pauses nothing, and recovery reads production
Re-establish eligibility each time and spend one bounded recovery retry against live production evidence
Data, records, and correctness
Release the session lock before slow work, on the request that holds it
Restore concurrent requests for one user, and make accidental mutex behavior explicit
Data, records, and correctness
Tests refuse the production schema by name, and a wiring check enforces the guard
Guard the dangerous database before bootstrap and test that every integration entry point uses the guard
Observability and capacity
Watch the work, not the process, with a window longer than the stall
Alert on missing work and checkpoints, accepting a detection window long enough to see delayed evidence
Data, records, and correctness
Authority over a shared record follows the person it is about, not the account that pays
Separate billing from authority and make handover, departure, and deletion obey the same rule
Deploy, release, and provisioning
Assert enforcement at the script, the workflow, and the ruleset
Test script/workflow failure propagation and verify active required-check registration before claiming merge is blocked
No decisions match.
Try a different phrase or choose another topic.
Beyond the decision
How it fits together.
The system, the risks, and the operating tools
behind the individual records.
The platform case study ↗
A twenty-year-old reservations platform, modernized while the business kept running.
02 /The agent threat model ↗
Assets, threats, controls, and the risks that remain when agents touch production systems.
03 /The operational checklist ↗
The questions I ask before a system is ready to hold real money.
04 /The observability stack ↗
Sanitized Prometheus, Grafana, and Alertmanager configuration you can inspect and reuse.
A note on evidence
Every claim has a boundary.
These decisions are used in running applications or tools used for real work. Private systems are sanitized. Public originals are linked. A shipped fix and a measured production improvement are separate claims.
Read the evidence boundary ↗