I like the part of the system that only misbehaves under load.
I am Patrick Donahue, a senior software engineer in Seattle. Most of what I write about lives in the unglamorous middle of an application: data models that survive contact with reality, query plans that quietly stop being true as a table grows, and background jobs that fail loudly instead of silently.
Nearly every essay here starts with a real production problem and a number attached to it. A page that counted 800 million rows. 319 N+1 queries across a fleet. A petabyte of media sitting on the wrong storage tier. A goroutine leak with no stack trace, no panic, and no failing test.
Currently open to a full-time engineering role. More about meStatic analysis for bugs that leave no evidence
Goroutine leaks are a genuinely nasty class of bug: no stack trace, no panic, no failing test, just a process that slowly gets worse until someone restarts it. I have been building tools for that, and benchmarking them somewhere honest.
tether
Treats every go statement as a proof obligation: either find a termination
witness, or report it with a concrete escape path. I pointed it at the entire Go 1.25
standard library, which is the only honest way to find out whether a tool like this is real.
- 91 → 21diagnostics after seven fixed false-positive classes, with no true positives lost
- 0findings in non-test stdlib code
- 1real WaitGroup hang found in the
database/sqltests
weave
The Loom-shaped gap in Go. Runs one goroutine at a time through a stateless DFS with replay, so a concurrency bug that shows up once every ten thousand runs becomes something you can reproduce on purpose and then shrink to its smallest failing schedule.
- vector clocksrace detection across explored schedules
- delta debuggingminimises a failing interleaving automatically
- CHESSpreemption bounding to keep the search finite
The four things I keep coming back to
Written over several years and a lot of production incidents. Grouped by the problem rather than the technology, because the technology keeps changing and the problem does not.
Postgres that stops being the problem
The most common reason a growing app feels bad is a query that was fine at ten thousand rows and is not fine at ten million, and nobody noticed the week it changed.
Knowing when the honest answer is 'I don't know'
A confident wrong answer is worse than an abstention. Most of my recent work is about building systems that can refuse.
Reading the bill instead of accepting it
Cloud spend is treated like weather. It is usually a lifecycle rule written years ago against an access pattern nobody re-verified.
Failing loudly, on purpose
Silent failure is the bug that costs you a customer eighteen months later. Idempotency, gap-free logs, and gates that fail closed rather than open.
Latest writing
- The Agent Wanted to Call It Healthy. I Made It Abstain.
Shipping real software with coding agents, where the value is the judgment layer on top.
- Is It a Regression, or Just a Flaky Test? Teach Your CI to Tell the Difference
A red build tells you something failed. It doesn't tell you the thing that matters — whether a human needs to fix it now, or whether the test will be green next run. Here's a small, dependency-free flake engine that answers that from run history.
- A False Signal Is Worse Than a Missed One: Citing and Abstaining in LLM Document Extraction
When you turn public documents into sales signals with an LLM, the expensive failure isn't the signal you miss — it's the one you invent. Two disciplines fix it: cite every signal to a verbatim source span, and abstain when the document supports nothing.
- Data Sovereignty Is an Access-Control Pattern (and It Should Fail Closed)
Turning ‘the community owns its data’ into enforceable, audited code: default-deny access, owner-granted sharing, no super-role, and an append-only audit trail — over a territorial-monitoring dataset.
- When ‘Compliant’ Is a Legal Claim: Computing EUDR Deforestation Risk — and Knowing When to Abstain
Scoring forest loss inside a plot boundary is the easy half. The engineering that matters is refusing to certify on bad geodata.
- The Number Was Green, So the Agent Stayed Quiet
Putting an eval harness and an abstain gate around an LLM analysis agent so it's trustworthy on noisy marketing data
- Never Bill a Sentence the Chart Didn't Write: Evidence Gates for Clinical Charge Capture
Why an AI that finds missed hospital reimbursement has to cite the chart and abstain — and why that guardrail belongs in deterministic code, not in a prompt.
- Designing a Fail-Closed Access Gate in TypeScript
A worker walks up to a turnstile. Should it open? The safe answer to that question is a small, pure, boring function — and a tamper-evident record of every time you asked.
A little context: I studied history, which nobody hires an engineer for, and which turned out to be four years of practice at the thing this job actually is. You get handed incomplete, contradictory sources and have to produce a defensible account of what happened, while being explicit about which parts you are confident in and which parts you are inferring. That is a debugging session. It is also what reading an unfamiliar codebase feels like.
I have spent the last several years as the senior engineer on a small team, which is a particular kind of education: there is nobody to escalate to. If the database is slow at 2am, that is yours, and you also get to be the one who explains it afterward.
Off the clock I build a lot of small strange things, mostly because the cost of building a small strange thing collapsed, and I host them myself because owning the deploy path teaches you things that reading about it does not. The longer introduction →