Invoice automation looks like a data-entry problem and turns out to be a reliability problem. The happy path—ingest a PDF, read the fields, match the PO, push it to the ERP—is the easy 60%. The work that decides whether the pipeline is trustworthy is everything around it: the malformed scan that no OCR pass will ever parse, the duplicate that arrives twice because a vendor re-sent it, the carrier portal that returns a 503 at 2 a.m. A durable invoice engine is one that gives every one of those documents a known terminal state and a paper trail—never a silent drop, never a double-provision.
The backbone is a small orchestrator: a workflow is an ordered sequence of typed steps (intake → extract → normalize → validate → approval → provision), and each step runs inside a retry loop with its own policy. The single most important design decision is to make failure modes explicit in the type system rather than collapsing them into a bare exception. Three signals carry the whole story:
for attempt in range(1, policy.max_attempts + 1):
await sleep(policy.delay(attempt)) # exponential backoff, capped
try:
await step.fn(ctx)
metrics.record(step.name, ok=True, ...)
break
except SkipWorkflow as skip: # clean early stop (dedupe)
return WorkflowResult(ok=True, reason=skip.reason)
except TerminalError as term: # never retry; dead-letter now
last_error = term
break
except Exception as exc: # transient; consume a retry
last_error = exc
if not succeeded:
store.dead_letter(invoice_id, step.name, str(last_error), attempts, payload)
A TerminalError—an unparseable document, an unknown
purchase order—dead-letters immediately, because retrying cannot change the
outcome and burning a retry budget on it just delays the inevitable. A
RetryableError (a connector 503, an OCR timeout) consumes a retry
with exponential backoff, then dead-letters only if the budget is exhausted. And
SkipWorkflow is deliberately not a failure: a deduped
duplicate is a clean success, which keeps your success-rate SLO honest instead of
penalizing the system for doing exactly the right thing.
Idempotency is the other half of trustworthy intake. The same content
fingerprint—canonical vendor plus invoice number plus amount—maps to
exactly one row behind a UNIQUE constraint, so re-ingesting a
document is a no-op rather than a second payable. This is the same
at-least-once→effectively-once discipline that background-job systems live
or die by; I wrote about the mechanics of getting it right in
Sidekiq Idempotency and Reliability,
and the same fingerprint-and-dedupe pattern transfers directly here.
Then there is observability, which is not an afterthought you bolt on but the
thing that lets you operate the pipeline at all. Every attempt emits a structured
JSON log line carrying a correlation id, the step name, the attempt number and a
duration, so a single invoice can be traced end to end with a grep.
Those same records roll up into per-step metrics and an SLO report—success
rate, p50/p95 latency, retry count, dead-letter count—checked against
explicit targets and rendered as a PASS/FAIL verdict. The CLI exits non-zero when
the SLO is breached, so the report doubles as a CI gate: a regression that pushes
the dead-letter count over budget fails the build, not the month-end close.
Connectors are where automation projects quietly accumulate fragility, so it
pays to give them a single, boring contract. In this engine every downstream
system—the ERP, the ticketing queue, the carrier portal—implements the
same async send(ctx) → ConnectorResponse interface, and the
provisioning step fans out to all three concurrently with a single
asyncio.gather. That uniformity buys two things. It makes
provisioning genuinely concurrent rather than three serial round-trips, which is
where the latency budget actually goes; and it makes swapping a mock for a real
NetSuite, ServiceNow, or vendor-portal client a mechanical change behind a stable
boundary instead of a rewrite. The retry policy lives on the step, not inside the
connector, so the flakiest hop—the carrier portal, in practice—gets
the same disciplined backoff as everything else without special-casing. When you
treat every integration as an interchangeable adapter, adding the fourth and
fifth downstream system stops being a project and becomes a class.
The dead-letter queue deserves more respect than it usually gets. It is not a graveyard—it is a replay buffer. Each dead-lettered item stores the offending step, the error, the attempt count, and the original payload, so an operator can fix the upstream cause and replay from the stored document rather than chasing it through email. Designing for replay from the start is what separates a pipeline you can run unattended from one that pages you nightly.
Two adjacent lessons round this out. First, instrument latency at the step level, not just end to end—the p95 that matters is almost always one slow connector hop, and a per-step histogram tells you which one before a customer does. Second, when the state store grows, the same care you apply to job reliability applies to schema changes: evolving the invoice and dead-letter tables without taking the pipeline down is its own discipline, the kind I covered in Zero-Downtime Migrations at Scale and in the query-shape work behind Hunting N+1 Systematically. A workflow engine is only as durable as the store underneath it.
Run it
The full source is on GitHub — github.com/tachyurgy/invoice-automation-engine (MIT). Cloned fresh, it runs with its base toolchain and nothing else. Here is an actual run:
$ python3 app.py
{"ts": "2026-06-07T04:08:38.099Z", "level": "INFO", "logger": "iae", "msg": "pipeline.start", "invoices": 6, "seed": 1}
{"ts": "2026-06-07T04:08:38.099Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-001", "step": "intake", "attempt": 1, "duration_ms": 0.12}
{"ts": "2026-06-07T04:08:38.099Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-001", "step": "extract", "attempt": 1, "duration_ms": 0.02}
{"ts": "2026-06-07T04:08:38.099Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-001", "step": "normalize", "attempt": 1, "duration_ms": 0.02}
{"ts": "2026-06-07T04:08:38.099Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-001", "step": "validate", "attempt": 1, "duration_ms": 0.01}
{"ts": "2026-06-07T04:08:38.099Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-001", "step": "approval", "attempt": 1, "duration_ms": 0.01}
{"ts": "2026-06-07T04:08:38.289Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-001", "step": "provision", "attempt": 1, "duration_ms": 189.25}
{"ts": "2026-06-07T04:08:38.290Z", "level": "INFO", "logger": "iae", "msg": "workflow.complete", "correlation_id": "inv-001", "final_step": "provision"}
{"ts": "2026-06-07T04:08:38.290Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-002", "step": "intake", "attempt": 1, "duration_ms": 0.07}
{"ts": "2026-06-07T04:08:38.290Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-002", "step": "extract", "attempt": 1, "duration_ms": 0.03}
{"ts": "2026-06-07T04:08:38.290Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-002", "step": "normalize", "attempt": 1, "duration_ms": 0.03}
{"ts": "2026-06-07T04:08:38.290Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-002", "step": "validate", "attempt": 1, "duration_ms": 0.02}
{"ts": "2026-06-07T04:08:38.290Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-002", "step": "approval", "attempt": 1, "duration_ms": 0.02}
{"ts": "2026-06-07T04:08:38.386Z", "level": "WARNING", "logger": "iae", "msg": "step.retryable_failure", "correlation_id": "inv-002", "step": "provision", "attempt": 1, "max_attempts": 4, "will_retry": true, "error": "carrier-portal: transient upstream error (HTTP 503)"}
{"ts": "2026-06-07T04:08:38.597Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-002", "step": "provision", "attempt": 2, "duration_ms": 189.54}
{"ts": "2026-06-07T04:08:38.597Z", "level": "INFO", "logger": "iae", "msg": "workflow.complete", "correlation_id": "inv-002", "final_step": "provision"}
{"ts": "2026-06-07T04:08:38.597Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-003", "step": "intake", "attempt": 1, "duration_ms": 0.13}
{"ts": "2026-06-07T04:08:38.597Z", "level": "WARNING", "logger": "iae", "msg": "step.retryable_failure", "correlation_id": "inv-003", "step": "extract", "attempt": 1, "max_attempts": 3, "will_retry": true, "error": "OCR engine timeout (transient)"}
{"ts": "2026-06-07T04:08:38.620Z", "level": "WARNING", "logger": "iae", "msg": "step.retryable_failure", "correlation_id": "inv-003", "step": "extract", "attempt": 2, "max_attempts": 3, "will_retry": true, "error": "OCR engine timeout (transient)"}
{"ts": "2026-06-07T04:08:38.663Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-003", "step": "extract", "attempt": 3, "duration_ms": 0.31}
{"ts": "2026-06-07T04:08:38.664Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-003", "step": "normalize", "attempt": 1, "duration_ms": 0.12}
{"ts": "2026-06-07T04:08:38.664Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-003", "step": "validate", "attempt": 1, "duration_ms": 0.11}
{"ts": "2026-06-07T04:08:38.664Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-003", "step": "approval", "attempt": 1, "duration_ms": 0.07}
{"ts": "2026-06-07T04:08:38.815Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-003", "step": "provision", "attempt": 1, "duration_ms": 150.94}
{"ts": "2026-06-07T04:08:38.816Z", "level": "INFO", "logger": "iae", "msg": "workflow.complete", "correlation_id": "inv-003", "final_step": "provision"}
{"ts": "2026-06-07T04:08:38.816Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-004", "step": "intake", "attempt": 1, "duration_ms": 0.14}
{"ts": "2026-06-07T04:08:38.816Z", "level": "ERROR", "logger": "iae", "msg": "step.terminal", "correlation_id": "inv-004", "step": "extract", "attempt": 1, "error": "unparseable document: missing invoice_number, amount"}
{"ts": "2026-06-07T04:08:38.817Z", "level": "ERROR", "logger": "iae", "msg": "workflow.dead_letter", "correlation_id": "inv-004", "step": "extract", "attempts": 3, "error": "unparseable document: missing invoice_number, amount"}
{"ts": "2026-06-07T04:08:38.817Z", "level": "INFO", "logger": "iae", "msg": "workflow.skip", "correlation_id": "inv-005", "step": "intake", "reason": "duplicate of inv-001 (fingerprint=acme robotics|AR-7781|4200.00)"}
{"ts": "2026-06-07T04:08:38.817Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-006", "step": "intake", "attempt": 1, "duration_ms": 0.05}
{"ts": "2026-06-07T04:08:38.817Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-006", "step": "extract", "attempt": 1, "duration_ms": 0.04}
{"ts": "2026-06-07T04:08:38.817Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-006", "step": "normalize", "attempt": 1, "duration_ms": 0.04}
{"ts": "2026-06-07T04:08:38.817Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-006", "step": "validate", "attempt": 1, "duration_ms": 0.04}
{"ts": "2026-06-07T04:08:38.817Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-006", "step": "approval", "attempt": 1, "duration_ms": 0.03}
{"ts": "2026-06-07T04:08:38.909Z", "level": "INFO", "logger": "iae", "msg": "step.success", "correlation_id": "inv-006", "step": "provision", "attempt": 1, "duration_ms": 92.03}
{"ts": "2026-06-07T04:08:38.910Z", "level": "INFO", "logger": "iae", "msg": "workflow.complete", "correlation_id": "inv-006", "final_step": "provision"}
================================================================
OBSERVABILITY / SLO REPORT
================================================================
Workflows processed : 6 (ok=5, failed=1)
Success rate : 83.3%
Total retries : 3
Dead-letter queue : 1
Overall p50 latency : 0 ms
Overall p95 latency : 189 ms