Shadow usage export to OpenMeter (Phase 1)

Engineering behaviour, verified facts, and limits for the observe-only usage export. Excluded from the documentation site, search, and LLM corpus.

Phase 1 proves that committed Platinum usage can reach an external aggregator asynchronously, survive delivery failure, and produce matching totals. It does not make that aggregator authoritative for anything.

What stays authoritative

Platinum owns billable lifecycle intervals, request classification, prices, exemptions, wallet balances, the credit ledger, enforcement, and every Stripe interaction. The aggregator receives physical quantities and nothing else. No code path in this integration reads or writes a price, a balance, a ledger row, or an enforcement decision, and no customer-facing total changes.

If the whole integration were deleted, native billing would produce identical money.

Pinned versions

Every OpenMeter release ever cut is a v1.0.0-beta.N; there is no GA and never has been. beta.232 is the newest of that de-facto production channel, not a stable release, and there is no semver contract between betas. Diff api/openapi.yaml before moving. The project's own quickstart tracks a mutable latest, which we do not: an unpinned aggregator makes a replay or recovery result unreproducible.

Metering needs the server, the sink worker, Kafka, ClickHouse, Postgres (meter definitions live there, not in ClickHouse), and Redis. The balance worker, billing worker, notification service, jobs runner and Svix are the commercial billing surface and are absent, with credits.enabled: false. Verified: the remaining six come up healthy and serve ingest and query.

Redis is optional upstream and mandatory here. The default dedup store is in-process memory that a restart erases, and the ClickHouse events table is a plain MergeTree with no TTL and no storage-level dedup, so the Redis expiration is the only thing between a replayed event and a doubled aggregate.

Shape

Capture writes one row inside the native settlement transaction. A separate worker delivers it. A reconciler compares totals. Nothing on the request, lifecycle, or enforcement path ever calls the aggregator.

settleRequestBucket tx ──► billing_usage_outbox ──► exporter ──► OpenMeter
        │                          │                                │
   native billing            (or) export_gaps                  meter query
        └──────────────── reconciler compares all three ────────────┘

Capture

The hook is the last statement of settleRequestBucket, immediately after the cumulative claim advances. That position is not cosmetic. Everything before it is what makes an unknown COMMIT outcome safe to retry, and it also means the existing replay guard returns before the hook on a confirmed replay, so one settled interval yields exactly one export row without a dedup table of its own.

The transaction grows by at most one statement, and by none when capture is off.

There is no try/catch and no SAVEPOINT around the insert. A failed statement poisons a Postgres transaction; catching it in JS and continuing only fails at COMMIT with a worse error, and a savepoint would double the statement count inside a 2 s budget to survive a case only a bug or a full disk produces.

Quantities are deltas

The native receipt is cumulative. The exported quantity is the advance, to - from, enforced by a table constraint. Re-exporting a growing running total would inflate a SUM meter on every send.

Delivery

Claims use FOR UPDATE SKIP LOCKED with a lease, round-robin across organizations. Leases outlast the HTTP call and acknowledgements are conditional on still holding one, so a slow acceptance arriving after takeover cannot mark rows another worker is mid-flight on. Blue-green rollout runs two control planes briefly; single-writer is never assumed.

Only a definitive per-row rejection advances a row toward quarantine. Timeouts, resets, black holes and 5xx are unknown outcomes: the aggregator may already hold the bytes. Counting those against a row would quarantine an entire backlog for something no row did wrong, so an outage is bounded by the backlog ceiling instead of by an attempt counter.

Verified behaviour of the pinned release

Measured against the running stack, not read from documentation.

Four of these changed the implementation:

• A 204 is not evidence. A value the aggregator cannot parse is accepted and then dropped, so it never reaches a total and nothing but reconciliation notices. This is the reason the reconciler exists.
• An absent user must be an absent key, not a null. Internal traffic legitimately has no user; sending null would have made every unattributed request a poison event.
• A batch rejection is not itemised, and one bad event takes the batch with it. So a rejected batch is resent one event at a time. Without that, a single poison row blocks every healthy tenant that shared its batch.
• source must be a function of the deployment lane alone — never a host, pod or pid — or a retry after a rollout lands as a second aggregate.

The documented-inclusive / measured-half-open discrepancy on to is why the reconciler ignores the response bounds entirely and filters every window by its own windowStart against a half-open period. That is correct under either behaviour and stays correct if a future beta changes its mind.

Measured cost

Measured 2026-09-12 on one 8 vCPU / 15.6 GiB evaluation host running the six pinned services, the control-plane code, and Postgres, all together. Single-host evaluation numbers. Not a capacity model and not a production sizing; the host is not an approved deployment target.

What sets the export rate

Export volume follows committed SETTLEMENTS, not requests. One settlement writes one outbox row however many requests it covers. A stream is one (sandbox, org, user, UTC hour, price, policy), and on the periodic flush the buffer refuses to settle a stream until it is 60 seconds old (requestBuffer.ts, run(): !captureActive && now - firstAt < 60_000), so in steady state

settlements/s  ~=  active streams / 60      capped at 256 per flush

Measured, driving real traffic through reserveRequest:

953 200 requests produced 8 035 events. A control plane cannot exceed about 167 settlements/s, because the buffer holds at most 10 000 streams and each settles once a minute. Ten times the 26.8/s figure therefore needs a second process, which is how the 259.7/s row above was produced.

The cost of capture, per settlement

settleRequestBucket called directly, so the only difference between the arms is the one INSERT. Settlement runs on a private max-one connection by design (requestSettlementDb.ts), so concurrency queues rather than parallelises and c1 is the production shape.

+0.48 ms p50, +2.26 ms p95 against a 3 s transaction deadline, and a 30 % reduction in the theoretical per-connection settlement ceiling (646 -> 449/s). At the rates a real fleet produces that ceiling is not the binding constraint: 157.5/s measured against 449/s available is 3.5x headroom on one process. It would matter on a control plane whose streams settle far more often than once a minute, which is not the shape this buffer produces.

At the full-path level the difference is not measurable: 26.85 settlements/s with capture off against 26.83 with it on, over 240 s each, because the rate is set by the flush cadence rather than by the cost of a settlement.

Export throughput

A tick claims and delivers until the queue is empty or its 10 s budget is spent. Measured by seeding a 200 000-row backlog and draining it through the real aggregator:

The shipped defaults sustained 261.4 delivered/s against a live 259.7 settlements/s -- keeping up, with a backlog averaging 121 rows and an oldest pending age of 2 s, and draining fully afterwards. They are keeping up at capacity, not with headroom: ten times the measured rate is roughly where batch 100 at concurrency 1 runs out. Raising them to 500 and 4 moves the ceiling to ~2 300/s.

Before the drain loop this was batch_size / tick — 50/s at the defaults, chosen by a sleep rather than by anything about the workload. The per-round cost is dominated by the aggregator round trip (~350-400 ms for a 100-event POST), so throughput scales with batch size and with how many batches are in flight, not with local CPU.

End-to-end parity

Across the whole campaign — every phase, every outage, every restart, and three concurrent exporters draining one outbox — ClickHouse held 551 547 events under 551 547 distinct ids. Nothing was stored twice.

Where the load actually lands

Per-container, sampled once a second at 157.5 settlements/s:

Kafka saturates first, by an order of magnitude over everything else. Control-plane RSS 117 MiB peak; native Postgres connections peaked at 8 with two control planes running (PT_DB_POOL_MAX=8). Redis held 47 916 dedup keys with zero evictions. Kafka consumer lag peaked at 1 370 under steady load and returned to 0; under the 200 000-row burst the sink was still catching up for minutes afterwards, which is the visibility lag, not loss.

The backlog probe

Every process samples for itself, so this cost is paid N times. Bounded by the partial index and by the configured ceiling:

Dependency outages, measured

Each case: 30 000 rows pending, the dependency stopped BEFORE the first delivery tick, restored ~90 s later, then drained and compared against the aggregator's own totals.

Two things generalise from that table.

Native billing was unaffected in every case, which is trivially true because nothing here touches it — and the run confirms it rather than assuming it.

No outage of any length quarantined anything, and attempt never advanced. defer_count did: it reached 6 during the OpenMeter outage, escalating the retry interval toward its 60 s cap. That is what a transport failure is allowed to cost.

A 204 really is not evidence

With the SINK WORKER stopped, the API accepted all 30 000 events and the exporter recorded 30 000 successful deliveries. Zero of them were queryable until the worker came back. Transport acceptance and aggregation are different events, and only a comparison notices the gap.

The sink worker can wedge, silently, and stay "healthy"

Restarting Kafka and then ClickHouse under load left the sink worker consuming nothing, permanently: the last log line was a broken-pipe notice, consumer lag froze at 90 000, three runs' worth of events stayed stranded in Kafka, and /healthz kept returning 200 the whole time. Ingest kept returning 204 and the exporter kept reporting success. The last three rows of the table above are all that one wedge.

Recovery is an operator action — docker compose -p ptom-shadow restart sink-worker — after which lag drained at roughly 540 events/s, reached zero, and all five cases then reported exact parity: 90 000 against 90 000. Nothing was lost; it was unqueryable for 20 minutes while every health signal the aggregator publishes said otherwise. Nothing in that health surface would have raised it. The reconciliation sweep and its freshness gauge are the only signals that would have, which is the concrete reason Phase 1 ships them rather than leaving reconciliation unscheduled.

Storage budget and the honest trade-off

The outbox insert shares the native settlement transaction. A table that could not accept a row would fail the billing write with it. Shadow metering therefore cannot promise all three of zero loss, unlimited outage tolerance, and zero impact on billing.

The chosen policy: at limits.usage_export_max_backlog undelivered rows, capture sheds. Native settlement continues untouched and the skipped quantity is summed into billing_usage_export_gaps, keyed by organization, meter and hour, so the record is bounded by (orgs × hours) rather than growing like the thing being shed. A period with any shed quantity reports as gap and can never report as match, however well the numbers line up.

Because the shed quantity is summed rather than discarded, a gap row is also a coarse replayable source for a later backfill.

The ceiling is evaluated from a cached estimate refreshed off the settlement path, never counted inside the transaction, which runs on one connection under a 2 s statement timeout and a 3 s deadline. A stale estimate fails toward capture.

The estimate is per process, and so is the sampler. captureMode() reads module memory in whichever process is settling, so a sampler that runs only on the elected leader protects only the leader: every follower, both halves of a blue-green swap, and any process that has just lost the lease would enforce no ceiling at all and keep inserting. startBacklogSampler is therefore started outside globalLeader.runAsLeader in server.ts, in every process.

A shared published value was rejected for the same reason the defect existed: something has to publish it, that something is a leader again, and the staleness window lands on exactly the handovers that broke it. Nothing is handed over, so a leader change and a blue-green overlap are not special cases.

The cost of that choice is N bounded index probes per interval rather than one. The probe is a LIMIT-bounded index-only scan over the partial index on state <> 'delivered', so it reads only undelivered rows and stops at the operator's own ceiling: measured at 1.9-3.8 ms average, 20 ms worst case, under 500 undelivered rows. limits.usage_export_backlog_sample_seconds trades ceiling freshness against that cost.

Proved across real operating-system processes in ceiling.multiproc.int.test.ts, because it cannot be proved inside one: a single test runner shares one module registry, so "the leader" and "the follower" would be the same variable. The suite's third role runs no sampler -- exactly the pre-change follower -- and must capture straight past the ceiling, which is what shows the other two assertions can fail.

Operator actions before exhaustion, in order: confirm the aggregator is reachable and delivery is enabled; check oldest pending age rather than depth alone, since a flat depth with rising age means delivery is stuck, not idle; raise the ceiling only if disk allows; if the aggregator will be down longer than the backlog allows, accept shedding and expect gap periods for that window. Do not delete pending or quarantined rows to reclaim space.

Reconciliation

reconcilePeriod and reconcileAll were reachable from each other and from tests, and from nothing else. A leader-scheduled sweep now runs them.

A period is one UTC hour, because usage_ts is a UTC hour: that is the finest grain at which native and captured quantities are comparable at all, and any coarser grain would hide an hour-sized discrepancy inside a day-sized total.

Bounded on the four axes that can grow, all of them configuration:

A sweep that hits the organization ceiling reports reduced coverage rather than comparing a subset and calling the result parity. The four verdicts are reported side by side, never collapsed into a percentage: gap and pending have to be visible next to match, or a fleet that is not being observed reads as a fleet with nothing wrong.

Measured: 25 (organization, hour) comparisons over a 2-hour window in 499 ms, returning 5 match and 20 mismatch.

The 20 were real. Those organizations' outbox rows had been deleted between phases of this campaign, so native and the aggregator both reported 238 560 and the captured record reported nothing. The sweep refused to call that a match even though two of the three numbers agreed -- which is what "capture stopped and nobody noticed" looks like from the outside.

The 5 matches are an hour whose three totals agreed to the event: 119 220 native, 119 220 captured, 119 220 aggregated, per organization.

Metrics, and what is deliberately not in them

Every exported series is a fleet total or one of four fixed statuses. No organization, user, sandbox, period or event id ever becomes a label: a per-tenant label set is unbounded cardinality and publishes the customer list into whatever scrapes it. The identifiers an operator needs to act on a mismatch are in the sweep's log line and in GET /internal/admin/billing/metering-source, behind the admin token.

Alert on, roughly in order of what they mean:

• platinum_usage_export_sampler_age_s above three sample intervals, or -1, on a process whose platinum_usage_export_enabled{gate="capture"} is 1. The backlog ceiling is not being enforced there.
• platinum_usage_export_reconcile_age_s above three sweep intervals, or -1 with the flag on. Nothing is proving the export is correct, and a silent drop then looks exactly like a healthy fleet.
• platinum_usage_export_reconcile_periods{status="mismatch"} > 0.
• platinum_usage_export_oldest_pending_s climbing past the sweep's settlement delay: delivery is falling behind and periods will start reporting pending rather than a verdict.
• platinum_usage_export_rows{state="quarantined"} > 0: a human has to resolve a payload the aggregator will never accept.
• platinum_usage_export_drained == 0 across consecutive scrapes: ticks are hitting their drain budget, so the backlog is growing.

Retention

Delivered rows accumulate for ever, and every one is paid for three times: by the partial index behind the backlog probe, by the delivery scan, and by the reconciliation read.

Deleting them on age would be wrong. The captured side of a parity comparison is read from this table, so a plain DELETE would make every pruned period reconcile as a mismatch against its own history -- retention manufacturing the discrepancy it exists to keep cheap to find. A pruned row's quantity is therefore folded into billing_usage_export_rollup by the same statement that removes it, and the reconciler reads outbox plus rollup. There is no instant at which a row is gone and unaccounted for.

Never pruned, at any age: pending, delivering, quarantined. The effective floor is never below the aggregator's deduplication window plus the exporter's own margin, nor below the reconciler's lookback, whatever limits.usage_export_retention_days is set to. 0, the shipped default, deletes nothing.

How fast it removes them, which is a separate question

What may be deleted and how much of it actually is are different questions, and only the first one has floors. A pruner whose ceiling sits below the ingest rate reports a successful run every hour and still loses ground for ever, which is the shape of the billing_events growth already on record in this codebase.

The drain rate was measured on PostgreSQL 16.11 against a 950,000-row outbox across 10,000 organizations, one row per organization-hour so every row in a batch is its own rollup group -- the worst case for the fold:

Throughput is flat from 1k to 10k and collapses above it, so a bigger statement buys nothing and only lengthens the lock hold. The batch size is therefore a lock-hold bound and throughput comes from the run budget and the cadence:

• a run works until it drains, hits limits.usage_export_retention_run_budget_ms (5 s), or runs out of eligible history;
• a run that DRAINED sleeps an hour -- there is no urgency in pruning history once there is none left to prune;
• a run that stopped on its budget sleeps limits.usage_export_retention_busy_interval_seconds (30 s) instead;
• limits.usage_export_retention_pace_ms (25 ms) is paid between statements, never inside one, so the connection is back in the pool for the pause. This, not the batch size, is what stops a long catch-up from holding one slot of a PT_DB_POOL_MAX=8 pool continuously;
• a statement slower than 750 ms halves the batch, to a floor of 100. Under contention the answer is a shorter transaction, not a longer one.

Measured end to end on that rig: 49,833 rows per 5.06 s run, one run every 35.1 s while behind, i.e. 1,421 rows/s or 5.1M rows/hour sustained, at a 14.4% duty cycle on one connection. Against the rig's measured settlement rates that is 9.0x headroom at 157.5/s and 5.5x at 260/s.

When it is not keeping up

Both ways this fails are failures of absence, so both are measured as the age of the oldest row rather than as a count of anything that happened:

• platinum_usage_export_retention_lag_s -- seconds of delivered history sitting behind the cutoff that nothing has removed. 0 while retention keeps up. One index probe (min(delivered_at) on the partial retention index), so it stays O(log n) on the table size it exists to prevent.
• platinum_usage_export_oldest_delivered_age_s -- reported whether or not retention is enabled. With retention off, which is the shipped default, this is the age of a table nothing prunes and the only gauge that shows it growing.
• platinum_usage_export_retention_drained -- 0 means every run is stopping on its budget.

alert-cron pages on both: usage_export_retention_behind (lag over PT_ALERT_USAGE_EXPORT_LAG_SEC, 6 h) and usage_export_retention_off (capture on, no retention window, oldest row over 30 days).

The one switch

billing.metering_source chooses which system is authoritative for metering and billing, and the admin billing panel renders it as a single two-position switch.

It accepts exactly one value today. The integration is observe-only: usage is exported after native settlement has already priced and debited it, one meter has a producer, and no reader anywhere consults the aggregator for a balance, an invoice, a spend limit or a quota. openmeter is therefore refused at write time with the reason, rather than accepted and then ignored -- a stored value the fleet is not obeying would show an operator "openmeter" over native billing.

The selectable set comes from the server's availability list rather than from a constant in the panel, so when OPENMETER_CAN_BE_AUTHORITATIVE flips the same control becomes live: same key, same audit trail, same switch. What has to exist first is listed in openmeter/meteringSource.ts.

Underneath the switch the panel shows what the shadow export is actually doing -- sampler state, the three gates, wired meters, backlog depth and oldest row, quarantine, and the last reconciliation verdict with its age -- so "observe-only" is a state an operator can see rather than a claim they have to take on faith.

Running the tests

The integration suite needs a database with both migration lanes applied, not just bun run db:migrate. The hand-written lane in apps/api/migrations/ builds credit_ledger's unique idempotency key concurrently, and without it a replayed debit silently succeeds instead of collapsing. Several suites depend on that key without asserting it, so on a drizzle-only database they pass while proving nothing.

This matters here specifically: the outbox insert shares the transaction whose replay safety that key underwrites. The outbox's own uniqueness is its primary key in 0086, in the drizzle lane, so it does not inherit that gap -- and it must stay there rather than moving to the hand-written lane.

Real-stack files (e2e.int.test.ts, outage.e2e.int.test.ts) skip unless PT_USAGE_EXPORT_URL is set, and the outage file also needs PT_OM_RIG_SSH because it stops and starts real containers; PT_OM_RIG_DIR points at the checkout's infra/openmeter-rig when the rig is not at /opt/pt-om-rig.

Reach a rig on a remote host over a tunnel -- it binds loopback only and has no authentication:

ssh -N -L 49888:127.0.0.1:49888 <rig-host> &
PT_USAGE_EXPORT_URL=http://127.0.0.1:49888 \
PT_OM_RIG_SSH=<rig-host> PT_OM_RIG_DIR=<checkout>/infra/openmeter-rig \
DATABASE_URL=... bun test src/openmeter/e2e.int.test.ts

Limits

• Not exactly-once. Delivery is at-least-once; ingest is idempotent on (id, source) within the aggregator's dedup window. That combination is effectively-once only for a retry that lands inside the window of its first accepted ingest. A row whose first send has aged out is held back as dedup_window_exceeded rather than resent, because past the window a resend is a silent double count. Redis loss collapses the window to zero.
• Per-event lookup expires before aggregates do. GET /api/v1/events only covers ~32 days, so the "did this specific id arrive?" recovery path is unavailable for anything older, even though its quantity is still in the meter total.
• Does not fix pre-commit loss. Request counts held in memory are lost on an abrupt process exit before Postgres commits. Shadow capture is downstream of that commit and cannot observe what never committed. Tracked as #933.
• Parity has a 7-day clock on one leg. The native archiver removes billing_events and billing_request_watermarks past its replay fence, so native-versus-captured must be checked inside that window. Captured-versus- remote has no such limit.
• A period is never final. Failed batches are retained while the process lives, so an hour can settle much later. Report parity as of a time, with the undelivered count; do not stamp periods closed.
• Stream ids are per-process. A control-plane restart mints a new stream for the same sandbox-hour, so one hour legitimately produces several events. This is correct for a SUM meter and tests must not assert one event per hour.
• The sink worker has no honest health signal. Measured: after a Kafka restart followed by a ClickHouse restart it stopped consuming for good while /healthz returned 200 and ingest kept returning 204. Monitor Kafka consumer lag and the reconciliation verdicts; do not monitor the sink's own health.
• A control plane cannot exceed ~167 settlements/s, because the request buffer holds at most 10 000 streams and each settles about once a minute. Higher export rates need more control-plane processes, not a bigger batch.
• Beta dependency. See the pinning section.

Enablement

flags.usage_export_capture_enabled and flags.usage_export_delivery_enabled are separate and both default off.

Capture off is genuinely inert: no outbox write, no added statement, no polling, no network call. Delivery off keeps capturing and stops all network activity, which is how an aggregator is paused without losing usage — bounded by the backlog ceiling.

First enablement covers new committed usage only. Earlier usage requires an explicit backfill, which Phase 1 does not implement.

PT_USAGE_EXPORT_URL, PT_USAGE_EXPORT_TOKEN and PT_USAGE_EXPORT_DEDUP_WINDOW_HOURS are boot-time environment values. The endpoint is not a runtime flag because repointing a fleet mid-period would split one period across two destinations. The token is never a config row: config rows are readable from the admin surface.

The deployed dedup window and PT_USAGE_EXPORT_DEDUP_WINDOW_HOURS must agree. The constant is a correctness bound, not a tuning knob: it decides when a resend stops being idempotent.

Not built

Wired: request usage only. The contract names cpu_seconds, ram_gib_seconds and storage_gib_hours so it can extend without a schema change, and no producer emits them. A resource gets an exporter when it has a stable cumulative native claim to anchor id = <meter>:<ref>:<from>:<to> and its delta; until then there is nothing honest to export, and none of its lifecycle accounting is this integration's to redo.

Also not built: backfill of pre-enablement usage, an automated requeue for quarantined rows, and any customer-facing read of aggregator data. Retention is built but ships disabled.