One database per region

Status: live for the us-east region on real clusters since 2026-09-17 (integration branch on the us-east CP). Production EU control planes are unchanged; the production database only gained four additive tables and a publication.

Why

A control plane pays one network round trip per sequential statement, and a sandbox lifecycle is a chain of them (create ≈ 19 after the round-trip cuts, stop ≈ 14). From Orangeburg to the Frankfurt database that is 90 ms each: +1.4 s on create for a US user versus an EU user, and no amount of statement surgery gets it under about +0.6 s. The fix is that each region's control planes talk to a database in their own region for everything on the sandbox path.

Measured on the rig (regional database local, global database at Frankfurt distance): create 822 ms, stop 318 ms, start 306 ms, GET 21 ms, exec 17 ms — the same numbers as a single local database. A PlanetScale cluster in AWS us-east-1 is 10–12 ms from Orangeburg (measured), which puts a US create at about 1.05 s against about 1.1 s for EU today.

The rule

Every table has exactly one writer side. apps/api/src/db/ownership.ts is the list; ownership.test.ts fails when a table is added without a decision.

One writer per row is what makes this a plain replication setup and not a sync problem: there is nothing to merge.

What a control plane sees

Two connection strings:

• DATABASE_URL — its own region's database. Every read, and every write to a regional table.
• PT_DATABASE_URL_GLOBAL — the home region's database. Writes to global tables only, through dbGlobal / sqlGlobal (db/global.ts, pool PT_DB_POOL_GLOBAL_MAX, default 4). Unset → same handle as db; every deployment today.

PT_HOME_REGION (default eu-west): only home-region CPs can hold the global leader lease and run the global workers (billing sampler, Stripe, invoices, webhook retry, CAS GC, usage export). A regional CP logs once that the scope is not eligible and runs regional work only.

Reads never change. Writes to global tables are the whole job:

• db/ownership.lint.test.ts fails on any db.insert|update|delete(<globalTable>) in source that is not dbGlobal, minus a shrinking allowlist (the billing settlement paths that mix regional and global writes in one transaction — see Open).
• On a regional database the global copies carry REVOKEd write grants for the application role (bun scripts/regional-grants.ts revoke <role> | psql <regional>), so a misrouted write fails loudly instead of forking the data. The application role must not be the table owner or a superuser for this to bite (on the rig: platinum_app, not platinum_admin).

Replication

Home database: CREATE PUBLICATION pt_global FOR TABLE <global tables> (list from scripts/regional-grants.ts), replica identity FULL on tables without a primary key. Regional database: CREATE SUBSCRIPTION pt_regional_<region> ... PUBLICATION pt_global. The subscription applies as its owner, not the application role, so the REVOKEs do not affect it. Requires wal_level = logical on the publisher.

PlanetScale Postgres documents publications and replication slots (outbound). It does not document acting as a subscriber. Until that is confirmed with them, the regional database must be one that can run CREATE SUBSCRIPTION (RDS/Aurora, Neon, Crunchy, self-managed). Fallback if a subscriber is impossible: read global tables through dbGlobal with the existing caches (auth, templates, config, webhooks), at one home round trip per cache miss.

Failure behaviour

• Home database unreachable: regional CPs keep serving the sandbox path in full. Reads of accounts and keys come from the local copy; global writes are audit (buffered), webhook deliveries (retried), key last-used (fire-and-forget) — nothing on the critical path awaits the home database. Replication resumes and catches up when the home database is back; slot retention bounds how long.
• Regional database unreachable: that region is down, others are unaffected.
• A CP crashes: nothing lives in process; the two databases hold every committed fact.

What is centralized, and what is asked for

A customer and an operator both expect one place to look. Three different things sit behind that expectation, and they do not need the same answer.

Money is already one place. credit_accounts, billing_invoices, billing_orders, billing_payment_methods, spend limits and the ledger are global tables. Regional metering never touches them directly: it writes billing_settlement_outbox in its own transaction and the shipper applies each row in the global database exactly once (billing_settlement_claims). An org's balance and its period usage are one number wherever the compute burned. Account work — top-ups, invoices, spend-limit sweeps, order expiry — runs under the home region's leader only.

Identity and configuration are already one place. Organisations, users, API keys, the template catalogue, org secrets, webhooks and app_config are global; every region reads them from a local replica and writes them through dbGlobal.

The index is not, and that is deliberate. sandboxes, hosts, host_commands and the metering rows are regional, because that is what keeps a create at 0.9 s instead of 4 s. So:

Fan-out costs one extra round trip (~90 ms EU↔US), bounded by PT_FANOUT_TIMEOUT_MS. It needs each region's API origin to be resolvable: the regions map (placement.regions) either carries api_origin per region or the origin is derived as <region>.<PT_API_HOST>, which is why the home region — served from the apex — must name its origin explicitly.

If regions ever outnumber what a fan-out should ask on a page render, the next step is a small global projection of the index (id, org, region, state, timestamps) fed by the same outbox pattern as audit and settlement — lists and counts from one table, detail and actions still from the owning region. It is not needed at two regions and it buys eventual consistency it would then have to reconcile.

Watching it

Two things can fall behind without any request failing: the subscription (stale keys, tiers, templates) and the settlement shipper (unbilled usage). Both are on the admin-only /metrics of every control plane (regionHealth.ts):

The replication gauges appear only where PT_DATABASE_URL_GLOBAL names a different database; the outbox gauges appear everywhere (at home they read the local table, which stays empty because home writes settle directly).

The catalogue is global, so anything that walks it and hands hosts work must apply the residency rule the create path applies (templateResidency.residencySql): the prewarm pass and the on-demand materializer did not, and the US host was ordered to materialize Frankfurt-only templates every ~11 s (322 failures in the 30 min before the fix, 0 after). The seed baker checks residencyFor before baking.

Scale

• Regions add a database and a subscription each; the home database publishes once, N subscribers. Watch pg_replication_slots retention on the publisher: a region that falls behind holds WAL.
• Per region, CP count × pool size is the regional database's connection budget, same arithmetic as today (db/client.ts).
• Global write volume from all regions is small and off the critical path: audit rows, deliveries, ledger entries.
• Cross-region reads: GET /sandboxes lists the region behind the regional API the SDK already pins (api_url on create). A request for a sandbox that lives elsewhere is proxied to its region (crossRegionProxy.ts).

Open

1. Billing. Done for all three writers. The flush (billingFinalize), the uptime sampler (billing.ts) and the request meter (requestMeter.ts) record what they owe as targeted rows in billing_settlement_outbox (period_org accrual, period_user accrual, debit) inside their regional transaction, and billingSettlementShip applies them in the global database exactly once (billing_settlement_claims). Writers that own their transaction ship right after commit, in order, stopping at the first failure so one settlement's rows land or wait together; the region-leader shipper loop retries the rest with backoff; a replayed receipt ships what its interrupted attempt owed. The sampler, status sampler, CAS GC and usage export run under the region leader, one per region; only the webhook retry loop stays global. The sampler's rotation cursor moved from app_config to the regional region_state. Debit keys are byte-identical to the pre-outbox ledger keys, so intervals in flight at deploy dedup. Contract change, documented in requestSettlement.int.test.ts: a global-side failure no longer rolls back the regional receipt; the money is held in the outbox and applied once when it can be.
2. Template builds are regional operations on a global row (templates.state/host_id/build_logs). Every write in api/templates.ts, the build ratification in api/hosts.ts and the seed baker's capture climb-down now go through dbGlobal, so a regional control plane can register and build templates (before this, POST /v1/templates on us-east answered permission denied for table templates). Two things still point at the clean end state — a regional template_builds table with templates holding only the catalogue:
   • templates.host_id has a foreign key to hosts(id), a regional table. A host-pinned template therefore cannot be written to the global catalogue from a regional control plane: the referenced host does not exist there. Unpinned templates (everything that goes through CAS and template_regions) are unaffected.
   • The rest of the build state (state, build_logs, build_phase) is regional work recorded on a global row, so it costs a cross-region write per transition.
3. Subscriber choice for the US database (above).
4. The rig's app role and grants live only on us-cp; production roles need the same split (owner ≠ application role).

As deployed (2026-09-17)

• US database: PlanetScale prod-us-east, AWS us-east-1 (the us-east-2.pg.psdb.cloud hostname is a gateway name), PS-20 HA, 100 direct connections. Roles: platinum_admin_us (postgres + pg_create_subscription: migrations, subscription) and platinum_app_us (pg_read_all_data plus per-table INSERT/UPDATE/DELETE on regional tables; global copies revoked via scripts/regional-grants.ts). Both lanes applied: drizzle through src/db/migrate.ts and migrations/*.sql with PGSSLMODE=require. Skipping the second lane leaves the schema drifted from prod's and blocks the initial copy (webhooks.secret NOT NULL).
• Home database: production prod in Frankfurt. CREATE PUBLICATION pt_global for the global tables that exist there (41). Roles platinum_repl (postgres + REPLICATION, the publisher side) and platinum_global_writer (pg_read_all_data + pg_write_all_data, the US CP's PT_DATABASE_URL_GLOBAL). Migrations 0101–0104 applied (additive tables only); 0101 had to be re-stamped above the ledger head because prod had taken a later migration first.
• Replication: CREATE SUBSCRIPTION pt_us_east ... WITH (copy_data = true) on the US database as platinum_admin_us; initial copy of 40 tables took about two minutes; lag holds at 0 s; the slot on prod retains bytes, not megabytes. A PlanetScale cluster can subscribe (role membership in pg_create_subscription is all it needs) and publish (a role with REPLICATION, granted through their postgres role).
• CP environment (/etc/platinum/node.env): DATABASE_URL = app role with ?sslmode=verify-full and no sslrootcert parameter (postgres.js forwards unknown parameters to the server), PT_DATABASE_URL_GLOBAL = writer role, PT_DB_POOL_MIN=4, PT_HOME_REGION=eu-west, PT_S3_PEER_* = the home bucket for template replication. The edge on the host uses the app role too.
• Host re-home: a host whose key was minted in another database re-registers with a fresh htok_ from POST /v1/admin/hosts/bootstrap-tokens placed in ADMIN_TOKEN of /etc/platinum/host.env; the new key is written through dbGlobal and replicates back for host auth.
• Templates: the catalogue is global; residency rows (template_regions) are regional; chunks fan out as <prefix>/chunks/<sha[0:2]>/<sha>; a control plane's own region falls back to PT_S3_BUCKET, the home region to PT_S3_PEER_BUCKET. The warm-pool baker bakes only resident templates — before that gate it cordoned the only US host through the boot-health guard.

Measured, real clusters, US vs production EU

US = us-east (OGB1) on its own PlanetScale us-east-1 cluster with Frankfurt as the global database; EU = production, unchanged code. Milliseconds, medians.

Where a create's time goes now (host agent, us-east): materialize 11 ms, clone rootfs 30-41 ms, TAP 21 ms, start VM 98 ms, guest ready 551 ms, control plane ~250 ms.

Three things bought this, and all three apply to EU unchanged:

1. Fewer database round trips per route (#1110-#1113, #1120, #1040) — what makes a regional database worth having.
2. The guest kernel is staged decompressed (#1138). Cloud Hypervisor was handed a bzImage, so every microVM decompressed 15 MB into 66 MB on its single vCPU at every boot: 720 ms to guest-ready, against 552 ms for the byte-identical kernel pre-decompressed. infra/deploy-hosts.sh converts a host's staged kernel on roll and keeps the bzImage for rollback.
3. A resume no longer waits for a best-effort keepalive (#1139). The net-keepalive launcher ran sleep 0.5 inside the guest so the host could see it had started; the host resume phase read 627 ms while the guest answered vsock in 40 ms.
4. A cloned rootfs is grown without a precautionary fsck (#1141). resize2fs refuses a filesystem that needs checking, so the check runs only when it asks: 41 ms off every create that grows.

Kernel choices measured on the US node (5 boots each, guest-ready p50), for the record: