This is the deploy plan for the multi-region work. It assumes
docs/us-east-readiness.md for what was built and whether it works; this
file is only about getting it there without a surprise.
Every stage has a gate. A gate that does not pass is a stop, not a note.
push to main → Development (deploy-cp.yml, env=development, profile=dev)push to staging → Stagingpush to prod → Production
main is the integration line. The promotion workflows fast-forwardmain → staging, then staging → prod; a branch can only deploy to the
environment it is. So nothing reaches production except by having sat on main,
then staging, first. Two other workflows fire on main: deploy-hosts.yml
(hosts/**, apps/edge/**) and db-migrations.yml.
db-migrations.yml is a verification job. It migrates a throwaway
pt_shadow database inside the runner — it does not touch dev, staging or
production. What actually migrates a real database is the deploy, which runs
db:migrate and then assert-journal-applied.ts on the target.
There are 21 pull requests. They are all CI-green and all mergeable against
their own base, which is not the same as mergeable after each other: main is
45 commits ahead, main enforces strict up-to-date status checks, and
replaying the queue on top of main conflicts on seven of them.
Land it as one integration merge.us-east/integration is already the
resolved merge of all 21, it is what every suite ran against, and main has been
merged into it once — one conflict resolution instead of twenty-one. The slice
PRs stay as the review record and close as "landed via ".
The alternative is a serial train: for each PR, update the branch, wait for 16
checks, merge, and watch that merge invalidate the other twenty. That is ~21 CI
cycles, and every branch update is a fresh chance to hit the migration trap
below. It buys a per-slice merge commit that a squash discards anyway.
drizzle reads one watermark, once, before it applies anything:
select id, hash, created_at from drizzle.__drizzle_migrations order by created_at desc limit 1if (!last || Number(last.created_at) < migration.folderMillis) …apply
An entry whose when is at or below that watermark is not an error and not a
warning. The loop does not enter, no row is inserted, and the process exits 0
saying "up to date". The SQL never runs.
This already happened. The stacked branches numbered their migrations from
a base main later overtook, picking when values above main's 0101
(snap_refs_capture_token) and 0102 (billable_since). us-east applied the
branch migrations first, its watermark jumped past both of main's entries, and
they became permanently unreachable — sandboxes was missing two columns and
nothing failed. assert-journal-applied.ts is the only reason anyone knew.
Only an environment that applied a branch's migrations before main's is
affected. Staging and production have main's entries and none of the
branch ones, so the new migration lands normally there.
Repair (dev regions only): back up and delete the ledger rows above the
last entry the journal and the database agree on, then re-run db:migrate.
\copy (select id,hash,created_at from drizzle.__drizzle_migrations where created_at > <last agreed>) to '/tmp/ledger-backup.csv' csv headerdelete from drizzle.__drizzle_migrations where created_at > <last agreed>;bun run db:migrate && bun src/db/assert-journal-applied.ts
This is only safe because every statement being re-applied is idempotent.
Never raise the when of an entry main has already shipped to get around
this. On every database that already applied it, it would run a second time —
and ALTER TABLE … ADD COLUMN has no IF NOT EXISTS.
The five multi-region tables (template_regions, billing_settlement_outbox,
billing_settlement_claims, audit_outbox, region_state) are now one
migration regenerated on top of main's head, not five renumbered by hand. A
hand-renumbered snapshot still describes the schema before main's 0101 and
0102, so the next db:gen reads their columns as missing and emits a DROP.
Every statement in it is CREATE TABLE IF NOT EXISTS, CREATE INDEX IF NOT EXISTS or a guarded DO block, so it re-applies as a no-op where the tables
already exist.
The file wins. The process is only a fallback for a host whose env file has
no DATABASE_URL. On us-cp the file says
postgres://platinum_admin@127.0.0.1:5432/platinum — the local two-database
netem rig — while the control plane itself runs against the us-east PlanetScale
database. So every deploy to that host has been migrating the rig, and the
database the region actually serves from never saw the new migrations. Nothing
reported it: the migrate step succeeded, against the wrong database.
That is how us-east ended up serving on a schema two columns behind its own
code. Deploying a tree whose billingFinalize selects billable_since onto it
made every create fail with
{"error":"column \"billable_since\" of relation \"sandboxes\" does not exist"}
Before any deploy, on every control-plane host:
grep -m1 '^DATABASE_URL=' /etc/platinum.env | sed 's#://[^@]*@#://***@#'sudo tr '\0' '\n' < /proc/$(pgrep -f 'bun.*server' | head -1)/environ \ | grep -m1 '^DATABASE_URL=' | sed 's#://[^@]*@#://***@#'
If those two name different databases, stop: fix the file before deploying, or
the migration lands somewhere nobody is looking. The file's URL must also be
migration-capable — the CP's own runtime role is a PlanetScale pscale_api_*
credential with no DDL rights, so it cannot repair a ledger even if it is the
one the process uses.
Everything multi-region is conditional on a separate global database, which dev
does not have, so dev must behave exactly as before. Run against dev:
verify/feature-matrix-e2e.sh expect the same score as before the mergeverify/e2e-deep.sh 45 pass / 0 fail / 3 skippedverify/stability-e2e.sh 4 scenarios
Gate: no check that passed before the merge fails after it. A single-region
deployment must not be able to tell this landed.
Fast-forward staging → prod. us-east is not serving customer traffic yet;
this stage only puts the multi-region code under EU.
Gates:
assert-journal-applied.ts on the production database.
EU's own suites at their pre-merge scores.
platinum_global_db_circuit_open is 0 and
platinum_settlement_outbox_pending is not growing.
Rollback: revert the integration merge on main and fast-forward again. The
migration does not need reverting — five unused tables and one column are inert
to code that does not read them. Do not drop them to "clean up"; a re-land
would then re-create them and the ledger would disagree.
A second host and a second control plane in us-east. One of each is a
regional outage waiting for a hardware fault, and no code fixes that.
The private VLAN with Hivelocity (ticket open — the reply they need is the
private NIC ens8 02:01:01:b0:00:02 172.16.8.2/22, node port 117052
84:16:0c:bb:4c:d0 172.16.8.10/22, VLAN 2039; 1e:00:ef:00:07:4a is the
VPS public NIC and is not the one to tag).
Decide the admin credential (below), then re-run verify/region-e2e.sh on
both regions and expect 11/11.
Run verify/region-outage-e2e.sh against us-east one final time: 12/12.
placement.regions → api_origin — set 2026-09-18: eu-west →
https://api.platinum.dev (EU serves the apex, so the derived
eu-west.api.platinum.dev never resolved), us-east →
https://us-east.api.platinum.dev. Datacenters preserved verbatim.
Per-region bucket — deliberately NOT set. Declaring buckets arms the
template-residency gate, which then refuses a placement whose template is
not resident in the target region. That is a production placement behaviour
change and belongs in its own change with its own gate, not in this one.
The admin credential. The fan-out forwards the caller's own credential by
design. api_keys and bauth_session are global tables read from each
region's replica, so a dashboard session or API key authenticates in every
region. ADMIN_TOKEN is a per-deployment environment variable and does not:
an ops token from us-east gets a 401 from EU. Either share one token across
the deployments or drive the admin with a session. Until one is chosen, the
fan-out check in verify/region-e2e.sh stays red, and that is honest.
A region's dashboard listing is opt-in (?regions=all). The picker
(GET /v1/regions) is not — it merges siblings by default. If the frontend
never sends regions=all, a user with boxes in both regions sees half of
them and nothing reports an error.
Session replication lag: a user who signs in on EU and immediately hits
us-east can be rejected until the session row replicates.
PT_FANOUT_TIMEOUT_MS is 3 s. A sibling slower than that is reported
unavailable, never waited for.