Volume fan-out: map, measurements and plan

Status: in progress, October 2026. This file maps how a volume reaches many sandboxes today, records what the prod E2E of the parent/children workload measured, and lists the work that makes that workload fast. The E2E lives in verify/good-volumes/e2e/fanout/parent-children.ts; every item below is judged by re-running it.

The workload

One parent sandbox and N child sandboxes, at most X alive at once (N > X). Nothing is declared at create: each sandbox attaches its volumes after it is running, at paths chosen then, and changes them during its life.

• results: each child writes into children/<name>/, the parent reads all of them live.
• tasks: the parent writes task.json, every child reads it from a read-only mount.
• Results outlive every sandbox: a new parent, or the file API with no sandbox, reads them all.

Map

                  ┌──────────────────────── control plane (apps/api) ─────────────────────────┐
  POST attach ──▶ │ api/volumes.ts attach → vol/mounts.ts prepareMounts → host command         │
                  │ vol/commits.ts submitCommit: base == head and whole volume → fast append   │
                  │                              otherwise → side commit + merge job           │
                  │ vol/mergeQueue.ts + vol/gateway.ts: one merge batch in flight per volume,  │
                  │                              on one gateway host                           │
                  │ vol/driveSync.ts: head moved → volume.sync to every live mount behind head │
                  │ vol/driveLog.ts: closed-file changes → volume.logapply to whole-volume     │
                  │                              RW drive mounts (subdir mounts excluded)      │
                  │ scheduler.ts: soft preference for the host that last mounted the volume   │
                  └───────────────┬─────────────────────────────────────────▲──────────────────┘
                     host commands│                                         │commit / merge results
  ┌───────────────────────────────▼──── host (hosts/host-agent) ────────────┴──────────────────┐
  │ vold: one ublk device per mount = private CoW overlay (epoch-N.ovl) over a base image    │
  │       base image = 1 MiB content-addressed blocks: L1 host cache → regional L2 → S3      │
  │ vol_attach.go: ch-remote add-disk (virtio-blk hotplug) → guest mounts ext4 by serial     │
  │ commit: freeze guest fs → hash dirty blocks → upload → CP head swap                      │
  │ vol_sync.go: commit own changes, wait, diff against head, apply file by file in guest    │
  │ vol_logapply.go: apply other writers' logged file changes guarded by base version        │
  │ volgw: merges side commits into head with userspace ext4 (pt-volfs)                      │
  └──────────────────────────────────────────────────────────────────────────────────────────┘
                                 S3: P/vol/<org>/{blocks,maps,idx}/…, commits/<id>/{image.json,index.bin}
                                 Postgres: volumes.head_commit_id, volume_mounts, merge jobs, drive log

Limits that matter here: 8 mounts per sandbox (policy, vol/mounts.ts:24), 1024 ublk devices per host (vold/ublk/limit.go), no limit on sandboxes per volume. No live migration exists; moves are stop → copy → start.

What prod measured (2026-10-17, api.platinum.dev)

Lag is child write + sync returned → first parent ls/cat that shows it. All correctness checks passed in every run: 0 missed writes of 1 064, 116/116 payload checksums, results intact after parent stop/start, after parent deletion and through the file API.

Failures seen under load:

1. About 11% of attaches at ~40 concurrent returned 503 database connection reset, retry.
2. A retried attach whose first attempt had succeeded got 409 path_exists: attach is not idempotent.
3. Attach p50 4.8 s, p90 11 s, max 17 s; moving the parent's mount took 24 s (subdir) to 62 s (whole).

Why it degrades: the causal chains

Traced through the code after the E2E (file:line as of origin/main 8e4897180).

Subdir children (the parent/children pattern).

1. A child's commit is always a side commit: fast append needs subPath === '' (apps/api/src/vol/commits.ts:281).
2. Merges run one batch at a time per volume on its gateway host (commits.ts:417-467, vol/mergeQueue.ts).
3. Each head move sends volume.sync to every live drive mount behind head, including children whose own folder did not change (vol/driveSync.ts:24-50).
4. Each such sync commits, waits for its merges (hostApi.ts:260-329), then derives a sub_path image of the new head (hostApi.ts:343 → mounts.ts:135-190).
5. Sub_path derivations take the same per-volume gateway mutex as merges (hosts/host-agent/internal/volgw/subimage.go:55, merge.go:135): ~40 builds queue ahead of the next merge, per head move, so work grows with writers².
6. The drive change log, which shows a closed file to other mounts in ~1.5 s, refuses subdir writers (vol/driveLog.ts:325, mounts.ts:466), so the parent only sees a child after all of the above. Without the log policy a child's own commits also stall at 3 merges in flight instead of 64 (internal/vold/commit.go:389-398).

Whole-volume children. Same chain, and every writer is also a receiver of the other 39 writers' log entries and syncs. Each apply holds that mount's own commits (vol_logapply.go:812, vol_sync_chunk.go:1035), and after three unsettled syncs the hold escalates to 20 s (vol_sync.go:196-210). Writers stall each other: writers × mounts.

Attach latency. waitForDepartingWriters waited until no writer was leaving the volume (mounts.ts:302-317). Children leaving all the time meant the wait never emptied and rode its 15 s bound.

Attach 503 then 409. The attach commits its row, then re-reads while it waits; a dropped connection on a re-read became a 503 for a mount that existed, and the retry hit the overlap check (api/volumes.ts:817).

Detach under load. A running sync holds the mount gate through its merge wait and settle tries (vol_sync.go:384-473), and detach waits for the gate (vol_attach.go:697).

Plan

Each workstream names the E2E number it moves. Status as of this branch set.

Not planned: live migration to colocate sharers. Colocation caps one volume's fan-out at one host and makes it one failure domain, and each host only pays the cold fetch once (256 MiB in ~0.8 s measured). E1 and E2 give the locality without moving running VMs.

Rollout order

1. CP: A1-A3, B1 (gated by vol_v 7, inert until hosts advertise it), C1, C5. main → Development; run the E2E on dev.
2. Hosts: B2, C2-C4 to Development hosts; run the E2E again (both MODEs).
3. Promote CP and host builds to staging, E2E, then prod, E2E.
4. D1 SDK release once A1 is live on prod.

Targets

At 40 concurrent subdir children: child write → parent sees it p50 ≤ 3 s, p99 ≤ 8 s; attach p90 ≤ 3 s; 0 attach 503s reaching the client; every E2E check true.

Results on Development (after #1653)

The E2E runs from a driver sandbox on dev (a laptop that sleeps voids runs: the harness now reports client_paused_ms and voids them). Dev's two hosts hold about 30 children next to their other tenants, so dev runs use X=28 and MIN_SHARED=28; the 32+ bar needs Staging.

Where the remaining time goes, from the parent's host commands (/internal/admin/commands.json): fan-out → pickup p50 40-70 ms, pickup → applied p50 40-140 ms. The rest is on the writer: a drive-log mount spaces its flush commits by driveLogFlushGap (500 ms per merge still waiting, at most 2 s), so with 1-4 merges pending a child's sync waits 0.5-2 s before its commit starts. That spacing is now a control-plane setting (vol.drive_log_flush_gap_min_ms / _max_ms), to be tuned against merge load on dev.

The tighter gap helps sub_path writers and hurts whole-volume ones: a sub_path commit's merge syncs nobody else (markSubCurrent), a whole-volume one syncs every other whole-volume mount, so more of them cost more than they save. The setting now applies to sub_path drive-log mounts only.

Session-workspace bars on Volumes (2026-10-17, Development)

These are the "Depots" proposal's bars, measured on today's volumes with verify/good-volumes/e2e/fanout/depot-bars.ts, depot-indep.ts and depot-chunks.ts. They feed P4 (prewarm) and P5 (chunking) of volume-node-share-plan.md. The workspace is 72,000 files, 2.0 GB, 121,743 entries, in a 2 CPU / 4 GiB box.

Cross-session dedup was measured on 5 independent sessions that wrote the same 20,000 files (3.16 GB logical) each. depot-chunks.ts hashed each raw image in fixed windows and skipped all-zero windows. 5× is the ceiling.

Takeaways:

• Today's 1 MiB blocks share almost nothing between independent sessions: 1.15× when the write order differs.
• A fixed 64 KiB window barely helps. Most of the gain comes from aligning chunks to files.
• That confirms the P5 scope: file-aligned or content-defined (FastCDC) chunks, rather than smaller fixed blocks.
• The gap between 4 KiB and the 5× ceiling is filesystem metadata (inode tables, directories, bitmaps), which file-level dedup would not store at all.