Volume node share plan: one volume system — replicate per node, share on the node, converge across nodes

Context

Platinum volumes are today a private copy-on-write block disk per sandbox mount (ublk + guest ext4) over 1 MiB content-addressed blocks in S3, with a shared per-host block cache (L1) and a regional cache (L2). Blocks are fetched once per node, but files are not shared: each sandbox has its own filesystem, so a write reaches other sandboxes only through commit → control-plane change log → host command → guest apply (prod old code p50 23 s; dev after #1653/#1658/#1665 p50 ~2.5 s at 28 writers). Cost and fan-out grow per sandbox (every mount is a commit/sync participant).

Two needs drove this: (1) parent/child agents sharing one volume across 32–1000 sandboxes (the @zeeg ask), (2) session workspaces that outlive their box and restore fast (the "Depots" proposal). Decision: one system — Volumes — that meets both; Depots' storage ideas (file-level dedup, FastCDC, packs, catalog-first lookups) move into the volume engine instead of a second primitive.

Requirements (R1–R10): mount anywhere/anytime into unlimited sandboxes; outlives every box (S3 truth); fast mount regardless of file count; ms visibility on a node, sub-second across nodes; cheap turn-end saves; history/restore/fork/browse/diff; ≥5× storage across sessions; native FS semantics; no creds/root in the box; unlimited volumes. Physics: two kernels cannot share one writable ext4 → same-node sharing needs one host filesystem per node; cross-node is eventual (or leased); a new bare-metal node takes 3–45 min to join (capacity, not storage, must hide that).

Consistency contract (what we promise, enforced, tested):

• same node: one real filesystem — POSIX, flock/fcntl, atomic rename, O_APPEND.
• across nodes: whole-file atomic versions, one global order per volume (deterministic last-writer-wins using the existing merge rules R1–R13, conflict copies kept), convergence; strict isolation only via write leases.
• durability chosen per volume / per fsync: local → peer (default) → s3.

Target architecture

S3 per region            truth: content-addressed chunks (→ file-aligned / FastCDC in P5)
Regional cache nodes     L2 chunks + per-volume SEQUENCER (primary + standby, term-fenced)
Node replica (per volume, per compute node)
  vold: ONE RW mount (ublk + epoch overlays) mounted by the host kernel, host-side freeze
  nodefs: FUSE-over-vsock server → every sandbox on the node (no VM device)
  journal + applier in vold (survives host-agent restarts)
Sandboxes                attach `share: "node"` (default later) or `"private"` (today's disk)

Live changes: replica → sequencer (one ordered, resolved log) → all replicas, ~5–15 ms. One checkpointer turns the log into S3 commits via the existing head CAS. Participants scale with nodes, not sandboxes.

Phases (each flag-gated, reversible, additive migrations, own vol_v step)

Ground rules: share is a request; CP records share_effective and silently falls back to private when flag/host vol_v/guest cap vol1.nodefs/mode don't allow it. Kill switch flags.vol_node_share_enabled; drain flags.vol_node_share_drain converts every node attachment back to private (built in P1d). Org canary via limits.org_overrides[org].volNodeShare (pattern: volV1Enabled, apps/api/src/config.ts).

P0.5 — finish the measurements (1–2 days). Fix verify/good-volumes/e2e/fanout/depot-bars.ts / depot-indep.ts to run long steps via POST /v1/sandboxes/:id/processes (exec kills its children — the stuck run). Measure R3 (100k-file restore warm + cold node), R5 (turn-end pt-vol commit time + uploaded_bytes from commit history), R7 (depot-chunks.ts: 4 KiB/64 KiB/1 MiB dedup over independent sessions, after POST /internal/admin/vol/gc/run). Feeds P5 scope.

P1a — nodefs production-grade (~3 eng-wk). From branch feat/vol-nodefs-p0 (hosts/host-agent/internal/nodefs/*, hosts/invm-agent/internal/nodefsrelay/*): length-prefixed framing over CH's STREAM vsock (P0 used SEQPACKET); reconnect with session epochs + per-session reply cache keyed by FUSE unique (exactly-once create/rename/unlink) + handle re-open by path; auth by per-sandbox listener <jail>/vsock.sock_18050 (pattern cmd/host-agent/vol_helper.go:18-95) + allowed (volume, subdir, ro) set from the mount payload; per-sandbox fairness queues; user.* xattrs. Gate: 10k random disconnects, 0 lost/duplicated writes; real-vsock write→read p50 < 2 ms, p99 < 20 ms on a dev host.

P1b — host replica mount in vold (~3–4 eng-wk, parallel to P1a). internal/vold/daemon.go:240 Open with a replica pseudo-jail; host-kernel ext4 mount of the ublk device inside a long-lived helper (mount namespace survives host-agent restarts; re-adopt from state file); host Guest impl (syncfs + FIFREEZE/FITHAW on the host mountpoint) for internal/vold/commit.go:236-300; host commands volume.replica.open/close; vold.VolCapability → 8. Gate: replica commit byte-identical to a private commit; freeze p99 < 50 ms with 30 writers.

P1c — CP model/API/SDK, lands dark (~2 eng-wk, start now). Migration 0140_vol_node_replicas: volume_node_replicas(id, volume_id, org_id, host_id, state, generation/epoch, mounted_commit_id, last_commit_id, applied_seq, hb_at) unique (volume, host); volume_mounts.share_requested/share_effective/replica_id/share_fallback_reason. vol/mounts.ts (parseSandboxVolumes, fallback rule, isSameAttach/sameMountSpec in api/volumes.ts), api/sandboxes.ts, OpenAPI, packages/sdk-ts/src/volumes.ts (VolumeMountSpec.share()), sdk-py, CLI. Gate: flag off → full volumes suite + SDK parity unchanged.

P1d — lifecycle on one node (~4 eng-wk). cmd/host-agent/vol_attach.go (volPrepareBoot:72, volAfterBoot:336, volCmdMount:603, volCmdUnmount:676, volFinalCommitAndClose:1431), pause.go (volQuiesceForPause:1246 drains relay — no device to strip; volPrepareForRestore:1954, volEnsureMountedAfterRestore:2279, volAfterRestore:2294 reconnect), invm-agent relay + cap vol1.nodefs (procmgr.go:1359); CP volMountsForCommand (vol/mounts.ts:545-612) maps node attachments onto the replica; cross-host resume pinned to the replica host or drained to private first. Placement in P1: node attachment only on the replica's host, else fallback. Gate: single-node fan-out E2E (parent-children.ts --share node) p99 < 20 ms; 100/100 pause→resume and 50/50 memory restores with files open; 0 D-state; drain round-trips.

P2 — replica = the participant (~3–4 eng-wk). Replica appends to the existing drive change log and receives volume.logapply/volume.sync applied into the host FS (not via guest agent); guests see changes via FUSE invalidation (nodefs/notify_linux.go). Fix existing gaps: generation/epoch fence on log append (vol/driveLog.ts ~339-350 has none), durable cursors (today in-memory vol_drivelog.go:529). vol/driveLog.ts:455, vol/driveSync.ts:44 select replicas; vol_v 9. Gate (2 dev hosts): same-node < 20 ms, cross-node p50 < 1.5 s, participants = nodes.

P3 — sequencer stream (~5–6 eng-wk). Per-volume sequencer on the regional-cache owner node (rendezvous owner + standby, internal/regionalcache/handoff.go, auth regionalcache/server.go:278) — no host↔host traffic, so the "S3 only" policy (apps/api/src/migration.ts:189) stands. Replica uplink ops {replica, epoch, client_seq, path, op, base{seq|vid}, inline ≤64 KiB | chunk refs in L2}; sequencer plans each op/batch with the existing planner (internal/volimg/merge.go, KeepBoth per mode) against its path index → resolved actions with seq; replicas replay in order (temp+fsync+rename, fileSeq < seq idempotence, per-volume apply barrier for atomic batches). Checkpointer (volgw fold job, volgw/fold.go) applies (b, b'] to head through submitCommit CAS (vol/commits.ts:221-300) setting durable_seq; foreign writers (API uploads, private mounts) enter via mergeQueue.ts as foreign batches → one ordering authority. Tables volume_replicas (epoch, applied_seq), volume_sequencer(term, primary, standby, lease_until, last_ckpt_seq); term CAS on Postgres now(), standby persists before ack, primary self-fences. Leases: new write_leases(volume_id, prefix, holder, epoch, term, expires_at) (replaces dead volume_write_leases, db/schema.ts:3100) granted as log events; non-holders EROFS. Durability: peer default = sequenced + standby disk + chunks on 2 L2 nodes; s3 = wait durable_seq (reuse vol/fastack.ts). GC pins log-referenced chunks (vol/gc.ts). vol_v 10, flags.vol_seq_stream. Gate: cross-node p50 < 50 ms, p99 < 200 ms; flag off falls back to P2 cleanly.

P4 — placement & cold start (~5 eng-wk). scheduler.ts:720 single preferHostId → replica-set affinity (= ANY(hosts with live replica)), per-node cap + spill policy; vol/mounts.ts:369. Prewarm: vold RPC pulls a volume's hot chunk list into L1 from L2 at replica open (pattern templatePrewarm.ts). Warm spare hosts per region (adminCapacity.ts → policy; real Scaleway provision api/cluster.ts:990). Gate: 200 children / 5 nodes: ≥90% on replica nodes; warm attach < 200 ms, cold-node attach < 3 s.

P5 — storage engine: chunking & dedup (~8 eng-wk, scope from P0.5). Per-volume format_version; file-aligned / FastCDC chunks + packs + catalog-first lookups (Depots design); readers for both formats ship one vol_v before writers; only new volumes. Commit uploads changed files only. Gate: ≥5× over 10 sessions (depot-chunks.ts/Depots repos), turn-end save ≤ 2 s and KB-sized. Plus diff endpoint (commit index compare) and one-volume-per-session limits/lifecycle (config.ts vol limits).

P6 — multi-region (scope later). Home region per volume; async chunk replication to read replicas; writes routed home.

P7 — moves & rebalance (~3–4 eng-wk). move: true = pause → resume on the replica's node; rebalancer consolidates (reconciler.ts); lift "volume mounts pin a sandbox" (startCapacityRelocation.ts:41-53) for node attachments. Gate: move < 5 s with files open, 0 lost writes.

Effort: P1 ≈ 12–14 eng-wk, P1–P4 ≈ 26–30, P5–P7 +16–20.

Races (cross-node) and resolution

Same file two nodes → higher seq wins, loser kept as conflict copy (R2). Rename vs write → R7/R8 guards on mv source; write after mv recreates (R1). Delete vs write → version-guarded delete, else delete_lost. Dir rename + child writes → path semantics, late child resurrects parent (R4). rmdir non-empty → rejected, children re-materialized (R10). O_APPEND/SQLite multi-node → require a path lease (documented); SQLite db+wal+journal one batch (merge.go SQLite group), -shm node-local. Sequencer failover → term-fenced standby. Replica crash mid-apply → temp+rename + durable cursor. Partition → local-level writes journaled (cap 1 GiB/10 min), peer fsync EIO after 30 s, fenced after 10 s, journal salvaged as a KeepBoth side commit. Clock skew → never used for ordering. Dup/reorder → dedupe (replica, epoch, client_seq), cursor resume. Join under load → checkpoint snapshot + stream from b+1, log retained to slowest cursor. GC vs lagging replica → pin replica mountedCommitId + log-referenced chunks.

Verification

• Invariants checked every phase: convergence (all replicas == head == log replay), no acknowledged write lost per durability level, no torn file, same-node linearizability, Go planner == vol/mergeModel.ts on every logged op, no stale-epoch/term event applied.
• Tooling: extend verify/good-volumes/e2e/lib/model.ts + replay.ts into a sequence-order oracle; add node↔sequencer / ↔Postgres / ↔L2 partitions to e2e/lib/hostctl.ts; new scenarios/replica.ts Jepsen-style (kill vold/agent/sequencer, partitions, faketime skew, S3 blackout, joins under load; Porcupine per-file register checker); reuse torture (SQLite integrity, git fsck) and redteam (stale epoch).
• Per-phase E2E: verify/good-volumes/e2e/fanout/parent-children.ts (--share node, subdir/whole modes) run from the dev driver sandbox; staging for ≥32/200-child gates; 3 consecutive clean rounds per lane before promote.
• Unit/int: apps/api bun test src/vol/ against Postgres 16 (both migration lanes), tsc --noEmit; Go go vet + go test for internal/vold, internal/volgw, internal/nodefs, cmd/host-agent in Linux (never --privileged FUSE tests on Docker Desktop that abort foreign FUSE connections).
• Rollout per phase: main → Development (one internal org override → global dev) → promote-staging (one org, 48 h watch) → promote-prod (flag off) → canary org → widen. Watch: nodefs op p50/p99, reconnects, relay aborts, FUSE timeouts, D-state guests, fallback-reason breakdown, freeze duration, sequencer term changes, lag (head seq − applied_seq), pause/restore success.

Gate results on Development

P1d, one node (2026-10-18)

The harnesses are in verify/good-volumes/e2e/fanout/. They ran from the dev driver sandbox, except node-share-chaos.ts, which needs ssh to the hosts. The guests used template ns-python, which has the current in-VM agent.

Findings along the way:

• Every /snapshot is disk-only (host-agent snapshot.go, because of two CH v52 bugs), so a restore is a cold boot. The memory path is pause → resume, which the 100-cycle gate covers.
• Rollout prerequisite: node share needs a guest with the current in-VM agent (cap vol1.nodefs). Hosts run with PT_PLATFORM_AGENT off, so templates built before P1d (pt-base) keep an agent without nodefs, and every node request falls back with guest_lacks_nodefs. Turn on PT_PLATFORM_AGENT=1, or rebuild templates, before widening node share.
• Placement: with parallel creates and no replica yet, children landed on other hosts and ran with replica_elsewhere. #1677 makes placement prefer the live replica's host.
• Handle leak: a guest that disappeared mid-listdir left a directory fd open under the replica, so volume.replica.close failed volume_busy every 2 min. Fixed in #1681.
• A host-agent restart during a pause/resume cycle (a host roll at 23:25 on 2026-10-17) cold-booted the child it was resuming, and a resume answered 503 relocation_snapshot_pending until the snapshot was off-host. The data stayed intact.
• The drain endpoint acts only while flags.vol_node_share_drain is on, unless the call passes force: true.