Node share: one hot copy of a volume per compute node

Status: proposal, October 2026. Follows volume-fanout-plan.md, whose fixes brought child → parent visibility on one volume from p50 23 s to ~2 s. This document is about the next order of magnitude: milliseconds for sandboxes on the same node.

Why it is seconds today, even on one node

The blocks of a volume are already on the node (the host L1 cache is shared by every mount). The files are not. Each sandbox mounts its own ext4 over its own copy-on-write block device, and two kernels cannot share one read/write block device: ext4 is not a cluster filesystem. So a child's write lives in the child's filesystem until it is committed, logged on the control plane, fanned out as a host command, picked up by the parent's host and applied into the parent's guest file by file. Every hop is a control-plane round trip, the same on one node as across two.

Milliseconds need one filesystem per node that every sandbox on that node talks to at the file level.

What we already learned (the virtio-fs era, #759 → #1268)

Platinum served shared volumes into guests through virtio-fs (virtiofsd, vhost-user) for two months. The device itself, not JuiceFS behind it, caused:

• restores wedged: CH cannot snapshot a vhost-user handshake; a stale fs device aborts restore; prod resumes 36/38 → 1/13 once the share was hot-plugged (f4e8c6883);
• ghost PCI devices after --restore: 8/8 memory restores wedged (82b61ecfa);
• single-connection virtiofsd killing CH at boot (1eba0b0c6, c97d7aea4), starvation by builds (24035d8a1);
• D-state pileups on a degraded mount (1139 stuck processes in one guest, 3a7ad7bda);
• no flock/fcntl across guests: virtiofsd negotiates neither lock capability (511517d16);
• a symlink TOCTOU from guest to host paths (5c5b1f8e3).

Pause, resume, memory snapshots and cross-host resume are core to Platinum. A design that adds a vhost-user device to a running VM repeats all of the above. The node share therefore uses no new VM device.

Design

                       S3 (blocks, images, commits)  ◄── source of truth
                                  ▲
                    commit (one participant per node) │ lazy block fetch, prewarm
                                  │                    ▼
  node ┌──────────────────────────────────────────────────────────────────────┐
       │ vold: ONE read/write mount of volume V  (ublk + epochs, as today)    │
       │ host kernel ext4 at /var/platinum/vol/V/<gen>   ← the node replica   │
       │                                                                      │
       │ nodefs server (host-agent): file ops on the replica, per-guest       │
       │   channels over vsock, RESOLVE_BENEATH, per-node locks, O_APPEND,    │
       │   invalidation pushed to every guest that caches a path              │
       └───────▲──────────────────▲──────────────────▲───────────────────────┘
          vsock│             vsock│             vsock│   (existing per-VM channel)
       ┌───────┴──────┐  ┌────────┴─────┐  ┌─────────┴────┐
       │ parent VM    │  │ child VM     │  │ child VM     │
       │ FUSE /agents │  │ FUSE /out    │  │ FUSE /out    │   invm-agent serves
       │ (whole vol)  │  │ (children/a) │  │ (children/b) │   FUSE, no new device
       └──────────────┘  └──────────────┘  └──────────────┘

The node replica. For each (volume, node) with at least one share attachment, vold opens one read/write mount of the head, exactly as it opens a sandbox mount today (lazy 1 MiB blocks from L1 → L2 → S3, epochs, commits), and the host kernel mounts it. A new node's cold start is opening the image and faulting blocks (256 MiB read cold in ~0.8 s measured), plus an optional prewarm of the volume's hot blocks.

Into the guest: FUSE over vsock, not virtio-fs. invm-agent mounts a FUSE filesystem at the requested path and forwards each op over the guest's existing vsock channel to the nodefs server on the host. Nothing is hot-plugged into the VM, so there is no device for a snapshot to capture or a restore to wedge on. Across pause, resume, snapshot and cross-host resume the channel reconnects as invm-agent's channels already do, and handles are re-walked by path (a node's replica of the same volume has the same paths, not the same inode numbers). Expected cost per op: a FUSE round trip plus a vsock round trip, tens of microseconds.

Coherence on a node. Every guest on the node reads and writes the same host page cache. The server pushes FUSE_NOTIFY_INVAL_* to guests caching a path another guest changed, so guests may cache attributes and data and still see a sibling's write at once. O_APPEND writes go to the host fd with O_APPEND (the server, not a client's cached size, picks the offset: the lost-append class from the JuiceFS era cannot occur). flock and fcntl locks are held by the server, so they work across sandboxes on the node.

Subdirectories and read-only. A subdir attachment is the server rooting that channel at children/<name> with openat2(RESOLVE_BENEATH); no sub_path image is ever derived. A read-only attachment is a channel that refuses writes.

Durability and versions. The node replica is one mount on the control plane. Commits freeze the host filesystem (FIFREEZE on the host mount: milliseconds, no guest RPC, so the guest-freeze stall in volume-fanout-plan.md is gone), upload as today, and move head. A guest fsync reaches the host ext4 and schedules a commit, as a FLUSH does today. History, tags, restore, fork and file access with no sandbox are unchanged.

Across nodes. Replicas on different nodes converge exactly as two mounts do today (merge, per-file last-writer-wins, the drive change log, volume.sync), but the participants are nodes, not sandboxes, and changes are applied into the host filesystem directly instead of through each guest's agent. 40 children on 2 nodes are 2 sync participants, not 41. The semantics a user sees:

Colocation. The scheduler places a sandbox that will use V on a node with a live replica of V and room, and spills to a new node (one cold start) when none has room:

• at create, from volumes or from a mount-free hint (affinity: {volumes: ["results"]}), so a caller that decides mount paths after boot still lands next to the data;
• on hot attach to a sandbox elsewhere: a replica opens on its node (cold start, always correct); with move: true the sandbox is instead paused and resumed on the replica's node (memory snapshot + cross-host resume, seconds of pause), which a vsock channel survives and a virtio-fs device would not;
• the rebalancer consolidates paused and stopped sandboxes onto their volumes' nodes.

True live migration (CH send/receive-migration, rootfs moving with it) is not needed for any of this and stays out of scope until a workload needs zero-pause moves.

Two modes, one volume. share: "node" (this document) for data several sandboxes use together; share: "private" (today's per-mount CoW ext4) for heavy local I/O, builds, SQLite, where a native disk beats any FUSE path. A volume can be attached both ways at once; both are commit participants.

Plan

Rollout follows the release lanes: Development → Staging → Production, each gated by the fan-out E2E (verify/good-volumes/e2e/fanout/parent-children.ts).

Risks

• FUSE metadata cost. Small-file-heavy work (git, npm) is slower over any FUSE than on a local ext4; share: "private" stays for it. P0 measures it.
• Host server as a new failure point. It runs inside host-agent's restart handoff like vold; a server restart is a channel reconnect, never a guest hang (no D-state: the guest FUSE daemon answers EIO/retries on a timeout, it does not block forever).
• Cross-node semantics stay eventual. Same-node is strong; across nodes it is still per-file last-writer-wins. Placement makes the strong case the common one.

P0 results (October 2026)

The nodefs spike exists as a library with tests. It is not yet wired into vold, the control plane or mount commands.

• Guest: hosts/invm-agent/internal/nodefsrelay. It mounts fuse.nodefs on the guest's own /dev/fuse and copies FUSE messages 1:1 to a SOCK_SEQPACKET socket, with 4 copiers per direction. Notifications get a writer of their own. A lost channel aborts the connection through fusectl, so callers get ENOTCONN and never hang.
• Host: hosts/host-agent/internal/nodefs. It runs on go-fuse v2.11.0 (fuse.NewServer with the magic mountpoint /dev/fd/N serves a socket as-is, with splice and passthrough off). The node type is ours; file handles are go-fuse's LoopbackFile.
• Tests run the real relay against the real server over an AF_UNIX seqpacket socketpair (the same message semantics as vsock seqpacket) in a privileged container. They are go test ./internal/nodefs/ (NODEFS_BENCH=1 for the numbers) and go test ./internal/nodefsrelay/. Result: 18 + 4 tests, green 15× in a row and 5× under -race.

What works, tested between two guests on one root:

• write+close → open+read visibility: 0 stale reads in 2000 rounds, with same-size rewrites, growing files, rename-over and delete/recreate.
• O_APPEND from 8 writers across 2 guests: no lost or torn lines.
• flock, F_GETLK, F_SETLK, F_SETLKW conflict across guests.
• A vanished guest's locks are released.
• A read-only root refuses 12 kinds of change with EROFS.
• A subdir root survives a hostile raw-FUSE client: .., names containing /, walking or opening through symlinks, a directory swapped for a symlink, a bind mount under the root, device MKNOD, opening a FIFO, LINK of a forged nodeid.
• New files are owned by the calling guest uid.

Numbers. These come from Docker Desktop on an Apple-silicon Mac (arm64, 12 vCPU, kernel 6.10) over a unix socketpair, not vsock on a node, so treat them as the transport-free cost:

The P0 gate is p50 < 2 ms and p99 < 20 ms, and both pass before vsock is added. Against the host file system directly, small-file create is about 30× slower and stat goes from about 1 µs to about 65 µs. That is the FUSE metadata cost the Risks section predicts; share: "private" stays for git and npm.

What P0 learned and fixed:

• Notification deadlock in the relay. FUSE_NOTIFY_INVAL_INODE waits for the page lock of a READ in flight. With every copier blocked in a notification, that READ's reply could never be delivered. Notifications now get a writer of their own.
• No AUTO_INVAL_DATA. With file attrs uncached, it sent a GETATTR before every read(2), which made 1 MiB reads take 8 ms instead of 1 ms. Close-to-open holds without it.
• Inode numbers are recycled at once. go-fuse merges nodes by (ino, gen), so the node's generation is the inode's birth time.
• go-fuse substitutes handles. For a GETATTR that carries no file handle, go-fuse fills in any handle that is still open. That made a just-deleted file stat as present while its RELEASE was still in flight. Such GETATTRs are now answered by path.
• Test-only self-deadlock. Go's os.Open of a FUSE file issues a FUSE POLL from non-preemptible runtime code. In one process that is both client and server, a GC stop-the-world then deadlocks. The tests use raw syscalls on mounts. Guest programs talk to a server on another machine, so production cannot hit this.

Open issues (P1 and later):

• Reconnect after pause/resume or a host-agent restart. The relay aborts today. P1 must keep /dev/fuse open and reconnect, re-walking nodes by path on the new session. The tree is path-based already and holds no host fds, so a node survives a reconnect as long as its path exists. Open file handles must be reopened by path, and locks are lost, so they need a policy.
• Inode identity across hosts. Replicas of one volume on two nodes have the same paths but different inode numbers and birth times. Moving a guest (cross-host resume) therefore means nodeids go stale, which ESTALE plus re-lookup handles for names but not for open handles. Within a host, a recycled inode number inside one coarse clock tick still aliases. ext4's FS_IOC_GETVERSION fixes that but costs an open per lookup.
• Invalidations are asynchronous. Correctness does not depend on them (attrs uncached, ESTALE re-walks). A reader that keeps a file open sees an in-place rewrite once the notification lands (≤ 3 ms in tests). A paused guest's queue (4096) drops entries; the count is exposed as Session.Dropped.
• fstat(2) of a file another guest deleted. While this guest has the file open, fstat returns ESTALE, as on NFS without silly rename. The kernel sends no handle for stat or fstat, so the server cannot tell them apart. Reads and writes through the fd keep working.
• Locks are OFD locks on the host fd. Two opens of a file in one guest process conflict, whereas POSIX locks would not. Closing one fd does not drop the locks taken through another. A guest F_SETLKW that gets interrupted keeps the host thread waiting until the lock is granted.
• Hardening still missing: xattrs are not served (ENOSYS, which also saves a GETXATTR per write); no per-org channel auth by CID; no limit on inodes or handles per guest; go-fuse prints "Short read" to the global logger on every channel close.
• The first POLL of each mount costs a guest Go program one non-preemptible round trip (the server answers ENOSYS once, and the kernel remembers).

P1a results (October 2026)

nodefs is now built for a real node. It is still a library; the per-sandbox listener, vold and the control plane come in P1b to P1d.

Transport. Cloud Hypervisor's hybrid vsock is stream-only, so every FUSE message now travels in a nodefswire frame: a 24-byte versioned header (length, type, seq, ack) and then the message. The two modules each carry a byte-identical copy (hosts/invm-agent/internal/nodefswire, hosts/host-agent/internal/nodefs/nodefswire), and TestGuestRelayCopyIsCurrent checks that they still match. go-fuse still reads one message at a time, from an internal seqpacket pair per session. After the handshake, both ends switch the socket to blocking calls on their own threads. Through Go's netpoller, every frame cost a park and a wakeup that had to wait for a free P. On a 2-vCPU node that added about 200 µs per round trip (stat took 255 µs instead of 60 µs).

Reconnect, exactly once. A session outlives its connections:

• Every request has a Seq. The server owns a request as soon as it reads it, and it acknowledges it.
• The relay keeps every request that is not yet acknowledged. After a reconnect it resends only what follows the server's Seq in WELCOME, and the server drops duplicates.
• Replies work the same way in the other direction: the server keeps them until the relay acknowledges them. These kept replies are the reply cache.
• So no request reaches go-fuse twice. That covers CREATE, RENAME, UNLINK, MKDIR, an O_APPEND WRITE, and FORGET, which has no reply.
• A lost connection never fails a call. The relay redials with backoff for ReconnectWindow (default 30 s). After that, or as soon as the server refuses the session, it aborts the mount: callers get ENOTCONN.

What a guest keeps across a disconnect:

• Within Linger (default 60 s), everything: nodes, open handles, locks, and files that a peer deleted while this guest had them open.
• After Linger the session goes dormant. Its host fds close, which releases its locks. On the next use, each open file is reopened by the node's current path, and only if that path still names the same inode (number and birth time). A file that was deleted or replaced in the meantime answers ESTALE from then on. Locks are not reacquired, as after an NFS server reboot without lock recovery.
• After DormantTTL (default 24 h), or when the server restarts, the session is gone. A resume is then refused and the relay aborts the mount, so P1d must remount.

Auth. Config.Authorize(peer) returns the attachments a peer may use, as (id, root, subdir, read-only). The peer label is whatever the caller says the connection is: in P1d, the per-sandbox listener it arrived on. The guest never supplies it. The HELLO names an attachment id, and anything not granted is refused. A resume must come from the same peer, with the session's 128-bit token, and must still be authorized. A changed grant ends the session.

Fairness. At most MaxInFlight (default 64) requests per session are with go-fuse at once. The rest wait in the session's queue, which holds up to MaxQueued (8 MiB); past that, the guest's connection is not read. Under a flood of 64 readers from guest A with a limit of 4, A peaked at 4 in flight, and B's stat stayed answered (p50 3 ms, p99 under 120 ms on a 2-vCPU VM where both guests share the CPU).

xattrs. Only user.* is served, on files and directories. Every other namespace answers EOPNOTSUPP. The relay answers a GETXATTR outside user.* itself, so the security.capability query the kernel makes before every write costs no round trip.

Found and fixed:

• go-fuse panics on a short request. A request shorter than its opcode's struct, or a READ or xattr request asking for a multi-GiB reply buffer, crashed host-agent (slice bounds out of range in handleRequest). It also crashed every other guest's session with it. guard_linux.go checks each opcode's size and caps before go-fuse sees the request. The framing fuzz found this bug.
• A created file's handle belonged to its directory's node (P0). Its invalidations named the directory, and a reopen would have opened the directory's path.

Tests (go test ./internal/nodefs/..., real FUSE in a VM):

• All P0 tests pass over the stream framing. With the new tests, that is 27 tests and a fuzz seed corpus (plus the latency bench), and 8 in nodefsrelay.
• The suite passed 3× in a row and once under -race.
• Disconnect fuzz: 4 workers ran create O_EXCL, pwrite, append, rename NOREPLACE, unlink, mkdir, rmdir and fsync while the connection was cut at random points from either side, including mid-handshake and mid-frame. At 10 000 disconnects (NODEFS_FUZZ_DISCONNECTS=10000; the default is 300), the result was 10 649 ops, 0 lost, 0 duplicated, and no hung caller. The host tree matched the model.
• Framing fuzz: 300 hostile connections. Each one ended only its own session. The server and a good guest on the same hub carried on.

Numbers. These come from a 2-vCPU dev sandbox VM (kernel 6.8) over an AF_UNIX stream socketpair. P0's code was run on the same VM for comparison. The VM is noisy: runs vary by about 1.5×.

The P1a gate is write→read p50 < 2 ms and p99 < 20 ms on real vsock. This VM passes it over a socketpair. Each round trip now costs two more thread hops than P0: server reader to go-fuse, and go-fuse to the reply pump. That is most of the remaining gap, and the throughput cost too.

Not done in P1a:

• The real vsock listener per sandbox. Hub.Serve(ln, peer) exists; the wiring is P1d.
• Measuring on a real vsock node.
• Pause handling. A paused guest whose connection stays open never detaches, so it holds its locks until resume.
• Session state does not survive a server restart.
• No keepalive. A peer that is wedged but still connected is not noticed.
• An INTERRUPT queued behind a full MaxQueued waits.
• go-fuse still prints "Short read" to the global logger when a session ends.

P1d: lifecycle on one node (October 2026, not yet deployed)

P1d wires P1a (nodefs) and P1b (vold replicas) into the control plane and the host. It is behind flags.vol_node_share_enabled and the org override volNodeShare, both off by default.

Control plane (apps/api/src/vol/nodeReplicas.ts). An attach whose share rule allows node places the volume's replica on the sandbox's host, in the attach transaction: the live replica there, the same replica row reopened one epoch on, or a new row at epoch 1, with volume.replica.open enqueued. The attachment is a volume_mounts row with share_effective 'node', replica_id and no device, and the host is told volume.share.attach. The last attachment to end closes the replica (volume.replica.close {commit: true}); an idle reaper closes any that the immediate paths missed. The replica is the commit participant: its intents, commits, reports, head waits, metadata reads and drive log appends arrive under its replica id with its epoch as the generation, and are authorized by volume_node_replicas (host, epoch, live). A stale epoch, another host or a closed replica gets 409 mount_fenced.

Command and payload contracts:

Placement rule (P1). A volume is node-shared on at most one host at a time. A node request on another host runs private (share_fallback_reason: replica_elsewhere), and so does one next to a replica that is closing (replica_closing).

Moves. A boot on the replica's host lists the attachment as a share and never re-points it. A boot anywhere else places the attachment again: it opens a replica on the new host when the volume has no live replica elsewhere, and otherwise the row becomes private in place (replica_elsewhere) and mounts as a private disk. The old replica closes through the idle reaper. Every memory image records the shares its guest has mounted (vol-shares.json next to vol-meta.json). An image with a share that this boot no longer serves exactly (detached, drained, or on another replica) cold-boots, so a guest never comes back with a FUSE mount that nothing serves at a path that a private disk now needs.

Host (hosts/host-agent/cmd/host-agent/vol_share.go). There is one nodefs.Hub per process and one listener per sandbox at <jail>/vsock.sock_18050. The listener path is the peer label, and the authorizer gives that sandbox only its own attachments, rooted at <replica mnt root>/<volumeId>/mnt and its subdir. The share registry is written to <dataRoot>/vol-shares.json on every change. Each boot's volMounts replaces that sandbox's set, and attach and detach change one entry. The host handles each stage this way:

• Cold boot: it closes the old VM's sessions and mounts every share after the guest is up. A share that cannot be mounted refuses the boot, like a disk.
• Pause: the shares stay registered and their sessions linger, then go dormant.
• Memory resume on the same host: the relay resumes its session. Three seconds later the host re-issues vol_share_mount, which does nothing when the relay is alive and replaces it when it was aborted.
• Agent restart: the host reloads the registry, rebinds the sockets and remounts the same way. The old sessions are gone, so the relays' resumes are refused and their mounts abort. Processes that held files there get ENOTCONN.
• Replica close: the host first ends every session still rooted on the replica, dormant ones included, so the unmount is not busy.
• Delete: the host ends the sandbox's sessions and drops it from the registry.

Guest (hosts/invm-agent/cmd/invm-agent/volshare.go). The guest dials vsock CID 2, port 18050, and mounts with nodefsrelay.Mount. It refuses the same protected paths that vol_mount refuses, refuses to overlap a volume disk or another share, and never unmounts a share that a process still holds. It advertises vol1.nodefs only when /dev/fuse exists.

Drain. When flags.vol_node_share_drain is on, the region leader converts node attachments back to private every 15 s (POST /internal/admin/vol/node-share/drain {volume_id?, force?} runs one pass on demand). On a running sandbox it first detaches the share in the guest (convert: true), then turns the same row private and mounts it at head once the replica's final commit is in. On a settled stopped sandbox it converts the row in place, and the next start mounts it as a disk.

Not in P1d. The replica does not apply other writers' changes (P2). A private mount's commits reach node attachments only after the replica closes and reopens. Placement does not prefer the replica's host (P4). A parked attachment, one whose sandbox is stopped, keeps its replica open. An upgrade of the in-guest agent ends the relays that the old agent process held.

P1d latency and handle fixes (October 2026)

Measured on Development (two 1-vCPU guests sharing a node volume): a 300-entry listdir took 18 ms, and simple ops had a p99 of 25–47 ms against a p50 of 0.2–2.5 ms. volume.replica.close also failed volume_busy for good after its sandboxes were deleted.

Leaked directory fds. A guest's open directory was go-fuse's loopback dir stream, which is closed only on RELEASEDIR. A guest that goes away never sends one, and session teardown closed only regular files. So CloseRoot, ClosePeer and CloseAttachment left a directory fd under the replica mount. Directory handles now belong to the session (dir_linux.go): teardown closes them, dormancy releases them, and the next READDIR reopens them by path at the same offset. Test: TestAnEndedSessionHoldsNoFdUnderItsRoot.

Readdir. READDIRPLUS did a host lookup and made an inode for every entry. About 22 entries fit a 4 KiB page, so 300 entries took OPENDIR plus 14 READDIRPLUS. It saves nothing here: file attributes are never cached, so ls -l costs one GETATTR per entry either way. It is off, and a listing is now 4 READDIR pages. Listings and file pages are also cached in the guest and validated by the host at every open (cache_linux.go): FOPEN_KEEP_CACHE / FOPEN_CACHE_DIR are set only if the object's (size, mtime, ctime) stamp equals the one recorded when the cache was filled. That stamp must have been settled, meaning the coarse clock had passed its ctime, and no older handle may be open. Otherwise the open drops the cache. So close-to-open coherence does not depend on notifications. The cache tests run with notifications off, and a mutant that trusts unsettled stamps fails them.

Round trips. Read-only opens set FOPEN_NOFLUSH. Locks are OFD locks released at RELEASE, so FLUSH did nothing. open+read of a small file is now GETATTR (may_open), OPEN, GETATTR, GETATTR when the file is unchanged, against 6 trips before. FUSE_NOTIFY_INVAL_ENTRY is no longer sent for a name the peer never looked up. The kernel ignores it, but first takes the directory's i_rwsem exclusively, which stalls the peer's lookups.

The tail. Nothing on the request/reply path waits on a timer. Acks ride on requests and replies; the 1 ms ack coalescers only free retained messages. A guest's relay runs in invm-agent with GOMAXPROCS=1. In TestNodefsGuestCPU, each guest's relay and workload are processes pinned to one CPU. A busy agent (GC-heavy load, NODEFS_GUEST_CHURN=1) reproduces Development's tail exactly: stat p99 4.9 ms, create p99 25 ms, open+read p99 16 ms, ping-pong p99 45 ms. The idle relay alone does not. Under 10% CPU bursts, raising the relay's GOMAXPROCS to 2–4 cut ping-pong p99 only from 7.9 to about 5.5 ms, so it is not changed. Fewer round trips per op is what reduces the exposure.