ADR: Shared Volume mount architecture
ADR: Shared Volume mount architecture
Status: accepted for the read-only private beta. The mounted read/write extension is blocked pending a durable mutation-identity protocol.
Shared Volumes are object-backed filesystems, not block devices. Object storage
under PT_VOLUME_S3_* is authoritative and is isolated from CAS. Each volume
uses the stable-key namespace <prefix>/orgs/<org-id>/volumes/<volume-id>/data/.
The guest never receives object-store credentials.
Options evaluated
| Design | Credential isolation | Hot attach and reconnect | Kernel/runtime needs | Cache consistency and performance | Decision |
|---|---|---|---|---|---|
| Host object-FUSE + virtio-fs | Long-lived S3 credentials remain in a root-only host file; each guest sees only one attachment directory | Cloud Hypervisor add-fs; host metadata and CP attachment rows replay the chain after agent/VM restart | host FUSE3, rclone, virtiofsd, guest virtio-fs | Extra FUSE + virtio-fs hop; the current supported mount is read-only | Selected for read-only hot attach without guest credentials |
| Authenticated guest FUSE over vsock, S3 on host | Host keeps S3 credentials, but a new privileged file protocol and guest client become security-critical | Protocol could reconnect, but neither replay nor hot-plug exists in the current guest agent | new guest FUSE client plus a versioned, authenticated vsock filesystem protocol | Could centralize cache policy; every filesystem operation crosses the new RPC boundary | Rejected for beta: materially larger new trusted surface and no proven reconnect implementation |
| Proven S3 FUSE with short-lived prefix credentials in guest | Guest receives credentials. Prefix scope reduces blast radius but does not prevent exfiltration during the lease | Client-dependent; expiry, refresh, clock skew, and restore replay become guest-visible failure modes | guest FUSE package and network egress to S3 | One fewer host/guest hop, with the selected client's object-store cache semantics | Rejected: violates the requirement that guests never hold object-store credentials |
Cloud Hypervisor and the host agent already have a post-ready virtiofsd /
add-fs path. It was previously disabled for the generic /mnt/pt-share
feature because it was attached to every VM and interacted badly with snapshot
restore. Shared Volumes use the hardened post-ready path only, one daemon and
tag per attachment, and are reattached after restore; they are never embedded in
a CH memory snapshot. A mount failure is visible as an attachment error and
never fails the sandbox boot.
The host runs the S3 FUSE backend with its own configuration. The backend is
rooted to the exact volume + attachment subpath; virtiofsd exports only that
directory. No guest environment, API response, host command payload, or process
argument contains S3 credentials. The implementation uses a protected host
credential file rather than command-line secrets.
Semantics
Shared Volumes may be mounted read-only concurrently across hosts. Mutations use the direct file API. Mounted read/write is unavailable: the existing rclone S3 request shape cannot distinguish a same-path, same-size new write from a retry after a lost response. The control plane therefore cannot safely make quota reservation commit idempotent.
The current rclone mount uses a per-attachment loopback cache filesystem under the Platinum data root, bounded to 10 GiB. Closed handles and inactive cache entries are evicted on a one-second cadence; this bounds the measured close-then-reopen visibility path but does not invalidate a still-open file. Clean detach/stop fences guest file descriptors across mount namespaces and only then removes the backend. The write-back observations below describe the disabled prototype, not a supported durability contract.
Each attachment has an owned and waited rclone process plus a supervised
virtiofsd transport. rclone runs in a bounded systemd scope; host admission
also caps attachment process slots from physical memory (one slot per 8 GiB,
clamped to 1-32) while preserving existing dirty-cache owners during restart.
Recovery is
CP-authoritative: lifecycle graph replacement commits existing attachment rows
back to pending in the same transaction that publishes the sandbox as
running, and dispatch happens after commit. A host-agent restart rebuilds
from credential-free attachment metadata. Failed or timed-out cleanup preserves
the backend and reports an attachment error instead of claiming a clean detach.
ro is enforced at the FUSE mount boundary and host attachment layer. Subpaths
are canonical, relative, and traversal-free before any host command is created.
Mount paths reject system directories and collisions. Local Volumes retain their
existing sparse-ext4/virtio-blk single-writer, host-pinned behaviour.
Snapshot policy
Sandbox snapshots/backups record attachment metadata only. Same-sandbox restore
reattaches recorded volumes after guest readiness. Clone/fork must choose
omit, reattach, or clone; it never implicitly copies volume contents.
Shared Volume snapshots reject active read/write attachments, claim a
volume-wide mutation barrier, copy objects into a new immutable generation, and
publish the manifest last. The resulting generation is consistently restorable
because writers are quiesced; it is not an atomic snapshot of concurrent
writes. Read-only attachments may remain mounted. Local snapshots require the
volume to be detached, pass read-only filesystem checks, and use a verified
reflink capture; hosts without reflink support return an explicit error.
Failure modes and containment
| Failure | Observable behaviour | Containment and recovery |
|---|---|---|
| rclone exits or its FUSE mount disappears | guest I/O fails and attachment health no longer matches the published mount identity | the sole waiter records the exit; recovery validates the owned process, RC endpoint, source, mount id, and operation generation before adoption or replacement |
| virtiofsd exits or its vhost-user socket disappears | the guest virtio-fs mount stops serving | the supervised per-attachment daemon is restarted only for its current attachment generation; stale supervisors cannot publish over a replacement |
| S3 is unavailable during a read or detach | reads fail and detach remains pending rather than discarding owned state | teardown refuses destructive cleanup when it cannot prove the attachment is drained |
| host data disk approaches exhaustion | a new attachment is denied before allocating cache | each attachment uses a 10 GiB ext4 cache boundary beneath DATA_ROOT; admission reserves 20 GiB host free space and never falls back to root's rclone cache |
| host-agent restarts | running VMs continue, while attachment state may briefly be reconciling | heartbeat/poll start first; bounded background recovery re-adopts only kernel- and identity-proven mounts and reconciles the control plane |
| Cloud Hypervisor device hot-plug races lifecycle rebuild | attach may return pending or be replayed after guest readiness | stable attachment ids, explicit device ids, operation generations, and host fencing make stale completion inert; lifecycle replay reconstructs current rows only |
an API, stale command, or recovered attachment requests rw | the request fails before provider or guest work | config, control-plane admission, both host constructors, and startup recovery remain fail-closed |
| a future gateway retries an ambiguous PUT | same-size writes can alias without a durable mutation identity | do not advertise gateway v1 or enable RW until the provider outcome and reservation can be recovered without guessing |
The per-attachment disk cache is a hard filesystem boundary. rclone and virtiofsd run in validated, generation-specific transient scopes. rclone has a 512 MiB hard memory limit and both have bounded task counts; virtiofsd does not yet have a separate hard memory limit and is contained only by the aggregate VM slice. Recovery preserves cleanup ownership until the exact process group and cgroup are empty. The private beta retains conservative attachment limits and host monitoring until broader workload and fault-injection measurements justify expansion.