Shared Volume RW gateway evidence — 2026-08-27

Source base: 88514dc56448018f80351c7d4470f7dbcb2f97f0.

The candidate was tested on the designated Tailscale-connected Linux/KVM host against the isolated local control plane and the designated Shared Volume test bucket. Production was not changed. The gateway and Admin RW switch are back OFF after testing; the host advertises gateway v0 with a null fingerprint.

Result

The root-only durable virtual-multipart spool makes the restricted mounted-RW MVP viable while retaining rclone v1.75 as the filesystem frontend. File bytes flow between the loopback gateway and S3, never through the control plane.

The proven MVP is deliberately narrow:

• one active or pending RW attachment per volume, plus existing RO attachments;
• sequential create/truncate/write/fsync/close, copy, rename and delete;
• no multi-writer claim, random-write or POSIX-locking guarantee;
• exact host capability/fingerprint, tier, organization, global/Admin and emergency-lock gates, all fail-closed and default OFF;
• reads remain available during control-plane failure; new mutations fail;
• root-only, attachment-scoped spool and credentials with no direct rclone-to-S3 write path.

Linux/KVM gold path

Two independent sandboxes mounted the same Shared Volume at /mnt/gold.

1. Sandbox A attached RW and sandbox B attached RO. A second RW attach raced and received HTTP 409 shared_volume_rw_attachment_limit.
2. A wrote a file; B read the same SHA-256. B's write failed read-only.
3. New-file, overwrite-grow, overwrite-shrink, copy, rename and delete changed logical bytes/object counts as expected. Delete reserved zero growth and committed the negative provider-confirmed delta.
4. The roles were detached and swapped. Former reader B wrote through RW; former writer A read the same SHA-256 through RO and could not write.
5. Organization quota was lowered to zero. dd conv=fsync returned EIO before the provider accepted bytes; the RO mount never saw the rejected object. Authorized delete still worked. Restoring quota allowed a new write.
6. The emergency write lock produced the same deterministic EIO while reads and deletes continued.
7. With the isolated control plane stopped, both mounts kept reading and a new RW fsync failed EIO. After restart, writes resumed and the RO sandbox saw the result.
8. Restarting the host recovered the live RW and RO mounts, preserved prior objects, accepted a new write and exposed it through RO.
9. A fault-injected rejected create originally left four open multipart rows. The final recovery logic classified definitive rejection separately from ambiguity, released its zero-byte reservations and aborted every orphan open upload on restart. A subsequent write and RO read succeeded.
10. Final detach, host restart with no attachments, reattach, persistence read, final write/read and detach all passed.

The RW rclone process contained no provider credential in argv, environment or its mode-0600 root config. Its only S3 endpoint was the attachment's loopback listener. The bounded ext4 cache was 10.46 GB usable in a 10.74 GB image; the observed spool was 16,086 bytes. Upload/checker concurrency was one.

Automated validation

Behavioral coverage includes atomic quota races, one-RW admission races, new/overwrite/copy/delete accounting, multipart replacement/retry/abort/complete, stale generation rejection, prefix and tenant isolation, lost responses, ambiguous reservation preservation, quota reduction, emergency lock, bounded spooling, detach, restart recovery, direct API barriers and existing RO/direct file lifecycle compatibility.

Safe final state

• no live Shared Volume attachment remained;
• all tested capabilities were closed and no multipart upload remained open;
• the global switch was false and the test organization override was removed;
• the host advertised shared_volume_write_gateway_v=0 and no fingerprint;
• a final RW admission attempt returned HTTP 503 shared_volume_rw_mount_disabled.

See docs/internal/shared-volume-rw-gateway-runbook.md for rollout, rollback, emergency-lock, metrics and reservation-recovery procedures.