Shared Volume recovery acceptance: 2026-09-06

Verdict

Incomplete / failing acceptance. Do not interpret unit coverage as a complete two-host lifecycle pass. PR #800's sandbox-scoping regression was reproduced and fixed. PR #829 contains useful fixes but does not cover the restart and cleanup failures below. This branch preserves its two commits and adds the recovery corrections found during testing.

Environment and limits of the evidence

• Two real Linux sandbox hosts, reached from a Mac-hosted local Development rig.
• Dedicated JuiceFS PostgreSQL metadata service and object storage; sandbox CAS storage is separate from JuiceFS storage.
• Existing local control-plane processes were retained. These are older review builds, not a newly deployed current-main control plane. Host-agent changes were built from current-main plus #829 and the corrections in this branch.
• Fixtures were isolated in a new test organization. Existing user fixtures were not intentionally deleted. Two additive committed schema changes were needed for sandbox idempotency and the normal host probation boot.
• No staging or production promotion was performed.

Behavioral results

An earlier fault-injection attempt stopped only the mount service's shell wrapper. That attempt is not counted as a storage outage. The subsequent test explicitly verified both JuiceFS processes in stopped state. Guest exec dispatch overlapped agent loss, so these observations do not establish a precise blocked-write latency or a maximum recovery-time guarantee.

Integrity evidence

The two 100-file checksum manifests had these SHA-256 digests on both guests:

46adbad75e9ffb738fac2423648163c238144c4746ad1b7f3451f67652278e40
0f227cd677964cff73d6852cb1dae9453d27b93ef4172a015b3cbc75e633cc09

Guest verification used sha256sum -c as a standalone command under set -e, then counted the 200 successful entries. Post-outage file checks included:

1141370ca6503f1057556be8855bb2a55acf72b30f8ff5d3dc24b26a10c0cb16
f78c3e13ba82dc925e9b3582ad5d03864b1b6702c785ba558001f8b4fbc93e39
60ddb8c022fcd87c160f81a400c6629e761a40a4e6ea2536112606044ba89f5c

Corrections in this branch

• Keep durable sandbox-delete identity until the shared RW marker is retracted.
• Scope sandbox-delete attachment discovery to that sandbox, as #800 does for start; unrelated corrupt records must not prevent unrelated cleanup.
• Resume exact started detach operations after the in-memory map is lost, including crashes before the retry-state marker is written. Recheck and flush the guest/export before removing transport.
• Do not generically reattach an epoch owned by a started or manual-required detach receipt during startup.
• Accept the persisted JuiceFS process-scope family when stopping inherited daemons. Prove absence using existing scope generations, not a newly generated unit name that never existed.
• Re-read intentional-stop state after the daemon wait finishes; a flag sampled before waiting misses a detach request arriving while the process is alive.
• Isolate deletion unit tests from real host attachment roots.

Remaining blockers

1. Live-RW checkpoints need a guest/transport combination supporting the required freeze protocol, or a separately designed and validated quiesce mechanism. Silently skipping freeze would not prove checkpoint consistency.
2. Failed checkpoint barriers require an identity-checked recovery/cancellation path; none was used to clear these fixtures by hand.
3. A crashed, acknowledged detach command did not automatically converge through the retained local control plane. This needs current-main CP reproduction.
4. Sandbox cleanup encountered an unreaped virtiofsd zombie after a failed recovery attach. Repeated SIGKILL cannot reap a zombie. Process wait ownership on failed attach remains a separate issue from ordinary successful detach.
5. Complete Shared Volume cleanup and checkpoint restore checksums are not proven. Remaining test records and barriers must not be presented as cleaned just because sandbox rows are deleted.

Automated and client checks

• API: 44 focused files, 327 passing tests, using a separate PostgreSQL test DB.
• TypeScript SDK: 132 passed, 2 skipped; skipped E2E tests are not counted as proof.
• Python SDK: 132 passed.
• CLI: 8 focused tests passed; MCP: 10 focused tests passed.
• Live TypeScript SDK: create/read/rename/usage and guest execution.
• Live Python SDK and CLI: volume listing.
• Real curl: health, volume, template, and sandbox reads.
• Linux host-agent race suite and vet were run on candidate iterations; final branch CI must still validate the exact published revision.

Final readback

• The final host code passed go test -race -count=1 ./..., go vet ./..., and a Linux build. The host-agent package race run completed in 85.217 seconds.
• Both test hosts run the same final binary SHA-256: 5f174a35772ecab6045b4e0a70492cfc1804abe1442828ae88e92bad3ccc49f8.
• Both host-agent and JuiceFS mount services are active. This is not full recovery: one host advertises JuiceFS volume/snapshot capability 0/0; the other advertises 3/1. The control plane reports filesystem recovery active.
• All three test sandbox rows are deleted and the local placement volume is deleted. Four Shared Volume rows remain: three available and one failed isolated create/delete probe. Two checkpoint rows remain creating.
• Shared deletes return dependency conflicts; deleting the failed probe returns a filesystem-recovery fence. Checkpoint deletion returns not-found for these incomplete checkpoints. No database fence or receipt was manually cleared.
• The original disabled checkpoint feature flag was restored. Recovery of the remaining storage state is still required; this report does not claim quotas, cross-host markers, snapshots, or Shared Volume data have all been removed.

These are targeted checks, not a claim to have exhaustively tested every API, CLI, SDK, quota boundary, outage duration, or concurrent ordering.