Shared Volume local acceptance

verify/juicefs-local-lifecycle.ts exercises a dedicated local Development control plane connected to two real hosts and the configured JuiceFS service. It does not modify the guard on the hosted Development acceptance script.

Run its default dry run first. Execution requires these environment variables:

The operator must first verify that the local CP's database, host inventory, JuiceFS metadata service and S3 configuration are the isolated Development rig. Loopback routing alone does not prove data isolation. Install and verify the candidate host binaries separately; the runner checks the CP SHA and actual sandbox placement, not a binary's on-disk hash or running process identity. The template must support managed processes; exec-backgrounded nohup jobs are killed when the exec RPC returns and are not a live-writer test.

The dedicated organization needs two Local Volume slots for host pinning, two Shared Volume slots for source/fork, two RW attachments, a third mount on one guest, checkpoint/restore admission, and at least 8,000 snapshot/source inode budget. Existing fixtures count against limits. No gate or quota is changed by the runner.

bun verify/juicefs-local-lifecycle.ts --dry-run
bun verify/juicefs-local-lifecycle.ts --execute-local

The runner uses curl for lifecycle requests and the current TypeScript SDK for guest execution. Before the concurrent workload it proves A-write→B-read and B-write→A-read checksums for exact declared bytes. It verifies the declared bytes of 200 independently written files from both hosts (not merely matching hashes of possibly incomplete views), idempotent resource and operation identity, unchanged attachment epochs across checkpoint, live managed-writer PID/handle survival, and a checkpoint-time acknowledged sequence of immutable, fsync-flushed files. The fork must contain every exact file in both acknowledged prefixes; writer liveness alone is not sufficient. It then verifies fork checksums and isolation, restart placement/readiness/checksums, and exact historical attachment detach receipts. Sandbox deletion waits for the original Local Volume pin attachment to be released; HTTP 404 after DELETE only proves the immediate API tombstone. Shared Volume/snapshot deletions wait for their durable operation to succeed. Local Volume deletion is admission-only and still needs independent host receipt and physical cleanup verification. Writer loops deliberately do not retry filesystem errors. An unsupported checkpoint fails acceptance; it is not relabelled a passing idle checkpoint. Writers use immutable sequence-file creation and a separately fsync-flushed acknowledgement file so the observer cannot accept an unacknowledged or transient prefix. The guest must advertise Python os.fsync or sync -f; Python is detected at runtime and is never assumed. Writers have a ten-minute process deadline if a run fails.

Optional PT_LOCAL_JUICEFS_FAULT_HOOK names an executable receiving a milestone and JSON of exact fixture IDs, admitted operation IDs and managed writer handles/PIDs. Milestones include attached, rw_verified, checkpoint_admitted, sandbox_stopped, detach_admitted, and sandbox_delete_admitted. It receives no token. The hook must independently verify service ownership, restrict the fault to the designated Development hosts, arm a bounded automatic restore before causing an outage, and emit timing/restoration evidence. The runner SIGKILLs the hook at 60 seconds; killing the hook is not proof that its remote fault was restored. An admitted operation is not proof of which host-side phase was interrupted: correlate exact operation receipts and host timestamps. Re-run separate fault schedules rather than claiming one run covered every interruption boundary.

JSONL output records fixture IDs and checksums. On failure it intentionally retains fixtures for diagnosis. Cleanup must target those exact resources via the public lifecycle APIs after recovery, not delete a namespace or bucket.

PASS_PUBLIC_LIFECYCLE is not complete physical-cleanup evidence. Independently verify each host has no fixture attachment records, exports, virtiofsd scopes, RW markers, guest devices, or checkpoint barriers; verify JuiceFS canonical directories and quotas are gone and metadata attachment counts are zero. Retained operation receipts may be required for safe retry and are not live attachments. Object-store garbage collection/trash retention is a separate storage policy and must not be bypassed by deleting bucket contents.

Interrupted recovery

Development super-admin endpoints expose the opaque filesystem recovery tuple: GET /v1/admin/juicefs/filesystem-recovery and POST /v1/admin/juicefs/filesystem-recovery/resume with incident_id, source_host_id, and lease_id. Resume preserves the same global fence and requires prior uncertain lifecycle operations to be resolved first. It cannot reset a partially established repair. Older ambiguous incidents additionally need fresh authenticated source identity and explicit incident-free reports from every other filesystem member, then force complete canonical-root repair. There is no force override or direct fence-clearing option.

Checkpoint cancellation uses POST /v1/admin/juicefs/operations/:id/resolve with {"action":"cancel"}. It releases the volume only after exact writer-resume proof; unresolved clone commands and unsafe/missing host evidence remain fenced. Detach/delete recovery uses {"action":"resume"} against the original operation and host receipt.