Recovering the final filesystem acknowledgement
Recovering the final filesystem acknowledgement
The final source-host acknowledgement is the last global recovery step. A lost
response must not issue a second mutation or release control-plane fences without
proof. Its original operation ID is <filesystem-recovery-lease-id>-incident-ack.
For an ambiguous final ACK, the control plane now queues only
volume.juicefs_status, preserving the exact original incident, root and agent
generations, manifest completeness flags, counts, and digests. It accepts only
the source host's matching root_recovery_incident_ack success receipt and its
complete nested acknowledgement proof. Every volume/project/attachment barrier
is rechecked before the same transaction completes recovery.
Automatic status recovery is bounded to six lookups, at least 15 seconds apart
after the initial lookup. A durable manual host receipt stops it immediately.
The resulting incident_ack_receipt manual fence has an explicit Development
super-admin verification surface:
POST /v1/admin/juicefs/filesystem-recovery/verify-ack
{
"incident_id": "<current-incident-id>",
"source_host_id": "<current-source-host-id>",
"lease_id": "<current-filesystem-recovery-lease-id>"
}This requests a read-only reconstruction of the same ACK receipt. It does not
recapture roots, reissue the unfence mutation, adopt another incident, or clear a
generic namespace_mutation/unknown manual fence. A host whose durable incident
was never acknowledged cannot fabricate success: the epoch and every resource
remain fenced until positive proof exists. A repeated request while its lookup
is pending is idempotent.
juicefsFilesystemAckRecovery.int.test.ts exercises real PostgreSQL response
loss, bounded retries, manual verification, concurrent requests, direct ACK
completion, and substituted host/operation/generation/manifest evidence. Host
tests separately prove that manual reconstruction cannot acknowledge a merely
required incident.