Restoring the JuiceFS filesystem from a metadata backup

Internal runbook. Every command here was executed against the live Development filesystem on 2026-09-18; the transcript with real output is in verify/juicefs-meta-restore/.

1. What is actually at risk

JuiceFS is two halves that fail independently.

Nothing in the object key records which file a block belongs to, or where in that file it sits. Lose the metadata and the bucket is a complete, intact, permanently unreadable pile of numbered byte ranges. A backup of the object store is not a backup of the filesystem.

2. The three copies of the metadata, and which one is authoritative

Copy 2 is real and has been running unnoticed all along — the review that opened this work was right that the repository contains no metadata backup, and wrong that the filesystem had none. Verified 2026-09-18: 56 objects, 3.3 MB, under platinum-dev-juicefs/meta/, oldest dump-2026-09-02-065516.json.gz, newest dump-2026-09-18-233328.json.gz. It is not relied on because:

• it shares a bucket and a prefix with the data it describes, so one lifecycle rule, one credential compromise, or one rm -r of the prefix takes both;
• JuiceFS thins it on its own schedule (keep all < 2 d, daily to 14 d, weekly to 8 w, monthly after) — we do not choose the retention;
• JuiceFS silently skips the automatic backup once the filesystem exceeds ~1 000 000 inodes at the default 1 h interval;
• it stops the moment the last client unmounts, and tells nobody;
• it is unchecksummed and its age is monitored by nothing.

Copy 1 fixes each of those: its own prefix, our retention (30 days with a 14-dump count floor), a SHA-256 sidecar, a verified read-back after upload, a cadence that does not depend on a mount being up, and limits.juicefs_meta_backup_max_age_hours on the control plane, which pages when it stops.

2a. Configuration

Hosts — through the encrypted Development profile, rendered by infra/env.sh host-juicefs and applied by infra/scripts/apply-host-juicefs-env.sh. All optional; the backup script exits 0 unless the first is 1.

Control plane — process environment, deliberately separate from PT_S3_* so one credential cannot reach both namespaces. Without all five the health rail reports metadata_backup.observable: false rather than guessing.

PT_JUICEFS_META_BACKUP_BUCKET, PT_JUICEFS_META_BACKUP_PREFIX, PT_JUICEFS_META_BACKUP_S3_ENDPOINT, PT_JUICEFS_META_BACKUP_S3_ACCESS_KEY, PT_JUICEFS_META_BACKUP_S3_SECRET_KEY, and optionally PT_JUICEFS_META_BACKUP_S3_REGION. A list-only credential is sufficient and is what should be used.

The ceiling is the app_config key limits.juicefs_meta_backup_max_age_hours (default 26 h against the 6 h timer: one missed run is silent, two consecutive misses page). 0 reports the age without ever calling it stale.

3. Consistency: what a dump does and does not promise

juicefs dump is not a snapshot. The JuiceFS documentation states that if files are modified during the export, the dump "will contain information from different points in time, which might prove unusable for some applications (like databases)", and recommends suspending writes when consistency matters.

Our dumps are taken against a live filesystem with no quiescing. Therefore:

• Per-inode consistent. Any single file's attributes and chunk→slice map come from one read and are coherent.
• Not filesystem-wide consistent. Two files may be from instants seconds apart. A rename that moved a subtree mid-dump can appear from both ends or from neither. A multi-file write that must be all-or-nothing (a database's data file and its WAL, a git object and its ref) can be restored torn.
• This is accepted deliberately: quiescing means stopping every writer in the fleet, and a backup that only runs in a maintenance window is a backup that does not run.

To take a consistent dump for a migration or a planned cutover, stop every writer first — flip flags.juicefs_emergency_write_lock, confirm emergency_rw_outstanding is 0 on GET /v1/admin/juicefs/health, then run the backup unit by hand.

4. RPO and RTO

RTO scales with metadata size, not with data size: no object is copied. A filesystem 1 000× this one's inode count should be expected to take minutes, not seconds, for the load; the object bucket is untouched either way.

5. What is unrecoverably lost

State this plainly to anyone asking "did we lose anything":

1. Every write between the last dump and the loss. Up to the RPO above. Files created in that window exist in the bucket as orphaned blocks that no metadata references; they are unreachable and will be reaped by juicefs gc --delete. Files modified in that window revert to their state at dump time. Files deleted in that window come back.
2. Open-file and session state. Sessions and Sustained are restored empty (verified: nextSession 33 → 0, nextTrash 38 → 0). A file that was deleted but still held open by a running sandbox is simply gone.
3. Trash contents past the dump. Whatever entered .trash after the dump.
4. The object-storage secret. juicefs dump omits it deliberately (that is why a metadata backup is not itself a credential). It must be re-supplied — see step 5 of §7.
5. Counter continuity. See §6. This one is not merely lost; it is actively dangerous.

6. HAZARD: juicefs load regresses the slice counter

This is the single most important thing in this document.

A JSON load does not restore the dumped counters. It recomputes them from the records it restored. Measured on 2026-09-18 loading a real dump of this filesystem:

Dumped counters: {UsedSpace:27152384 UsedInodes:472 NextInode:80898 NextChunk:98305 NextSession:33 NextTrash:38 ...}
Loaded counters: {UsedSpace:27140096 UsedInodes:469 NextInode:78871 NextChunk:98042 NextSession:0  NextTrash:0  ...}

nextChunk went backwards by 263. nextInode went backwards by 2 027.

Why that matters

Object keys are derived from the slice id: chunks/<id/1000/1000>/<id/1000>/<id>_<index>_<len>. Slice ids 98 042 … 98 304 have already been issued on this filesystem, and objects for them already exist in the bucket. A restored filesystem that is then written to will hand those same ids out again, and the client will PUT to keys that already hold somebody else's bytes.

Concretely, once a restored filesystem accepts its first write:

• new blocks overwrite existing objects in the bucket at colliding keys;
• any restored inode whose chunk map still points into the re-issued range now reads the new file's bytes — silent cross-file corruption, with no error anywhere;
• juicefs gc accounting disagrees with reality, and gc --delete can delete an object that two different inodes believe they own.

Nothing warns you. The filesystem mounts, reads correctly, and corrupts on write.

Required mitigation — do this BEFORE the restored filesystem takes any write

Mount read-only (--read-only) until the counters have been advanced. Then raise them above anything the object store has ever used:

1. Read the counters recorded in the dump you restored from — they are the Counters block of the JSON, and they are the high-water marks of the lost filesystem:

   gzip -dc restore.dump.json.gz | python3 -c 'import json,sys; print(json.load(sys.stdin)["Counters"])'
   # {'nextChunk': 98305, 'nextInodes': 80898, 'nextSession': 33, 'nextTrash': 38, ...}

2. Find the highest slice id actually present in the bucket, which can be higher than the dump's counter if writes continued after the dump. The key layout is hierarchical, so walk the highest prefix rather than listing everything:

   # highest <id/1000/1000>, then highest <id/1000> under it, then the highest key
   aws s3 ls s3://<bucket>/<prefix>/chunks/           | sort -V | tail -1
   aws s3 ls s3://<bucket>/<prefix>/chunks/<a>/       | sort -V | tail -1
   aws s3 ls s3://<bucket>/<prefix>/chunks/<a>/<b>/   | sort -V | tail -1

3. Set the restored engine's counters to **max(dump counter, bucket high-water)

   • a generous margin** (10 000 is cheap; ids are 64-bit). JuiceFS CE 1.4.1 exposes no command for this — juicefs config has no counter flags and juicefs load has no --keep-counters — so it is a direct write to the metadata engine:

   -- names in the SQL engine are nextChunk / nextInode (note: the JSON dump
   -- spells the inode counter "nextInodes")
   UPDATE jfs_counter SET value = 108305 WHERE name = 'nextChunk';
   UPDATE jfs_counter SET value =  90898 WHERE name = 'nextInode';
   SELECT name, value FROM jfs_counter ORDER BY name;

4. Only now drop --read-only and let writers back in.

A restored filesystem that will never be written to (a forensic read-only copy, the rehearsal in §8) does not need this. A restored production filesystem always does.

7. The restore procedure

Prerequisites: the juicefs binary at exactly the version the filesystem was running (1.4.1 here — a newer client can refuse an older MetaVersion, and an older one cannot read a newer dump), network reach to the object bucket, and the object credentials from a root-only carrier.

Step 1 — Provision an empty metadata engine

juicefs load requires an empty database and does not roll back a partial import. Create a fresh one; never load into a database that has ever held a JuiceFS filesystem.

sudo -u postgres psql -v ON_ERROR_STOP=1 <<'SQL'
CREATE ROLE jfs_restore LOGIN PASSWORD :'pw';   -- password from a root-only file, never argv
CREATE DATABASE juicefs_restore OWNER jfs_restore;
SQL

For a real production restore this is a new Scaleway managed PostgreSQL instance in fr-par, with sslmode=verify-full and the pinned root CA, matching infra/scripts/juicefs-identity.sh's accepted identity shape. The metadata engine can never be far from the hosts — see the platinum-juicefs-metadata-topology note: a cross-continent metadata engine fails the 10 s mount deadline.

Step 2 — Fetch the newest dump and verify it

# newest object under platinum-dev-juicefs-meta-backups/<fs>/ ending .dump.json.gz
aws s3 cp s3://<bucket>/<backup-prefix>/<fs>/<stamp>-<host>.dump.json.gz        restore.dump.json.gz
aws s3 cp s3://<bucket>/<backup-prefix>/<fs>/<stamp>-<host>.dump.json.gz.sha256 restore.dump.json.gz.sha256

# the sidecar is not decoration: a truncated dump loads without complaining
test "$(awk '{print $1}' restore.dump.json.gz.sha256)" = "$(sha256sum restore.dump.json.gz | awk '{print $1}')" \
  && echo 'SHA-256 MATCH'
gzip -t restore.dump.json.gz && echo 'gzip integrity OK'

Step 3 — Load

export META_PASSWORD="$(cat /root/restore-pgpass)"   # never in argv
juicefs load 'postgres://jfs_restore@127.0.0.1:5432/juicefs_restore?sslmode=disable' restore.dump.json.gz

Read the two counter lines it prints and record them — they are the input to §6. Expect secret key was removed; please correct it with 'config' command.

Step 4 — Confirm the filesystem identity

juicefs status 'postgres://…/juicefs_restore'

Setting.UUID must equal 846d38da-b680-47d6-a8b8-09dd83d4ecaa and Setting.Name must equal platinum-dev-juicefs. A different UUID means you loaded the wrong dump: the UUID is what ties this metadata to platinum-dev-juicefs/chunks/ in the bucket, and a mismatch will read garbage or nothing.

Setting.Bucket, BlockSize (4096), Compression (none), TrashDays and MetaVersion must also match — the block size in particular, because it is baked into every object key already written.

Step 5 — Re-attach the object secret

The dump omits it. Use the helper, never a hand-typed command:

sudo PT_JUICEFS_S3_SECRET_KEY=… infra/scripts/juicefs-restore-set-object-secret.sh \
  'postgres://jfs_restore@127.0.0.1:5432/juicefs_restore?sslmode=disable'
# -> secret-key: updated

Step 6 — Mount READ-ONLY and prove it reads real files

export ACCESS_KEY=… SECRET_KEY=… AWS_REGION=fr-par
juicefs mount -d --read-only --backup-meta 0 --no-usage-report --no-agent \
  --cache-dir /srv/restore-cache --cache-size 512 \
  'postgres://…/juicefs_restore' /mnt/restore

--read-only is what holds the §6 hazard shut. --backup-meta 0 stops the restored client writing its own hourly dump into the production object prefix.

Then prove it, rather than assuming it:

find /mnt/restore -xdev -type f | wc -l       # compare with the dump's file count
sha256sum /mnt/restore/<a real tenant file>   # compare with the live copy if one exists
juicefs fsck 'postgres://…/juicefs_restore'   # no --repair: read-only validation

fsck reports Found N blocks / Used by M slices. Missing objects are named individually; an empty report means every block the restored metadata references is present in the bucket.

/.config at the mount root will differ from the live filesystem. It is JuiceFS's own virtual view of the running client's options (cache dir, read-only flag, version), not tenant data.

Step 7 — Advance the counters (§6), then cut over

Only after §6: repoint the fleet by writing the new metadata URL into /etc/platinum/juicefs.env on every host via infra/scripts/apply-host-juicefs-env.sh, and restart platinum-juicefs-mount.service.

Step 8 — Re-verify the fingerprint

Changing the metadata URL changes the host fingerprint. juicefs_bootstrap_fingerprint in infra/scripts/juicefs-identity.sh frames the metadata identity (role, host, port, database, TLS parameters — password excluded) into a SHA-256 alongside the filesystem name, endpoint, region, bucket, prefix and CA digest. After a restore onto a new database host, every host's hosts.juicefs_config_fingerprint will differ from the recorded expected_config_fingerprint, and GET /v1/admin/juicefs/health will report every host as compatibility: "mismatched" — which correctly makes them unschedulable for shared volumes.

The recorded expectation must be re-established from the restored bootstrap owner. Do not "fix" this by editing fingerprints on individual hosts: a fleet that disagrees about which metadata engine it is talking to is exactly what the fingerprint exists to catch.

8. Rehearsing this without touching production

Everything in §7 steps 1–6 is safe to run against the live filesystem as long as you never point step 5 or 7 at the production metadata URL. The dump is read-only; the load goes into a database you created; the mount is read-only and reads the same objects any client reads.

juicefs-restore-set-object-secret.sh refuses outright if the target metadata host equals PT_JUICEFS_META_URL's host, because that is the one step that writes to a format record.

The full rehearsal, with real output, is in verify/juicefs-meta-restore/.

9. UNVERIFIED: Scaleway RDB backup and PITR

The managed PostgreSQL instance behind juicefs_dev may have automated backups and point-in-time recovery enabled. This has not been verified, and nothing in this repository configures, checks or alerts on it.

Until someone confirms it from the Scaleway console or API, treat variant C in §4 as having no known RPO and no known RTO, and do not count it as a recovery path. What is proven today is variant A.

Confirming it is worth doing: RDB PITR would cut the RPO from 6 h to minutes, and it is the only copy that survives the dump job itself being broken.