Development JuiceFS host foundation

This is an additive, default-off Development-only backend. It does not change Local Volumes or the existing object-FUSE Shared Volume backend.

Host rollout accepts a root-only carrier through PT_HOST_JUICEFS_ENV_FILE. The carrier must be either exactly PT_JUICEFS_VOLUMES_ENABLED=0, or the complete set below with PT_JUICEFS_ENVIRONMENT=development. The deploy script validates it locally, copies it through SSH with mode 0600, and writes the password to the separate root-only /etc/platinum/juicefs-meta-password file. It never prints values.

PT_JUICEFS_VOLUMES_ENABLED=1
PT_JUICEFS_ENVIRONMENT=development
PT_JUICEFS_BOOTSTRAP_OWNER=0
PT_JUICEFS_META_URL=postgres://juicefs_dev@host:5432/juicefs_dev?sslmode=verify-full&sslrootcert=/etc/platinum/juicefs-meta-ca.pem&max_open_conns=4&max_idle_conns=1&max_life_time=3600
PT_JUICEFS_META_PASSWORD=...
PT_JUICEFS_META_TLS_CA_BASE64=... # canonical base64 of the PEM bundle, single line
PT_JUICEFS_META_TLS_CA_SHA256=... # sha256 of the decoded PEM bytes
PT_JUICEFS_S3_ENDPOINT=...
PT_JUICEFS_S3_REGION=...
PT_JUICEFS_S3_BUCKET=...
PT_JUICEFS_S3_PREFIX=platinum-dev-juicefs # exactly equal to filesystem name
PT_JUICEFS_S3_ACCESS_KEY=...
PT_JUICEFS_S3_SECRET_KEY=...
PT_JUICEFS_MOUNT_ROOT=/var/lib/platinum/juicefs/dev
PT_JUICEFS_CACHE_DIR=/var/lib/platinum/juicefs/cache/dev
PT_JUICEFS_CACHE_SIZE_MIB=4096
PT_JUICEFS_FREE_SPACE_RATIO=0.1
PT_JUICEFS_LOG_DIR=/var/lib/platinum/juicefs/log/dev
PT_JUICEFS_FILESYSTEM_NAME=platinum-dev-juicefs
# PT_JUICEFS_FILESYSTEM_UUID is optional until the lease-bound CP bootstrap
# records it. Host carriers remain owner=0; only a one-shot host command may
# format an absent filesystem.

JuiceFS CE 1.4.1 does not enforce an arbitrary third path component passed to --bucket: it uses endpoint/bucket as storage and writes physical objects beneath the formatted filesystem name. This rig therefore requires PT_JUICEFS_S3_PREFIX == PT_JUICEFS_FILESYSTEM_NAME, passes only endpoint/bucket to JuiceFS, and includes that equal physical prefix in the carrier fingerprint, IAM policy, local rig and live verifier. A Development format whose metadata records a bucket-path suffix is refused; migrate it only with mounts stopped and an explicit backup/readback plan.

JuiceFS is applied only in shared_volume_mode=configure. The host roll carries an optional-capability contract with three modes: preserve (the default, and what every push to main runs) touches no JuiceFS credential, unit or mount; configure renders and installs the carrier and converges the client prerequisites; disable retracts the credential after the control-plane feature flag is off, and refuses while any attachment receipt, scoped virtiofsd or bind export survives. An ordinary binary roll therefore never decrypts a JuiceFS profile, never requires DOTENV_PRIVATE_KEY, and cannot change host JuiceFS state.

host-install.sh and a configure deploy-hosts.sh roll both use the pinned JuiceFS CE 1.4.1 installer with the official immutable release archive and committed per-architecture SHA-256. They install platinum-juicefs-bootstrap.service and platinum-juicefs-mount.service; the latter is a root-only foreground FUSE client with a bounded cache, free-space floor, local metrics listener, bounded start/stop, and no lazy unmount. It pins JuiceFS --heartbeat 1 for the smallest supported Development quota-refresh interval. This bounds the normal cross-client quota convergence delay, but it does not claim mathematically zero simultaneous-writer overshoot; acceptance records the observed EDQUOT/ENOSPC boundary and measured delay instead.

Development Object Storage prerequisite

Do not bootstrap against the currently supplied platinum-shared-volumes bucket: the inspected bucket is empty and has no bucket policy in the platinum-prod Scaleway project, which also contains a sibling bucket. Scaleway Object Storage IAM is project-scoped and its IAM conditions do not provide a safe object-prefix restriction, so a JuiceFS application key for that project could reach the sibling. Before any Development rollout, create or designate a separate Development Scaleway project and dedicated JuiceFS-only bucket, then grant a least-privilege application only to that project/bucket. Its documented object-key policy boundary is ${PT_JUICEFS_FILESYSTEM_NAME}/; the verifier must prove both positive access there and denied access to a sibling namespace. No IAM application, policy, or API key is created by these scripts.

The host profile is valid only for a dedicated juicefs_dev PostgreSQL logical database, a least-privilege juicefs_dev role, authenticated TLS sslmode=verify-full (or verify-ca only where the Development endpoint has no stable DNS name), and a root-only CA bundle whose decoded SHA-256 is bound into both host fingerprints. The CA is materialized at /etc/platinum/juicefs-meta-ca.pem (0600); the URI must name that exact path. The metadata password remains in a separate 0600 file and is supplied only to the short-lived CE child as META_PASSWORD, never in its argv or the systemd environment. The canonical two-host carrier uses four open and one idle connection per JuiceFS client; validation caps open connections at four and requires idle not exceed open. With two canary mounts this leaves room under the measured role cap for one bounded maintenance/repair command; it is not an unlimited pool. It also requires bounded max_open_conns, max_idle_conns, and max_life_time URL values. The bootstrap identity hashes a canonical metadata coordinate that retains the role and database/host/TLS mode but removes the password. Thus password rotation does not split a fleet; a role, database, host, TLS, endpoint, bucket, physical prefix, mount-root, cache-policy, binary-version, or filesystem-UUID change does. bootstrap.json is root-only and atomically written; an unversioned early Development identity is migrated only after a live name/storage/bucket readback plus the physical filesystem-name prefix check. A versioned mismatch fails closed.

Bootstrap is invoked only by a CP-issued, lease-bound one-shot command on one Development host. It also takes a local flock, reads existing metadata first, and only formats when absent. It never uses juicefs format --force; a concurrent formatter is accepted only after the winner's name, UUID (when configured), storage and bucket/physical-prefix identity verify. Non-secret identity/status is written to /var/lib/platinum/juicefs/bootstrap.json.

Every later fresh or upgraded host must receive volume.juicefs_join, never a format flag. Join re-reads the shared metadata configuration, verifies the filesystem UUID and CP bootstrap fingerprint, writes local root-only identity, then starts and proves the mount. The operation has a durable receipt tied to the operation ID, UUID, and bootstrap fingerprint, so a lost response can be recovered with volume.juicefs_status but a changed replay is refused.

The host heartbeat advertises juicefsVolumeV=3 only when the exact service is active, its mount is actually fuse.juicefs, cache free-space floor is met, configuration is complete and Development-only, and its non-secret v3 fingerprint matches (JuiceFS version/wire, verified UUID, hashed metadata identity, canonical object namespace, mount/cache safety settings). A bounded root-only metadata status probe and a descriptor-confined hidden object-store sentinel write/fsync/readback must also be fresh; a timeout or failed probe retracts capability. A mount-generation supervisor fences all attachment transports before a root client restart, then rebuilds only exact durable attachments. Any uncertain flush/device/guest state remains fenced (v0). Host syncfs calls are caller-bounded and deduplicated per exact export. A kernel I/O thread can remain blocked after that timeout, but it cannot spawn unbounded retries or produce a detach/cleanup receipt: the attachment/export is retained as flush_uncertain and the host stays fenced until recovery.

juicefsGitV=1 is conditional: the Development Git-workspace flag, exact fingerprint, root mount, hidden Git implementation, and usable Git binary must all be healthy. Otherwise it is 0, so the scheduler cannot route a partial Git feature. Workspaces use an independent hidden git directory/index and a guest-visible work tree only; linked worktrees and guest-readable host remotes are never used.

juicefsMetrics heartbeat schema v1 remains accepted during rollout. A fully healthy enabled host emits schema v2 with the same unlabelled aggregate fields plus cache used/configured/free bytes and pressure; metadata/object dependency health and age; active client count; mount recovery attempt/success totals; and fixed attach/detach/checkpoint/Git latency and failure totals. Counters are capped at 2^53-1, active attachments at 100000, and no paths, IDs, filenames, URLs, or credentials appear in the shape. The control plane clears absent or partial telemetry rather than retaining a stale healthy status.

Fresh installation and an existing-host Development configure roll both run the same idempotent configure-fusermount-apparmor.sh helper before enabling the mount unit, and both run it before either agent binary is replaced, so a first-enable failure leaves the running process and its on-disk executable on the previous known-good version. It reloads Ubuntu's supported fusermount3 local include for only the canonical /var/lib/platinum/juicefs/dev root; non-Development rolls, and every preserve roll, do not write, enable, disable, or remove any JuiceFS/AppArmor surface.

Each attach resolves /orgs/<org-id>/volumes/<volume-id>/live and rejects a symlinked or escaping subpath. It bind-mounts just that directory into a root-only attachment export and starts a dedicated, cgroup-scoped virtiofsd. The global JuiceFS root, metadata URL and storage credentials never enter the sandbox, control-plane command payload, API response, heartbeat, or metric labels.

Attachment desired state and operation receipts are fsync-then-rename durable records. A detach retry without state proves guest mount, Cloud Hypervisor device, scoped virtiofsd, and bind export are absent before succeeding. On a sandbox stop/start or host-agent restart, a readable fresh guest with no mount and no device is safe to replay; an unreadable guest/device inventory is fenced. A stale device is removed and re-proven absent before a new transport is attached. The host never overwrites a durable attachment record whose complete stable identity/generation differs from a new attach command.

The host Git command primitives create an independent normal repository per workspace, status/diff/log, explicit checkpoint/push, controlled CAS merge, restore, and retained mapping. They use a hidden canonical bare repo under /system/git/<project-id>.git and ephemeral host-only remotes; guest .git/config never stores a host root. The capability stays fail-closed at juicefsGitV=0 unless the runtime conditions above are all satisfied.

Canonical project deletion is a separate, durable operation. The control plane first proves that no live or retained workspace remains. The host then verifies the root-only organization marker and an empty hidden workspace directory using descriptor confinement, flushes the canonical repository, invokes only juicefs rmr --skip-trash on its stable-ID-derived directory, and proves both directory absence and quota removal before returning {deleted:true, quotaReleased:true}. A missing repository is an idempotent success only after the same quota-absence proof; a mismatched marker, unreadable directory, or remaining hidden workspace fails closed.

Volume deletion is fenced the same way every other destructive operation is. The control plane mints an exclusive delete lease -- the durable delete operation row -- inside the transaction that holds the organization advisory lock and compare-and-sets the volume into deleting; the host refuses a destructive delete whose payload does not carry that lease. The host's own attachment map and its durable attachment state directory are per-host records and cannot see a writer on a sibling host, so every read-write attach also publishes a small root-only marker under orgs/<org>/volumes/<id>/attachments on the shared filesystem before the export bind exists, and retracts it only once finalize-detach has proven local absence. That directory sits outside the guest-visible live generation, so it is neither exported nor counted against the volume's hard quota. Deletion fails closed on any marker; a host may reclaim only a marker carrying its own host ID, and only after it has proven it holds no attachment for the volume.