Instant cold resume architecture and delivery plan

Status: Draft for implementation

Primary metric: API resume request to first successful sandbox.exec

Durable source: Object Storage

Related component: Regional L2 CAS cache

1. Decision

Platinum will build three cooperating resume paths:

1. Keep one bounded, predictive, fully materialized replica for sandboxes that are likely to resume soon. This is the fastest path.
2. Add an optional, horizontally scalable regional L2 CAS cache between host-local storage and Object Storage. This reduces repeated network fetches and origin load.
3. Add demand hydration for arbitrary cold state. Start the VM after the critical rootfs and memory state is ready, then fetch the rest through L1, L2, or Object Storage while the VM runs.

The current eager restore remains the permanent fallback.

The L2 cache alone is not the instant-resume feature. It can make a remote read faster, but the current restore path still constructs the complete rootfs and memory files before launch.

2. Why this boundary is required

The current S3 snapshot restore in snapshot_s3.go does this:

1. Download the sidecar.
2. Materialize the complete rootfs.
3. Materialize the complete memory ranges file.
4. Read the complete memory file sequentially to warm the host page cache.
5. Start Cloud Hypervisor with a local file:// UFFD source.

The older cold-restore benchmark moved about 1.2 GB of compressed data from Object Storage in 3 to 5 seconds, but spent about 185 seconds materializing 24 GB locally. The complete restore was about 190 seconds. These are historical measurements, not a current production SLO. See perf-2026-05-28-uffd-zstd.md.

Platinum also already has a true fast path. After a stop snapshot upload, the control plane sends sandbox.warm_chunks to as many as two same-family peers. Despite its name, the host fully materializes rootfs, memory, and sidecar data. Resume then prefers a host with that full replica.

That mechanism can be fast, but it is not bounded correctly today:

• Fan-out is fixed at two peers.
• A measured Kortix replica is approximately 24 GiB, so one stopped sandbox can use approximately 48 GiB across two peer hosts.
• Replica cleanup is manual and byte-bounded LRU is not complete.
• Replica bytes are not part of normal sandbox disk accounting.

The first implementation task is therefore to make the existing full-replica path bounded and predictive. The L2 cache then covers shared misses. Demand hydration is what removes complete materialization from the user-visible cold path.

3. Product contract

Measure one timeline for every resume:

API accepted
  -> target selected
  -> state source selected
  -> VM process started
  -> guest agent ready
  -> first successful sandbox.exec
  -> background hydration complete

Do not use chunk latency or materialization time as the product success metric. Those are diagnostic metrics. The product metric is time to the first successful command.

Proposed SLOs for initial engineering gates are:

These are targets, not measured production claims. Product promises must be set only after the real end-to-end path passes load and failure tests.

4. Target architecture

The target host chooses one of three lanes.

Lane A: predicted full replica

Use an already complete rootfs, memory file, and sidecar on the selected compute host. This remains the lowest-risk subsecond path.

Use it only for recent or high-probability resumes. Store at most one replica by default. Enforce regional, per-host, and global byte limits with LRU and reuse scoring.

Lane B: demand-hydrated cold resume

Create a local writable target, fetch only the state required to boot and serve the first command, then hydrate the remaining state in the background.

Every remote read follows:

compute-host L1 -> regional L2 -> Object Storage

The VM must converge to a normal local representation. After convergence, runtime, backup, archive, resize, snapshot, and migration paths must not depend on the cache node or network.

Lane C: eager fallback

Use the existing complete materialization path when demand hydration is disabled, unsupported, unhealthy, or outside its safety gates. This fallback must remain tested for the lifetime of the lazy path.

5. Immutable resume descriptor

Add ResumeDescriptorV2. The database stores only the active descriptor key, ETag, generation, and logical metadata. The immutable descriptor in Object Storage binds all state required for one resume generation.

Example shape:

{
  "version": 2,
  "sandboxId": "sbx_...",
  "generation": "1786663000123-0123456789abcdef0123456789abcdef",
  "createdAt": "2026-08-13T00:00:00Z",
  "s3Bucket": "platinum-development",
  "rootfs": {
    "manifestKey": "<prefix>/vms/<sandbox>/generations/<generation>/rootfs.manifest",
    "manifestSHA256": "...",
    "logicalSize": 21474836480,
    "chunkSize": 1048576
  },
  "memory": {
    "manifestKey": "<prefix>/vms/<sandbox>/generations/<generation>/memory.manifest",
    "manifestSHA256": "...",
    "logicalSize": 4294967296,
    "chunkSize": 1048576
  },
  "sidecar": {
    "key": "<prefix>/vms/<sandbox>/generations/<generation>/sidecar.tar",
    "sha256": "...",
    "size": 4096
  }
}

Publish in this order:

1. Upload new immutable chunks.
2. Upload generation-scoped rootfs and memory manifests.
3. Upload the sidecar and optional critical-set traces.
4. Upload the descriptor last.
5. Atomically compare-and-swap the database pointer to the new descriptor.
6. Best-effort warm the predicted peer and L2 only after step 5 succeeds.

Implemented in the regional-cache stack: steps 1 through 5, descriptor ETag verification, bounded sidecar SHA verification, manifest-body SHA pinning, and a database compare-and-swap fence bound to the exact sandbox.stop command id. Legacy snapshot references remain readable during rollout. Critical-set traces and safe-age orphan-generation collection remain follow-up work; neither is on the current resume critical path.

Never acknowledge durable stop or archive completion from L1, L2, or a full peer replica. Object Storage and the database pointer remain the durability boundary.

6. Rootfs demand hydration

The first prototype will use a read-only CAS source behind Linux ublk, with a local writable destination managed by dm-clone.

The 1 MiB dm-clone region matches the current CAS chunk size. On the first read from an uncopied region:

1. Fetch and verify the CAS chunk through L1, L2, or Object Storage.
2. Serve the requested region.
3. Copy it into the local destination.
4. Mark the region copied.
5. Continue background hydration in trace-first order.

Writes always go to the local destination. The remote source is immutable. When every region is copied, switch permanently to the normal local device and remove the remote source from the runtime path.

This is a canary design, not a committed production dependency. ublk_drv, kernel version, dm-clone, Cloud Hypervisor block reconnect, and crash recovery are hard capability gates. A vhost-user-blk process is the fallback prototype if the deployed kernel cannot support ublk safely.

Do not use a plain remote NBD root disk as the production design. A dead daemon or network path must not put guest I/O into an unbounded or uninterruptible wait.

Rootfs lazy mode must fall back to eager restore if:

• The source process, device, or network cannot fail within a bounded deadline.
• Cloud Hypervisor device state or reconnect fails under active I/O.
• Archive, backup, resize, e2fsck, migration, or re-stop becomes incorrect.
• Kernel or module requirements are unsafe to operate fleet-wide.
• Steady-state workload overhead exceeds 10%.

7. Memory demand hydration

Replace the current local-file-only UFFD source with a CAS-aware source.

On a guest page fault:

1. Find the containing memory CAS chunk.
2. Resolve it through L1, L2, or Object Storage.
3. Verify and decode the chunk once.
4. Satisfy the faulting page.
5. Coalesce concurrent faults for the same chunk.
6. Prefetch nearby or recorded boot-critical pages.
7. Continue bounded background convergence.

Known zero chunks must map to zero pages without network or disk I/O.

The current 1 MiB chunk format can amplify a 4 KiB page fault. Benchmark smaller memory frames or a sub-chunk index, such as 64 to 256 KiB frames, before locking the v2 memory format.

Memory demand hydration continues only if:

• Cold first-command P99 is 2 to 5 seconds through L2 and direct Object Storage.
• Workload P95 slowdown after launch is less than 10%.
• A cache or origin failure never wedges the VM.
• Background hydration always converges or aborts safely.
• Re-stop, snapshot, and restart preserve correct memory state.

8. Regional L2 cache

The L2 is a fetch accelerator for immutable encoded CAS chunks. It does not store live rootfs, volumes, mutable manifests, archives, or authoritative state.

8.1 Cache host

Use an ordinary Linux storage server with:

• A small operating-system partition.
• One or more SATA SSD or NVMe data roots.
• A private network or approved encrypted overlay to compute hosts.
• mTLS or short-lived host tokens.
• No Cloud Hypervisor, sandbox scheduler, or compute-host registration.
• A native Go platinum-cache-agent systemd service.

Media type is an operator choice. The prior SATA test proved that SATA and NVMe both saturate a 1 Gbps link. It did not prove SATA suitability at 10 Gbps or high random concurrency. Production hardware must pass the actual full-path benchmark.

8.2 Horizontal topology

Use a separate cache_nodes registry. Hosts fetch a versioned regional topology and use weighted rendezvous hashing for direct node selection.

Start with one owner per chunk during the canary. Before multi-node production, add:

• Controlled topology-generation rollout to avoid a fleet-wide remap.
• Two-choice routing or limited replication for measured hot keys.
• Per-node and per-host concurrency, byte, and fill limits.
• Request coalescing by SHA.
• Immediate fail-open behavior to Object Storage.

The admin panel may show cache nodes near compute hosts, but cache nodes must remain a different resource type. An operator can register, activate, drain, and remove nodes. Capacity and throughput scale by adding nodes; durability does not depend on them.

8.3 Admission and eviction

Do not fill L2 for every Object Storage miss. Unique personalized dirty chunks can evict popular template and base-image data.

Version one uses second-hit admission with a bounded frequency sketch. Prefer chunks that are requested by multiple hosts or multiple resumes. Later use TinyLFU-style admission if measurements justify it.

Eviction is byte-bounded. Admit below 75% target usage, stop admission at 85%, and evict back below 75%. Exact thresholds remain configuration with safe clamps.

Predictive warm uses a separate budget and must never displace the proven reusable hot set without an explicit score comparison.

8.4 Storage engine evolution

Loose files are acceptable only for the first canary. A file per 1 MiB SHA creates an inode, directory, and metadata problem at large scale.

The cache-agent storage interface must support two engines:

type ChunkStore interface {
    Get(ctx context.Context, sha [32]byte) (io.ReadCloser, ChunkMeta, error)
    Put(ctx context.Context, sha [32]byte, encoded io.Reader, n int64) error
    Delete(ctx context.Context, sha [32]byte) error
    Stats() StoreStats
}

• LooseStore: canary engine, easy to inspect and repair.
• PackStore: append-only segment files plus an indexed SHA-to-offset map, compaction, checksums, and crash-safe index rebuild. Build it before inode or metadata limits become material.

The HTTP protocol must also permit pipelined or batched multi-chunk reads later. One HTTP request for every 1 MiB chunk is not the earth-scale endpoint.

9. Request classes and failure behavior

Use two request classes.

Critical reads

These block the first command or an active page fault.

• Coalesce by SHA.
• Use a short node deadline.
• Optionally hedge once to Object Storage after a measured percentile threshold.
• Never retry across all cache nodes.
• A failed L2 should add less than 250 ms to P99 before origin fallback.

Background reads

These hydrate or prewarm future state.

• Use normal timeouts and bounded concurrency.
• Yield to critical traffic.
• Stop during cache pressure, origin throttling, or compute-host pressure.
• Never delay a stop acknowledgement or foreground resume.

Use a shared per-node circuit breaker so 64 chunk workers do not each wait on the same dead cache. A 404 is a normal miss. Corruption opens the breaker and the host must fetch and verify the chunk from Object Storage.

10. Scheduling and placement

Placement should use reusable bytes, not only a Boolean replica flag.

For each candidate host, calculate a bounded affinity score from:

• Complete local replica availability.
• Local L1 bytes useful to the descriptor.
• Regional L2 useful-byte estimate.
• Expected origin bytes.
• Host disk pressure, active hydration work, CPU family, and ordinary capacity.

Cache affinity can break a tie. It must not place a sandbox on a pressured or incompatible host.

11. Security model

• Cache nodes listen only on a private network or approved encrypted overlay.
• Hosts use optional mTLS and short-lived Ed25519 data grants bound to the requesting host and region.
• Version one uses one purpose-bound data grant for both bounded GET and verified PUT; it does not claim separate read and fill permissions.
• Every host verifies plaintext SHA-256 before publication or use.
• Cache nodes verify fills before atomic publication.
• Limit requests, bytes, concurrency, and fills per host and tenant.
• If practical, authorize reads only for hashes reachable from an active descriptor assigned to the requesting host.
• Cache nodes receive no Object Storage credentials in version one.
• Audit register, activate, drain, remove, and topology changes.

A compromised authorized host can otherwise use known hashes to retrieve data or can churn the cache with valid verified objects. Content hashes are integrity identifiers, not access control. A future split of read and fill authority would be additional containment, not a property of the initial implementation.

12. Observability

Record these fields for every resume:

• Descriptor generation and chosen lane.
• Target-selection duration.
• L1, L2, and Object Storage request and byte counts.
• Useful byte-hit ratio, not only object-hit ratio.
• Critical rootfs and memory bytes before first command.
• Time to VM start, guest ready, first successful command, and convergence.
• Foreground and background latency separately.
• Fault count, coalescing ratio, zero-page count, and prefetch accuracy.
• Cache-node CPU, disk, network, inode, queue, eviction, admission, and error metrics.
• Origin requests and bytes avoided.

Use tracing across API, control plane, host agent, cache agent, Object Storage, UFFD, and Cloud Hypervisor. A fast chunk benchmark must never conceal a slow first command.

13. Pull-request plan

Each PR merges disabled and backward-compatible.

PR 1: establish truth

• Add the end-to-end resume timeline and first-successful-command metric.
• Add rootfs block and memory-fault trace collection.
• Label current L1, full-replica, and Object Storage lanes.
• Produce one reproducible current-production-shaped baseline.

PR 2: bound the existing full-replica path

• Account replica bytes per sandbox, host, and region.
• Replace fixed fan-out two with zero or one predicted target by default.
• Add byte quota, LRU, reuse score, region checks, and deletion on supersede.
• Keep a manual emergency purge path with audit output.

PR 3: add ResumeDescriptorV2

• Add generation-scoped immutable manifests and descriptor.
• Publish descriptor last and compare-and-swap the database pointer.
• Keep readers compatible with current snapshot references.
• Add orphan-generation collection after a safe age (follow-up after publication telemetry proves the retention window).

PR 4: add regional L2

• Implement platinum-cache-agent, registry, token flow, L2 client, breaker, admission, eviction, metrics, and operator API.
• Insert it into the eager path after L1 and before Object Storage.
• Start with LooseStore; keep the global and regional flags off.
• Run corruption, overload, node-loss, topology-change, and origin-stampede tests.

PR 5: remove avoidable eager work

• Elide known zero chunks.
• Bound asynchronous L1 and L2 fill work.
• Replace complete memory warm with measured critical-set prefetch.
• Preserve crash-safe publication and eager fallback.

PR 6: rootfs lazy canary

• Prototype ublk plus dm-clone behind capability and sandbox flags.
• Add local convergence, kill switch, and eager retry.
• Test daemon death, packet loss, cache loss, origin loss, reboot, snapshot, backup, archive, resize, migration, and re-stop.

PR 7: rootfs production hardening

• Add boot-trace prefetch, priority I/O, topology churn handling, and recovery.
• Prove Cloud Hypervisor state and device reconnect under in-flight I/O.
• Roll out to Development, Staging, then a small Production cohort.

PR 8: memory demand hydration

• Add the CAS-aware UFFD source, zero-page mapping, coalescing, and critical prefetch.
• Add background convergence and eager failover.
• Test correctness across process-heavy and memory-heavy workloads.

PR 9: scaled cache and placement

• Add hot-key replication or two-choice routing from measured skew.
• Add PackStore and batched reads when metadata or request count requires them.
• Add useful-byte placement affinity.
• Automate scale recommendations; keep provisioning an explicit operator action.

14. Acceptance and kill gates

Continue the L2 rollout only when the real Platinum path shows:

• At least 30% or 2 seconds improvement in P95 first-successful-command time for the target cohort.
• More than 60% useful byte-hit ratio for that cohort.
• More than 50% reduction in Object Storage bytes or requests.
• Failed L2 adds less than 250 ms at P99 before origin fallback.
• Zero corrupt bytes reach L1, a materialized file, or the guest.
• Stable behavior below 75% target cache occupancy.

Limit or stop the L2 rollout when:

• Local materialization and guest readiness still consume more than 70% of cold time.
• Long-stopped sandbox useful-byte hit rate is below 30%.
• Cross-provider or cross-region L2 is not faster than region-local Object Storage.
• Metadata or CPU saturates before disk or network.
• Node loss causes origin throttling or additional resume failures.
• Meeting the SLO requires cache capacity close to the durable corpus.

Continue demand hydration only when failure injection proves that loss of cache, origin, agent, or network never wedges the VM and always reaches eager fallback or a clean failure.

15. Keep the architecture boundary

Use lazy memory hydration, copy-on-write block sources, hot-page prefetch, byte-bounded local caches, parallel range reads, and streaming decompression where measurements justify them.

Do not introduce another authoritative object system for this feature. The hard work is snapshot publication, demand hydration, failure fallback, scheduling, and observability. A generic cache product does not remove those responsibilities.

Relevant primary references:

• Linux dm-clone documentation
• Linux ublk documentation

16. Final implementation order

instrument first-command latency
  -> bound and predict existing full replicas
  -> publish immutable resume descriptors
  -> add optional horizontal L2
  -> remove avoidable eager work
  -> canary converging lazy rootfs
  -> add CAS-aware UFFD memory
  -> scale topology and placement from measurements

This design gives Platinum a fast path for likely resumes, a shared regional cache for repeated state, and a bounded cold path for arbitrary old sandboxes. It does not make cache availability a durability or live-I/O dependency.