Regional L2 CAS cache component specification

Status: Draft for implementation

Scope: Optional regional cache between compute-host storage and Object Storage

Durability rule: Object Storage remains authoritative

Delivery model: A disabled-by-default PR stack, followed by measured rollout

This document specifies the L2 component. It is not the complete instant-resume design. See Instant cold resume architecture for the bounded full-replica, demand-hydration, and eager-fallback plan.

1. Decision

Platinum will add a horizontally scalable regional L2 cache for immutable CAS chunks.

The read path will be:

compute-host L1 cache -> regional L2 cache -> Object Storage

The L2 layer will improve cold-host and cross-host chunk fetches. It will not replace Object Storage. It will not provide live sandbox disks.

It also will not make an arbitrary long-stopped sandbox resume instantly by itself. The current host still materializes the complete rootfs and memory files before VM launch. The product metric is API resume to first successful sandbox.exec, not only L2 fetch or file-materialization latency.

One region can have zero, one, or many cache nodes. A region with no healthy cache node will use the current path without an error.

The first version will use a small Platinum Go service. It will not deploy Alluxio, SeaweedFS, Ceph, or CacheLib.

2. Goals

The feature must:

• Reduce latency and variance for repeated remote CAS reads.
• Keep Object Storage as the durable cross-host source.
• Work in selected providers and regions only.
• Scale capacity and throughput by adding cache nodes.
• Fail open to Object Storage when a node is slow, full, corrupt, or unavailable.
• Store only immutable data that a host can verify by SHA-256.
• Expose capacity, health, hit rate, latency, and cost to operators.
• Avoid scheduling sandboxes on cache nodes.
• Support SATA SSD and NVMe hardware without changing the protocol.
• Reduce time to the first successful command for the selected resume cohort.

3. Non-goals

The first version will not:

• Store a running sandbox rootfs or persistent volume.
• Mount a remote block device into a running sandbox.
• Replace Object Storage for durability.
• Store mutable manifests, archives, backups, or build contexts.
• Guarantee that a cached chunk survives a node or disk failure.
• Rebalance existing cache data when the topology changes.
• Provide cross-region cache reads.
• Purchase or provision provider resources through Platinum.
• Change sandbox disk quotas or fix guest rootfs pressure.
• Guarantee instant resume without selective replicas or demand hydration.

4. Current behavior

Today, Materializer.fetchChunk in materialize.go uses this order:

1. Read the chunk from the compute host's local CAS directory.
2. On a miss, fetch the chunk from Object Storage.
3. Decode the object and verify its SHA-256.
4. Store the verified plaintext chunk in the local CAS.
5. Write or clone the chunk into the materialized file.

CAS chunks are fixed at 1 MiB. A manifest contains the plaintext SHA-256 for each chunk. This makes remote cache responses independently verifiable.

The local cache already has pressure-triggered eviction in disk_reclaim.go. That eviction can remove cold, referenced chunks because Object Storage remains authoritative.

Manifest freshness currently comes from Object Storage. The L2 layer must not alter that behavior in the first version.

5. Resulting storage model

For a stopped sandbox, the durable snapshot still goes to Object Storage. A later resume checks L1, then L2, then Object Storage. A successful Object Storage read becomes an asynchronous L2 fill candidate.

Template and stopped-snapshot producers also warm L2. Each distinct chunk becomes a candidate only after Object Storage confirms that immutable chunk. The producer waits for the bounded cache attempt before publishing the authoritative manifest. A cache failure is recorded and ignored, so durability and publication still depend only on Object Storage. This makes a fresh snapshot eligible for an L2 hit on its first cross-host resume; admission, capacity, and topology still decide whether it hits.

6. Architecture

There is no cache-to-Object-Storage path. Cache nodes will not receive Object Storage credentials in version one.

The host owns the read-through policy:

1. Try L1.
2. Select one L2 owner for the chunk.
3. Try L2 once.
4. On any safe bypass condition, use the existing Object Storage path.
5. Decode the remote object and verify the plaintext SHA-256.
6. Publish the chunk into L1.
7. If Object Storage supplied the chunk, enqueue an asynchronous L2 fill.

The producer path is separate:

1. Hash and encode a local chunk.
2. Confirm or write the immutable chunk in Object Storage.
3. Attempt one bounded synchronous PUT to the selected L2 owner.
4. Suppress further producer fills to a failing owner for 30 seconds.
5. Publish the manifest after all authoritative chunks and bounded fill attempts finish.

Producer PUT failures never open the read breaker and never fail the upload.

This design limits the cache node's authority. A compromised cache node can serve bad or stale bytes, but the host rejects them by hash.

7. Cache-node topology

7.1 Separate node type

Cache nodes must not use the existing hosts table. That table represents compute capacity and participates in scheduling and reconciliation.

Add a separate cache_nodes table and a separate cache-agent registration flow. The admin application can show cache nodes beside compute hosts, but the data models must remain distinct.

7.2 Regional ownership

The control plane returns only ready cache nodes in the compute host's region. No cache node means the feature is inactive for that host.

Each cache node has a stable node ID and configured capacity weight. Hosts use weighted rendezvous hashing over:

chunk SHA-256 + cache node ID

The configured weight must remain stable. Live free space must not change the hash ring because that would remap the hot set continuously.

The canary gives one chunk one selected L2 owner. A miss goes directly to Object Storage. Hosts do not probe every cache node.

Before a multi-node production rollout, add controlled topology-generation rollout and measured hot-key handling. A hot chunk can use limited replication or two-choice routing. Do not replicate the complete cache or turn it into a durability tier.

Adding or removing a node changes ownership for part of the key space. Platinum will accept the resulting cold misses during the canary. Later roll topology generations out gradually so all compute hosts do not remap and stampede Object Storage at once. It will not synchronously copy old cache data during a topology change.

A draining node is removed from new topology generations immediately. It continues to serve reads from hosts holding an older generation, rejects new fills, and exits only after the 60-second topology TTL plus its in-flight requests have elapsed.

7.3 Horizontal scaling

Operators add a cache node when one or more conditions remain true:

• Regional cache occupancy stays above 75%.
• Eviction removes useful hot data.
• Cache hit rate falls because the working set does not fit.
• Network or disk utilization reaches the tested safe limit.
• P95 cache latency breaches the regional target.

A single node is a valid starting point. Two nodes improve capacity and aggregate throughput. They are not required for durability because Object Storage is the fallback.

8. Data-plane protocol

The cache service will expose a versioned HTTPS API on the provider-private network or an approved private overlay.

8.1 Fetch

GET /v1/chunks/{sha256}
Authorization: Bearer <short-lived-host-token>

Responses:

• 200 application/octet-stream: exactly one raw-v1 or zstd-v2 CAS object.
• 404: normal cache miss.
• 429: overloaded; bypass the node briefly.
• 503: unavailable; bypass the node.

The host must limit the body, decode it, then verify plaintext length and SHA-256.

8.2 Fill

PUT /v1/chunks/{sha256}
Authorization: Bearer <short-lived-host-token>
Content-Type: application/octet-stream

The host sends the encoded object bytes received from Object Storage. This preserves the current raw-v1 or zstd-v2 wire format and avoids expanding compressed chunks on L2 storage and the network.

The cache service must:

• Accept at most one CAS chunk per request.
• Reject a body larger than ChunkSize + 64 KiB.
• Decode with a 1 MiB plaintext limit.
• Verify the decoded plaintext SHA-256 against the path.
• Write to a temporary file in the selected cache root.
• Publish with an atomic rename.
• Return 201 for a new object and 204 for an existing object.
• Return 507 when admission is disabled by capacity pressure.

Fills run through a host queue bounded by both object count and encoded bytes. The initial byte limit is 64 MiB with four workers. A full queue drops the fill. It must never delay sandbox materialization.

Version one uses second-hit admission with a bounded frequency sketch. A chunk fetched only once is not admitted unless predictive warm explicitly selects it. This prevents one-use personalized state from evicting shared templates and base-image chunks.

8.3 Corruption report

POST /v1/chunks/{sha256}/invalidate
Authorization: Bearer <short-lived-host-token>

The service must decode and rehash the local object before deletion. It must not trust a host to delete an arbitrary valid chunk.

8.4 Service endpoints

The service also exposes:

• GET /healthz for process health.
• GET /readyz for admission and storage readiness.
• GET /metrics for Prometheus metrics.

9. Cache storage engine

The first canary will use ordinary files. It stores the encoded CAS object received from Object Storage. This matches the existing wire format and keeps the binary simple.

Default layout:

/var/platinum/cache/chunks/<first-two-hash-characters>/<full-sha256>

The service can accept multiple cache roots. It will map each hash to one root with rendezvous hashing. A disk failure will therefore remove only that disk's cache shard.

RAID is not required for cache durability. RAID 1 is permitted for operational simplicity, but it halves usable cache capacity. RAID 0 should not be required by the design.

9.1 Capacity control

Every node must have an explicit maximum cache byte value. Filesystem capacity alone is not a safe limit.

The default control law is:

• Stop admitting fills at 85% of the configured maximum.
• Evict until usage falls below 75%.
• Continue serving verified hits during eviction.
• Keep filesystem reserve outside the configured maximum.

The service will update a chunk's modification time at most once every five minutes after a successful hit. Eviction will remove the oldest sampled files first. This provides an approximate LRU without a separate database or full ordering scan.

At startup, the service will rebuild its byte counter with a directory walk. It can serve existing hits during the walk, but it must reject fills until the count is complete.

An open file remains readable after unlink. Eviction must still coordinate with in-flight fills to avoid publishing an object that was just selected for deletion.

The service does not need to fsync cache contents. A crash can discard a recent fill because the host verifies reads and Object Storage remains authoritative.

Loose files are not the final scale engine. A file for every 1 MiB SHA creates inode, directory, and metadata overhead at high object counts. Define a ChunkStore interface in the first implementation. Add an indexed append-only pack/segment engine before metadata or inode use becomes material. The protocol must also permit batched or pipelined multi-chunk reads later.

9.2 Media independence

The service must not contain SATA- or NVMe-specific logic. Hardware is an operator choice that must pass the same acceptance benchmark.

The prior 1 Gbps SATA test reached about 112 MiB/s and was network-limited. It does not prove 10 Gbps SATA behavior. Production sizing must use an end-to-end result on the selected network and media.

10. Failure behavior

L2 failure must increase Object Storage load, not sandbox failure rate.

10.1 Host timeout policy

Initial limits implemented by the host cache client:

• Foreground read timeout: 200 ms total.
• Slow-response threshold: 100 ms.
• Active reads per cache node and compute-host client: 8. Queued reads remain inside the materializer's outer one-second L2 budget.
• Background fill timeout: 2 seconds.
• L2 retries per chunk: zero.
• Breaker open duration: 30 seconds.
• Half-open requests: one.

The per-node read limit is sized so eight concurrent 1 MiB chunks fit inside the slow-response budget on the minimum 1 Gbps node class. Without that admission gate, a normal 64-worker materialization burst can manufacture a slow-response streak and eject an otherwise healthy SATA node. These values are starting limits. The rollout benchmark can make them smaller. Environment overrides must have safe minimums and maximums.

Critical reads that block the first command or an active memory fault need a separate deadline. Failed L2 should add less than 250 ms at P99 before direct Object Storage fallback. A two-second L2 wait is not acceptable on an instant-resume path.

10.2 Circuit breaker

The breaker is per cache node, not per chunk.

It opens when:

• Three transport failures occur inside two seconds.
• A chunk GET returns 429 or 503 during a materialization burst.
• A host receives a corrupt chunk.

A 404 is a normal miss and does not open the breaker.

A fill response of 429 or 507 suppresses fills for 30 seconds. It does not open the read breaker because the node can still serve existing chunks.

When the breaker is open, every worker skips L2 and uses Object Storage. This avoids 64 concurrent chunk workers waiting on the same dead cache node.

10.3 Failure matrix

11. Control-plane changes

11.1 Database

Add cache_nodes with these fields:

• id, name, region, and optional zone.
• endpoint and protocol_version.
• state: registering, ready, draining, unhealthy, or gone.
• capacity_bytes, max_bytes, and used_bytes.
• capacity_weight and optional network_mbps.
• provider, provider_instance_id, provider_region, and provider_sku.
• price_per_hour_micros_usd and the original provider quote in metadata.
• agent_version, last_heartbeat, and metadata.
• Creation and update timestamps.

Add cache_node_bootstrap_tokens with the existing single-use bootstrap pattern. Use a distinct ctok_ prefix. Extend API-key identity with a structural cache_node_id field and cache-node scopes. The API-key binding must remain usable after a cache-node row is removed, so no foreign key may null that identity field.

11.2 Internal API

Add:

• POST /internal/cache-nodes/register.
• POST /internal/cache-nodes/:id/heartbeat.

The heartbeat reports health, disk usage, request load, eviction state, and version. Use a five-second heartbeat and consider a node unhealthy after 30 seconds without a successful heartbeat. The control plane computes readiness from heartbeat freshness and explicit state.

Constrain cache-node credentials with an explicit allowlist. A bootstrap token can only register a node. A registered cache-node key can only heartbeat for its own node ID. It cannot call compute-host, admin, or sandbox routes.

11.3 Host discovery

Extend the existing compute-host heartbeat exchange. The control plane builds the regional endpoint set from ready nodes with a heartbeat no older than 30 seconds and caches that endpoint calculation for five seconds. It returns a fresh approximately 45-second Ed25519 data grant on every host heartbeat. The grant is stored separately from endpoint selection and is bound to the authoritative host ID and region.

The host never uses an expired grant. An empty topology clears L2 immediately; malformed updates cannot replace the last valid configuration, whose grant still provides a hard expiry. Any L2 error or unavailable topology bypasses directly to authoritative Object Storage.

Old host agents ignore the new response field. This permits an expand-first rollout.

The compute-host heartbeat response uses an additive field shaped like this:

{
  "regionalCacheConfig": {
    "version": 1,
    "region": "fr-par",
    "endpoints": [
      {
        "url": "https://10.0.4.12:7443",
        "weight": 100
      }
    ],
    "grant": "v1.<kid>.<payload>.<signature>",
    "grantExpiresAt": "2026-08-12T12:00:45Z"
  }
}

11.4 Admin API

Add:

• GET /v1/admin/cache-nodes.
• POST /v1/admin/cache-nodes/bootstrap-tokens.
• GET /v1/admin/cache-nodes/bootstrap-tokens.
• DELETE /v1/admin/cache-nodes/bootstrap-tokens/:id.
• PATCH /v1/admin/cache-nodes/:id for label, weight, ready, or draining only.
• POST /v1/admin/cache-nodes/:id/disable.
• DELETE /v1/admin/cache-nodes/:id.

These endpoints manage Platinum registration only. They do not purchase, delete, or resize provider resources.

12. Authentication and security

Enrollment consumes a one-time ctok and returns a distinct node-scoped control-plane API key. That node key authenticates heartbeat only; it is never a data-plane bearer.

Compute hosts use a control-plane-signed Ed25519 grant with canonical wire form v1.<kid>.<payload>.<signature>. The signed payload contains the fixed purpose platinum.regional-cache.data.v1, authoritative compute-host ID, region, issue time, and expiry. Grants target approximately 45 seconds and the verifier refuses a signed lifetime above 60 seconds. Version one authorizes bounded chunk GET and verified PUT with the same data purpose; there is no invalidation operation.

The control plane holds the signing seed. Cache nodes receive only the active and previous public keys, allowing bounded rotation overlap. Authenticated heartbeats rotate that public keyring and persist it atomically under the systemd StateDirectory.

Security requirements:

• Bind the data API to the approved private network or overlay.
• Require TLS in staging and production.
• Never place Object Storage credentials on a cache node.
• Accept only lowercase, 64-character hexadecimal chunk paths.
• Limit request body size before allocation.
• Rate-limit by host token and node capacity.
• Never expose arbitrary filesystem paths or Object Storage keys.
• Bound compressed input and decoded output to prevent decompression bombs.
• Decode and verify every remote chunk on the compute host.
• Apply per-host and per-tenant request, byte, concurrency, and fill quotas.
• Where practical, authorize hashes reachable from an active resume descriptor assigned to the requesting host; a SHA is not an authorization boundary.
• Log registration, activation, drain, and removal events.
• Never log tokens or provider credentials.

13. Host-agent changes

Create hosts/host-agent/internal/regionalcache with:

• Topology parsing and expiry.
• Weighted rendezvous selection.
• An HTTP client with fixed safe limits.
• A per-node circuit breaker.
• Bounded asynchronous fill workers.
• Metrics and structured outcomes.

Modify Materializer.fetchChunk only after the L1 miss. Preserve the current Object Storage retry and quarantine-recovery logic.

Refactor the existing CAS codec into one bounded DecodeAndVerifyObject helper. Both Object Storage and L2 responses must use it. On an Object Storage hit, retain a defensive copy of the encoded object for the asynchronous L2 fill before fetchChunk returns the verified plaintext chunk.

The result type must identify these sources:

local | l2 | s3 | quarantine

Add per-chunk singleflight before remote fetch. The current materialization singleflight is per final file, so two unrelated materializations can still request the same missing hash.

Manifest reads and ETag checks must continue to use Object Storage.

Quarantine recovery must continue to report the live Object Storage sweep error. It must not fill L2 because doing so would hide a missing live CAS object.

14. Cache-agent binary

Add hosts/host-agent/cmd/cache-agent. Build it as platinum-cache-agent from the existing Go module.

The installer performs the one-time registration. The binary owns:

• Authenticated heartbeat and serving-policy/keyring refresh.
• The chunk data API.
• Atomic publication and hash verification.
• Bounded raw-v1 and zstd-v2 decoding through a shared CAS helper.
• Admission, eviction, and optional scrub.
• Data-plane metrics.
• Graceful drain and shutdown.

Essential runtime configuration after one-time enrollment:

PT_CACHE_LISTEN
PT_CACHE_ROOT
PT_CACHE_CAPACITY_BYTES
PT_CACHE_CONTROL_PLANE_URL
PT_CACHE_NODE_ID
PT_CACHE_CONTROL_PLANE_API_KEY_FILE
PT_CACHE_GRANT_PUBLIC_KEYRING_FILE
PT_CACHE_REGION
PT_CACHE_TLS_CERT_FILE
PT_CACHE_TLS_KEY_FILE

The one-time bootstrap token is installer input, not runtime configuration. Pass it with --token-stdin or a root-owned mode-0600 regular file. Never export it, place it in an environment file or command argument, persist it after enrollment, or reuse it as the node heartbeat key or a data-plane bearer.

15. Feature control

The host-side feature is dark by default and requires:

PT_REGIONAL_CACHE_ENABLED=1

Disabling the flag bypasses L2. Control-plane operators can remove a region from service by draining or disabling its nodes; a region with no ready, fresh node receives an empty topology and hosts immediately bypass L2.

Canary enablement must support an explicit host-ID allowlist before region-wide use.

16. Observability

16.1 Host metrics

Add:

• pt_cas_chunk_fetch_total{source,outcome}.
• pt_cas_chunk_fetch_duration_seconds{source}.
• pt_cas_chunk_fetch_bytes_total{source}.
• pt_l2_breaker_state{node}.
• pt_l2_fill_total{outcome}.
• pt_l2_fill_queue_depth.
• pt_l2_corruption_total{node}.

Materialization logs add L2 hit, miss, bypass, error, bytes, and wait duration. They keep the existing local and Object Storage counters.

16.2 Cache-node metrics

Add:

• Used, maximum, and filesystem-free bytes.
• Objects stored and evicted.
• Request count by method and status.
• Read and fill latency histograms.
• Encoded and decoded bytes served and accepted.
• Hash-validation failures.
• Admission and eviction state.
• Active requests and rate-limit events.

16.3 Operator view

Add a focused Cache Nodes panel to the existing admin application. Follow the current lifted-admin style; do not refactor the whole admin application.

Show:

• Node state, region, endpoint, version, and last heartbeat.
• Used and maximum bytes.
• Hit rate, request rate, throughput, and P95 latency.
• Eviction and corruption counts.
• Monthly estimated node cost.
• Activate, drain, and remove actions.

17. Technology decision

The first version will use a Platinum Go service because the storage contract is small: immutable 1 MiB objects addressed by SHA-256.

Open-source software does not remove hardware or operational cost. It only changes the software implementation cost.

18. Component PR stack

Do not deliver this as one large PR. Merge each PR disabled and backward-compatible. The complete order, including replica control and demand hydration, is in Instant cold resume architecture.

PR 1: Specification

• Add this document.
• Confirm interfaces and acceptance gates.

PR 2: Cache-agent service

• Add cache protocol, ChunkStore, LooseStore, validation, second-hit admission, eviction, and metrics.
• Add unit and local integration tests.
• Do not connect production hosts.

PR 3: Control-plane registry and authentication

• Add migrations, schema, internal API, admin API, topology, and signed tokens.
• Add audit events and config defaults.
• Keep the global kill switch off.

PR 4: Host-agent L2 client

• Add discovery, routing, timeouts, breaker, singleflight, and asynchronous fill.
• Insert L2 after L1 and before the existing Object Storage path.
• Preserve manifest, Object Storage, and quarantine behavior.
• Keep the feature off.

PR 5: Admin operator surface

• Add the Cache Nodes panel.
• Add registration, activate, drain, and remove flows.
• Add accessibility and responsive checks.

PR 6: Deployment and benchmark assets

• Add systemd units, health checks, and provider-neutral setup guidance.
• Add end-to-end materialization and first-successful-command benchmarks.
• Add a fault-injection runbook.
• Do not hardcode one provider or region.

19. Tests

19.1 Unit tests

• Hash-path validation and request-size limits.
• Atomic publish under duplicate concurrent fills.
• Multi-root hash placement.
• High-water admission and low-water eviction.
• Weighted rendezvous stability and expected remapping.
• Token scope, region, expiry, and key rotation.
• Circuit-breaker state transitions.
• Topology expiry and disabled behavior.

19.2 Integration tests

• L1 hit makes no L2 or Object Storage request.
• L2 hit makes no Object Storage chunk request.
• L2 miss fetches Object Storage, verifies bytes, fills L1, and queues the encoded object for L2.
• L2 corruption is rejected and repaired from Object Storage.
• Connection refusal falls back within the latency budget.
• Silent packet drop falls back within the latency budget.
• Full L2 serves hits and rejects fills without blocking materialization.
• A topology change does not fail an in-flight materialization.
• Feature disabled produces the existing behavior and metrics.
• Quarantined Object Storage recovery still works.

19.3 End-to-end tests

• Materialize a real CAS manifest through the actual host-agent path.
• Resume a stopped sandbox on a different compute host.
• Remove the selected L2 node during a materialization.
• Serve a deliberately corrupt chunk from L2.
• Fill a node past its high watermark.
• Add and drain a second node.
• Restart a cache node during load.

20. Benchmark and rollout gates

Prior lab work used the cache as an Object Storage origin. It did not exercise a deployed L1 to L2 to Object Storage path. The feature cannot rely on those numbers as production proof.

The end-to-end benchmark must report:

• API resume to first successful sandbox.exec.
• L1, L2, and Object Storage bytes and hit counts.
• Useful byte-hit ratio for the target resume cohort.
• Complete materialization time.
• P50, P95, and P99 chunk latency.
• Aggregate throughput at 1, 16, 64, and 128 workers.
• Host CPU, cache-node CPU, disk utilization, and network utilization.
• Object Storage fallback rate.
• Object Storage bytes and requests avoided.
• Performance during refused connections, packet drops, overload, and corruption.

Required rollout gates:

• Disabled mode matches the current path.
• L2 failure does not exceed the agreed fallback latency budget.
• Every corrupt response is rejected before publication or materialization.
• A cache hit provides a material user-visible or infrastructure benefit.
• The selected SATA or NVMe configuration passes sustained concurrency tests.
• The node can operate below 75% target occupancy without constant eviction.
• Operator metrics distinguish L1, L2, and Object Storage work.
• P95 first-command time improves by at least 30% or 2 seconds for the target cohort.
• Useful byte-hit ratio exceeds 60% and origin bytes or requests fall by at least 50%.
• A failed L2 adds less than 250 ms at P99 before Object Storage fallback.

Limit or stop the rollout if local materialization and guest readiness still consume more than 70% of cold time, if long-stopped useful-byte hit rate is below 30%, or if the cache must approach the durable corpus size to meet its SLO.

21. Rollout

Use the normal release order:

feature branch -> main/Development -> staging -> prod

Rollout stages:

1. Merge all code with the feature disabled.
2. Register one Development cache node.
3. Enable one Development compute host.
4. Run correctness, load, and fault tests.
5. Enable all Development hosts in one region.
6. Repeat with one Staging cache node and a canary compute host.
7. Run a controlled production benchmark on one canary host.
8. Enable one production region only if the gates pass.
9. Add a second cache node only when capacity or throughput data requires it.

Rollback is the kill switch. Hosts then return to L1 and Object Storage. Cache-node contents can remain until later cleanup because they are not authoritative.

22. Cost and sizing

The cache duplicates a hot subset of Object Storage. It does not automatically lower the Object Storage bill.

Use encoded hot-object bytes because L2 preserves the Object Storage wire format. Do not size the node from allocated sandbox disk.

For each region:

required_nodes = ceil(regional_hot_encoded_bytes / target_usable_bytes_per_node)

monthly_cache_cost = required_nodes * monthly_cost_per_node

infrastructure_overhead_percent =
  100 * monthly_cache_cost / current_monthly_compute_and_storage_cost

The operator view must use live configured node cost. It must not embed a stale vendor price.

Do not increase customer storage pricing from modeled cache capacity alone. Measure regional use and decide whether the cache is platform overhead or a paid performance tier.

23. Acceptance criteria

The feature is complete when:

• A region can run with zero, one, or many cache nodes.
• Cache nodes cannot receive sandbox placements.
• Only immutable encoded CAS objects enter L2.
• Object Storage remains authoritative for writes and manifests.
• Hosts verify every L2 response by SHA-256.
• Slow, dead, full, and corrupt nodes fall back safely.
• L2 fills never block materialization.
• Adding or draining a node needs no sandbox downtime.
• Operators can see capacity, hit rate, latency, failures, and cost.
• Disabled mode preserves the existing behavior.
• End-to-end benchmarks prove the selected hardware and network path.

24. Explicit first implementation choice

Start with one modest cache node in one Development region. Use the selected storage media only after the full-path benchmark.

Build horizontal scaling into the protocol and registry from the first PR. Do not buy a large node merely to avoid implementing topology, eviction, and observability.

The architecture is therefore:

bounded compute-host L1
    -> optional horizontally scalable regional L2
    -> authoritative Object Storage

This component reduces origin load and cold-fetch variance. Selective bounded full replicas and demand-hydrated rootfs and memory provide the instant-resume behavior.