Inference router (internal)

/v1/inference serves named models through an ordered upstream chain. Kortix managed models (deepseek-v4.1-flash, glm-5.3-flash) are the first customers: the Kortix API sends every managed-model request here with one pt_live_ key.

Surface

• Auth: API key with inference:write (inference:read for /models).
• Optional header x-platinum-session: the caller's conversation id. Zen requires one and caches prompts by it. Default: <org>:<user|key>.
• The response always carries the public model id and usage.cost (USD, Platinum's price). Upstream identity (provider, upstream cost) is removed.

Chain and failover (apps/api/src/inferenceModels.ts)

Each model lists Morph direct → OpenCode Zen → OpenRouter. The router moves to the next upstream on:

• a network error;
• HTTP 401, 402, 403, 404, 408, 409, 425, 429 or any 5xx;
• an error object as the first SSE data: frame (read for at most 10 s);
• a streamed request with no response headers in 30 s (PT_INFERENCE_STREAM_HEADER_TIMEOUT_MS), on every step but the last.

Other 4xx (400 context_length_exceeded, 413, 422) return to the caller at once. When every upstream fails, the caller gets the last status with {error:{message,type,code}}. A step without its key is skipped; a model with no usable step answers 503 model_unavailable.

OpenRouter is confined to a zero-data-retention endpoint pool (only, zdr: true, data_collection: 'deny', max_price). A pool member with from is used only from that date: the end of its subprocessor notice.

Price and metering

• Price per model in µ$ per 1M tokens (price): input, cached input, output.
• billing_events rows: event_type 'inference.tokens', size classes input_mtok, cached_input_mtok, output_mtok (divisor 1M in USAGE_DIVISORS), storage_tier = public model id.
• The debit's settlement metadata carries upstream, attempts and upstreamCostMicros. Every served request also logs one line: [inference] served org=… model=… upstream=… charge_micros=… upstream_cost_micros=….
• The spend gate (402 on an empty wallet in an enforced org) runs before any upstream call. Inference POSTs are exempt from the org write rate limit and from the audit safety net.

Enable

• Global: flags.inference_enabled.
• One org: PATCH /internal/admin/org-overrides/:orgId with {"inference": true} (false turns one org off while the flag is on).
• Keys: PT_INFERENCE_MORPH_API_KEY, PT_INFERENCE_ZEN_API_KEY, PT_INFERENCE_OPENROUTER_API_KEY in the lane's dotenvx profile.

Add or change a model

Edit INFERENCE_MODELS: id, limits, price, chain. A price change is a deploy. Keep price equal to what the Kortix catalog bills (packages/llm-catalog MANAGED_MODELS[].pricing in kortix-ai/suna).

Next: own GPUs in the mix

Today every named model runs on the third-party chain (Morph direct → OpenCode Zen → OpenRouter). The GPU box path (PT_INFERENCE_UPSTREAM_URL) serves only models outside the registry.

To move a share of traffic (for example 20%) onto our own GPUs:

1. Add an upstream kind for our own endpoints to ChainStep, with the box's model id and a cost table at the GPU's real cost.
2. Give the model a weighted first step: our GPUs take the first attempt for the configured share of requests. Every other request starts at Morph. Every request keeps the full chain behind its first step, so a GPU failure fails over to the vendors.
3. Pin a conversation (x-platinum-session) to one first step, so prompt caches stay warm.

Callers change nothing: the model id, price and usage.cost stay the same. The settlement metadata (upstream, upstreamCostMicros) shows the margin per upstream.