Inference

Inference is the compute contract layer of the Agent OS. It defines the InferenceBackend trait — the interface every hardware and software inference backend must implement to serve agent-loop compute. The Life Agent OS owns this contract; hardware vendors (Apple Silicon, CUDA, Groq, Tenstorrent) compete underneath it.

The strategic positioning: the POSIX of agent silicon. POSIX did not build operating systems — it owned the interface that every OS had to implement. CUDA is NVIDIA's moat not because Nvidia builds great GPUs, but because it owns the runtime contract that every ML framework targets. Spec E is the same play applied to agent-loop compute.

Biological analogy

Inference corresponds to the motor cortex / neuromuscular junction in biological systems. The motor cortex issues movement commands through a standard electrochemical contract that every muscle fiber implements, regardless of which muscle group is being activated. The cortex does not care whether the signal goes to a bicep or a quadricep — the contract is uniform.

InferenceBackend plays the same role: the kernel (the motor cortex) issues a StepContext through a standard Rust trait. Whether the step runs on an M4 Max via MLX, a CUDA cluster via vLLM, or an in-process mock during testing, the kernel sees identical behavior. The backend is the muscle; the trait is the neuromuscular junction.

Architecture

Inference is structured as two shipped crates today, with five more planned:

The InferenceBackend trait

pub trait InferenceBackend: Send + Sync {
    /// Stable identifier — e.g. "mlx", "vllm", "groq", "tt-wormhole".
    /// Must be unique per deployment. Used in routing, metrics, and logs.
    fn backend_id(&self) -> &str;

    /// Cheap static query. Called frequently; must not block.
    fn capabilities(&self) -> &BackendCapabilities;

    /// Execute one model step. Returns a token stream that closes with
    /// `Token::Done` on success or `InferenceError::Backend { code }` on failure.
    fn step<'a>(
        &'a self,
        ctx: StepContext<'a>,
    ) -> Pin<Box<dyn Stream<Item = Result<Token, InferenceError>> + Send + 'a>>;
}

step is the only non-trivial method. It accepts a StepContext (the compiled prompt, KV cache handle, routing hint, budget constraints) and returns a token stream. Consumers iterate the stream and accumulate tokens until they see Token::Done or an error.

BackendCapabilities

pub struct BackendCapabilities {
    pub supports_tool_use: bool,
    pub supports_speculative_decoding: bool,
    pub max_context_tokens: u32,
    pub supports_streaming: bool,
}

The default implementation of supports_speculative_decoding panics with a clear message. This is a deliberate lock: L5-D3 in the Spec E locked decisions. The #[should_panic] test in inprocess_smoke anchors this contract.

CloseCode

The close codes are distinct from lifegw §6.5 wire codes numerically, though they represent the same semantic categories:

ToolAwait — the reconnect pattern

ToolAwait is the most important close code for agent-loop compute. When a model produces a tool call, the backend cannot continue inference until the tool result is known. Rather than blocking, the backend closes the stream with CloseCode::ToolAwait and includes the tool call in the final InferenceError. The caller:

This pattern means inference backends never need to know about Praxis, MCP, or any tool implementation. The agent loop in Arcan orchestrates the reconnect. The backend is stateless between reconnects.

Routing

InferenceRouter distributes requests across multiple registered backends:

let mut router = InferenceRouter::new();
router.register(InProcessInferenceBackend::new("mock"));
// future: router.register(MlxBackend::new(mlx_config));
// future: router.register(VllmBackend::new(vllm_config));

let stream = router.step(ctx).await?;

Routing decisions use RoutingHint (caller preference) and WorkloadClass (the classification of the current request — interactive, batch, speculative). The router selects the backend that best matches the workload class given current backend capabilities and load.

KV cache

InMemoryKvCache is the E-Sub-A reference implementation. It stores KV entries keyed by (session_id, backend_id) with a configurable capacity and LRU eviction. When eviction occurs, the backend emits CloseCode::KvEvicted on the next step request for that session.

inference-autonomic (E-Sub-E) will replace InMemoryKvCache with a Lago-backed durable cache. Anima rotation events become KV invalidation signals — when an agent rotates its identity, its cached KV state is automatically purged.

Locked decisions (E-Sub-A)

Test coverage

At E-Sub-A ship: 24 tests — 20 unit tests for core types and routing logic, 1 conformance scaffold test, and 3 inprocess_smoke tests including the #[should_panic] lock for L5-D3 (speculative decoding default-impl panic contract). All 14 CI checks passed: Test Linux (11m), Test macOS (13m), Build Release (24m), Lint (4m), MSRV (4m), Security Audit (3m), plus 8 fast-path checks.

Relation to other subsystems