Together AI Brings Continuous Compilation to Production Inference

Together AI announced Polymorphic Inference on November 4, 2026, introducing Together Compiler as its first implementation. The agent-driven system continuously retunes inference for changing models, traffic and hardware, and already runs across many of the company’s top production inference workloads and some popular serverless endpoints.

Together AI reports that the compiler delivered 1.3x to 3x the tokens per second of highly optimized open-source baselines in under 24 hours across tests involving Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash. Those results frame the announcement’s central proposition: optimization can be an ongoing search process tied to a deployment, rather than a round of manual tuning frozen at launch.

Changing Models, Traffic and Hardware

Polymorphic Inference is Together AI’s term for serving infrastructure that adapts to faster model releases, diverse architectures, evolving traffic shapes and service-level requirements, and more frequent hardware changes. The contrasting static approach starts from a fixed engine configuration. A configuration tuned for one setting is not necessarily suitable for another, particularly when model behavior and request distributions change.

The company’s production observations supply concrete examples. Together AI reports that prompts during one country’s working hours reached about 97,000 tokens, compared with 70,000 elsewhere, and that prompt lengths rose 80% in the two weeks after one model release as usage shifted toward agentic workloads. Customer bursts can also double prompt lengths and reduce cache hit rates, changing the conditions under which an existing configuration was tuned.

The announcement also revises Together AI’s earlier emphasis on megakernels, which fuse operations into a single kernel. After putting early versions into production, the company concluded that fusion alone is not a universal design goal: mixture-of-experts architectures and dynamic workloads can require different techniques. Kernels developed around fixed or random traffic can likewise be miscalibrated for production traffic whose data-dependent routing affects which optimizations win.

How Continuous Compilation Works

Together Compiler takes the traffic shape, model and hardware as inputs, then uses agents to generate and measure candidate optimizations for an inference engine. A persistent engine loads the model once and hot-reloads candidate forward passes without restarting the full engine. Parallel agents generate candidates and communicate through a message board, keeping experiments queued while other agents are reasoning or using tools.

Two measurement tiers balance experiment volume against realism. Fast engines load a subset of layers for rapid iteration and low-variance timings; slower engines load more layers to test whether candidates hold up on realistic data distributions. A separate cost model combines hardware specifications, microbenchmarks and measured forward passes to prune the parallelism-configuration search before agents spend time optimizing candidates.

The technical paper says winning candidates undergo verification tests for KV-cache contamination and output quality before they can be served. The checks run after the optimization budget is exhausted, when the final source code is prepared for integration into the serving engine.

The launch post’s startup comparison reports 467 seconds to serving for its status-quo sequence, versus 9.0 seconds with Together Compiler’s setup. Those figures cover engine startup during optimization rather than individual request latency, and help explain why hot-reloading is central to the system’s experiment throughput.

Evaluation Scope and Customer Access

For the engine comparisons, the paper used leading open configurations as of October 30, 2026, and kept the baseline parallelism strategy fixed when isolating the effect of agentic optimization. Trainium2 was the disclosed exception to the standard evaluation workload: it used a synthetic multi-turn dataset because the implementation did not yet support higher concurrencies within the available HBM capacity.

Together AI attributes the optimization results partly to 252-fold higher experiment throughput than baseline agentic recursive-self-improvement optimization systems.

Beyond language-model inference, the paper reports 1.5x to 6.4x speedups on tested weather, biology, time-series and partial-differential-equation models while maintaining bit-exact outputs on the stated evaluations.

End customers can request early access through a limited beta. Together AI plans one-click optimization of dedicated endpoints, while its research roadmap includes speculative decoding that adapts by domain, request, token and traffic load, plus dynamic routing across workers and deployment configurations for heterogeneous serverless traffic.

The launch makes deployment-specific measurement, rather than a universally optimal kernel, the organizing principle of Together AI’s inference stack. The immediate question for customers is whether the limited beta can reproduce the reported gains on their own models, traffic and hardware while passing the same quality checks—the variation that Continuous Compilation is designed to address.