Baidu Releases ERNIE 5.5 Flash Base for Further Training

Baidu’s ERNIE Team announced ERNIE-5.5-Flash-Base on November 5, 2026, releasing downloadable weights for a text-only mixture-of-experts model with 280 billion total parameters and 14 billion activated per token. The checkpoint ends pre-training, before mid-training or post-training, giving developers a common starting point for domain adaptation, instruction tuning and reinforcement-learning research.

The release also provides a fixed baseline for controlled comparisons of training methods. Baidu said a post-trained ERNIE 5.5 Flash model will follow this base version.

Architecture and Training System

The model card lists 43 layers, a hidden size of 4,096 and 64 attention heads. Thirty-seven layers use Heavily Compressed Attention: their cache holds one entry for each 128-token block, plus a short window of recent uncompressed tokens. Six Multi-head Latent Attention layers preserve token-level access to the full context and use an indexer to choose the highest-scoring earlier positions. Baidu says this sparse selection keeps per-query core-attention cost constant as context grows.

All layers except the first use a mixture-of-experts block with 512 routed experts, of which 10 are activated, plus one shared expert. The experts operate in a 2,048-dimensional latent space rather than the model’s full hidden width. Baidu says that roughly halves expert dispatch and combination traffic and expert FLOPs relative to full-width experts. The design also includes four parallel residual streams through manifold-constrained hyper-connections and a MultiMax output head with a learnable piecewise temperature.

Pre-training used 33.5 trillion tokens, the Muon optimizer and FP8 mixed precision. The context used during training was gradually extended to 128K tokens; fine-tuning and inference support 1 million tokens. Alongside multilingual web data, the corpus includes code and records of software development such as commits, issues, pull requests, reviews, generated tests and CI-verified fixes. Curated long documents, sequences with semantically similar distractors, and synthetic retrieval and question-answering data supply long-context training. Those choices align the base release with repository-level reasoning, agentic software engineering and extended inputs.

The accompanying FlashMask repository documents attention kernels for Hopper and Blackwell GPUs, with Blackwell configurations supporting head dimensions up to 576 for query and key and 512 for value. Baidu reports speed gains of up to 175% in the forward pass and 74% in the backward pass over FlashAttention-4 MaskMod on SM100 hardware. The distributed Muon system also combines dense and MoE-parameter synchronization into one collective operation and spreads optimizer state more evenly across nodes, without changing the optimizer’s updates.

Baidu’s Benchmark Results

Baidu evaluated ERNIE against the publicly released DeepSeek-V4-Flash-Base, Hy3-preview-Base and MiMo-V2.5-Base checkpoints with the same harness, prompts and few-shot examples, and the same LLM judge for free-form question answering. Selected results from its table show where the model leads and where it trails:

Baidu’s comparison is task-dependent rather than a clean sweep: ERNIE leads the MMLU and HumanEval+ results shown, while Hy3 leads FullStackBench and DeepSeek leads InfiniteBench. The table supports strong exam and coding showings within Baidu’s evaluation while preserving those exceptions.

Deployment and License Conditions

Downloads are available in FP8 and BF16 safetensors formats. The model card documents production serving with SGLang and vLLM through the completions API, matching the checkpoint’s base-model status. NVIDIA’s Megatron Bridge recipe adds a supervised fine-tuning workflow covering data preparation, training and export back to Hugging Face format. Its full-model recipe uses 16 nodes with four GB300 GPUs per node. NVIDIA validates the recipes on Blackwell GB200 and GB300 systems; the components support Hopper, but Hopper is not validated.

The custom ERNIE 5.5 Flash Base Model License Agreement v1.0 imposes territorial and commercial conditions. Its territory excludes the European Union, United Kingdom and South Korea, and activities including use, modification and distribution outside that territory are not authorized. Non-commercial use is free of model license charges. Commercial use at or below $50 million in annual revenue from related products or services requires a one-time notice and prominent “Powered by ERNIE” attribution. Above that threshold, users need Baidu’s separate prior written authorization.

The agreement also bars use of the licensed materials, derivative models or their outputs to improve an unrelated AI or machine-learning system. That restriction matters for distillation and synthetic-data workflows, which the license expressly addresses when defining derivative models and limits on output use.

ERNIE 5.5 Flash Base gives researchers and developers a fixed checkpoint for testing later training methods without repeating Baidu’s pre-training run. Its design concentrates on controlling active computation and long-context cost, but the release’s practical reach is bounded by the hardware needed for full-model fine-tuning and a license that restricts geography, larger-scale commercial use and transfers of capability to unrelated models.