Kimi K2
Training a trillion-parameter model without instability, and teaching it to use tools
Kimi K2, released by the Beijing company Moonshot AI in July 2025, has about one trillion parameters, of which 32 billion are used for each token. It was built to act as well as answer: to call tools, read their results and decide what to do next. Two problems stood in the way. Training a model this large tends to become numerically unstable, and good examples of tool use are scarce. Moonshot addressed the first with a new optimizer, MuonClip, and the second by generating large numbers of practice tool-use sessions. The report says pretraining on 15.5 trillion tokens ran without a single loss spike. [2]
Release: . Moonshot AI released the Base and Instruct weights on July 11, 2025. The technical report followed later.Model: 1T total · 32B activeWeight license: Modified MIT license; attribution condition for large-scale products
Many experts, few used at a time
K2 is a mixture-of-experts model. Each expert layer contains many feed-forward networks, the experts, and a router that chooses which of them process each token. K2’s model card lists 384 routed experts per layer, of which each token uses 8, plus 1 shared expert that every token uses. Its attention is multi-head latent attention, the compressed-cache design from DeepSeek-V2. [1]
The figure shrinks the layer to eight experts so every route is visible. Switch between tokens to see different experts selected, then change how many each token uses. More experts per token means more computation in the layer. The numbers identify networks, not specialties; the router is learned, and which experts a token picks depends on its context.
A shelf of eight books takes the same space whether you open two or all eight. Unused experts likewise still need storage, and attention, the shared expert and communication between chips still take time. The fraction of experts used is not a speed-up.
Stored experts versus experts used
Keeping a giant training run stable
An optimizer is the rule that turns training errors into weight changes. Most large models use one called AdamW. K2 instead uses Muon, which treats each weight matrix as a whole and reshapes its update so that no single direction dominates, a step called orthogonalization. Muon can learn more from each token of data, but the K2 team found that at scale it let attention scores grow until training became unstable. [2]
An attention score is the product of a query and a key, so if the weights that produce queries and keys keep growing, the scores can explode. K2’s fix, QK-Clip, tracks the largest attention score in each head during training. When one exceeds a threshold, QK-Clip shrinks that head’s query and key weights after the update, so future scores stay in range. The batch just processed is unaffected; the change applies from the next batch on. Muon combined with QK-Clip is called MuonClip. [2]
The arithmetic is simple. If a query–key product is 400 and the limit is 100, multiply both the query weights and the key weights by 0.5. The product shrinks by 0.5 × 0.5 = 0.25, to 100. The real implementation treats the parts of MLA that are shared across heads differently from the per-head parts.
Stability matters because a training run compounds millions of updates. A method that looks fine in a short experiment can blow up at full scale. The evidence that it works is a smooth loss curve over the entire run, followed by good results on held-out tests.
Keep the next update in range
Practicing tool use
Examples of a model using tools well are hard to find in ordinary text. K2’s launch describes a pipeline that manufactures them. It generates tool descriptions, tasks and agents, runs the agents against simulated or real tools, and keeps only the sessions that meet each task’s grading rubric. In each session, the tool descriptions tell the model what inputs an action needs, and each tool result is evidence for the next action. [3]
Take a small file-search task. The first call asks for a file that does not exist. A useful training session includes the error, a directory listing, a corrected path and a final answer based on the file’s contents. A dataset of clean, successful answers alone would never teach recovery from mistakes.
That is why the quality of the simulation matters. If a simulated tool always succeeds, the model gets no practice with missing data, confusing outputs or rejected inputs. If the grader rewards a confident final answer without checking it, the pipeline can select convincing failures. These are risks of the method in general, not observed problems with K2.
The model and the software around it
The July release included Base and Instruct checkpoints. The first Instruct model did not have the long-thinking mode or the image input that later K2 releases added, so this page describes the original version only. [3]
A model alone does not run tools. An application supplies the tool runner, permissions, context management and error reporting. The model writes a request for an action, the surrounding software carries it out, and the result comes back as text. An agent benchmark therefore measures the checkpoint and its environment together. Reproducing a score needs the same tools and evaluation setup, not only the same model.
Simplifications and limits
- Muon and mixtures of experts predate K2. The release’s own contributions are QK-Clip and making the combination work at this scale.
- The eight-expert router and the clipping arithmetic are teaching examples.
Sources and further reading
Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.
- Kimi-K2-Instruct model cardArchitecture table, active and total parameters, expert counts and modified MIT license.
- Kimi K2 technical reportSections 2.1 and 3.1; MuonClip/QK-Clip mechanism and agent trajectory synthesis.
- Kimi K2 official launchJuly release scope, data synthesis and initial non-thinking/text-only model boundaries.