DeepSeek-V3

How DeepSeek trained a 671-billion-parameter model without wasting its hardware

DeepSeek-V3, released in December 2024, kept DeepSeek-V2’s architecture, with its compressed attention cache and many small experts, and scaled it to 671 billion parameters, 37 billion of them used for each token. [1] The hard part was training a model that large without leaving expensive hardware idle. The paper reports that the final training run took 2.788 million GPU-hours, a figure that drew wide attention. This page explains three techniques behind that efficiency: a way to keep experts evenly loaded, arithmetic in 8-bit numbers, and a training objective that predicts more than one token ahead.

5 sections · about 8 minutes · Training

Release: 2024-12-26. DeepSeek released the original V3 weights on December 26, 2024. The later V3-0324 and V3.1 checkpoints are separate revisions.Model: 671B total · 37B activeWeight license: DeepSeek Model License for original weights; MIT for code

A sparse model can still have traffic jams

In a mixture-of-experts model, the experts are spread across many chips. Suppose eight chips each hold one expert, and one training step sends 80 tokens to them. If each chip receives 10, they all finish together. If one receives 45 and several receive 5, the busy chip becomes a queue and the others wait. The model does little work per token on average, yet much of the hardware sits idle.

The usual remedy adds a penalty to the training loss whenever traffic is uneven. That creates a conflict. A strong penalty pushes the router to send tokens to experts that suit them less, which can make the model worse at predicting text. DeepSeek-V3 handles the conflict differently, with a routing bias. [2]

What matters is how tokens spread across the whole batch, not the route of any single token. A diagram of one token choosing its experts explains selection, but it cannot show whether the hardware is well used.

Steering traffic without changing what experts contribute

Each V3 expert layer has 1 shared expert and 256 routed experts, of which each token uses 8. To pick the 8, the router scores every expert. V3 adds a per-expert bias to those scores, used only for the choice. After each training step, the bias of an overloaded expert goes down and the bias of an underused expert goes up, nudging later tokens toward idle experts. Once the experts are chosen, their outputs are weighted by the original scores, without the bias. Because this replaces the usual balancing penalty, it is called auxiliary-loss-free balancing. A very small penalty, computed per sequence, remains as a safeguard. [3]

Take a choice between two experts. Expert A scores 0.62 and expert B 0.60. A is overloaded, so its bias is −0.04; B is underused, with a bias of +0.01. The adjusted scores are 0.58 and 0.61, so B is chosen. B’s output is then weighted by its original score, 0.60. The bias changed which expert ran, not how much its output counts.

It is like choosing a slightly less convenient checkout line because your usual one is backed up.

Separate choosing an expert from weighting its output

Simplified routing with a load-balancing bias. It omits normalization of the selected scores, the distributed implementation and the small per-sequence penalty. The bias changes which experts are chosen; the original scores still weight their outputs.

Training in 8-bit numbers

Neural-network arithmetic usually uses 16- or 32-bit numbers. V3 was trained with mixed precision: most large matrix multiplications use FP8, an 8-bit floating-point format, while operations sensitive to rounding stay in higher precision. The paper reports this as the first validation of FP8 training on a model this large. [3] The released weights are stored in FP8, with a script to convert them to the 16-bit bf16 format. [1]

The figure shows the simplest effect of fewer bits, which is storage. Twenty billion numbers take 40 GB at 16 bits, 20 GB at 8 bits and 10 GB at 4 bits. These are made-up weights stored uniformly. The arithmetic says nothing about whether a model keeps its quality at a given precision.

The difficulty with 8 bits is range. An 8-bit format has only 256 distinct values, so it cannot represent very large and very small numbers at once. V3 gives each small block of numbers its own scale factor, and it adds up the results of multiplications in higher precision so that many small contributions are not rounded away. [3] The time saved depends on these details and on hardware support, not on the bit count alone.

Bits set the size of an ideal weight array

Simplified storage for a made-up array of 20 billion weights stored uniformly. It excludes scale factors, mixed precision, activations, the KV cache, sharding and any effect on accuracy. Halving the bits halves the ideal storage; it does not automatically halve training time.

Predicting more than one token ahead

V3 also trains with a multi-token prediction objective. Besides predicting the next token, the model has an extra module that predicts the token after that. The module is chained to the main model so that each extra prediction can use the tokens before it, which keeps the order of the text intact. Earlier work predicted several future tokens with independent output heads; the paper contrasts its chained design with that approach. [3] [4]

Take the training text “the brass key opened the door.” Ordinary training asks the model, having read up to “brass”, to predict “key”. The extra module takes the model’s internal state at “brass”, combines it with the embedding of the real next token, “key”, and predicts “opened”. During training “key” is known, because it is in the text. The extra module only adds training signal; it gives the main model no view of the future when it generates.

The paper’s motivation is that the extra prediction makes the training signal denser and may encourage representations that plan ahead. The module can also be reused during generation to propose a draft token that the main model then checks, a technique called speculative decoding. How much faster that makes generation depends on how often the drafts are accepted. [4]

What the training-cost figure covers

The paper’s training cost, 2.788 million H800 GPU-hours, or about $5.6 million at an assumed rental price of $2 per GPU-hour, was widely quoted as the cost of building DeepSeek-V3. The paper states that the figure covers only the official training run. It excludes earlier research, the experiments that tested the design, data collection, buying the hardware and the continuing cost of serving users. [3] Comparing it with a company’s total spending compares two different things.

The broader point is to measure the cost you actually care about. For training, look at useful arithmetic, expert traffic, numerical stability and idle hardware. For deployment, look at weight memory, conversation caches and response time under real load. V3’s techniques work as a set, because fixing one bottleneck tends to expose the next.

Simplifications and limits

The precision figure is a generic storage calculation, not V3’s full training recipe.
Auxiliary-loss-free balancing still includes a very small per-sequence balancing penalty.
The original December 2024 weights use DeepSeek’s model license; later revisions of the family may use different licenses.

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

Official DeepSeek-V3 repositoryOriginal specifications, downloadable FP8 weights and license distinction.
Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-ExpertsRouting biases and load-balancing motivation.
DeepSeek-V3 technical reportSections 2.1–2.2 and 3.3; bias selection, complementary loss, MTP and cost scope.
Better & Faster Large Language Models via Multi-token PredictionEarlier multi-token prediction formulation; distinction from V3’s sequential modules.
DeepSeek official change logDecember 26, 2024 V3 release entry.