BitNet b1.58 2B4T

Weights with three values

A ternary weight has three possible values: −1, 0 and +1. Multiplying an input by such a weight means negating it, dropping it or keeping it. A matrix multiplication, the main work in a language model, then reduces mostly to additions and subtractions, with a single multiplication by a scale factor at the end. Software written for this can exploit it.

The 2B4T model replaces the ordinary linear layers of a transformer with BitLinear layers. Each layer’s weights are divided by their average absolute value and rounded to −1, 0 or +1, and each layer’s inputs are stored as 8-bit integers. The model was trained from scratch under these rules. [1] The “1.58” in the name is log₂ 3 ≈ 1.58, the number of bits of information needed to tell three equally likely values apart. Real files pack the values in other ways, so a weight does not literally occupy 1.58 bits on disk.

Training keeps a higher-precision copy of each weight, called the latent weight, and rounds it every time the layer runs. In the figure, the scale is fixed at 0.5. A latent weight of 0.65 divided by 0.5 is 1.3, which rounds to +1, so the layer uses +1 × 0.5 = 0.5. Drag the latent weight down: below 0.25 it rounds to 0, and the value the layer uses jumps. Between jumps, small changes to the latent weight change nothing in the calculation. That flatness is the problem the next section deals with.

Learning through rounding

Training adjusts each weight in the direction that reduces the error, using the gradient: how much the error changes when the weight changes slightly. Rounding breaks this. A small change to the latent weight usually leaves the rounded value unchanged, so the gradient through rounding is zero almost everywhere and undefined at the jumps. An optimizer that followed it would never move.

Quantization-aware training works around this with a stand-in gradient. The most common, the straight-through estimator, passes the gradient through the rounding step as if the rounding were not there. Each training step keeps the latent weights in higher precision, rounds them for the forward calculation, measures the error, and applies the stand-in gradient to the latent weights. The stand-in is an approximation that works in practice; rounding itself is still not differentiable. The approach comes from the earlier BitNet b1.58 paper, which the 2B4T model scales up. [2]

Suppose a latent weight moves from 0.20 to 0.24 over several steps. With a scale of 0.5, it still rounds to 0, so the model’s output does not change. One more step, to 0.30, crosses the threshold and the weight becomes +1. Many small updates can build up before any visible change. The latent weights exist only during training; the released model ships the rounded values.

The model needs matching software

The model card describes the checkpoint: about 2 billion parameters, 4 trillion training tokens, its context length and the software it is meant to run with. [3] The release does not show that any existing model can be rounded to three values and keep its quality. 2B4T works because it was trained under the restriction.

A general-purpose inference engine might unpack the three-valued weights into ordinary 16-bit numbers and run a normal matrix multiplication. That gives the right answer but throws away most of the savings. Software written for ternary weights can pack them tightly and replace multiplications with additions. Microsoft’s bitnet.cpp provides such software for CPUs, and its repository records the model’s April 14, 2025 release. [4]

Think of directions written in a code of “left”, “stay” and “right”. The short code saves space only if the program reading it understands it. If the program expands each direction into a long record before using it, much of the saving is lost. Efficiency depends on the whole path from the file to the arithmetic.

Fewer bits do not prove a better model

Two kinds of claim are easy to mix up. “A three-valued weight takes less storage than a 16-bit number” is arithmetic. “This trained model answers a question correctly” is an empirical result. The first cannot establish the second.

The report compares 2B4T with similar-sized models trained in full precision, after its own fine-tuning on instructions and preference training with DPO. [1] Those comparisons depend on the chosen tasks, prompts, data and software. They do not show that ternary training is better at every size or for every task.

Weight storage is also only part of the memory a running model needs. Embeddings, scale factors, the KV cache, temporary values and software workspace all add to it, and training needs far more memory than the final rounded weights. A careful test reports the memory actually used and the quality of the answers, not only the nominal bit width.

Simplifications and limits

• The figure fixes the scale for teaching; the real model computes it from each weight matrix.
• Bits of information per weight, the storage format on disk and total runtime memory are three different quantities.
• This page computes no speed-up or accuracy estimate.

Follow the idea

gpt-oss-20b and gpt-oss-120bGemma 2 · 9B and 27B

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. BitNet b1.58 2B4T technical reportArchitecture, absmean/absmax quantization, training stages, and inference sections.
2. The Era of 1-bit LLMsOriginal ternary BitNet b1.58 method and its relationship to the later checkpoint.
3. Official BitNet-b1.58-2B-4T model cardModel configuration, license, and deployment scope.
4. Official BitNet inference frameworkApril 14, 2025 release note and specialized inference implementation.