gpt-oss-20b and gpt-oss-120b

Stored weights and active weights

A mixture of experts replaces a layer’s single feed-forward network with a set of alternative networks, the experts, and a router that picks a few of them for each token. gpt-oss-120b has 128 experts per layer and 36 layers; gpt-oss-20b has 32 experts per layer and 24 layers. Both use 4 experts per token, and both read and write text only. [1]

Picture a workshop with 128 workbenches, where each job visits four. Counting the benches a job visits tells you part of the work it needs, but all 128 benches still take up floor space. Unused experts likewise save arithmetic for a token without disappearing from memory. And because attention and other shared parts run for every token, 4 out of 128 is not the fraction of the whole model that is active.

The model names round the sizes. The model card gives the figures below, where “active” means the parameters used for one token. [5]

A checkpoint is the set of stored weight files. Its size does not include the extra working memory the model needs while it answers.

Two attention savings, and a sink

Experts do nothing to reduce the cost of reading the conversation. For that, gpt-oss alternates two kinds of attention layer: full attention over the whole context, and local attention over only the previous 128 tokens. Both sizes also use 64 query heads that share 8 sets of keys and values, which is grouped-query attention. The window reduces how many positions some layers read; grouped-query attention reduces how many keys and values are stored for each position. The full-attention layers keep a direct path to early context. [5]

A subtler change concerns how attention distributes its weight. Attention turns its scores into weights that always sum to one across the visible tokens, so a head must spread a full unit of weight even when none of the tokens is useful. gpt-oss gives each head a learned attention sink: an extra score that takes part in that normalization but carries no value. Weight assigned to the sink is effectively discarded, so the weights on real tokens can sum to less than one. The reference code adds the sink to the normalization and then drops its entry before combining the values. [6]

Take three tokens with equal scores and values of 2, 6 and 10. Without a sink, each gets one-third of the weight and the output is 6. Add a sink with the same score, and each of the four options gets one-quarter, so the output is (2 + 6 + 10) / 4 = 4.5. Nothing extra was retrieved and the cache is no smaller. The head simply contributed less to the result, which lets a head with nothing useful to add stay quiet.

Four-bit expert weights

The expert weights were released in MXFP4, a four-bit floating-point format in which each small group of values shares a scale factor. Counting those scale factors, the model card reports 4.25 bits per expert parameter, and it says the expert weights were already in this format during post-training. [5] Software must decode the format exactly; it is not interchangeable with other four-bit formats. OpenAI publishes reference implementations and recommends checking that other implementations agree with them. [2]

A simple estimate gives a lower bound. Twenty-one billion weights at 16 bits need 42 GB; at 4 bits they need 10.5 GB, if every weight used four bits and there were no scale factors. Neither assumption holds for gpt-oss. Some weights are kept at higher precision, and the KV cache, activations, scale factors and working buffers all take memory too.

Switch between bit widths in the figure. The storage arithmetic changes exactly, but accuracy does not appear, because storage arithmetic cannot predict rounding error. OpenAI’s deployment guide lists the supported implementations and their actual, larger memory needs. [3]

Handing work to a tool

A text model can write “I used a calculator” without any calculator running. A working tool system needs a clear handoff: who is speaking, which tool should receive the request, when the model stops to wait, and how the result comes back. The model only writes the request; the application performs the action.

Harmony is the conversation format gpt-oss was trained on. It marks each message’s role, such as user, assistant or tool; its channel, such as analysis for reasoning and final for the answer; its recipient; and how it ends. Handing off to a tool ends a message differently from finishing an ordinary message or finishing the assistant’s turn. [4] Those distinctions tell the application what to do next: call a function, keep generating, or show the answer.

For a simple arithmetic task, the application receives a calculator request, runs the calculation, adds the result to the conversation as a tool message, and lets the model continue. The application must still check the request’s arguments and decide which tools are allowed. A structured request is only as safe as the system that enforces its structure.

The table gives the meaning of each ending, not the literal tokens. One rule is easy to get wrong. Keep the current turn’s analysis messages while its tool calls are still in progress, but drop the analysis from earlier, completed turns when the user sends a new message. Dropping the current reasoning in the middle of a tool exchange can break the model’s behavior. The raw analysis is not meant to be shown to the user as the answer. [7]

Choosing how much to reason

The models accept three reasoning-effort settings: low, medium and high. Their post-training included supervised learning and reinforcement learning for reasoning and tool use. [5] The settings are instructions the models were trained to follow. They change how long the reasoning tends to be, but they are not exact token limits, and they do not switch to a different set of weights.

Suppose two settings produce the same final sentence, but one writes many more reasoning tokens and makes two tool calls. Their response time and cost can differ greatly. The real budget includes generated tokens, attention over a growing context, the tools’ own response times and any retries. The parameter count shows none of this.

Longer output does not prove better reasoning. Intermediate steps can be wrong, and they may not describe how the answer was actually produced. Judge each setting on the same tasks, using tasks with a dependable check, and count the failures as well as the successes.

To compare implementations, keep the model, prompt, effort setting, tools and sampling settings the same. Check both the format of tool calls and the task results, because a correctly formatted response can still be wrong. OpenAI’s compatibility checks and evaluation tools test for these separate failures. [2]

Simplifications and limits

• Both sizes share a page because they share the same mechanisms; their parameter and expert counts differ.
• The four-bit chart is an ideal lower bound. It does not claim that the whole model uses four bits or that it fits on a particular device.
• This page does not guess at the full training data, which OpenAI has not published.
• The attention sink changes how attention mixes values. It is not an extra cached token, a way to compress context, or a guarantee of faithful reasoning.
• Reported checkpoint sizes, ideal packed-weight sizes and peak serving memory are three different quantities.

Follow the idea

Mixtral 8×7BDeepSeek-V3DeepSeek-R1

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Introducing gpt-ossRelease date, expert counts, sizes, text modality, and public weight license.
2. Verifying gpt-oss implementationsMXFP4 expert release, reference implementations, and protocol correctness.
3. Running gpt-oss with TransformersDeployment memory and implementation caveats.
4. OpenAI Harmony response formatRoles, channels, recipients, and end-of-message versus handoff markers.
5. gpt-oss model cardArchitecture and post-training sections; variable effort and limitations.
6. OpenAI gpt-oss reference attention implementationsdpa and AttentionBlock: learned sink normalization, zero value contribution, GQA, and alternating local/full masks.
7. Handling raw chain of thought in gpt-ossCurrent-turn reasoning across tool calls, completed-turn history, and separation from final user-facing output.