Tülu 3

Three stages of post-training

A pretrained model predicts text. Post-training turns it into an assistant, usually in stages. In supervised fine-tuning, the model is trained on example responses an assistant should give. In preference optimization, it is trained on pairs of responses so that the preferred one becomes more likely than the rejected one. In reinforcement learning, the model writes new answers of its own, each answer is scored, and the model is updated to make high-scoring answers more likely.

Tülu 3 started from Meta’s Llama 3.1 base models and used all three stages: supervised fine-tuning on curated data, preference tuning with direct preference optimization (DPO), and reinforcement learning with verifiable rewards, or RLVR. None of the ingredients was new. The contribution was a complete, tested recipe that anyone could inspect and rerun. [1]

Think of learning arithmetic. Worked solutions show a method. Being told which of two explanations is clearer teaches style. Having your own attempt at a new problem marked right or wrong gives feedback on what you produced. The three kinds of feedback overlap, but none replaces the others.

A reward is only as good as its checker

For the question “What is 3 × 2?”, a simple checker can pull out the last number in the answer and compare it with 6. That rule is easy to audit, but it ignores everything else in the answer. An explanation with wrong arithmetic in the middle can still end in 6 and earn the full reward.

In the figure, choose the “Flawed reasoning” response with the Final number rule. Then switch to Exact string and choose the “Formatted” response. The first rule is too lenient about reasoning; the second is too strict about format. Both give the same verdict every time, and neither measures everything a reader cares about.

Tülu 3 used checks only where an answer can be checked: math problems from the GSM8K and MATH datasets, and instructions with testable constraints, such as a required word count. For those prompts, the check’s result replaced a reward model’s score. The reinforcement-learning algorithm was proximal policy optimization (PPO), with a penalty that keeps the model from drifting too far from its starting checkpoint. It did not use GRPO, the group-based method described on the DeepSeek-R1 page. [2]

How a reward changes future answers

A language model assigns a probability to each token, and therefore to each complete response. In reinforcement learning, the model writes a response, receives a score, and adjusts its weights so that high-scoring responses become more likely. The update works through the probabilities of the tokens the model wrote, so the checker can be any program at all, as long as it returns a score.

Suppose the model solves a certain kind of problem one time in ten. The successful attempts show which way to shift its behavior. If every attempt fails, a reward of 0 or 1 says nothing about which step went wrong. If every attempt succeeds, there is nothing left to learn. Choosing problems of the right difficulty therefore matters as much as the algorithm.

Rising reward during training does not prove the model got better. A run of attempts can be lucky, a checker can be gamed, and a model can improve on its training problems without improving on new ones. A sound experiment keeps the problems used for training separate from those used for testing.

What an open recipe makes possible

The Tülu project published its post-training data and a codebase covering supervised fine-tuning, DPO and RLVR. [3] That makes controlled experiments possible. Keep the same base model and the same fine-tuned checkpoint, change only the reward rule, and compare the two results on a test set that neither run trained on.

Comparing two unrelated chatbots cannot answer that kind of question. Their difference could come from the architecture, the pretraining data, the prompt template, the sampling settings, or test questions that leaked into training. With a shared recipe, researchers can change one thing at a time.

The model weights remain under Meta’s Llama 3.1 license, because Tülu 3 is built on Llama; publishing the training code does not change that. The model card names the base model and lists the released artifacts. [4] “Open” covers several separate things: weights, data, code, intermediate checkpoints and evaluation methods. A release can provide some of them without the others.

Simplifications and limits

• The checker in the figure is a teaching example, not the released verifier.
• RLVR describes where the reward comes from, not which algorithm uses it; Tülu 3 used PPO.
• This page covers the original 8B and 70B models from November 2024. The 405B model came later.

Follow the idea

DeepSeek-R1Olmo 3

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Tülu 3: The next era in open post-trainingNovember 21, 2024 launch, recipe stages, RLVR scope.
2. Tülu 3 technical reportSections 5–6: DPO, PPO, verifiable reward objective, ablations and KL controls.
3. Tülu project and released recipeReleased data, algorithms, and code.
4. Llama-3.1-Tülu-3-8B model cardBase-model provenance and model license.