Phi-3 · mini
How a small model trained on carefully chosen and generated text kept up with much larger ones
Most progress in language models has come from making them larger and training them on more text. Phi-3-mini, released by Microsoft in April 2024, tested another route. It is a 3.8-billion-parameter model, small enough to run on a phone, trained on text that was filtered and written to teach reasoning. This page explains that data strategy, what it bought, and what it cost.
Release: . Microsoft released the Phi-3-mini weights on April 23, 2024. The small, medium and 3.5 models came later.Model: 3.8B dense · Phi-3-miniWeight license: MIT
Why training data matters more for a small model
Microsoft released Phi-3-mini on April 23, 2024, with downloadable weights on Hugging Face. It has 3.8 billion parameters. The same announcement described larger Phi-3 models, which arrived later. [1]
A model learns from whatever its training text contains, and training time is limited. Imagine teaching a small model to write short programs from a scrape of public code. The scrape contains useful algorithms, but also boilerplate copied thousands of times, auto-generated files and abandoned experiments. Every copy of boilerplate uses training time that a more instructive example could have used. A large model can absorb that waste and still learn a lot. A small model can represent less, so what it reads matters more.
The idea behind “textbook-quality” data
Phi-3 continues a line of Microsoft research that began with the 2023 paper “Textbooks Are All You Need”. That work trained small coding models on code filtered for educational value, plus textbook-style lessons and exercises written by a larger model. The paper warned that naively generated text tends to repeat itself, and it varied the topics and intended audiences in its prompts to make the generated lessons more diverse. [2] The general idea that data quality matters is much older; this research pushed it hard for small models.
Compare two ways of teaching the same sum. One example says only “the sum is 45.” Another lists the numbers, shows a loop adding them up, and tests the result. Both end at 45, but the second shows every step along the way. A model trained to predict the next token gets to practice each of those steps.
Variety matters as much as clarity. A thousand near-identical addition problems teach nothing about empty lists, wrong input types or overflow. A good synthetic set varies the underlying problem, not just the variable names. Generated text also has to be checked, because a well-formatted wrong explanation teaches the wrong thing.
Turn raw text into useful practice
Phi-3’s training recipe
The Phi-3 report describes training mini on 3.3 trillion tokens in two phases. The first phase used heavily filtered web text, chosen to teach general knowledge and language. The second used web text filtered even more strictly, mixed with synthetic data aimed at reasoning and other skills. The architecture itself is ordinary: a transformer with the same block structure as Llama 2. The report says the result scores about as well as Mixtral 8×7B and GPT-3.5 on several benchmarks, including 69% on MMLU, a broad multiple-choice knowledge test. [3]
Filtering and generation play different roles. Filtering selects from text that already exists. Generation creates new text or rewrites existing text into another form. A web page might supply a concept, for example, and a generated exercise might turn it into a question with a worked solution. Checking the solution keeps errors out of the training data.
The report is open about the cost. Phi-3-mini does poorly on tests of factual recall such as trivia questions, which the authors attribute to its small size. A curriculum aimed at reasoning can raise reasoning scores and still leave gaps in what the model knows. [3]
How small the weights are
The model card lists Phi-3-mini as a 3.8-billion-parameter model under the MIT license, which permits almost any use. The card has since been updated for a June 2024 revision of the instruction-tuned model, and its newer benchmark scores belong to that revision, not to the April release. [4]
The figure starts from 3.8 billion numbers. At 16 bits (2 bytes) each, the weights take 7.6 GB. At 8 bits they take 3.8 GB, and at 4 bits, 1.9 GB. The report shows a 4-bit version running offline on an iPhone 14. [3]
Storing weights in fewer bits is called quantization: each weight is rounded to the nearest value the smaller format can represent. The rounding changes the model’s outputs, sometimes noticeably. The figure’s buttons change only the storage arithmetic. Whether a 4-bit Phi-3 still answers well has to be measured.
Count the weight bytes
Telling a skill gap from a knowledge gap
When a small model gets a question wrong, ask which of two things failed: the reasoning or a fact. Suppose a question gives a distance and a speed and asks for the travel time on a newly opened train line. A model might do the division perfectly and still know nothing about the line. Handing it a reliable timetable fixes the missing fact. It does not help a model that cannot do the division.
That distinction is Phi-3’s main lesson: choosing the training data is part of designing the model. A curriculum built for reasoning produced a small model that reasons well for its size and recalls facts poorly. Such a model is useful when its strengths and gaps fit the task, for example when the facts it needs can be supplied in the prompt.
Simplifications and limits
- This page covers mini, the first Phi-3 model with public weights.
- Careful data selection does not prove the curriculum is optimal, and it does not give the model complete factual knowledge.
- The storage figure counts weights only.
Sources and further reading
Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.
- Microsoft’s Phi-3 release announcementApril 23 public mini release, 3.8B size and distribution channels.
- Textbooks Are All You NeedSection 2: filtered educational data, synthetic textbooks/exercises and diversity constraints.
- Phi-3 Technical ReportSections 2 and 6: mini data recipe, token count, architecture and factual-knowledge weaknesses. Later Phi-3.5 additions are not backdated.
- Microsoft’s Phi-3-mini-4k-instruct model cardMini checkpoint definition, current public weights, intended use and MIT license.