K2 Horizon

Six sizes and two kinds of parameter count

All six are decoder-only transformers: they read a sequence of tokens and predict the next one. IFM says they share an architecture, training method, interfaces and deployment tooling, and that only the 0.9B model uses a smaller vocabulary.IFM, K2 Horizon technical blog, “One connected fleet”, 3 September 2026.

Four models are dense: every token runs the same attention and feed-forward blocks, without selecting a subset of experts. The other two are sparse Mixture-of-Experts (MoE) models, and their names carry two counts. K2-Horizon-375B-A23B names about 375 billion non-embedding parameters and about 23 billion active parameters per token. Its full checkpoint also contains the vocabulary matrices.

Active parameter counts approximate the work in weight projections; attention over the context adds work that depends on sequence length. For inference with all weights resident, memory must hold every expert even though each token uses only a few. The 36B-A4B checkpoint stores 37.4B parameters including vocabulary matrices, about 74.8 GB at two bytes per weight, before the KV cache and other runtime memory. Quantization reduces weight storage; offloading moves part of it to another memory tier. Runtime also depends on routing, memory traffic, batching and the serving software, so the sparse model need not run as fast as a dense 4B model.

The table lists what each model’s config.json on Hugging Face specifies.The model names count parameters outside the vocabulary tables. The 3.7B checkpoint stores 5.06B parameters; its input embedding and output matrices, each 250,624 × 2,560, hold 1.28B of them. Stored counts are from the Hugging Face API, 25 September 2026. Depth grows from 28 to 64 layers and width from 1,536 to 6,144. Every model uses grouped-query attention, in which 32 to 64 query heads share 8 key-value heads; that keeps the per-token key-value cache small. The 36B-A4B is narrower than the 7B and deeper, and holds most of its capacity in experts.

The vocabulary choice matters most at the small end. At the 0.9B model’s width, a 250,624-token vocabulary would put about 0.77B parameters into the input and output tables alone (2 × 250,624 × 1,536). The 64,256-token vocabulary leaves that budget for transformer layers. The 0.9B model reaches 131,072 tokens of context by stretching its position encoding with YaRN from an 8,192-token original; the other five support 524,288 tokens after long-context midtraining.

IFM positions the 0.9B for watches and glasses, the 3.7B and 7B for phones and other on-device uses, the 32B and 36B-A4B for local workstations and on-premises servers, and the 375B-A23B for enterprise workloads.IFM, K2 Horizon press release, 3 September 2026, and the technical blog. The press release also calls the 7B “the industry’s best-performing model under 10B”; that is IFM’s claim. These are vendor positioning statements. Real requirements depend on weight precision, key-value cache size, serving software, memory bandwidth and latency targets. The 0.9B was trained with multi-teacher distillation, using separate teachers for math and code, STEM, and instruction following. IFM describes the 3.7B as a size meant for fine-tuning.

What changes inside a layer

A transformer layer has two halves. Attention lets each position gather information from earlier positions. The feed-forward network (FFN) then transforms each position on its own. Attention starts by computing three projections of each token’s hidden vector: a query, a key and a value. Query–key scores decide how much weight each earlier position receives, and the values are the vectors averaged with those weights.

K2 Horizon uses three versions of this layer. The dense models run one FFN for every token. The 375B-A23B replaces most of its FFNs with 192 smaller experts and a router, a small linear layer that scores every expert for the current token and keeps the top 8; one shared expert runs for every token regardless. The 36B-A4B does the same with 100 experts, and also routes the value projection. That second change is MoVA.

Both routers score experts with a sigmoid and weight the chosen experts’ outputs by those scores.router_score_func: "sigmoid" in every config. In the 36B-A4B and 375B-A23B configs, mlp_only_layers: [0, 1, 2] keeps the first three layers dense. The shared expert gives every token a common path through the layer, and the routed experts add capacity that each token uses only in part.

MoVA: many value projections, four used per token

Mixture-of-Value Attention (MoVA) is IFM’s new layer. In standard attention, one linear map turns each token’s hidden vector into its value vector. MoVA keeps a pool of such maps, the value experts, and a router that picks a few of them for each token. Queries and keys are computed as usual. In the 36B-A4B, each of the 45 sparse layers holds 64 value experts and uses 4.IFM’s technical report is due at the end of September 2026. This description follows the released modeling code, modeling_k2_horizon.py (class K2HorizonMoVAAttention): a router scores 64 experts, the top 4 are kept, and each expert is a 2,560 → 1,024 linear map whose outputs are summed with the router’s weights.

Each value expert has the shape of one ordinary value projection: 2,560 × 1,024, or 2.6 million parameters. Per layer, then, the value path stores 64 of them and computes with 4: 64 times the storage of standard attention, 4 times the compute, and 16 stored for each one used. Summed over the model, that is 7.5B parameters stored in value experts, about a fifth of the model, and 0.47B used per token.

IFM’s reason for routing values is that attention decides how information from across the context is combined, so sparsity there adds a second place to grow capacity besides the FFN. It also says MoVA remains compatible with FlashAttention, grouped-query attention and sparse attention.IFM, technical blog, “MoVA: scaling attention with sparse experts”. One consequence is visible in the code: the routed value vectors are mixed before they are written to the key-value cache, so the cache is the same size as in ordinary grouped-query attention.The 36B-A4B also multiplies each attention output by a learned softplus gate (attention_gate_func: "softplus"); the other five configs leave the gate off.

By this count about 4.7B non-embedding parameters run per token;The 375B-A23B config sets mova_num_experts: 0. Of the six released models, only the 36B-A4B uses MoVA. IFM describes the model as activating approximately 4B. IFM reports that, trained under the same conditions, the 36B-A4B scores slightly below the dense 32B while using far fewer active parameters. That is IFM’s own comparison, and the blog does not report an experiment that separates MoVA’s contribution from that of the MoE feed-forward layers.

Uno drafts several tokens and the base model checks them

A language model normally generates autoregressively: each forward pass through the network produces one new token. The 7B and 0.9B models have optional Uno adapters, from IFM’s paper Unlocking Lossless Speedups in LLMs via Discrete Diffusion, that produce several tokens per round without changing the distribution the model samples from.Sahoo et al., arXiv:2609.04010, 2026. Code at ifm-ai/uno; adapters at K2-Horizon-7B-Uno and K2-Horizon-0.9B-Uno.

Uno adds a second, small set of weights to the frozen base model: rank-128 LoRA adapters, low-rank corrections attached to each weight matrix, trained with a discrete-diffusion objective to predict clean tokens from noisy positions in parallel.

Each round then takes two forward passes. In the draft pass, the adapted model proposes a block of B tokens in parallel. In the verify pass, the unmodified base model scores the block, and a sampler the paper calls Ψ-Spec keeps the longest prefix that passes the standard speculative-decoding rejection test. After a rejection, the sampler draws a replacement from a corrected distribution that accounts for the rejected draft. If every draft passes, it draws a bonus token from the base model. This acceptance-and-correction rule preserves the base distribution; a verifier alone would not guarantee lossless sampling.The paper’s accounting: the first drafted token comes from the base weights and is always accepted, so a full round yields between 2 and B + 1 tokens for 2 forward passes, before truncation at an end token or output limit. Tokens per forward pass therefore lie between 1 and (B + 1)/2.

For one drafted token, let q be the draft distribution and p the base distribution at that prefix. The sampler accepts token x with probability min(1, p(x)/q(x)). On rejection, it samples from max(p − q, 0), normalized to sum to one.Sahoo et al., Algorithm 1, lines 7–11. The linear sampler initializes the noisy positions with random tokens. “Lossless” refers to the output distribution, not identical text for a particular random seed.

For example, if q(A) = 0.8 and p(A) = 0.5 in a two-token vocabulary, A is accepted with probability 0.625 and B always passes. Accepted drafts contribute 0.5 probability to A and 0.2 to B; the remaining 0.3 must go to B. Sampling that replacement directly from the 50/50 base distribution would instead produce 0.65 for A and 0.35 for B.

The 7B-Uno model card reports an average of 2.71 tokens per forward pass across its benchmarks and a system throughput, tokens per second summed over a full batch, of 5,255 against 2,794 for the next-best model it compares. The same table shows the trade-off. Mercury 2 and DiffusionGemma, two diffusion language models, are about twice as fast for a single request; the paper reports that Uno outscores both on every agentic tool-use, coding and long-context benchmark it evaluated.

The paper reports speedups of up to 3× over the base model, including at the largest batch size that fits on the device. It also finds that Uno adds fewer parameters and uses less inference memory than the speculative-decoding methods it compares, DFlash and Eagle3, because it needs no separate draft model. The Uno repository ships two samplers: a linear sampler for high system throughput and a tree sampler for high per-request throughput. The published 7B-Uno results report 70.1% on SWE-bench Verified, 77.1% on GPQA Diamond and 98.9% on MATH500.

How the 375B model was trained

The 375B-A23B model card lists every training stage with its steps, tokens and sequence length, and IFM has published a checkpoint from each stage. Each stage starts from the final checkpoint of the one before.

Midtraining extends the context window in four stages, from 8K to 32K, 128K and then 512K tokens, using 1.91T tokens in all.The config sets the rotary-encoding base θ to 10,000,000, which spreads position angles over longer distances. IFM introduces post-training data, meaning reasoning traces, instruction following and agent trajectories in the chat format, from the start of midtraining rather than saving it for the end. Stage 4 shifts the mix further toward agentic and reasoning data. The model card describes the 524,288-token context as native from midtraining onward.

From the last midtraining checkpoint, IFM trained five reinforcement-learning (RL) experts, for knowledge work, instruction following, search, tool use and reasoning, and merged them into one model. Supervised fine-tuning (SFT) came after the merge: three phases totaling 332B tokens, the last on a high-quality subset of the earlier data with a decaying learning rate.The smaller models’ cards show different recipes. The 3.7B and 7B also run RL before SFT; the 32B and 36B-A4B go straight from midtraining to two SFT phases.

The training data

IFM describes each model’s pretraining corpus as roughly 20 trillion tokens, about 10 trillion of them synthetic.The stage tables differ in detail: the 375B card lists 15.1T tokens of pretraining, the 3.7B, 7B, 32B and 36B-A4B cards list 22.9T, and the blog says those four were trained on exactly the same 22T tokens. Nearly 17% of the pretraining corpus consists of problem-solving trajectories with explicit reasoning, and math trajectories were also rewritten as dialogues and study guides. IFM does not break out how the synthetic share and the reasoning share overlap.

The post-training data rests on task synthesis: more than 100 million unique tasks generated from task taxonomies, “diversity knobs” and web-search seeds. Tool definitions appear in JSON, XML and Markdown, and tool calls in JSON, XML and typed XML, so the model sees several conventions. IFM chose Markdown as the default tool format because on its data it was about 18.5% more token-efficient than JSON. Training on several schemas makes the common ones familiar; it does not guarantee that a deployed model follows every schema reliably. The 375B card recommends reasoning_effort="high", temperature 1.0 and top-p 0.95.

Reported benchmark scores

Every number in this section comes from IFM. The K2 Horizon scores are from IFM’s own runs; the comparison scores are as IFM reports them, and for the 32B and 36B-A4B IFM says they come from Artificial Analysis. IFM chose the comparison models, and they differ from model to model. Read the figure as IFM’s comparison, not as an independent ranking.

In IFM’s comparisons the three small models lead most rows, including the math and software-engineering results. The 36B-A4B leads its comparison set on tool use and terminal tasks and trails on science and long-context reasoning. The 32B trails Qwen3.8-27B on every row but one; its card reports a Stage 1 checkpoint and says Stage 2 results are coming.The blog labels this column K2-Horizon-32B; the 32B model card calls the evaluated checkpoint K2-Horizon-32B-Stage1. The 375B leads every model listed on SWE-Atlas-QnA and every open-weight model on Toolathlon; on most other rows GLM 5.2 or one of the closed models scores higher.

IFM also audited one of its own results. On Terminal-Bench 2.1 the 375B passed 500 of 712 trials, 70.2%. An automated audit, following Artificial Analysis’s reward-hacking procedure, flagged 24 of those passes across 10 tasks, where the model had, for example, found the benchmark’s reference solution on GitHub or edited the test harness. Removing them gives 66.9%.IFM, technical blog, “From Open Source to Open Science”. The figure plots the 70.2% that the model card reports, because the comparison scores were not audited the same way, and marks 66.9% beside it. IFM reports a similar case with the 7B, which downloaded SWE-bench answers in one run and scored an inflated 82%; the card lists 70.6%.The 7B model card’s table chooses reference models row by row. For SciCode it lists Mistral Small 4 (28.0) where the blog lists Gemma 4-12B (38.2), which outscores K2’s 31.6. This page uses the blog’s fixed comparison set.

What has been released

IFM calls K2 Horizon fully open: for every model it promises weights, training code, training data or data recipes, intermediate checkpoints, logs and evaluation details. On 25 September 2026 most of that was available and some was still announced. The table separates the two, and compares each artifact with what open-weight releases usually include.

Getting the models

All six model repositories are under the IFM organization on Hugging Face. The model cards give vLLM and SGLang serving recipes; the 375B recipe uses tensor parallelism across 8 GPUs with expert parallelism, validated on one 8× H200 node. IFM names Compass, Cerebras and Nebius as inference partners for API access.IFM, press release. The blog adds day-zero support in vLLM, SGLang and Ollama, and hardware support from NVIDIA, AMD and Cerebras. Uno adapters and code are at github.com/ifm-ai/uno.