Our iMatrix GGUF quantization of Ornith-1.0-35B reached 30,000 downloads within days of release. This post is the technical account behind that release: what Ornith is, why MoE models present a distinct quantization problem, how iMatrix calibration addresses it mathematically, and what our calibration choices mean in practice.

Ornith-1.0-35B: Architecture

Ornith-1.0-35B is a Mixture-of-Experts model built on the Qwen3.5 architecture. Despite the 35B parameter count, it is explicitly designed as the lightweight member of its model family — targeting efficient single-GPU deployment through sparse activation rather than dense computation. At inference time, only a subset of experts activates per token, meaning the effective compute per forward pass is substantially lower than the total parameter count suggests.

The model supports a 262,144-token context window and uses BF16 tensor precision natively. Its training pipeline departs from standard RLHF: Ornith employs reinforcement learning to learn not only solution rollouts but the scaffolding that drives those rollouts — search trajectories, intermediate reasoning structure, and self-correction loops. Output includes explicit reasoning traces enclosed in <think>...</think> blocks before the final answer, surfacing chain-of-thought without post-hoc prompting.

Benchmark results confirm its positioning as a coding and agentic model:

Benchmark	Score
SWE-bench Verified	75.6%
SWE-bench Pro	50.4%
Terminal-Bench 2.1 (Terminus-2)	64.2%
NL2Repo	34.6%

A 75.6% score on SWE-bench Verified places it among the strongest open-weight models for real-world software engineering tasks.

Why MoE models compress differently

Standard post-training quantization assumes relatively uniform weight importance across layers. Dense transformers partially justify this — all attention heads and FFN weights participate in every forward pass. MoE models violate this assumption structurally. Each token activates only a small subset of experts; the remaining experts receive no gradient signal for that token and their weights are never read during that pass. Across a large corpus, some experts specialize heavily and are activated frequently; others handle narrow distributions and are rarely triggered.

Naive uniform quantization — assigning the same bit-width to every weight tensor — is therefore particularly lossy for MoE models. Aggressively compressing a frequently activated expert degrades every token that routes through it. Compressing a rarely activated expert at the same rate wastes precision budget on weights that barely affect output distribution. The mismatch between parameter count and actual compute participation makes calibration-aware quantization more valuable here than in dense models.

iMatrix: activation-weighted quantization

iMatrix (importance matrix) quantization addresses this by computing a per-weight importance score from real activation data before quantization begins. The core insight is that quantization error on a weight w_j only matters in proportion to how much the corresponding activation x_j amplifies it at the output. A weight with large activation magnitude demands tighter quantization; one with near-zero activation can tolerate coarser rounding.

The importance-adjusted weight used during quantization scales as:

w_adj[j] = w[j] * sqrt(σ² + x[j]²)

where σ² is the variance of activations across the calibration corpus for that weight’s corresponding input channel, and x[j] is the per-element activation magnitude. The sqrt(σ² + x[j]²) term is effectively a smoothed RMS of the activation — it prevents any single outlier from dominating the importance score while still capturing channels that are consistently high-magnitude.

During quantization, this adjusted weight determines the rounding decision. Weights in high-importance channels are quantized with finer effective granularity — the quantizer is pushed to minimize error where activation magnitude would otherwise amplify it into the output. Weights in low-importance channels accept larger absolute quantization error because the network rarely sees significant signal through them.

Applied to a MoE model, this naturally allocates more precision budget to frequently activated expert weights and allows heavier compression of dormant expert weights — without any explicit per-expert routing logic. The calibration data itself encodes routing frequency through activation statistics.

Calibration data: what matters and what doesn’t

The importance matrix is only as good as the calibration corpus. An activation that never occurs in calibration will be scored as unimportant regardless of its true significance at deployment time. Several properties of calibration data affect outcome:

Diversity over volume. Studies on iMatrix calibration show that results stabilize quickly — 100K tokens from a diverse corpus produces importance scores close to those from 1M tokens. The marginal value of more data drops fast once rare activation patterns are covered. What matters is that calibration text spans the intended deployment distribution.

Real text over synthetic. Pseudo-random or high-temperature sampled text tends to activate unusual activation patterns not representative of normal inference. Standard natural language from pretraining-style corpora — code, prose, structured data — generalises better to real use.

Match the model’s strengths. For a coding and agentic model like Ornith, calibration data should include significant code coverage. A Wikipedia-only calibration corpus will underweight code-path activations, compressing weights that matter most for the primary use case more aggressively than the importance matrix would suggest.

For this release we used 2 million tokens from WikiText-103 across 128 calibration chunks. WikiText-103 is general natural language — adequate for a broad importance estimate but not tuned to Ornith’s coding specialisation. A code-heavy calibration corpus (e.g. The Stack or a mix of GitHub and prose) would likely produce tighter results on SWE-bench-style tasks at the same bit-width. That is a candidate for a follow-up release.

Quantization levels and the precision-size tradeoff

GGUF quantization operates on per-block weight groups. Lower bit-widths share a single scale factor across more weights, increasing the quantization error for any individual weight whose true value deviates from its block’s centroid. iMatrix does not change this block structure — it changes which weights end up in which precision tier across the full weight tensor.

Quantization	File size	Bits per weight (approx.)	Notes
Q2_K	11.66 GB	~2.6	Significant quality degradation on reasoning tasks
Q3_K_M	15.00 GB	~3.3	Acceptable for general use, noticeable on long chains
Q4_K_M	21.17 GB	~4.5	Recommended — near-full quality on most benchmarks
Q6_K	29.00 GB	~6.6	Near lossless for most tasks
Q8_0	36.90 GB	~8.5	Effectively identical to BF16 output

For a chain-of-thought model like Ornith, Q2 and Q3 quantizations carry more risk than on a direct-answer model. Reasoning traces are long and compounding — early token probability distortions propagate through the reasoning chain. Q4_K_M is the minimum we’d recommend for reliable SWE-bench-style task completion; Q6_K for users where output quality is the primary constraint.

Running the model

The quantized model runs on any llama.cpp-compatible runtime. The Q4_K_M variant fits on a 24 GB GPU and runs efficiently on Apple Silicon with large unified memory:

llama-cli -hf liodon-ai/Ornith-1.0-35B-GGUF-imatrix-GGUF:Q4_K_M

All quant variants are in the same repository: liodon-ai/Ornith-1.0-35B-GGUF-imatrix-GGUF. The full-precision base model is available from deepreinforce-ai under MIT license.