We built SLM-10M as a research experiment: can a sub-10M parameter transformer-based language model compete meaningfully on standard benchmarks? The answer turned out to be yes — SLM-10M currently tops the Open SLM Leaderboard in the under-10M tier.

Architecture

SLM-10M uses a compact transformer architecture with aggressive parameter sharing and sparse attention. We applied post-training quantization to produce GGUF-compatible checkpoints, making the model runnable on CPU-only hardware with no GPU required.

Why small models?

Large language models are powerful but inaccessible for edge deployment, low-resource research, and constrained hardware environments. The goal of our SLM research track is to push the boundary of what is achievable at 10M parameters — not as a toy, but as a genuinely capable compact model.

SLM-10M is freely available on HuggingFace under Apache 2.0. The training code and evaluation scripts are on GitHub.