Kimi-K3.1-Pro by Moonshot AI: interesting facts It's huge, and sparse • It has 2.8T total parameter...
Thomas Wolf(@Thom_Wolf) · 人工智能
Kimi-K3.1-Pro by Moonshot AI: interesting facts It's huge, and sparse • It has 2.8T total parameters, but only 104B are active per token. • Along with Kimi-K3, it's the largest real model release on the Hub. The next biggest are Qwen3.8-2.4T and DeepSeek-V4-Pro (1.6T). • It has 896 experts and sends each token to 16 of them. (DeepSeek-V4-Pro has 2 times less and Mistral-Large-4 has 7 times less) Clever attention for a 1M-token context • Most of its 93 layers use linear attention (KDA). Every 4th layer uses DeepSeek-style sparse attention, which only looks at the top 2048 tokens. • This is how it handles a 1M-token context (the same as the other frontier open models) • It uses a custom activation called SiTU-GLU and has a 401M-parameter vision encoder (MoonViT-V2), so it takes both text and images. Small on disk for its size • It was trained to run in 4-bit (MXFP4), so the whole repo is only ~1.56 TB. • Stored in regular BF16 it would be about 5.6 TB(!) Benchmarks (from the model card) • It goes back and forth with Claude Opus 5.5 and GPT-6 Astra. It's top of the card's table on BrowseComp (92.1), HLE with tools (70.0) and SWE Marathon (51.7). • It's weaker on pure reasoning without tools and on cyber security. • Moonshot ran most of their own coding evals using the Claude Code harness. License (a modified MIT) • It's free to use, modify and redistribute. • API businesses making over $20M a year need a deal with Moonshot. • Products with over 100M monthly users or over $20M monthly revenue must display "Kimi K3.1-Pro". On the Hub • It went public about 1.5 hours ago and already has 214 likes, about 9× Kimi-K3's first hour. It's #19 on the trending list. • It isn't gated, but loading it in transformers requires "trust_remote_code" Lisan al Gaib @scaling01 Kimi-K3.1-Pro is now open-weights huggingface.co/moonshotai/Kim… 🔗 View Quoted Tweet 💬 2 🔄 0 ❤️ 2 👀 314 📊 2 ⚡ Powered by xgo.ing