llm-atlas/research/GROK_RESEARCH_ROADMAP.md

# LLM Atlas — Grok 辅助检索路线（K3-Anchored）

> 生成来源：本机 Grok CLI Headless 模式，2026-07-28。
> 用途：扩展论文线索与检查遗漏，不直接作为正文证据。主代理需回到每篇一手来源核验后，才能把结论写入网站。

**Site:** Chinese educational site *LLM Atlas*
**Anchor primary source:** Kimi Team, *Kimi K3: Open Frontier Intelligence*, arXiv:[2607.24653](https://arxiv.org/abs/2607.24653) (2026-07) · DOI:[10.48550/arXiv.2607.24653](https://doi.org/10.48550/arXiv.2607.24653)
**Companion official sources:** [Kimi Linear](https://arxiv.org/abs/2510.26692) · [Attention Residuals](https://arxiv.org/abs/2603.15031) · [Kimi K2](https://arxiv.org/abs/2507.20534) · [Kimi K2.5](https://arxiv.org/abs/2602.02276) · [Kimi-VL / MoonViT](https://arxiv.org/abs/2504.07491) · [Muon is Scalable](https://arxiv.org/abs/2502.16982)

**Scope rules**
- Landmark *primary* papers only; year | title | canonical arXiv/DOI | problem solved | bridge to next idea.
- ≥80 papers; DeepSeek lineage given dense coverage; post-V3.2 only if **official** DeepSeek release.
- Uncertain / underspecified claims marked **`[UNCERTAIN]`**.
- No chapter prose — notes + tables only.

**Paper count (this doc):** ~95 listed entries (some multi-cited across modules).

---

## 0. K3 Spec Snapshot (from official report)

| Spec | Value (official abstract / Table 1) |
|---|---|
| Total / activated params | 2.8T MoE / **104B** activated |
| Context | **1M** tokens |
| Attention pattern | Hybrid **3 KDA : 1 Gated MLA** per block; final layer Gated MLA; **69 KDA + 24 MLA** |
| Depth connectivity | **Block Attention Residuals** (blocks of 12 layers) |
| FFN | **Stable LatentMoE**: **16 of 896** routed experts (+ shared experts) |
| Vision | **MoonViT-V2** (~401M), trained **from scratch** with NTP |
| Optimizer | **Per-Head Muon** (+ weight clipping) |
| Positional | **NoPE** on MLA; position via KDA decay/gating |
| Deploy quant (post-train QAT) | MoE expert weights **MXFP4**, activations **MXFP8** |
| Scaling claim | ~**2.5×** overall scaling efficiency vs Kimi K2 **`[claim from authors]`** |
| Post-train pillars | Multi-effort RL · Multi-Teacher On-Policy Distillation (MOPD) · agent environments · 1M agentic RL |

---

## 1. K3 Ingredient → Research Ancestry Map

| K3 ingredient | Direct primary | Immediate ancestry | Module |
|---|---|---|---|
| **KDA / Kimi Linear** | [2510.26692](https://arxiv.org/abs/2510.26692) | Gated DeltaNet → DeltaNet → GLA → Linear Transformer | M04–M05 |
| **Gated MLA** | K3 §2.1.2 + DeepSeek-V2 MLA [2405.04434](https://arxiv.org/abs/2405.04434) | MQA/GQA → KV compression → latent KV | M03, M05 |
| **Attention Residuals** | [2603.15031](https://arxiv.org/abs/2603.15031) | ResNet residuals → PreNorm dilution problem | M02, M06 |
| **Stable LatentMoE** | K3 §2.3 + LatentMoE [2601.18089](https://arxiv.org/abs/2601.18089) | Switch/GShard → DeepSeekMoE → aux-loss-free routing → QB | M07, M09 |
| **MoonViT-V2** | K3 §2.4; prior MoonViT in Kimi-VL [2504.07491](https://arxiv.org/abs/2504.07491) | ViT → native-res VL encoders; K3 drops SigLIP init | M14 |
| **Per-head Muon** | K3 §2.5; Muon scale-up [2502.16982](https://arxiv.org/abs/2502.16982); MuonClip in K2 [2507.20534](https://arxiv.org/abs/2507.20534) | AdamW → matrix orthogonalization optimizers | M08 |
| **Scaling / data** | K3 §3; Kaplan/Hoffmann; K2 data rephrase | Scaling laws + domain mix + rephrase | M08 |
| **Long context (1M)** | K3 §3 progressive extension + KDA CP | RoPE/YaRN → Ring/Ulysses → NoPE+KDA | M11 |
| **Multi-effort RL** | K3 §4 effort ∈ {low, high, max} | o1-style test-time compute; k1.5 | M13 |
| **Multi-teacher on-policy distillation (MOPD)** | K3 §4.1.3; Thinking Machines on-policy distill | R1 distillation; on-policy KD | M13 |
| **Agent environments** | K3 §4.2 (pluggable harness, AET, knowledge graph) | Tool-use / SWE / OSWorld lineage | M14 |
| **KDA systems** | K3 §5.1 FlashKDA, KDA Context Parallelism | FLA / DeltaNet CP | M11, M15 |
| **MoonEP** | K3 §5 (perfect balance EP; vs DeepEP) | DeepEP / expert parallel | M11, M15 |
| **1M agentic RL** | K3 §5 partial rollouts, external KV, resumable sandboxes | Long-horizon RL infra | M14–M15 |
| **MXFP4** | K3 QAT §4.1.4; OCP MX [2310.10537](https://arxiv.org/abs/2310.10537) | FP8 training → microscaling | M10 |

---

## 2. Module Graph & Prerequisites

```
M01 Foundations ──► M02 Residual/Norm/Pos
       │                    │
       ▼                    ▼
M03 Softmax KV path     M04 Linear/Delta/SSM path
       │                    │
       └────────┬───────────┘
                ▼
         M05 Hybrid Attention (KDA+Gated MLA, NoPE)
                │
                ▼
         M06 Depth Connectivity (AttnRes)
                │
         M07 Sparse MoE width (Stable LatentMoE)
                │
         M08 Scale / Data / Muon
                │
         M09 DeepSeek full stack (MLA→V3→R1→V3.2→V4)
                │
    ┌───────────┼───────────┐
    ▼           ▼           ▼
  M10 Quant   M11 Long-ctx  M12 Alignment (PPO/DPO)
    │        systems        │
    └───────────┬───────────┘
                ▼
         M13 Reasoning RL + MOPD + multi-effort
                │
                ▼
         M14 Agents + Vision (MoonViT-V2)
                │
                ▼
         M15 K3 Capstone Systems (MoonEP, 1M RL, MXFP4 serve)
```

**Suggested learning order:** M01→M02→(M03∥M04)→M05→M06→M07→M08→M09→(M10∥M11∥M12)→M13→M14→M15.

---

## M01 — Foundations: Sequence Models & Softmax Attention

**Prerequisites:** undergrad ML, basic seq modeling
**Why for K3:** defines the quadratic attention baseline that KDA/MLA hybridize.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2014 | Sequence to Sequence Learning with Neural Networks | https://arxiv.org/abs/1409.3215 | End-to-end neural transduction without alignments | Need better alignment / attention |
| 2015 | Neural Machine Translation by Jointly Learning to Align and Translate | https://arxiv.org/abs/1409.0473 | Soft attention over encoder states | Attention as differentiable lookup |
| 2017 | Attention Is All You Need | https://arxiv.org/abs/1706.03762 | Drop recurrence; multi-head self-attention + PE | Canonical Transformer stack |
| 2018 | BERT: Pre-training of Deep Bidirectional Transformers | https://arxiv.org/abs/1810.04805 | Bidirectional pretrain for understanding | Pretrain–finetune paradigm |
| 2018 | Improving Language Understanding by Generative Pre-Training (GPT) | https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf | Decoder-only LM pretrain | GPT lineage |
| 2019 | Language Models are Unsupervised Multitask Learners (GPT-2) | https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf | Zero-shot transfer via scale | Scaling narrative |
| 2020 | Language Models are Few-Shot Learners (GPT-3) | https://arxiv.org/abs/2005.14165 | In-context learning at 175B | ICL as emergent interface |

---

## M02 — Residual Stacks, Normalization, Position

**Prerequisites:** M01
**Why for K3:** PreNorm residual dilution motivates **AttnRes**; NoPE choice; RMSNorm in MoonViT-V2.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2015 | Deep Residual Learning for Image Recognition | https://arxiv.org/abs/1512.03385 | Train very deep nets via identity skip | Residual highway becomes default |
| 2016 | Layer Normalization | https://arxiv.org/abs/1607.06450 | Stabilize RNN/Transformer activations | LN → PreNorm stacks |
| 2019 | Root Mean Square Layer Normalization | https://arxiv.org/abs/1910.07467 | Cheaper, stable norm without mean centering | Modern LLM default (incl. MoonViT-V2) |
| 2021 | RoFormer: Enhanced Transformer with Rotary Position Embedding | https://arxiv.org/abs/2104.09864 | Relative PE via rotations | Dominant PE until long-ctx hacks |
| 2023 | YaRN: Efficient Context Window Extension of LLMs | https://arxiv.org/abs/2309.00071 | Extend RoPE LMs beyond train length | Positional interpolation family |
| 2024 | The NoPE Hypothesis (and related NoPE studies) **`[select primary carefully; see also K3 NoPE practice]`** | https://arxiv.org/abs/2404.12224 | When explicit PE is unnecessary | K3: NoPE on MLA; position via KDA |

**K3 note:** MLA layers use **NoPE**; positional signal comes from KDA’s channel-wise decay/gating (K3 §2.1, §3).

---

## M03 — Softmax Attention Efficiency & KV Compression → MLA

**Prerequisites:** M01–M02
**Why for K3:** **Gated MLA** is the global-attention “anchor” layer in the 3:1 hybrid.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2019 | Fast Transformer Decoding: One Write-Head is All You Need (MQA) | https://arxiv.org/abs/1911.02150 | Share KV across heads → smaller cache | Multi-query attention |
| 2023 | GQA: Training Generalized Multi-Query Transformer Models | https://arxiv.org/abs/2305.13245 | Interpolate MHA↔MQA | Production GQA default |
| 2022 | FlashAttention: Fast and Memory-Efficient Exact Attention | https://arxiv.org/abs/2205.14135 | IO-aware exact attention | Hardware-aware attention kernels |
| 2023 | FlashAttention-2 | https://arxiv.org/abs/2307.08691 | Higher occupancy / better parallelism | Training/prefill baseline |
| 2023 | LongNet: Scaling Transformers to 1B Tokens (dilated attn) | https://arxiv.org/abs/2307.02486 | Sparse patterns for extreme length | Sparse attention design space |
| 2024 | DeepSeek-V2: Strong Economical Efficient MoE LM (**MLA**) | https://arxiv.org/abs/2405.04434 | Compress KV into latent vector (MLA) | Latent cache for huge MoEs |
| 2024 | DeepSeek-V3 Technical Report (MLA at 671B) | https://arxiv.org/abs/2412.19437 | Scale MLA + DeepSeekMoE production-grade | Gated MLA inherits MLA |
| 2025 | DeepSeek-V3.2 (**DSA** on MLA) | https://arxiv.org/abs/2512.02556 | Fine-grained sparse attention + lightning indexer | Sparse selection over latent KV |

**K3 Gated MLA (official):** ungated MLA output gated by full-rank \(W_g\); gate matches full-rank KDA output gate (K3 §2.1.2).

---

## M04 — Linear Attention, Delta Rule, SSMs → KDA Ancestry

**Prerequisites:** M01
**Why for K3:** **KDA** = gated delta-rule recurrence with finer channel-wise forget + full-rank output gate.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2020 | Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention | https://arxiv.org/abs/2006.16236 | Kernelize attention → linear time / RNN form | Linear Transformer baseline |
| 2021 | Linear Transformers Are Secretly Fast Weight Programmers | https://arxiv.org/abs/2102.11174 | Delta-rule / fast weights view of linear attn | **Delta rule** update idea |
| 2023 | Mamba: Linear-Time Sequence Modeling with Selective SSMs | https://arxiv.org/abs/2312.00752 | Input-dependent SSM selection | Modern linear-time competitor |
| 2024 | Gated Linear Attention Transformers (GLA) | https://arxiv.org/abs/2312.06635 | Data-dependent gates + FlashLinearAttention | Hardware-efficient gated linear attn |
| 2024 | Parallelizing Linear Transformers with the Delta Rule (DeltaNet) | https://arxiv.org/abs/2406.06484 | Parallel train delta-rule linear transformers | Chunkwise delta training |
| 2024 | Transformers are SSMs / Mamba-2 (SSD) | https://arxiv.org/abs/2405.21060 | Unify Transformer & SSM via structured duality | State-space dual algorithms |
| 2025 | Gated Delta Networks: Improving Mamba2 with Delta Rule | https://arxiv.org/abs/2412.06464 | Combine **gating + delta rule** | Direct parent of **KDA** |
| 2025 | Kimi Linear: Expressive Efficient Attention (**KDA**) | https://arxiv.org/abs/2510.26692 | Channel-wise fine-grained KDA + hybrid with MLA | Productionized in K3 (with K3-specific gates) |

**K3 KDA deltas vs Kimi Linear (official §2.1.1):** lower-bounded log-decay (tile-stable chunkwise form); **full-rank** output gate (vs low-rank in Linear).

---

## M05 — Hybrid Attention Architectures (KDA + Gated MLA + NoPE)

**Prerequisites:** M03, M04
**Why for K3:** core sequence mixer: **3×KDA + 1×Gated MLA**, final Gated MLA.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2023 | Retentive Network (RetNet) | https://arxiv.org/abs/2307.08621 | Multi-scale retention recurrence | Hybrid linear/global motivation |
| 2024 | Griffin: Mixing Gated Linear Recurrences with Local Attention | https://arxiv.org/abs/2402.19427 | Mix local attention + linear recurrence | Hybrid design pattern |
| 2024 | Jamba: Hybrid Transformer-Mamba | https://arxiv.org/abs/2403.19887 | Production hybrid MoE | Hybrid at scale |
| 2024 | DeltaNet hybrids (sliding / global layers) *in* DeltaNet paper | https://arxiv.org/abs/2406.06484 | Sparse global layers restore quality | 3:1-style interleaving idea |
| 2025 | Kimi Linear (3:1 KDA:MLA hybrid) | https://arxiv.org/abs/2510.26692 | Match/beat full MLA quality at lower KV | Direct K3 attention recipe |
| 2026 | Kimi K3 (Gated MLA + refined KDA) | https://arxiv.org/abs/2607.24653 | Scale hybrid to 2.8T / 1M ctx | Capstone hybrid |

**Reading target:** derive why periodic full latent attention repairs linear-state information bottlenecks.

---

## M06 — Depth-wise Connectivity: PreNorm Dilution → Attention Residuals

**Prerequisites:** M02, M05
**Why for K3:** **Block AttnRes** is the depth information-flow upgrade.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2015 | Deep Residual Learning (again as residual baseline) | https://arxiv.org/abs/1512.03385 | Identity mapping depth | Fixed unit residual weights |
| 2016 | Highway Networks | https://arxiv.org/abs/1505.00387 | Learned gates on residual paths | Content-dependent depth flow |
| 2017 | DenseNet | https://arxiv.org/abs/1608.06993 | Concatenate all prior feature maps | Dense cross-layer reuse |
| 2021 | DeepNet: Stabilizing Extremely Deep Transformers | https://arxiv.org/abs/2203.00555 | Scale residuals for 1000-layer Transformers | Deep PreNorm issues |
| 2026 | Attention Residuals (AttnRes / Block AttnRes) | https://arxiv.org/abs/2603.15031 | Softmax over prior layer outputs; block variant for memory | **K3 uses Block AttnRes** (S≈12) |
| 2026 | DeepSeek-V4 mHC (manifold-constrained hyper-connections) *official later residual variant* | https://arxiv.org/abs/2606.19348 | Alternative residual upgrade at 1M-ctx MoE | Parallel residual research thread |

**K3 detail:** pseudo-query \(q_l=w_l\); Block AttnRes reduces overhead \(O(Ld)\to O(Nd)\).

---

## M07 — Sparse Width: MoE → DeepSeekMoE → LatentMoE → Stable LatentMoE

**Prerequisites:** M01, M08 (scaling intuition helps)
**Why for K3:** **Stable LatentMoE** width path (16/896 + Quantile Balancing).

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2017 | Outrageously Large Neural Networks (Sparsely-Gated MoE) | https://arxiv.org/abs/1701.06538 | Conditional computation at scale | MoE modern start |
| 2020 | GShard: Scaling Giant Models with Conditional Computation | https://arxiv.org/abs/2006.16668 | Automatic sharding + MoE at TPU scale | Systems MoE |
| 2021 | Switch Transformers | https://arxiv.org/abs/2101.03961 | Simplify to top-1 routing; trillion-param | Load-balance auxiliaries |
| 2022 | ST-MoE | https://arxiv.org/abs/2202.08906 | Stable training recipes for sparse models | Stability folklore |
| 2024 | DeepSeekMoE: Towards Ultimate Expert Specialization | https://arxiv.org/abs/2401.06066 | Fine-grained experts + shared experts | DeepSeek MoE DNA |
| 2024 | Mixtral of Experts | https://arxiv.org/abs/2401.04088 | Strong open sparse LM | Sparse quality reference |
| 2024 | DeepSeek-V2 (DeepSeekMoE @ production) | https://arxiv.org/abs/2405.04434 | Economical training with sparse FFN | MLA+MoE combo |
| 2024 | DeepSeek-V3 (aux-loss-free routing) | https://arxiv.org/abs/2412.19437 | Bias-based load balance without aux loss | Aux-loss-free lineage |
| 2026 | LatentMoE: Optimal Accuracy per FLOP/Parameter | https://arxiv.org/abs/2601.18089 | Route in latent cheaper expert path | Namesake of **LatentMoE** |
| 2026 | Kimi K3 Stable LatentMoE + **Quantile Balancing** | https://arxiv.org/abs/2607.24653 | Stabilize extreme sparsity (896 experts, k=16) | Production K3 FFN |

**K3 QB (official):** expert bias from margin quantiles via global histogram all-reduce (not token-side exact quantiles).
**Also cite:** Jianlin Su blog *Travels in MoE* (load balance via optimal assignment) — referenced by K3 as [111] (Chinese primary blog, not arXiv).

---

## M08 — Scaling Laws, Data Recipes, Optimizers (→ Per-Head Muon)

**Prerequisites:** M01
**Why for K3:** 2.5× scaling-efficiency claim; Per-Head Muon; cosine > WSD in their search.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2020 | Scaling Laws for Neural Language Models (Kaplan et al.) | https://arxiv.org/abs/2001.08361 | Power-law loss vs N, D, C | Classical scaling |
| 2022 | Training Compute-Optimal LLMs (Chinchilla / Hoffmann) | https://arxiv.org/abs/2203.15556 | Tokens–params balance | TPP retuning practice |
| 2014 | Adam: A Method for Stochastic Optimization | https://arxiv.org/abs/1412.6980 | Adaptive moments | Default baseline |
| 2017 | Decoupled Weight Decay Regularization (AdamW) | https://arxiv.org/abs/1711.05101 | Correct weight decay | Pre-Muon default |
| 2024 | Muon: An Optimizer for Hidden Layers (Keller Jordan et al.) | https://kellerjordan.github.io/posts/muon/ | Newton–Schulz orthogonalized momentum | Matrix-wise updates |
| 2025 | Muon is Scalable for LLM Training (Moonshot / Moonlight) | https://arxiv.org/abs/2502.16982 | Weight decay + update scale → Muon at LLM scale | Moonshot optimizer stack |
| 2025 | Kimi K2 (MuonClip / QK-Clip) | https://arxiv.org/abs/2507.20534 | Stabilize attention logits under Muon | Pretrain 15.5T zero spike claim |
| 2026 | Kimi K3 Per-Head Muon | https://arxiv.org/abs/2607.24653 | Orthogonalize QKV **per head** for balanced head scales | K3 default optimizer |

**Data / schedule landmarks**

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2020 | Exploring the Limits of Transfer Learning (T5) | https://arxiv.org/abs/1910.10683 | Unified text-to-text + C4 data | Data quality culture |
| 2022 | Training Compute-Optimal… (data side of Chinchilla) | https://arxiv.org/abs/2203.15556 | More tokens for given compute | Overtrain regimes |
| 2024 | DeepSeek LLM: Scaling Open-Source LMs with Longtermism | https://arxiv.org/abs/2401.02954 | Open scaling-law study + 67B | DeepSeek program start |
| 2025 | MiniCPM / WSD schedule discussion | https://arxiv.org/abs/2404.06395 | Warmup–Stable–Decay alternative | K3 finds **cosine better** after independent HPO |
| 2025 | Kimi K2 data rephrasing recipe | https://arxiv.org/abs/2507.20534 | Knowledge/math rephrase + fidelity checks | Inherited by K3 §3.1 |

---

## M09 — DeepSeek Lineage (Dense → MoE → MLA → V3 → R1 → V3.2 → V4)

**Prerequisites:** M03, M07, M08, M12–M13 for post-train pieces
**Policy:** include only **official** DeepSeek papers/reports.

### 9.1 Pretrain architecture stack

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2024 | DeepSeek LLM: Scaling Open-Source Language Models with Longtermism | https://arxiv.org/abs/2401.02954 | Open dense 7B/67B + scaling study | Foundation |
| 2024 | DeepSeekMoE: Ultimate Expert Specialization | https://arxiv.org/abs/2401.06066 | Fine-grained + shared experts | Sparse specialization |
| 2024 | DeepSeek-V2: MLA + DeepSeekMoE | https://arxiv.org/abs/2405.04434 | KV latent compression + economical MoE | **MLA birth** |
| 2024 | DeepSeek-V3 Technical Report | https://arxiv.org/abs/2412.19437 | 671B/37B act; **FP8**; **MTP**; **aux-loss-free routing**; **DualPipe** | Flagship pretrain stack |
| 2025 | DeepSeek-V3.2: DSA + scalable RL + agentic synthesis | https://arxiv.org/abs/2512.02556 | **DeepSeek Sparse Attention**; agent post-train scale | Long-ctx efficiency |
| 2026 | DeepSeek-V4: Million-Token Context Intelligence | https://arxiv.org/abs/2606.19348 | CSA+HCA hybrid attn; **mHC**; Muon; 1M ctx Pro/Flash | Official later 1M-ctx peer to K3 |

### 9.2 Math, RL, reasoning distillation

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2024 | DeepSeekMath (+ **GRPO**) | https://arxiv.org/abs/2402.03300 | Math continual pretrain + critic-free group RL | **GRPO** algorithm |
| 2025 | DeepSeek-R1: Incentivizing Reasoning via RL | https://arxiv.org/abs/2501.12948 | **R1-Zero** pure RL; **R1** multi-stage; **distillation** to dense 1.5B–70B | Reasoning RL template |
| 2025 | DeepSeek-R1 Nature version (same line) | https://doi.org/10.1038/s41586-025-09422-z | Peer-reviewed presentation of R1 | Archival citation |

### 9.3 V3 systems keywords → primary sections

| Keyword | Where (primary) | Problem solved | Bridge |
|---|---|---|---|
| **FP8 mixed precision** | V3 report §3.3 | Low-precision train at 671B | → MX / QAT serving |
| **MTP** (multi-token prediction) | V3 report | Stronger pretrain objective / draft head | Speculative / denser signal |
| **Aux-loss-free routing** | V3 report | Balance without aux loss interference | → K3 QB different mechanism |
| **DualPipe** | V3 report §3.2.1 | Overlap PP bubbles + MoE dispatch/combine | → MoonEP alternative EP design |
| **DSA** | V3.2 report | Sparse token selection under MLA | Contrast with KDA hybrid |
| **DeepEP** | https://github.com/deepseek-ai/DeepEP | Expert-parallel comm library | K3 MoonEP builds on / contrasts |

**Atlas teaching note:** treat DeepSeek as the *dense reference stack* for MLA/MoE/FP8/RL; treat K3 as the *hybrid linear + AttnRes + Stable LatentMoE* fork at larger sparsity and 1M agentic RL.

---

## M10 — Low-Precision Training & Deployment (FP8 → MXFP4)

**Prerequisites:** M08–M09
**Why for K3:** post-train **QAT** with **MXFP4 weights / MXFP8 activations** on MoE experts.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2018 | Quantization and Training of Neural Networks for Integer-Arithmetic-Only Inference | https://arxiv.org/abs/1712.05877 | QAT for int inference | QAT paradigm |
| 2020 | Training with Quantization Noise | https://arxiv.org/abs/2004.07320 | Noise injection for robust quant | QAT variants |
| 2022 | FP8 Formats for Deep Learning (Micikevicius et al.) | https://arxiv.org/abs/2209.05433 | FP8 train/infer formats | Hardware FP8 |
| 2023 | Microscaling Data Formats for Deep Learning (**MX / MXFP4**) | https://arxiv.org/abs/2310.10537 | Block scales + narrow element types | **OCP MX** family |
| 2024 | DeepSeek-V3 FP8 training framework | https://arxiv.org/abs/2412.19437 | Validate FP8 at extreme MoE scale | Pretrain low-prec |
| 2026 | Kimi K3 MXFP4 QAT from SFT onward | https://arxiv.org/abs/2607.24653 | Deploy-time memory for 2.8T experts | Serving recipe |

**OCP primary (non-arXiv):** OCP Microscaling Formats (MX) Specification v1.0 — https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf

---

## M11 — Long Context: Algorithms & Parallelism (→ KDA CP, 1M)

**Prerequisites:** M03–M05
**Why for K3:** progressive 8K→64K→256K→**1M**; **KDA Context Parallelism**; state-aware prefix cache.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2023 | ALiBi | https://arxiv.org/abs/2108.12409 | Length extrapolation via linear biases | PE-free-ish long ctx |
| 2023 | RoPE / YaRN (see M02) | https://arxiv.org/abs/2309.00071 | Extend rotary models | Common production path |
| 2023 | Ring Attention with Blockwise Transformers | https://arxiv.org/abs/2310.01889 | Sequence parallel for near-infinite context | Distributed long attn |
| 2023 | DeepSpeed Ulysses | https://arxiv.org/abs/2309.14509 | Sequence parallelism system opts | Cluster long-ctx train |
| 2024 | Linear Attention Sequence Parallelism (LASP) | https://arxiv.org/abs/2404.02882 | SP specialized for linear attn | Linear-SP ancestors |
| 2025 | LASP-2 / hybrid linear SP | https://arxiv.org/abs/2502.07864 **`[verify id if citing; K3 cites related]`** | Hybrid linear+softmax SP | Closer to KDA hybrids |
| 2025 | Context Parallelism for DeltaNet (Wang) | https://yywangcs.notion.site/DeltaNet-2a9fc9f5d8058013a498f34e0b25bd52 | CP for delta recurrence | Direct ancestor of **KCP** |
| 2026 | Kimi K3 KDA Context Parallelism + FlashKDA | https://arxiv.org/abs/2607.24653 · FlashKDA https://github.com/MoonshotAI/FlashKDA | Correct state transport under delta update \(M_t S_{t-1}\) | 1M train/prefill |

**K3 progressive extension (official):** pretrain grows 8K→64K; cooldown 256K→1M; needle/synthetic scatter tasks force full-window use.

---

## M12 — Alignment Foundations (RLHF / PPO / DPO)

**Prerequisites:** M01
**Why for K3:** base of modern post-train before GRPO / multi-effort / MOPD.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2017 | Proximal Policy Optimization Algorithms | https://arxiv.org/abs/1707.06347 | Stable policy gradient updates | RL workhorse |
| 2022 | Training Language Models to Follow Instructions with Human Feedback (InstructGPT) | https://arxiv.org/abs/2203.02155 | RLHF pipeline for LLMs | Industry alignment template |
| 2022 | Constitutional AI | https://arxiv.org/abs/2212.08073 | Principle-based AI feedback | RLAIF direction |
| 2023 | Direct Preference Optimization (DPO) | https://arxiv.org/abs/2305.18290 | Preference learning without RL loop | Offline preference |
| 2023 | Llama 2: Open Foundation and Fine-Tuned Chat Models | https://arxiv.org/abs/2307.09288 | Open RLHF stack details | Open alignment recipes |
| 2024 | SimPO / ORPO family (optional shortlist) | https://arxiv.org/abs/2405.14734 | Simpler preference objectives | Alt to DPO |

---

## M13 — Reasoning RL, Multi-Effort, On-Policy Distillation

**Prerequisites:** M12, M09
**Why for K3:** multi-domain RL × effort levels; **MOPD** consolidation.

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2022 | Chain-of-Thought Prompting | https://arxiv.org/abs/2201.11903 | Elicit intermediate reasoning | Test-time reasoning culture |
| 2023 | Let’s Verify Step by Step (process reward) | https://arxiv.org/abs/2305.20050 | Process vs outcome supervision | PRM path |
| 2024 | DeepSeekMath / **GRPO** | https://arxiv.org/abs/2402.03300 | Group-relative baseline, no critic | Memory-efficient RL |
| 2024 | OpenAI o1 announcement / “Learning to Reason with LLMs” | https://openai.com/index/learning-to-reason-with-llms/ | Scale RL + test-time compute | Multi-effort ancestor **`[blog primary; not full paper]`** |
| 2025 | Kimi k1.5: Scaling RL with LLMs | https://arxiv.org/abs/2501.12599 | Long-CoT RL scaling (Moonshot) | Kimi reasoning line |
| 2025 | DeepSeek-R1 / R1-Zero / distillation | https://arxiv.org/abs/2501.12948 | Pure RL emergence; then multi-stage + distill | Distill reasoning patterns |
| 2025 | On-policy distillation (Thinking Machines Lab) | https://thinkingmachines.ai/blog/on-policy-distillation/ **`[blog; K3 cites as Connectionism note]`** | Student on-policy w.r.t teacher | **MOPD** conceptual parent |
| 2026 | MiMo-V2-Flash Technical Report (multi-teacher mention lineage) | https://arxiv.org/abs/2601.02780 | Related multi-teacher post-train | Peer system cited by K3 |
| 2026 | Kimi K3 multi-effort RL + **MOPD** | https://arxiv.org/abs/2607.24653 | Domain experts {general, agentic, coding} × {low, high, max} → single policy | Capstone post-train |

**K3 MOPD (official):** on-policy student samples; multi-teacher logits/rewards consolidate specialists; top-k distill ablations showed no clear gain.

---

## M14 — Agents, Environments, Multimodal (MoonViT-V2)

**Prerequisites:** M13, M05
**Why for K3:** agent harness modularization; AET; native vision-in-the-loop; MoonViT-V2 from scratch.

### 14.1 Agent / tool / coding environments

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2023 | Toolformer | https://arxiv.org/abs/2302.04761 | Self-supervised tool API calls | Tool-use pretrain |
| 2023 | ReAct | https://arxiv.org/abs/2210.03629 | Interleave reason + act | Agent loop pattern |
| 2024 | SWE-bench | https://arxiv.org/abs/2310.06770 | Real GitHub issue resolution | Coding agent benchmark |
| 2024 | OSWorld | https://arxiv.org/abs/2404.07972 | Computer-use agents | GUI agent env |
| 2025 | Terminal-Bench | https://arxiv.org/abs/2502.14045 **`[confirm version K3 cites]`** | Hard CLI agent tasks | Terminal agents |
| 2025 | BrowseComp | https://arxiv.org/abs/2504.12516 | Browsing agents | Web agents |
| 2025 | Kimi K2 agentic data synthesis + joint RL | https://arxiv.org/abs/2507.20534 | Large synthetic tool trajectories | Kimi agent stack |
| 2026 | Kimi K2.5 Visual Agentic Intelligence | https://arxiv.org/abs/2602.02276 | Multimodal agents / swarm **`[swarm details in K2.5]`** | Vision agents |
| 2026 | Kimi K3 agent envs (pluggable harness, AET, knowledge graph) | https://arxiv.org/abs/2607.24653 | Avoid harness overfitting; long-horizon verifiers | Production agent RL data |

### 14.2 Vision encoders → MoonViT-V2

| Year | Title | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2020 | An Image is Worth 16x16 Words (ViT) | https://arxiv.org/abs/2010.11929 | Transformer vision backbone | ViT era |
| 2021 | Learning Transferable Visual Models From Natural Language Supervision (CLIP) | https://arxiv.org/abs/2103.00020 | Contrastive vision–language pretrain | SigLIP-style inits |
| 2023 | SigLIP | https://arxiv.org/abs/2303.15343 | Sigmoid contrastive VL | Prior MoonViT init |
| 2024 | LLaVA-OneVision | https://arxiv.org/abs/2408.03326 | Unified single/multi-image/video tasks | Native-res VL practice |
| 2025 | Kimi-VL Technical Report (**MoonViT**) | https://arxiv.org/abs/2504.07491 | Native-resolution MoonViT + MoE LM | Moonshot VL stack |
| 2026 | Kimi K3 **MoonViT-V2** from-scratch NTP | https://arxiv.org/abs/2607.24653 | Drop SigLIP init for stability; match quality | K3 vision path |

**K3 vision claim (official):** MoonViT-V2 ≈0.4B, 27 layers, RMSNorm, bias-free; shared image/video; 2×2 pixel-shuffle; up to 3584² in 1M context.

---

## M15 — Capstone Systems: KDA Kernels, MoonEP, 1M Agentic RL, Serve

**Prerequisites:** M05–M07, M10–M11, M13–M14
**Why for K3:** makes 2.8T hybrid + 1M agentic RL trainable and deployable.

| Year | Title / artifact | URL | Problem solved | Bridge to next |
|---|---|---|---|---|
| 2019 | Triton: Intermediate Language for Tiled NN Kernels | https://www.eecs.harvard.edu/~htk/publication/2019-mapl-tillet-kung-cox.pdf | Productive GPU kernels | FLA / custom attn |
| 2021 | GPipe | https://arxiv.org/abs/1811.06965 | Pipeline parallel | PP lineage |
| 2021 | Megatron-LM / efficient large-scale training | https://arxiv.org/abs/2104.04473 | 3D parallelism | Pretrain systems baseline |
| 2020 | ZeRO | https://arxiv.org/abs/1910.02054 | Shard optimizer states | Memory for huge models |
| 2024 | FLA: Flash Linear Attention library | https://github.com/fla-org/flash-linear-attention | Kernel zoo for linear/delta | KDA ops host |
| 2025 | DeepEP | https://github.com/deepseek-ai/DeepEP | Expert-parallel communication | MoonEP contrast |
| 2025 | DualPipe (in V3 report) | https://arxiv.org/abs/2412.19437 | Overlap MoE+PP | Pipeline bubbles |
| 2026 | FlashKDA | https://github.com/MoonshotAI/FlashKDA | CUTLASS chunkwise KDA | Train/prefill speed |
| 2026 | Kimi K3 **MoonEP** | https://arxiv.org/abs/2607.24653 | Perfect balance EP, static shapes, zero-copy, bounded redundant experts | 2.8T MoE train |
| 2026 | Kimi K3 1M agentic RL co-located system | https://arxiv.org/abs/2607.24653 | Partial rollouts, external KV retention, resumable sandboxes | Long-horizon RL |
| 2026 | Kimi K3 state-aware KDA prefix caching + MXFP4 serve | https://arxiv.org/abs/2607.24653 | Decode across hybrid state + quant experts | Production deployment |

---

## 3. Module → K3 Ingredient Checklist

| Module | Delivers toward K3 |
|---|---|
| M01–M02 | Transformer residual + PE literacy |
| M03 | MLA / Gated MLA |
| M04–M05 | KDA + hybrid 3:1 |
| M06 | Attention Residuals |
| M07 | Stable LatentMoE + QB vs aux-loss-free |
| M08 | Scaling 2.5× story + Per-Head Muon |
| M09 | DeepSeek dense comparator (MLA, FP8, MTP, DualPipe, GRPO, R1, DSA, V4) |
| M10 | MXFP4/MXFP8 QAT |
| M11 | 1M context + KCP |
| M12–M13 | Multi-effort RL + MOPD |
| M14 | Agent envs + MoonViT-V2 |
| M15 | MoonEP + 1M agentic RL systems |

---

## 4. DeepSeek Coverage Matrix (Atlas “strong track”)

| Topic | Primary | Module | Must-read density |
|---|---|---|---|
| DeepSeek LLM | 2401.02954 | M08/M09 | high |
| DeepSeekMoE | 2401.06066 | M07/M09 | high |
| V2 / MLA | 2405.04434 | M03/M09 | **core** |
| Math / GRPO | 2402.03300 | M13/M09 | **core** |
| V3 FP8 / MTP / aux-free / DualPipe | 2412.19437 | M09/M10/M15 | **core** |
| R1 / R1-Zero / distill | 2501.12948 (+ Nature DOI) | M13/M09 | **core** |
| V3.2 / DSA | 2512.02556 | M03/M09 | high |
| V4 (official later) | 2606.19348 | M06/M09/M11 | high (1M peer) |
| DeepEP | GitHub deepseek-ai/DeepEP | M15 | systems |

---

## 5. Uncertain / Careful Claims

| Claim | Status |
|---|---|
| K3 **2.5×** scaling efficiency vs K2 | Author-reported fit on OOD val curves (Fig. 7); **not independent replication** |
| Benchmarks vs Claude Fable 5 / GPT-5.6 Sol | Author suite; **external harness variance** possible |
| “First open 3T-class” marketing phrasing | Product language; verify total-param definitions (shared vs routed, embeddings, vision) |
| Exact MoonEP algorithm vs DeepEP | Described in K3 §5; **no standalone MoonEP paper** at time of roadmap |
| FlashKDA paper | Primarily **code artifact** + K3 description; limited standalone theory paper |
| LASP-2 arXiv number if used in syllabus | Double-check id before publishing student links |
| On-policy distillation “Connectionism” note | Blog-level primary; treat as concept source, not peer-reviewed algorithm paper |
| DeepSeek-V4 CSA/HCA vs K3 KDA | Parallel 1M-ctx designs; **do not equate** without side-by-side study |
| MoonViT-V2 “matches SigLIP init quality” | Author ablation claim (Fig. 6 + text) |
| Multi-effort exact reward formulas | Partially specified (thinking-token thresholds); full reward suite may be incomplete in report |

---

## 6. Compact Reading Paths (for site UX)

### Path A — “Understand K3 architecture in 12 papers”
1. Attention Is All You Need
2. FlashAttention-2
3. DeepSeek-V2 (MLA)
4. Gated DeltaNet
5. Kimi Linear (KDA)
6. Attention Residuals
7. DeepSeekMoE
8. LatentMoE
9. Muon is Scalable
10. Microscaling (MXFP4)
11. DeepSeek-V3 (systems)
12. **Kimi K3**

### Path B — “DeepSeek full stack (official only)”
DeepSeek LLM → DeepSeekMoE → DeepSeekMath/GRPO → V2/MLA → V3 → R1 → V3.2/DSA → V4

### Path C — “Post-train / agents”
InstructGPT → DPO → GRPO → R1 → k1.5 → K2 → K2.5 → K3 (MOPD + multi-effort + 1M agentic RL)

---

## 7. Suggested Atlas Module Metadata (CMS fields)

```yaml
module_id: M05
title_zh: 混合注意力：KDA 与 Gated MLA
prerequisites: [M03, M04]
k3_tags: [KDA, GatedMLA, NoPE, hybrid-3to1]
primary_papers: 6
estimated_hours: 8
```

---

## 8. Source Log (what this roadmap used)

| Source | Role |
|---|---|
| arXiv:2607.24653 K3 PDF text extract | Architecture, post-train, systems, bibliography |
| arXiv abstracts for DeepSeek V2/V3/V3.2/V4, R1, Math, MoE, LLM | Official lineage |
| arXiv:2510.26692 Kimi Linear | KDA definition |
| arXiv:2603.15031 AttnRes | Depth residual redesign |
| arXiv:2502.16982 / 2507.20534 | Muon / K2 |
| arXiv:2504.07491 / 2602.02276 | MoonViT / K2.5 |
| arXiv:2310.10537 + OCP MX | MXFP4 |
| arXiv:2601.18089 LatentMoE | Latent MoE naming/ancestry |

---

*End of structured research notes — LLM Atlas / K3-anchored roadmap.*