wuyang/llm-atlas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
llm-atlas/research/GROK_RESEARCH_ROADMAP.md
T
wuyang 7d5c5e4c54 feat: launch LLM Atlas research course
2026-07-28 21:55:32 +08:00

36 KiB
Raw Permalink Blame History

LLM Atlas — Grok 辅助检索路线（K3-Anchored）

生成来源：本机 Grok CLI Headless 模式，2026-07-28。 用途：扩展论文线索与检查遗漏，不直接作为正文证据。主代理需回到每篇一手来源核验后，才能把结论写入网站。

Site: Chinese educational site LLM Atlas Anchor primary source: Kimi Team, Kimi K3: Open Frontier Intelligence, arXiv:2607.24653 (2026-07) · DOI:10.48550/arXiv.2607.24653 Companion official sources: Kimi Linear · Attention Residuals · Kimi K2 · Kimi K2.5 · Kimi-VL / MoonViT · Muon is Scalable

Scope rules

  • Landmark primary papers only; year | title | canonical arXiv/DOI | problem solved | bridge to next idea.
  • ≥80 papers; DeepSeek lineage given dense coverage; post-V3.2 only if official DeepSeek release.
  • Uncertain / underspecified claims marked [UNCERTAIN].
  • No chapter prose — notes + tables only.

Paper count (this doc): ~95 listed entries (some multi-cited across modules).

0. K3 Spec Snapshot (from official report)

Spec Value (official abstract / Table 1)
Total / activated params 2.8T MoE / 104B activated
Context 1M tokens
Attention pattern Hybrid 3 KDA : 1 Gated MLA per block; final layer Gated MLA; 69 KDA + 24 MLA
Depth connectivity Block Attention Residuals (blocks of 12 layers)
FFN Stable LatentMoE: 16 of 896 routed experts (+ shared experts)
Vision MoonViT-V2 (~401M), trained from scratch with NTP
Optimizer Per-Head Muon (+ weight clipping)
Positional NoPE on MLA; position via KDA decay/gating
Deploy quant (post-train QAT) MoE expert weights MXFP4, activations MXFP8
Scaling claim ~2.5× overall scaling efficiency vs Kimi K2 [claim from authors]
Post-train pillars Multi-effort RL · Multi-Teacher On-Policy Distillation (MOPD) · agent environments · 1M agentic RL

1. K3 Ingredient → Research Ancestry Map

K3 ingredient Direct primary Immediate ancestry Module
KDA / Kimi Linear 2510.26692 Gated DeltaNet → DeltaNet → GLA → Linear Transformer M04–M05
Gated MLA K3 §2.1.2 + DeepSeek-V2 MLA 2405.04434 MQA/GQA → KV compression → latent KV M03, M05
Attention Residuals 2603.15031 ResNet residuals → PreNorm dilution problem M02, M06
Stable LatentMoE K3 §2.3 + LatentMoE 2601.18089 Switch/GShard → DeepSeekMoE → aux-loss-free routing → QB M07, M09
MoonViT-V2 K3 §2.4; prior MoonViT in Kimi-VL 2504.07491 ViT → native-res VL encoders; K3 drops SigLIP init M14
Per-head Muon K3 §2.5; Muon scale-up 2502.16982; MuonClip in K2 2507.20534 AdamW → matrix orthogonalization optimizers M08
Scaling / data K3 §3; Kaplan/Hoffmann; K2 data rephrase Scaling laws + domain mix + rephrase M08
Long context (1M) K3 §3 progressive extension + KDA CP RoPE/YaRN → Ring/Ulysses → NoPE+KDA M11
Multi-effort RL K3 §4 effort ∈ {low, high, max} o1-style test-time compute; k1.5 M13
Multi-teacher on-policy distillation (MOPD) K3 §4.1.3; Thinking Machines on-policy distill R1 distillation; on-policy KD M13
Agent environments K3 §4.2 (pluggable harness, AET, knowledge graph) Tool-use / SWE / OSWorld lineage M14
KDA systems K3 §5.1 FlashKDA, KDA Context Parallelism FLA / DeltaNet CP M11, M15
MoonEP K3 §5 (perfect balance EP; vs DeepEP) DeepEP / expert parallel M11, M15
1M agentic RL K3 §5 partial rollouts, external KV, resumable sandboxes Long-horizon RL infra M14–M15
MXFP4 K3 QAT §4.1.4; OCP MX 2310.10537 FP8 training → microscaling M10

2. Module Graph & Prerequisites

M01 Foundations ──► M02 Residual/Norm/Pos
       │                    │
       ▼                    ▼
M03 Softmax KV path     M04 Linear/Delta/SSM path
       │                    │
       └────────┬───────────┘
                ▼
         M05 Hybrid Attention (KDA+Gated MLA, NoPE)
                │
                ▼
         M06 Depth Connectivity (AttnRes)
                │
         M07 Sparse MoE width (Stable LatentMoE)
                │
         M08 Scale / Data / Muon
                │
         M09 DeepSeek full stack (MLA→V3→R1→V3.2→V4)
                │
    ┌───────────┼───────────┐
    ▼           ▼           ▼
  M10 Quant   M11 Long-ctx  M12 Alignment (PPO/DPO)
    │        systems        │
    └───────────┬───────────┘
                ▼
         M13 Reasoning RL + MOPD + multi-effort
                │
                ▼
         M14 Agents + Vision (MoonViT-V2)
                │
                ▼
         M15 K3 Capstone Systems (MoonEP, 1M RL, MXFP4 serve)

Suggested learning order: M01→M02→(M03∥M04)→M05→M06→M07→M08→M09→(M10∥M11∥M12)→M13→M14→M15.

M01 — Foundations: Sequence Models & Softmax Attention

Prerequisites: undergrad ML, basic seq modeling Why for K3: defines the quadratic attention baseline that KDA/MLA hybridize.

Year Title URL Problem solved Bridge to next
2014 Sequence to Sequence Learning with Neural Networks https://arxiv.org/abs/1409.3215 End-to-end neural transduction without alignments Need better alignment / attention
2015 Neural Machine Translation by Jointly Learning to Align and Translate https://arxiv.org/abs/1409.0473 Soft attention over encoder states Attention as differentiable lookup
2017 Attention Is All You Need https://arxiv.org/abs/1706.03762 Drop recurrence; multi-head self-attention + PE Canonical Transformer stack
2018 BERT: Pre-training of Deep Bidirectional Transformers https://arxiv.org/abs/1810.04805 Bidirectional pretrain for understanding Pretrain–finetune paradigm
2018 Improving Language Understanding by Generative Pre-Training (GPT) https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf Decoder-only LM pretrain GPT lineage
2019 Language Models are Unsupervised Multitask Learners (GPT-2) https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Zero-shot transfer via scale Scaling narrative
2020 Language Models are Few-Shot Learners (GPT-3) https://arxiv.org/abs/2005.14165 In-context learning at 175B ICL as emergent interface

M02 — Residual Stacks, Normalization, Position

Prerequisites: M01 Why for K3: PreNorm residual dilution motivates AttnRes; NoPE choice; RMSNorm in MoonViT-V2.

Year Title URL Problem solved Bridge to next
2015 Deep Residual Learning for Image Recognition https://arxiv.org/abs/1512.03385 Train very deep nets via identity skip Residual highway becomes default
2016 Layer Normalization https://arxiv.org/abs/1607.06450 Stabilize RNN/Transformer activations LN → PreNorm stacks
2019 Root Mean Square Layer Normalization https://arxiv.org/abs/1910.07467 Cheaper, stable norm without mean centering Modern LLM default (incl. MoonViT-V2)
2021 RoFormer: Enhanced Transformer with Rotary Position Embedding https://arxiv.org/abs/2104.09864 Relative PE via rotations Dominant PE until long-ctx hacks
2023 YaRN: Efficient Context Window Extension of LLMs https://arxiv.org/abs/2309.00071 Extend RoPE LMs beyond train length Positional interpolation family
2024 The NoPE Hypothesis (and related NoPE studies) [select primary carefully; see also K3 NoPE practice] https://arxiv.org/abs/2404.12224 When explicit PE is unnecessary K3: NoPE on MLA; position via KDA

K3 note: MLA layers use NoPE; positional signal comes from KDA’s channel-wise decay/gating (K3 §2.1, §3).

M03 — Softmax Attention Efficiency & KV Compression → MLA

Prerequisites: M01–M02 Why for K3: Gated MLA is the global-attention “anchor” layer in the 3:1 hybrid.

Year Title URL Problem solved Bridge to next
2019 Fast Transformer Decoding: One Write-Head is All You Need (MQA) https://arxiv.org/abs/1911.02150 Share KV across heads → smaller cache Multi-query attention
2023 GQA: Training Generalized Multi-Query Transformer Models https://arxiv.org/abs/2305.13245 Interpolate MHA↔MQA Production GQA default
2022 FlashAttention: Fast and Memory-Efficient Exact Attention https://arxiv.org/abs/2205.14135 IO-aware exact attention Hardware-aware attention kernels
2023 FlashAttention-2 https://arxiv.org/abs/2307.08691 Higher occupancy / better parallelism Training/prefill baseline
2023 LongNet: Scaling Transformers to 1B Tokens (dilated attn) https://arxiv.org/abs/2307.02486 Sparse patterns for extreme length Sparse attention design space
2024 DeepSeek-V2: Strong Economical Efficient MoE LM (MLA) https://arxiv.org/abs/2405.04434 Compress KV into latent vector (MLA) Latent cache for huge MoEs
2024 DeepSeek-V3 Technical Report (MLA at 671B) https://arxiv.org/abs/2412.19437 Scale MLA + DeepSeekMoE production-grade Gated MLA inherits MLA
2025 DeepSeek-V3.2 (DSA on MLA) https://arxiv.org/abs/2512.02556 Fine-grained sparse attention + lightning indexer Sparse selection over latent KV

K3 Gated MLA (official): ungated MLA output gated by full-rank (W_g); gate matches full-rank KDA output gate (K3 §2.1.2).

M04 — Linear Attention, Delta Rule, SSMs → KDA Ancestry

Prerequisites: M01 Why for K3: KDA = gated delta-rule recurrence with finer channel-wise forget + full-rank output gate.

Year Title URL Problem solved Bridge to next
2020 Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention https://arxiv.org/abs/2006.16236 Kernelize attention → linear time / RNN form Linear Transformer baseline
2021 Linear Transformers Are Secretly Fast Weight Programmers https://arxiv.org/abs/2102.11174 Delta-rule / fast weights view of linear attn Delta rule update idea
2023 Mamba: Linear-Time Sequence Modeling with Selective SSMs https://arxiv.org/abs/2312.00752 Input-dependent SSM selection Modern linear-time competitor
2024 Gated Linear Attention Transformers (GLA) https://arxiv.org/abs/2312.06635 Data-dependent gates + FlashLinearAttention Hardware-efficient gated linear attn
2024 Parallelizing Linear Transformers with the Delta Rule (DeltaNet) https://arxiv.org/abs/2406.06484 Parallel train delta-rule linear transformers Chunkwise delta training
2024 Transformers are SSMs / Mamba-2 (SSD) https://arxiv.org/abs/2405.21060 Unify Transformer & SSM via structured duality State-space dual algorithms
2025 Gated Delta Networks: Improving Mamba2 with Delta Rule https://arxiv.org/abs/2412.06464 Combine gating + delta rule Direct parent of KDA
2025 Kimi Linear: Expressive Efficient Attention (KDA) https://arxiv.org/abs/2510.26692 Channel-wise fine-grained KDA + hybrid with MLA Productionized in K3 (with K3-specific gates)

K3 KDA deltas vs Kimi Linear (official §2.1.1): lower-bounded log-decay (tile-stable chunkwise form); full-rank output gate (vs low-rank in Linear).

M05 — Hybrid Attention Architectures (KDA + Gated MLA + NoPE)

Prerequisites: M03, M04 Why for K3: core sequence mixer: 3×KDA + 1×Gated MLA, final Gated MLA.

Year Title URL Problem solved Bridge to next
2023 Retentive Network (RetNet) https://arxiv.org/abs/2307.08621 Multi-scale retention recurrence Hybrid linear/global motivation
2024 Griffin: Mixing Gated Linear Recurrences with Local Attention https://arxiv.org/abs/2402.19427 Mix local attention + linear recurrence Hybrid design pattern
2024 Jamba: Hybrid Transformer-Mamba https://arxiv.org/abs/2403.19887 Production hybrid MoE Hybrid at scale
2024 DeltaNet hybrids (sliding / global layers) in DeltaNet paper https://arxiv.org/abs/2406.06484 Sparse global layers restore quality 3:1-style interleaving idea
2025 Kimi Linear (3:1 KDA:MLA hybrid) https://arxiv.org/abs/2510.26692 Match/beat full MLA quality at lower KV Direct K3 attention recipe
2026 Kimi K3 (Gated MLA + refined KDA) https://arxiv.org/abs/2607.24653 Scale hybrid to 2.8T / 1M ctx Capstone hybrid

Reading target: derive why periodic full latent attention repairs linear-state information bottlenecks.

M06 — Depth-wise Connectivity: PreNorm Dilution → Attention Residuals

Prerequisites: M02, M05 Why for K3: Block AttnRes is the depth information-flow upgrade.

Year Title URL Problem solved Bridge to next
2015 Deep Residual Learning (again as residual baseline) https://arxiv.org/abs/1512.03385 Identity mapping depth Fixed unit residual weights
2016 Highway Networks https://arxiv.org/abs/1505.00387 Learned gates on residual paths Content-dependent depth flow
2017 DenseNet https://arxiv.org/abs/1608.06993 Concatenate all prior feature maps Dense cross-layer reuse
2021 DeepNet: Stabilizing Extremely Deep Transformers https://arxiv.org/abs/2203.00555 Scale residuals for 1000-layer Transformers Deep PreNorm issues
2026 Attention Residuals (AttnRes / Block AttnRes) https://arxiv.org/abs/2603.15031 Softmax over prior layer outputs; block variant for memory K3 uses Block AttnRes (S≈12)
2026 DeepSeek-V4 mHC (manifold-constrained hyper-connections) official later residual variant https://arxiv.org/abs/2606.19348 Alternative residual upgrade at 1M-ctx MoE Parallel residual research thread

K3 detail: pseudo-query (q_l=w_l); Block AttnRes reduces overhead (O(Ld)\to O(Nd)).

M07 — Sparse Width: MoE → DeepSeekMoE → LatentMoE → Stable LatentMoE

Prerequisites: M01, M08 (scaling intuition helps) Why for K3: Stable LatentMoE width path (16/896 + Quantile Balancing).

Year Title URL Problem solved Bridge to next
2017 Outrageously Large Neural Networks (Sparsely-Gated MoE) https://arxiv.org/abs/1701.06538 Conditional computation at scale MoE modern start
2020 GShard: Scaling Giant Models with Conditional Computation https://arxiv.org/abs/2006.16668 Automatic sharding + MoE at TPU scale Systems MoE
2021 Switch Transformers https://arxiv.org/abs/2101.03961 Simplify to top-1 routing; trillion-param Load-balance auxiliaries
2022 ST-MoE https://arxiv.org/abs/2202.08906 Stable training recipes for sparse models Stability folklore
2024 DeepSeekMoE: Towards Ultimate Expert Specialization https://arxiv.org/abs/2401.06066 Fine-grained experts + shared experts DeepSeek MoE DNA
2024 Mixtral of Experts https://arxiv.org/abs/2401.04088 Strong open sparse LM Sparse quality reference
2024 DeepSeek-V2 (DeepSeekMoE @ production) https://arxiv.org/abs/2405.04434 Economical training with sparse FFN MLA+MoE combo
2024 DeepSeek-V3 (aux-loss-free routing) https://arxiv.org/abs/2412.19437 Bias-based load balance without aux loss Aux-loss-free lineage
2026 LatentMoE: Optimal Accuracy per FLOP/Parameter https://arxiv.org/abs/2601.18089 Route in latent cheaper expert path Namesake of LatentMoE
2026 Kimi K3 Stable LatentMoE + Quantile Balancing https://arxiv.org/abs/2607.24653 Stabilize extreme sparsity (896 experts, k=16) Production K3 FFN

K3 QB (official): expert bias from margin quantiles via global histogram all-reduce (not token-side exact quantiles). Also cite: Jianlin Su blog Travels in MoE (load balance via optimal assignment) — referenced by K3 as [111] (Chinese primary blog, not arXiv).

M08 — Scaling Laws, Data Recipes, Optimizers (→ Per-Head Muon)

Prerequisites: M01 Why for K3: 2.5× scaling-efficiency claim; Per-Head Muon; cosine > WSD in their search.

Year Title URL Problem solved Bridge to next
2020 Scaling Laws for Neural Language Models (Kaplan et al.) https://arxiv.org/abs/2001.08361 Power-law loss vs N, D, C Classical scaling
2022 Training Compute-Optimal LLMs (Chinchilla / Hoffmann) https://arxiv.org/abs/2203.15556 Tokens–params balance TPP retuning practice
2014 Adam: A Method for Stochastic Optimization https://arxiv.org/abs/1412.6980 Adaptive moments Default baseline
2017 Decoupled Weight Decay Regularization (AdamW) https://arxiv.org/abs/1711.05101 Correct weight decay Pre-Muon default
2024 Muon: An Optimizer for Hidden Layers (Keller Jordan et al.) https://kellerjordan.github.io/posts/muon/ Newton–Schulz orthogonalized momentum Matrix-wise updates
2025 Muon is Scalable for LLM Training (Moonshot / Moonlight) https://arxiv.org/abs/2502.16982 Weight decay + update scale → Muon at LLM scale Moonshot optimizer stack
2025 Kimi K2 (MuonClip / QK-Clip) https://arxiv.org/abs/2507.20534 Stabilize attention logits under Muon Pretrain 15.5T zero spike claim
2026 Kimi K3 Per-Head Muon https://arxiv.org/abs/2607.24653 Orthogonalize QKV per head for balanced head scales K3 default optimizer

Data / schedule landmarks

Year Title URL Problem solved Bridge to next
2020 Exploring the Limits of Transfer Learning (T5) https://arxiv.org/abs/1910.10683 Unified text-to-text + C4 data Data quality culture
2022 Training Compute-Optimal… (data side of Chinchilla) https://arxiv.org/abs/2203.15556 More tokens for given compute Overtrain regimes
2024 DeepSeek LLM: Scaling Open-Source LMs with Longtermism https://arxiv.org/abs/2401.02954 Open scaling-law study + 67B DeepSeek program start
2025 MiniCPM / WSD schedule discussion https://arxiv.org/abs/2404.06395 Warmup–Stable–Decay alternative K3 finds cosine better after independent HPO
2025 Kimi K2 data rephrasing recipe https://arxiv.org/abs/2507.20534 Knowledge/math rephrase + fidelity checks Inherited by K3 §3.1

M09 — DeepSeek Lineage (Dense → MoE → MLA → V3 → R1 → V3.2 → V4)

Prerequisites: M03, M07, M08, M12–M13 for post-train pieces Policy: include only official DeepSeek papers/reports.

9.1 Pretrain architecture stack

Year Title URL Problem solved Bridge to next
2024 DeepSeek LLM: Scaling Open-Source Language Models with Longtermism https://arxiv.org/abs/2401.02954 Open dense 7B/67B + scaling study Foundation
2024 DeepSeekMoE: Ultimate Expert Specialization https://arxiv.org/abs/2401.06066 Fine-grained + shared experts Sparse specialization
2024 DeepSeek-V2: MLA + DeepSeekMoE https://arxiv.org/abs/2405.04434 KV latent compression + economical MoE MLA birth
2024 DeepSeek-V3 Technical Report https://arxiv.org/abs/2412.19437 671B/37B act; FP8; MTP; aux-loss-free routing; DualPipe Flagship pretrain stack
2025 DeepSeek-V3.2: DSA + scalable RL + agentic synthesis https://arxiv.org/abs/2512.02556 DeepSeek Sparse Attention; agent post-train scale Long-ctx efficiency
2026 DeepSeek-V4: Million-Token Context Intelligence https://arxiv.org/abs/2606.19348 CSA+HCA hybrid attn; mHC; Muon; 1M ctx Pro/Flash Official later 1M-ctx peer to K3

9.2 Math, RL, reasoning distillation

Year Title URL Problem solved Bridge to next
2024 DeepSeekMath (+ GRPO) https://arxiv.org/abs/2402.03300 Math continual pretrain + critic-free group RL GRPO algorithm
2025 DeepSeek-R1: Incentivizing Reasoning via RL https://arxiv.org/abs/2501.12948 R1-Zero pure RL; R1 multi-stage; distillation to dense 1.5B–70B Reasoning RL template
2025 DeepSeek-R1 Nature version (same line) https://doi.org/10.1038/s41586-025-09422-z Peer-reviewed presentation of R1 Archival citation

9.3 V3 systems keywords → primary sections

Keyword Where (primary) Problem solved Bridge
FP8 mixed precision V3 report §3.3 Low-precision train at 671B → MX / QAT serving
MTP (multi-token prediction) V3 report Stronger pretrain objective / draft head Speculative / denser signal
Aux-loss-free routing V3 report Balance without aux loss interference → K3 QB different mechanism
DualPipe V3 report §3.2.1 Overlap PP bubbles + MoE dispatch/combine → MoonEP alternative EP design
DSA V3.2 report Sparse token selection under MLA Contrast with KDA hybrid
DeepEP https://github.com/deepseek-ai/DeepEP Expert-parallel comm library K3 MoonEP builds on / contrasts

Atlas teaching note: treat DeepSeek as the dense reference stack for MLA/MoE/FP8/RL; treat K3 as the hybrid linear + AttnRes + Stable LatentMoE fork at larger sparsity and 1M agentic RL.

M10 — Low-Precision Training & Deployment (FP8 → MXFP4)

Prerequisites: M08–M09 Why for K3: post-train QAT with MXFP4 weights / MXFP8 activations on MoE experts.

Year Title URL Problem solved Bridge to next
2018 Quantization and Training of Neural Networks for Integer-Arithmetic-Only Inference https://arxiv.org/abs/1712.05877 QAT for int inference QAT paradigm
2020 Training with Quantization Noise https://arxiv.org/abs/2004.07320 Noise injection for robust quant QAT variants
2022 FP8 Formats for Deep Learning (Micikevicius et al.) https://arxiv.org/abs/2209.05433 FP8 train/infer formats Hardware FP8
2023 Microscaling Data Formats for Deep Learning (MX / MXFP4) https://arxiv.org/abs/2310.10537 Block scales + narrow element types OCP MX family
2024 DeepSeek-V3 FP8 training framework https://arxiv.org/abs/2412.19437 Validate FP8 at extreme MoE scale Pretrain low-prec
2026 Kimi K3 MXFP4 QAT from SFT onward https://arxiv.org/abs/2607.24653 Deploy-time memory for 2.8T experts Serving recipe

OCP primary (non-arXiv): OCP Microscaling Formats (MX) Specification v1.0 — https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf

M11 — Long Context: Algorithms & Parallelism (→ KDA CP, 1M)

Prerequisites: M03–M05 Why for K3: progressive 8K→64K→256K→1M; KDA Context Parallelism; state-aware prefix cache.

Year Title URL Problem solved Bridge to next
2023 ALiBi https://arxiv.org/abs/2108.12409 Length extrapolation via linear biases PE-free-ish long ctx
2023 RoPE / YaRN (see M02) https://arxiv.org/abs/2309.00071 Extend rotary models Common production path
2023 Ring Attention with Blockwise Transformers https://arxiv.org/abs/2310.01889 Sequence parallel for near-infinite context Distributed long attn
2023 DeepSpeed Ulysses https://arxiv.org/abs/2309.14509 Sequence parallelism system opts Cluster long-ctx train
2024 Linear Attention Sequence Parallelism (LASP) https://arxiv.org/abs/2404.02882 SP specialized for linear attn Linear-SP ancestors
2025 LASP-2 / hybrid linear SP https://arxiv.org/abs/2502.07864 [verify id if citing; K3 cites related] Hybrid linear+softmax SP Closer to KDA hybrids
2025 Context Parallelism for DeltaNet (Wang) https://yywangcs.notion.site/DeltaNet-2a9fc9f5d8058013a498f34e0b25bd52 CP for delta recurrence Direct ancestor of KCP
2026 Kimi K3 KDA Context Parallelism + FlashKDA https://arxiv.org/abs/2607.24653 · FlashKDA https://github.com/MoonshotAI/FlashKDA Correct state transport under delta update (M_t S_{t-1}) 1M train/prefill

K3 progressive extension (official): pretrain grows 8K→64K; cooldown 256K→1M; needle/synthetic scatter tasks force full-window use.

M12 — Alignment Foundations (RLHF / PPO / DPO)

Prerequisites: M01 Why for K3: base of modern post-train before GRPO / multi-effort / MOPD.

Year Title URL Problem solved Bridge to next
2017 Proximal Policy Optimization Algorithms https://arxiv.org/abs/1707.06347 Stable policy gradient updates RL workhorse
2022 Training Language Models to Follow Instructions with Human Feedback (InstructGPT) https://arxiv.org/abs/2203.02155 RLHF pipeline for LLMs Industry alignment template
2022 Constitutional AI https://arxiv.org/abs/2212.08073 Principle-based AI feedback RLAIF direction
2023 Direct Preference Optimization (DPO) https://arxiv.org/abs/2305.18290 Preference learning without RL loop Offline preference
2023 Llama 2: Open Foundation and Fine-Tuned Chat Models https://arxiv.org/abs/2307.09288 Open RLHF stack details Open alignment recipes
2024 SimPO / ORPO family (optional shortlist) https://arxiv.org/abs/2405.14734 Simpler preference objectives Alt to DPO

M13 — Reasoning RL, Multi-Effort, On-Policy Distillation

Prerequisites: M12, M09 Why for K3: multi-domain RL × effort levels; MOPD consolidation.

Year Title URL Problem solved Bridge to next
2022 Chain-of-Thought Prompting https://arxiv.org/abs/2201.11903 Elicit intermediate reasoning Test-time reasoning culture
2023 Let’s Verify Step by Step (process reward) https://arxiv.org/abs/2305.20050 Process vs outcome supervision PRM path
2024 DeepSeekMath / GRPO https://arxiv.org/abs/2402.03300 Group-relative baseline, no critic Memory-efficient RL
2024 OpenAI o1 announcement / “Learning to Reason with LLMs” https://openai.com/index/learning-to-reason-with-llms/ Scale RL + test-time compute Multi-effort ancestor [blog primary; not full paper]
2025 Kimi k1.5: Scaling RL with LLMs https://arxiv.org/abs/2501.12599 Long-CoT RL scaling (Moonshot) Kimi reasoning line
2025 DeepSeek-R1 / R1-Zero / distillation https://arxiv.org/abs/2501.12948 Pure RL emergence; then multi-stage + distill Distill reasoning patterns
2025 On-policy distillation (Thinking Machines Lab) https://thinkingmachines.ai/blog/on-policy-distillation/ [blog; K3 cites as Connectionism note] Student on-policy w.r.t teacher MOPD conceptual parent
2026 MiMo-V2-Flash Technical Report (multi-teacher mention lineage) https://arxiv.org/abs/2601.02780 Related multi-teacher post-train Peer system cited by K3
2026 Kimi K3 multi-effort RL + MOPD https://arxiv.org/abs/2607.24653 Domain experts {general, agentic, coding} × {low, high, max} → single policy Capstone post-train

K3 MOPD (official): on-policy student samples; multi-teacher logits/rewards consolidate specialists; top-k distill ablations showed no clear gain.

M14 — Agents, Environments, Multimodal (MoonViT-V2)

Prerequisites: M13, M05 Why for K3: agent harness modularization; AET; native vision-in-the-loop; MoonViT-V2 from scratch.

14.1 Agent / tool / coding environments

Year Title URL Problem solved Bridge to next
2023 Toolformer https://arxiv.org/abs/2302.04761 Self-supervised tool API calls Tool-use pretrain
2023 ReAct https://arxiv.org/abs/2210.03629 Interleave reason + act Agent loop pattern
2024 SWE-bench https://arxiv.org/abs/2310.06770 Real GitHub issue resolution Coding agent benchmark
2024 OSWorld https://arxiv.org/abs/2404.07972 Computer-use agents GUI agent env
2025 Terminal-Bench https://arxiv.org/abs/2502.14045 [confirm version K3 cites] Hard CLI agent tasks Terminal agents
2025 BrowseComp https://arxiv.org/abs/2504.12516 Browsing agents Web agents
2025 Kimi K2 agentic data synthesis + joint RL https://arxiv.org/abs/2507.20534 Large synthetic tool trajectories Kimi agent stack
2026 Kimi K2.5 Visual Agentic Intelligence https://arxiv.org/abs/2602.02276 Multimodal agents / swarm [swarm details in K2.5] Vision agents
2026 Kimi K3 agent envs (pluggable harness, AET, knowledge graph) https://arxiv.org/abs/2607.24653 Avoid harness overfitting; long-horizon verifiers Production agent RL data

14.2 Vision encoders → MoonViT-V2

Year Title URL Problem solved Bridge to next
2020 An Image is Worth 16x16 Words (ViT) https://arxiv.org/abs/2010.11929 Transformer vision backbone ViT era
2021 Learning Transferable Visual Models From Natural Language Supervision (CLIP) https://arxiv.org/abs/2103.00020 Contrastive vision–language pretrain SigLIP-style inits
2023 SigLIP https://arxiv.org/abs/2303.15343 Sigmoid contrastive VL Prior MoonViT init
2024 LLaVA-OneVision https://arxiv.org/abs/2408.03326 Unified single/multi-image/video tasks Native-res VL practice
2025 Kimi-VL Technical Report (MoonViT) https://arxiv.org/abs/2504.07491 Native-resolution MoonViT + MoE LM Moonshot VL stack
2026 Kimi K3 MoonViT-V2 from-scratch NTP https://arxiv.org/abs/2607.24653 Drop SigLIP init for stability; match quality K3 vision path

K3 vision claim (official): MoonViT-V2 ≈0.4B, 27 layers, RMSNorm, bias-free; shared image/video; 2×2 pixel-shuffle; up to 3584² in 1M context.

M15 — Capstone Systems: KDA Kernels, MoonEP, 1M Agentic RL, Serve

Prerequisites: M05–M07, M10–M11, M13–M14 Why for K3: makes 2.8T hybrid + 1M agentic RL trainable and deployable.

Year Title / artifact URL Problem solved Bridge to next
2019 Triton: Intermediate Language for Tiled NN Kernels https://www.eecs.harvard.edu/~htk/publication/2019-mapl-tillet-kung-cox.pdf Productive GPU kernels FLA / custom attn
2021 GPipe https://arxiv.org/abs/1811.06965 Pipeline parallel PP lineage
2021 Megatron-LM / efficient large-scale training https://arxiv.org/abs/2104.04473 3D parallelism Pretrain systems baseline
2020 ZeRO https://arxiv.org/abs/1910.02054 Shard optimizer states Memory for huge models
2024 FLA: Flash Linear Attention library https://github.com/fla-org/flash-linear-attention Kernel zoo for linear/delta KDA ops host
2025 DeepEP https://github.com/deepseek-ai/DeepEP Expert-parallel communication MoonEP contrast
2025 DualPipe (in V3 report) https://arxiv.org/abs/2412.19437 Overlap MoE+PP Pipeline bubbles
2026 FlashKDA https://github.com/MoonshotAI/FlashKDA CUTLASS chunkwise KDA Train/prefill speed
2026 Kimi K3 MoonEP https://arxiv.org/abs/2607.24653 Perfect balance EP, static shapes, zero-copy, bounded redundant experts 2.8T MoE train
2026 Kimi K3 1M agentic RL co-located system https://arxiv.org/abs/2607.24653 Partial rollouts, external KV retention, resumable sandboxes Long-horizon RL
2026 Kimi K3 state-aware KDA prefix caching + MXFP4 serve https://arxiv.org/abs/2607.24653 Decode across hybrid state + quant experts Production deployment

3. Module → K3 Ingredient Checklist

Module Delivers toward K3
M01–M02 Transformer residual + PE literacy
M03 MLA / Gated MLA
M04–M05 KDA + hybrid 3:1
M06 Attention Residuals
M07 Stable LatentMoE + QB vs aux-loss-free
M08 Scaling 2.5× story + Per-Head Muon
M09 DeepSeek dense comparator (MLA, FP8, MTP, DualPipe, GRPO, R1, DSA, V4)
M10 MXFP4/MXFP8 QAT
M11 1M context + KCP
M12–M13 Multi-effort RL + MOPD
M14 Agent envs + MoonViT-V2
M15 MoonEP + 1M agentic RL systems

4. DeepSeek Coverage Matrix (Atlas “strong track”)

Topic Primary Module Must-read density
DeepSeek LLM 2401.02954 M08/M09 high
DeepSeekMoE 2401.06066 M07/M09 high
V2 / MLA 2405.04434 M03/M09 core
Math / GRPO 2402.03300 M13/M09 core
V3 FP8 / MTP / aux-free / DualPipe 2412.19437 M09/M10/M15 core
R1 / R1-Zero / distill 2501.12948 (+ Nature DOI) M13/M09 core
V3.2 / DSA 2512.02556 M03/M09 high
V4 (official later) 2606.19348 M06/M09/M11 high (1M peer)
DeepEP GitHub deepseek-ai/DeepEP M15 systems

5. Uncertain / Careful Claims

Claim Status
K3 2.5× scaling efficiency vs K2 Author-reported fit on OOD val curves (Fig. 7); not independent replication
Benchmarks vs Claude Fable 5 / GPT-5.6 Sol Author suite; external harness variance possible
“First open 3T-class” marketing phrasing Product language; verify total-param definitions (shared vs routed, embeddings, vision)
Exact MoonEP algorithm vs DeepEP Described in K3 §5; no standalone MoonEP paper at time of roadmap
FlashKDA paper Primarily code artifact + K3 description; limited standalone theory paper
LASP-2 arXiv number if used in syllabus Double-check id before publishing student links
On-policy distillation “Connectionism” note Blog-level primary; treat as concept source, not peer-reviewed algorithm paper
DeepSeek-V4 CSA/HCA vs K3 KDA Parallel 1M-ctx designs; do not equate without side-by-side study
MoonViT-V2 “matches SigLIP init quality” Author ablation claim (Fig. 6 + text)
Multi-effort exact reward formulas Partially specified (thinking-token thresholds); full reward suite may be incomplete in report

6. Compact Reading Paths (for site UX)

Path A — “Understand K3 architecture in 12 papers”

  1. Attention Is All You Need
  2. FlashAttention-2
  3. DeepSeek-V2 (MLA)
  4. Gated DeltaNet
  5. Kimi Linear (KDA)
  6. Attention Residuals
  7. DeepSeekMoE
  8. LatentMoE
  9. Muon is Scalable
  10. Microscaling (MXFP4)
  11. DeepSeek-V3 (systems)
  12. Kimi K3

Path B — “DeepSeek full stack (official only)”

DeepSeek LLM → DeepSeekMoE → DeepSeekMath/GRPO → V2/MLA → V3 → R1 → V3.2/DSA → V4

Path C — “Post-train / agents”

InstructGPT → DPO → GRPO → R1 → k1.5 → K2 → K2.5 → K3 (MOPD + multi-effort + 1M agentic RL)

7. Suggested Atlas Module Metadata (CMS fields)

module_id: M05
title_zh: 混合注意力：KDA 与 Gated MLA
prerequisites: [M03, M04]
k3_tags: [KDA, GatedMLA, NoPE, hybrid-3to1]
primary_papers: 6
estimated_hours: 8

8. Source Log (what this roadmap used)

Source Role
arXiv:2607.24653 K3 PDF text extract Architecture, post-train, systems, bibliography
arXiv abstracts for DeepSeek V2/V3/V3.2/V4, R1, Math, MoE, LLM Official lineage
arXiv:2510.26692 Kimi Linear KDA definition
arXiv:2603.15031 AttnRes Depth residual redesign
arXiv:2502.16982 / 2507.20534 Muon / K2
arXiv:2504.07491 / 2602.02276 MoonViT / K2.5
arXiv:2310.10537 + OCP MX MXFP4
arXiv:2601.18089 LatentMoE Latent MoE naming/ancestry

End of structured research notes — LLM Atlas / K3-anchored roadmap.

Reference in New Issue View Git Blame Copy Permalink