feat: publish MoE deep dive
This commit is contained in:
@@ -140,6 +140,7 @@ const toc = [
|
||||
因此模型侧路由与系统侧 Expert Parallel 从来不能分开看。
|
||||
</p>
|
||||
</div>
|
||||
<a class="button primary" href="/moe/#deepseek">进入 MoE 专题:比较粗专家、细粒度专家与共享专家 →</a>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="mla">
|
||||
@@ -212,6 +213,10 @@ const toc = [
|
||||
根据近期负载上调冷门专家、下调热门专家;bias 只影响路由选择,不直接进入最终门控权重,
|
||||
从而把“系统要均衡”和“模型要准确”更松地解耦。
|
||||
</p>
|
||||
<p>
|
||||
<a href="/moe/#balance">专题版推导</a>会进一步区分 z-loss、辅助平衡损失、Loss-Free expert bias
|
||||
与 K3 Quantile Balancing;“aux-loss-free”并不等于系统里不存在任何平衡约束。
|
||||
</p>
|
||||
|
||||
<h3>FP8 训练真正难在哪</h3>
|
||||
<p>
|
||||
|
||||
Reference in New Issue
Block a user