diff --git a/PROGRESS.md b/PROGRESS.md
index fe66c78..bcedcb1 100644
--- a/PROGRESS.md
+++ b/PROGRESS.md
@@ -14,7 +14,7 @@
 | 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
 | Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
 | 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
-| DeepSeek 专题 | 完成二轮 | 83% | 真实专家负载 / MLA kernel / RL 训练 traces 与独立复现 |
+| DeepSeek 专题 | 三轮实证进行中 | 87% | 真实 latent-cache kernel、更大样本负载、FP8/pipeline 与 R1-like RL 复现 |
 | 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
 | 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
 | 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
@@ -41,7 +41,7 @@
 - [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
 - [x] 完成可检索、可按专题筛选的论文库页面。
 - [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
-- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 四联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等七十一个原创交互视图。
+- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 八联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等七十五个原创交互视图。
 - [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
 - [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
 - [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
@@ -169,11 +169,15 @@
 - [x] RTX 5090 K3 形状受控计时完成：fixed / varlen、BF16 / no-state / FP32 state 共 1,800 个 CUDA Event samples；fixed BF16 mean 2.6210 ms、P95 2.6437 ms，varlen BF16 mean 2.3335 ms、P95 2.3574 ms。
 - [x] 可复现 Dockerfile、运行脚本与机器可读 JSON 接入开源树；网站第四视图把作者 H20/GB200 表、本机 RTX 5090 值、exact suite、构建链、synthetic router 和 `A_log` 未决冲突分开显示。
 - [x] K3 FlashKDA 执行里程碑以源提交 `2ef846f`、不可变镜像 `20260729T053852Z-2ef846f` 发布；OCI digest `sha256:08ad5b55…32e6`，NAS、VPS/Tailscale、NPM、DNS、HTTPS、证书、门户与十六套生产 Chrome 回归全链路通过；保留 `20260729T044605Z-be2b291` 回滚。
+- [x] 启动 DeepSeek 三轮真实权重执行：固定官方 V2-Lite revision `604d5664…82de0`，审计 29.261 GiB / 4 shards，并以第一分片的完整层边界执行 layer 0–6。
+- [x] RTX 5090 连续 forward 闭环：4 条固定 prompt、90 个有效 token、6 个 MoE 层产生 3,240 次真实 top-6 路由；记录每层 `[4,27,576]` MLA 投影与 HF eager 展开 K/V shapes。
+- [x] 独立复跑 31/31 exact：来源、配置、tokenization、逐层 hidden hashes、MLA shapes、aggregate loads 与全部 token routes 完全一致；计时明确不比较。
+- [x] DeepSeek 新四视图本地真实 Chrome 回归通过：逐 token 路由、覆盖/均衡分账、latent/HF eager cache 计算、27 层执行断面、键盘 tabs 与 390px 移动端均无异常。
 
 ## 正在进行
 
 - [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
-- [ ] DeepSeek 三轮：真实专家负载、MLA kernel、FP8 / pipeline 与 R1-like RL traces，外加独立小模型复现。
+- [ ] DeepSeek 三轮下一闸门：真实 latent-cache kernel、更大样本 expert-load 置信区间、FP8 / pipeline traces 与 R1-like RL 小模型复现。
 - [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
 - [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
 - [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
@@ -289,6 +293,9 @@
 | 2026-07-29 | 论文库扩充到 486 篇 | 新增 DeepSeek-Coder/Coder-V2、ESFT、Prover-V1.5/V2 与 Engram 6 个旁支节点 |
 | 2026-07-29 | DeepSeek 二轮用四个独立实验闭环 | 稀疏容量、MLA 缓存、V3 协同与 RL 偏差分别操作；精确公式、作者报告和 teaching model 永久分级 |
 | 2026-07-29 | DeepSeek 二轮用不可变镜像 `20260729T031901Z-cd96dab` 发布 | OCI digest `sha256:4bfb3faf…fa3968`；复用 `12010→8080`、NPM host 31 / cert 41、门户 order 180 与公开 Forgejo；保留 `20260729T023329Z-a2c9298` 回滚 |
+| 2026-07-29 | DeepSeek 三轮以官方 V2-Lite 第一分片建立真实执行证据 | layer 0–6 是完整连续边界；layer 7 跨分片即停止，不把部分下载写成完整生成 |
+| 2026-07-29 | MLA 算法状态与框架物化格式分两张账 | V2-Lite latent 576 元素与 HF eager 5,120 元素同时报告；88.75% 不冒充 V2 论文 93.3% |
+| 2026-07-29 | 专家覆盖、负载均衡与语义永久分开 | 60–64/64 used 不推出 balanced；CV/Gini/effective 同报；expert ID 不跨层连线或命名 |
 | 2026-07-29 | K3 二轮按 32 张对象账与完整报告顺序重建 | total/active、2.5×、KDA state、深度来源、专家路由、视觉目标、轨迹、缓存与评测协议不再压成一页组件摘要 |
 | 2026-07-29 | K3 原生视觉事实回到 §2.4 / §3.3 核验 | 删除“先冻结语言模型再解冻”旧表述；明确 MoonViT-V2 从头训练，视觉/文本从开始共同 NTP |
 | 2026-07-29 | K3 Figure 1–16 / Table 1–5 全部建立课程视觉契约 | 每张图同时写支持范围与不可外推项；作者报告、论文、推导与 toy model 使用 R/P/D/T 标签 |
diff --git a/README.md b/README.md
index bd7b15a..69db2db 100644
--- a/README.md
+++ b/README.md
@@ -19,7 +19,7 @@
 
 当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
 语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
-以及 71 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
+以及 75 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
 8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
 第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
 497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
@@ -27,8 +27,11 @@
 `sm_120a` wheel，在 RTX 5090 上完成 6/6 官方参考 exact-match 和 K3 fixed / varlen 形状计时。详见
 [K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
 [checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
-DeepSeek 二轮专题以 24 张问题账、10 次技术转向、
-4 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4。
+DeepSeek 三轮专题以 24 张问题账、10 次技术转向、
+8 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
+并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
+MLA/HF eager cache shapes 与 `31/31` exact 独立复跑。详见
+[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md)。
 其余专题按进度账本持续扩建。
 
 ## 本地开发
diff --git a/experiments/deepseek/README.md b/experiments/deepseek/README.md
new file mode 100644
index 0000000..bb89198
--- /dev/null
+++ b/experiments/deepseek/README.md
@@ -0,0 +1,56 @@
+# DeepSeek real-weight execution probes
+
+The probes in this directory use official DeepSeek artifacts and keep their
+scope deliberately narrower than a full-model benchmark.
+
+## DeepSeek-V2-Lite truncated trace
+
+`v2_lite_trace.py` executes layers 0–6 from the official BF16 checkpoint. Those
+seven layers are fully contained in shard 1; layer 7 is split across shards 1
+and 2 and is therefore outside the default evidence boundary.
+
+Pinned model:
+
+```text
+deepseek-ai/DeepSeek-V2-Lite@604d5664dddd88a0433dbae533b7fe9472482de0
+```
+
+Required Python stack:
+
+```text
+torch==2.11.0+cu128
+transformers==4.41.2
+safetensors==0.8.0
+```
+
+The 2024 remote code does not import under Transformers 5.5 because
+`is_torch_fx_available` was removed. The probe imports the official files as a
+read-only local package; it does not patch the model source.
+
+Download the metadata, tokenizer, remote code, index, and first shard with the
+Hugging Face CLI, then run:
+
+```bash
+python experiments/deepseek/v2_lite_trace.py \
+  --artifact-dir /path/to/deepseek-v2-lite \
+  --output src/data/deepseek-v2-lite-trace.json
+```
+
+The result contains:
+
+- real tokenizer pieces and model-derived hidden states;
+- the actual `[B,T,576]` MLA compressed projection at each executed layer;
+- the expanded key/value tensors stored by the Hugging Face eager cache;
+- token-level top-6 routed expert IDs and weights for six MoE layers;
+- per-layer and per-prompt expert-load summaries;
+- explicit boundaries against global load, expert semantics, training traces,
+  full-model generation, and production serving claims.
+
+Run the probe twice and compare deterministic evidence while excluding timing:
+
+```bash
+python experiments/deepseek/compare_v2_lite_traces.py \
+  --first /path/to/trace-1.json \
+  --second /path/to/trace-2.json \
+  --output src/data/deepseek-v2-lite-trace-repro.json
+```
diff --git a/experiments/deepseek/compare_v2_lite_traces.py b/experiments/deepseek/compare_v2_lite_traces.py
new file mode 100644
index 0000000..9d14448
--- /dev/null
+++ b/experiments/deepseek/compare_v2_lite_traces.py
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""Compare deterministic evidence from two DeepSeek-V2-Lite trace runs."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--first", type=Path, required=True)
+    parser.add_argument("--second", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--captured-at", default=None)
+    return parser.parse_args()
+
+
+def read(path: Path) -> dict[str, Any]:
+    return json.loads(path.read_text())
+
+
+def sha256(path: Path) -> str:
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def main() -> None:
+    args = parse_args()
+    first = read(args.first)
+    second = read(args.second)
+    checks: list[dict[str, Any]] = []
+
+    def check(name: str, left: Any, right: Any) -> None:
+        checks.append({"name": name, "exact": left == right})
+
+    check("provenance", first["provenance"], second["provenance"])
+    check("configuration", first["configuration"], second["configuration"])
+    check("prompt_tokenization", first["prompts"], second["prompts"])
+    check(
+        "initial_hidden_hash",
+        first["execution"]["initial_hidden"]["sha256_fp32"],
+        second["execution"]["initial_hidden"]["sha256_fp32"],
+    )
+    check(
+        "final_hidden_hash",
+        first["execution"]["final_hidden"]["sha256_fp32"],
+        second["execution"]["final_hidden"]["sha256_fp32"],
+    )
+    for left, right in zip(
+        first["execution"]["layers"],
+        second["execution"]["layers"],
+        strict=True,
+    ):
+        layer = left["layer"]
+        check(
+            f"layer_{layer}_hidden_hash",
+            left["hidden_after"]["sha256_fp32"],
+            right["hidden_after"]["sha256_fp32"],
+        )
+        check(f"layer_{layer}_mla_shapes", left["mla"], right["mla"])
+        if "routing" in left:
+            check(
+                f"layer_{layer}_aggregate_load",
+                left["routing"]["aggregate_load"],
+                right["routing"]["aggregate_load"],
+            )
+            check(
+                f"layer_{layer}_token_routes",
+                [
+                    prompt["token_routes"]
+                    for prompt in left["routing"]["per_prompt"]
+                ],
+                [
+                    prompt["token_routes"]
+                    for prompt in right["routing"]["per_prompt"]
+                ],
+            )
+
+    captured_at = args.captured_at or datetime.now(timezone.utc).isoformat()
+    result = {
+        "schema_version": 1,
+        "captured_at": captured_at,
+        "first": {
+            "captured_at": first["captured_at"],
+            "sha256": sha256(args.first),
+        },
+        "second": {
+            "captured_at": second["captured_at"],
+            "sha256": sha256(args.second),
+        },
+        "timing_compared": False,
+        "checks": checks,
+        "exact_checks": sum(item["exact"] for item in checks),
+        "total_checks": len(checks),
+        "all_exact": all(item["exact"] for item in checks),
+    }
+    if not result["all_exact"]:
+        failed = [item["name"] for item in checks if not item["exact"]]
+        raise AssertionError(f"trace mismatch: {failed}")
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2) + "\n")
+    print(json.dumps(result, indent=2))
+
+
+if __name__ == "__main__":
+    main()
diff --git a/experiments/deepseek/v2_lite_trace.py b/experiments/deepseek/v2_lite_trace.py
new file mode 100644
index 0000000..4449c70
--- /dev/null
+++ b/experiments/deepseek/v2_lite_trace.py
@@ -0,0 +1,588 @@
+#!/usr/bin/env python3
+"""Trace real DeepSeek-V2-Lite MLA states and MoE routes from official weights.
+
+The script deliberately executes only the contiguous layers fully contained in
+the first official safetensors shard (layers 0 through 6). It loads one decoder
+layer at a time, so a 32 GB workstation GPU can produce model-derived hidden
+states without downloading or materializing the complete 15.7B-parameter model.
+
+No model source is patched. The pinned remote-code files are imported as a
+read-only local package, and every decoder layer is the official class.
+"""
+
+from __future__ import annotations
+
+import argparse
+import gc
+import hashlib
+import importlib.util
+import json
+import math
+import platform
+import statistics
+import subprocess
+import sys
+import time
+import types
+from datetime import datetime, timezone
+from itertools import combinations
+from pathlib import Path
+from typing import Any
+
+import torch
+import torch.nn.functional as F
+from safetensors import safe_open
+from transformers import AutoTokenizer, DynamicCache
+
+
+DEFAULT_PROMPTS = [
+    {
+        "id": "zh_explanation",
+        "label": "中文解释",
+        "text": "用通俗的语言解释，为什么稀疏专家模型可以拥有很多参数，但每个 token 只使用其中一小部分。",
+    },
+    {
+        "id": "en_architecture",
+        "label": "English architecture",
+        "text": "Explain how a compressed key-value latent changes the memory cost of autoregressive decoding.",
+    },
+    {
+        "id": "code",
+        "label": "Python code",
+        "text": "Write a Python function that returns the first repeated element in a list and explain its complexity.",
+    },
+    {
+        "id": "math",
+        "label": "数学推理",
+        "text": "若正数 x 满足 x 加上它的倒数等于 3，求 x 的平方加上倒数的平方。",
+    },
+]
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--artifact-dir", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--layers", type=int, default=7)
+    parser.add_argument("--device", default="cuda")
+    parser.add_argument("--captured-at", default=None)
+    return parser.parse_args()
+
+
+def sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for block in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(block)
+    return digest.hexdigest()
+
+
+def tensor_sha256(tensor: torch.Tensor) -> str:
+    value = tensor.detach().float().contiguous().cpu().numpy()
+    return hashlib.sha256(value.tobytes()).hexdigest()
+
+
+def load_official_modules(root: Path) -> tuple[Any, Any]:
+    """Import relative official files without Transformers' remote-code scanner."""
+    package_name = "deepseek_v2_lite_official"
+    package = types.ModuleType(package_name)
+    package.__path__ = [str(root)]
+    sys.modules[package_name] = package
+
+    loaded = {}
+    for leaf in ("configuration_deepseek", "modeling_deepseek"):
+        name = f"{package_name}.{leaf}"
+        spec = importlib.util.spec_from_file_location(name, root / f"{leaf}.py")
+        if spec is None or spec.loader is None:
+            raise RuntimeError(f"cannot load official module: {leaf}")
+        module = importlib.util.module_from_spec(spec)
+        sys.modules[name] = module
+        spec.loader.exec_module(module)
+        loaded[leaf] = module
+    return loaded["configuration_deepseek"], loaded["modeling_deepseek"]
+
+
+def percentile(values: torch.Tensor, q: float) -> float:
+    return torch.quantile(values.float(), q).item()
+
+
+def hidden_stats(hidden: torch.Tensor, mask: torch.Tensor) -> dict[str, Any]:
+    valid = hidden[mask.bool()]
+    rms = valid.float().square().mean(dim=-1).sqrt()
+    norms = valid.float().norm(dim=-1)
+    return {
+        "valid_tokens": valid.shape[0],
+        "hidden_rms": {
+            "mean": rms.mean().item(),
+            "p10": percentile(rms, 0.10),
+            "p50": percentile(rms, 0.50),
+            "p90": percentile(rms, 0.90),
+        },
+        "l2_norm": {
+            "mean": norms.mean().item(),
+            "p10": percentile(norms, 0.10),
+            "p50": percentile(norms, 0.50),
+            "p90": percentile(norms, 0.90),
+        },
+        "finite": bool(torch.isfinite(valid).all()),
+        "sha256_fp32": tensor_sha256(valid),
+    }
+
+
+def load_metrics(load: torch.Tensor) -> dict[str, Any]:
+    values = load.float()
+    mean = values.mean()
+    ordered = values.sort().values
+    count = values.numel()
+    indices = torch.arange(1, count + 1, dtype=torch.float32)
+    denominator = count * ordered.sum()
+    gini = (
+        ((2 * indices - count - 1) * ordered).sum() / denominator
+        if denominator
+        else torch.tensor(0.0)
+    )
+    probabilities = values / values.sum().clamp_min(1)
+    nonzero = probabilities[probabilities > 0]
+    entropy = -(nonzero * nonzero.log()).sum()
+    return {
+        "routes": int(values.sum()),
+        "used_experts": int((values > 0).sum()),
+        "zero_experts": int((values == 0).sum()),
+        "mean": mean.item(),
+        "std": values.std().item(),
+        "cv": (values.std() / mean).item() if mean else 0.0,
+        "gini": gini.item(),
+        "entropy_nats": entropy.item(),
+        "effective_experts": math.exp(entropy.item()),
+        "min": values.min().item(),
+        "p50": percentile(values, 0.50),
+        "p90": percentile(values, 0.90),
+        "max": values.max().item(),
+    }
+
+
+def route_trace(
+    tokenizer: Any,
+    input_ids: torch.Tensor,
+    attention_mask: torch.Tensor,
+    topk_ids: torch.Tensor,
+    topk_weights: torch.Tensor,
+    prompts: list[dict[str, str]],
+    expert_count: int,
+) -> dict[str, Any]:
+    batch, sequence, top_k = topk_ids.shape
+    valid_routes = attention_mask.bool().unsqueeze(-1).expand_as(topk_ids)
+    aggregate_load = torch.bincount(
+        topk_ids[valid_routes].cpu(), minlength=expert_count
+    )
+
+    per_prompt = []
+    used_sets: dict[str, set[int]] = {}
+    for batch_index, prompt in enumerate(prompts):
+        length = int(attention_mask[batch_index].sum())
+        ids = input_ids[batch_index, :length].tolist()
+        tokens = tokenizer.convert_ids_to_tokens(ids)
+        routes = topk_ids[batch_index, :length].cpu()
+        weights = topk_weights[batch_index, :length].float().cpu()
+        load = torch.bincount(routes.flatten(), minlength=expert_count)
+        used_sets[prompt["id"]] = set(torch.nonzero(load, as_tuple=False).flatten().tolist())
+        token_rows = []
+        for position, (token_id, token, experts, scores) in enumerate(
+            zip(ids, tokens, routes.tolist(), weights.tolist(), strict=True)
+        ):
+            ordered = sorted(
+                zip(experts, scores, strict=True),
+                key=lambda pair: pair[1],
+                reverse=True,
+            )
+            token_rows.append(
+                {
+                    "position": position,
+                    "token_id": token_id,
+                    "token": token,
+                    "decoded_piece": tokenizer.decode(
+                        [token_id],
+                        skip_special_tokens=False,
+                        clean_up_tokenization_spaces=False,
+                    ),
+                    "experts_by_weight": [
+                        {"expert": expert, "weight": weight}
+                        for expert, weight in ordered
+                    ],
+                    "selected_weight_sum": sum(scores),
+                    "top1_top2_margin": ordered[0][1] - ordered[1][1],
+                }
+            )
+        top_experts = sorted(
+            enumerate(load.tolist()), key=lambda pair: pair[1], reverse=True
+        )[:8]
+        per_prompt.append(
+            {
+                "id": prompt["id"],
+                "label": prompt["label"],
+                "tokens": length,
+                "load": load.tolist(),
+                "metrics": load_metrics(load),
+                "top_experts": [
+                    {"expert": expert, "routes": routes}
+                    for expert, routes in top_experts
+                    if routes
+                ],
+                "token_routes": token_rows,
+            }
+        )
+
+    jaccard = []
+    for left, right in combinations(prompts, 2):
+        a = used_sets[left["id"]]
+        b = used_sets[right["id"]]
+        jaccard.append(
+            {
+                "left": left["id"],
+                "right": right["id"],
+                "used_expert_jaccard": len(a & b) / len(a | b),
+                "shared_experts": len(a & b),
+                "union_experts": len(a | b),
+            }
+        )
+
+    valid_weights = topk_weights[valid_routes].float().cpu().view(-1, top_k)
+    weight_sums = valid_weights.sum(dim=-1)
+    ordered_weights = valid_weights.sort(dim=-1, descending=True).values
+    top_experts = sorted(
+        enumerate(aggregate_load.tolist()), key=lambda pair: pair[1], reverse=True
+    )[:12]
+    return {
+        "aggregate_load": aggregate_load.tolist(),
+        "aggregate_metrics": load_metrics(aggregate_load),
+        "top_experts": [
+            {"expert": expert, "routes": routes}
+            for expert, routes in top_experts
+            if routes
+        ],
+        "selected_weight_sum": {
+            "mean": weight_sums.mean().item(),
+            "p10": percentile(weight_sums, 0.10),
+            "p50": percentile(weight_sums, 0.50),
+            "p90": percentile(weight_sums, 0.90),
+        },
+        "top1_top2_margin": {
+            "mean": (ordered_weights[:, 0] - ordered_weights[:, 1]).mean().item(),
+            "p50": percentile(ordered_weights[:, 0] - ordered_weights[:, 1], 0.50),
+            "p90": percentile(ordered_weights[:, 0] - ordered_weights[:, 1], 0.90),
+        },
+        "per_prompt": per_prompt,
+        "prompt_pair_jaccard": jaccard,
+    }
+
+
+def nvidia_smi() -> dict[str, str]:
+    fields = [
+        "name",
+        "driver_version",
+        "memory.total",
+        "power.limit",
+        "clocks.max.sm",
+    ]
+    output = subprocess.check_output(
+        [
+            "nvidia-smi",
+            f"--query-gpu={','.join(fields)}",
+            "--format=csv,noheader,nounits",
+        ],
+        text=True,
+    ).strip()
+    return dict(zip(fields, [item.strip() for item in output.split(",")], strict=True))
+
+
+def main() -> None:
+    args = parse_args()
+    root = args.artifact_dir.resolve()
+    shard = root / "model-00001-of-000004.safetensors"
+    required = [
+        root / "config.json",
+        root / "configuration_deepseek.py",
+        root / "modeling_deepseek.py",
+        root / "model.safetensors.index.json",
+        root / "tokenizer.json",
+        shard,
+    ]
+    missing = [str(path) for path in required if not path.exists()]
+    if missing:
+        raise FileNotFoundError(f"missing official artifacts: {missing}")
+    if args.device.startswith("cuda") and not torch.cuda.is_available():
+        raise RuntimeError("CUDA requested but unavailable")
+    if not 1 <= args.layers <= 7:
+        raise ValueError("shard 1 fully contains only layers 0 through 6")
+
+    configuration, modeling = load_official_modules(root)
+    config = configuration.DeepseekV2Config.from_pretrained(root)
+    config._attn_implementation = "eager"
+    tokenizer = AutoTokenizer.from_pretrained(
+        root,
+        trust_remote_code=True,
+        local_files_only=True,
+    )
+    if tokenizer.pad_token_id is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    tokenizer.padding_side = "right"
+
+    encoded = tokenizer(
+        [prompt["text"] for prompt in DEFAULT_PROMPTS],
+        add_special_tokens=True,
+        padding=True,
+        return_tensors="pt",
+    )
+    input_ids = encoded.input_ids
+    attention_mask_cpu = encoded.attention_mask
+    device = torch.device(args.device)
+
+    start = time.perf_counter()
+    with safe_open(shard, framework="pt", device="cpu") as handle:
+        embedding = handle.get_tensor("model.embed_tokens.weight")
+        hidden = F.embedding(input_ids, embedding).to(device)
+    embedding_ms = (time.perf_counter() - start) * 1000
+
+    attention_mask = attention_mask_cpu.to(device)
+    sequence = input_ids.shape[1]
+    position_ids = torch.arange(sequence, device=device).unsqueeze(0)
+    causal_mask = modeling._prepare_4d_causal_attention_mask(
+        attention_mask,
+        input_ids.shape,
+        hidden,
+        0,
+    )
+    cache = DynamicCache()
+    layer_results = []
+    initial_hidden = hidden_stats(hidden, attention_mask)
+
+    for layer_index in range(args.layers):
+        prefix = f"model.layers.{layer_index}."
+        load_start = time.perf_counter()
+        with safe_open(shard, framework="pt", device="cpu") as handle:
+            state = {
+                key[len(prefix) :]: handle.get_tensor(key)
+                for key in handle.keys()
+                if key.startswith(prefix)
+            }
+        state_numel = sum(value.numel() for value in state.values())
+        state_bytes = sum(value.numel() * value.element_size() for value in state.values())
+        with torch.device("meta"):
+            layer = modeling.DeepseekV2DecoderLayer(config, layer_index)
+        layer.to_empty(device="cpu")
+        layer.load_state_dict(state, strict=True, assign=True)
+        del state
+        load_ms = (time.perf_counter() - load_start) * 1000
+
+        torch.cuda.reset_peak_memory_stats(device) if device.type == "cuda" else None
+        transfer_start = time.perf_counter()
+        layer = layer.to(device=device, dtype=torch.bfloat16).eval()
+        if device.type == "cuda":
+            torch.cuda.synchronize(device)
+        transfer_ms = (time.perf_counter() - transfer_start) * 1000
+
+        captures: dict[str, torch.Tensor] = {}
+
+        def capture_kv(_module: Any, _inputs: Any, output: torch.Tensor) -> None:
+            captures["compressed_kv"] = output.detach()
+
+        handles = [
+            layer.self_attn.kv_a_proj_with_mqa.register_forward_hook(capture_kv)
+        ]
+        if layer_index > 0:
+
+            def capture_gate(_module: Any, _inputs: Any, output: Any) -> None:
+                captures["topk_ids"] = output[0].detach()
+                captures["topk_weights"] = output[1].detach()
+
+            handles.append(layer.mlp.gate.register_forward_hook(capture_gate))
+
+        before = hidden_stats(hidden, attention_mask)
+        if device.type == "cuda":
+            torch.cuda.synchronize(device)
+        forward_start = time.perf_counter()
+        with torch.inference_mode():
+            outputs = layer(
+                hidden,
+                attention_mask=causal_mask,
+                position_ids=position_ids,
+                past_key_value=cache,
+                use_cache=True,
+            )
+        hidden = outputs[0]
+        if device.type == "cuda":
+            torch.cuda.synchronize(device)
+        forward_ms = (time.perf_counter() - forward_start) * 1000
+        after = hidden_stats(hidden, attention_mask)
+        for handle in handles:
+            handle.remove()
+
+        compressed = captures["compressed_kv"]
+        latent = compressed[..., : config.kv_lora_rank]
+        rope_key = compressed[..., config.kv_lora_rank :]
+        key_cache = cache.key_cache[layer_index]
+        value_cache = cache.value_cache[layer_index]
+        result: dict[str, Any] = {
+            "layer": layer_index,
+            "ffn": "dense" if layer_index == 0 else "moe",
+            "state_tensors": len(layer.state_dict()),
+            "state_numel": state_numel,
+            "state_bytes": state_bytes,
+            "timing_ms": {
+                "load_cpu": load_ms,
+                "transfer_to_device": transfer_ms,
+                "forward": forward_ms,
+            },
+            "peak_device_allocated_mib": (
+                torch.cuda.max_memory_allocated(device) / 2**20
+                if device.type == "cuda"
+                else None
+            ),
+            "hidden_before": before,
+            "hidden_after": after,
+            "mla": {
+                "compressed_projection_shape": list(compressed.shape),
+                "latent_shape": list(latent.shape),
+                "rope_key_shape": list(rope_key.shape),
+                "latent_rms": latent.float().square().mean().sqrt().item(),
+                "rope_key_rms": rope_key.float().square().mean().sqrt().item(),
+                "eager_key_cache_shape": list(key_cache.shape),
+                "eager_value_cache_shape": list(value_cache.shape),
+                "eager_cache_bytes": key_cache.numel() * key_cache.element_size()
+                + value_cache.numel() * value_cache.element_size(),
+            },
+        }
+        if layer_index > 0:
+            topk_ids = captures["topk_ids"].view(
+                input_ids.shape[0], input_ids.shape[1], -1
+            )
+            topk_weights = captures["topk_weights"].view(
+                input_ids.shape[0], input_ids.shape[1], -1
+            )
+            result["routing"] = route_trace(
+                tokenizer,
+                input_ids,
+                attention_mask_cpu,
+                topk_ids.cpu(),
+                topk_weights.cpu(),
+                DEFAULT_PROMPTS,
+                config.n_routed_experts,
+            )
+        layer_results.append(result)
+
+        del layer, compressed, latent, rope_key, key_cache, value_cache
+        captures.clear()
+        gc.collect()
+        if device.type == "cuda":
+            torch.cuda.empty_cache()
+
+    valid_token_count = int(attention_mask_cpu.sum())
+    latent_elements = config.kv_lora_rank + config.qk_rope_head_dim
+    eager_elements = config.num_attention_heads * (
+        config.qk_nope_head_dim + config.qk_rope_head_dim + config.v_head_dim
+    )
+    captured_at = args.captured_at or datetime.now(timezone.utc).isoformat()
+    index = json.loads((root / "model.safetensors.index.json").read_text())
+    result = {
+        "schema_version": 1,
+        "captured_at": captured_at,
+        "evidence_identity": "X / official BF16 weights and tokenizer, local truncated forward",
+        "boundary": {
+            "model": "DeepSeek-V2-Lite base",
+            "executed_layers": list(range(args.layers)),
+            "total_model_layers": config.num_hidden_layers,
+            "full_model_generation": False,
+            "training_or_global_expert_load": False,
+            "expert_semantics_inferred": False,
+            "prompt_sample": "four authored prompts; descriptive trace, not population estimate",
+            "cache_note": "HF eager materializes expanded K/V; latent-cache arithmetic is derived from official dimensions",
+        },
+        "provenance": {
+            "huggingface_model": "deepseek-ai/DeepSeek-V2-Lite",
+            "huggingface_revision": "604d5664dddd88a0433dbae533b7fe9472482de0",
+            "sha256": {
+                "config": sha256(root / "config.json"),
+                "modeling_code": sha256(root / "modeling_deepseek.py"),
+                "tokenizer": sha256(root / "tokenizer.json"),
+                "index": sha256(root / "model.safetensors.index.json"),
+                "shard_1": sha256(shard),
+            },
+            "checkpoint_tensor_bytes": index["metadata"]["total_size"],
+            "shard_1_bytes": shard.stat().st_size,
+        },
+        "environment": {
+            "python": platform.python_version(),
+            "platform": platform.platform(),
+            "libc": list(platform.libc_ver()),
+            "torch": torch.__version__,
+            "torch_cuda": torch.version.cuda,
+            "transformers": __import__("transformers").__version__,
+            "safetensors": __import__("safetensors").__version__,
+            "device": str(device),
+            "nvidia_smi": nvidia_smi() if device.type == "cuda" else None,
+            "matmul_allow_tf32": torch.backends.cuda.matmul.allow_tf32
+            if device.type == "cuda"
+            else None,
+        },
+        "configuration": {
+            "total_parameters_reported": "15.7B",
+            "activated_parameters_reported": "2.4B",
+            "layers": config.num_hidden_layers,
+            "hidden": config.hidden_size,
+            "attention_heads": config.num_attention_heads,
+            "qk_nope_head_dim": config.qk_nope_head_dim,
+            "qk_rope_head_dim": config.qk_rope_head_dim,
+            "v_head_dim": config.v_head_dim,
+            "kv_lora_rank": config.kv_lora_rank,
+            "routed_experts": config.n_routed_experts,
+            "active_routed_experts": config.num_experts_per_tok,
+            "shared_experts": config.n_shared_experts,
+            "expert_intermediate": config.moe_intermediate_size,
+            "first_dense_layers": config.first_k_dense_replace,
+            "router_scoring": config.scoring_func,
+            "router_topk_method": config.topk_method,
+            "normalize_selected_weights": config.norm_topk_prob,
+        },
+        "prompts": [
+            {
+                **prompt,
+                "token_ids": input_ids[index, : int(attention_mask_cpu[index].sum())].tolist(),
+                "tokens": tokenizer.convert_ids_to_tokens(
+                    input_ids[index, : int(attention_mask_cpu[index].sum())].tolist()
+                ),
+            }
+            for index, prompt in enumerate(DEFAULT_PROMPTS)
+        ],
+        "execution": {
+            "batch": input_ids.shape[0],
+            "padded_sequence": input_ids.shape[1],
+            "valid_tokens": valid_token_count,
+            "embedding_ms": embedding_ms,
+            "initial_hidden": initial_hidden,
+            "layers": layer_results,
+            "final_hidden": hidden_stats(hidden, attention_mask),
+        },
+        "cache_accounting": {
+            "dtype": "BF16",
+            "latent_elements_per_token_layer": latent_elements,
+            "latent_bytes_per_token_layer": latent_elements * 2,
+            "hf_eager_elements_per_token_layer": eager_elements,
+            "hf_eager_bytes_per_token_layer": eager_elements * 2,
+            "eager_over_latent_ratio": eager_elements / latent_elements,
+            "latent_reduction_vs_eager": 1 - latent_elements / eager_elements,
+            "components": {
+                "latent_content": config.kv_lora_rank,
+                "rope_key": config.qk_rope_head_dim,
+                "expanded_key": config.num_attention_heads
+                * (config.qk_nope_head_dim + config.qk_rope_head_dim),
+                "expanded_value": config.num_attention_heads * config.v_head_dim,
+            },
+        },
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, ensure_ascii=False, indent=2) + "\n")
+    print(json.dumps(result, ensure_ascii=False, indent=2))
+
+
+if __name__ == "__main__":
+    main()
diff --git a/research/DEEPSEEK_RESEARCH.md b/research/DEEPSEEK_RESEARCH.md
index d00358b..f909c8a 100644
--- a/research/DEEPSEEK_RESEARCH.md
+++ b/research/DEEPSEEK_RESEARCH.md
@@ -1,7 +1,7 @@
-# DeepSeek 技术谱系二轮正式研究账本
+# DeepSeek 技术谱系正式研究账本（含三轮真实权重执行）
 
 > 研究截止：2026-07-29  
-> 课程角色：DeepSeek 聚光专题二轮；与 MoE、长上下文、训练系统、数值、推理、Agent、评测专题互相链接，但不替代各专题完整推导。  
+> 课程角色：DeepSeek 聚光专题三轮；与 MoE、长上下文、训练系统、数值、推理、Agent、评测专题互相链接，但不替代各专题完整推导。
 > 证据规则：正文事实只来自一手论文、作者官方仓库和 Kimi K3 官方报告；Grok 产物仅见 `DEEPSEEK_GROK_LEADS.md`，不承担证据。  
 > 简化规则：所有二维图、成本滑条和训练曲线若非论文复跑，必须标“教学模型”。
 
@@ -557,7 +557,7 @@ total bytes = per-token-layer · L · T · B · bytes
 - 至少 24 张问题账；
 - 至少 20 个正文目录；
 - 60 个一手/官方阅读节点；
-- 四个独立可操作实验；
+- 八个独立可操作实验：四个公式/教学实验 + 四个真实工件实验；
 - DeepSeekMath 必须在主时间线中；
 - DAPO / Dr.GRPO 必须标后续公开研究；
 - MLA 实验必须把 RoPE cache 算进去；
@@ -568,3 +568,30 @@ total bytes = per-token-layer · L · T · B · bytes
 - tabs 支持键盘方向键；
 - toy model、作者报告和公式推导使用不同标签；
 - 专属 Chrome 回归并纳入全站回归。
+
+## 8. 第三轮真实权重执行增量
+
+第三轮固定官方 `deepseek-ai/DeepSeek-V2-Lite` revision
+`604d5664dddd88a0433dbae533b7fe9472482de0`，在 RTX 5090 上用官方 tokenizer、
+模型代码与 BF16 第一分片连续执行 layer 0–6。
+
+已闭合：
+
+- 4 条固定 prompt、90 个有效 token；
+- 6 个 MoE 层、每 token top-6，共 3,240 次真实 routed-expert 选择；
+- 每层 `[4,27,576]` MLA 压缩投影；
+- HF eager 实际 key `[4,16,27,192]`、value `[4,16,27,128]`；
+- V2-Lite 逐 token/层 BF16 的 1,152-byte latent 合同与 10,240-byte eager 物化账；
+- 两次独立执行在来源、tokenization、hidden hashes、MLA shapes、loads 与全部 routes 上
+  `31/31 exact`，计时排除。
+
+永久边界：
+
+- 覆盖率不等于均衡，必须同时看 CV、Gini 与 effective experts；
+- expert ID 不赋予语义，跨层同号 ID 不是同一参数；
+- `88.75%` 是 V2-Lite latent vs HF eager 的 shape 算术，不冒充 V2 报告 `93.3%`；
+- layer 7 因跨 shard 停止，不把 7/27 层写成完整模型生成；
+- 原始权重不进 Git，工件以 revision、SHA-256、脚本、trace 与复跑比较固定。
+
+完整方法、数值表与复现入口见
+`research/DEEPSEEK_V2_LITE_TRACE.md`。
diff --git a/research/DEEPSEEK_V2_LITE_TRACE.md b/research/DEEPSEEK_V2_LITE_TRACE.md
new file mode 100644
index 0000000..4f13e48
--- /dev/null
+++ b/research/DEEPSEEK_V2_LITE_TRACE.md
@@ -0,0 +1,340 @@
+# DeepSeek-V2-Lite 真实权重执行审计
+
+> 状态：DeepSeek 专题第三轮工件；2026-07-29  
+> 目的：把“MLA 会压缩缓存”“MoE 会选择专家”从论文概念推进到固定官方权重、真实 tokenizer、真实 hidden states 和逐 token 路由记录。  
+> 边界：这不是完整模型 benchmark、训练 trace、生产服务复现、专家语义分析或总体负载估计。
+
+## 0. 一页结论
+
+这次实验固定官方 `deepseek-ai/DeepSeek-V2-Lite` revision
+`604d5664dddd88a0433dbae533b7fe9472482de0`，使用官方 tokenizer、配置、模型代码和
+BF16 checkpoint 第一分片，在 RTX 5090 上依次执行 embedding 与 decoder layer 0–6。
+第 0 层是 dense FFN，第 1–6 层是完整 MoE 层；第 7 层横跨第一、第二分片，因此明确停止。
+
+四条最重要的结果：
+
+1. 4 条人工固定 prompt 共 90 个有效 token，在 6 个 MoE 层产生
+   `6 layers × 90 tokens × top-6 = 3,240` 次真实 routed-expert 选择。
+2. 每个执行层都实际观测到 MLA 的 `[B,T,576]` 压缩投影；其中 512 是
+   KV latent，64 是 decoupled RoPE key。
+3. 官方 Hugging Face eager 实现仍把 cache 物化为
+   key `[4,16,27,192]` 和 value `[4,16,27,128]`。以 BF16 计，
+   每 token、每层是 10,240 bytes；按配置保存 latent + RoPE key 只需 1,152 bytes，
+   相差 `8.8889×`，后者相对前者减少 `88.75%`。
+4. 两次独立执行的来源、配置、tokenization、hidden-state hashes、MLA shapes、
+   aggregate loads 与全部 token routes 共 `31/31` 项完全一致；计时明确不参与一致性检查。
+
+这里最反直觉的发现不是某个“神奇专家”，而是：
+
+- **覆盖不等于均衡。** 90 个 token 在单层可触达 60–64/64 个专家，但各层
+  CV 为 0.549–0.925，Gini 为 0.303–0.420。
+- **路由 ID 不等于语义标签。** `E8` 只表示某一层内部数组的第 8 号专家；跨层的
+  `E8` 是不同参数，四条 prompt 也不足以赋予“代码专家”等语义。
+- **算法压缩不等于实现已经压缩存储。** MLA 的低秩状态合同与 HF eager cache
+  的实际物化格式是两张账，必须同时报告。
+
+## 1. 证据身份
+
+### 1.1 官方工件
+
+| 工件 | 固定值 |
+|---|---|
+| Hugging Face model | `deepseek-ai/DeepSeek-V2-Lite` |
+| revision | `604d5664dddd88a0433dbae533b7fe9472482de0` |
+| checkpoint tensor bytes | `31,412,968,448`，约 29.261 GiB |
+| shard 1 bytes | `8,594,887,408`，约 8.005 GiB |
+| config SHA-256 | `f346286b0f1c8b044252fd54cb4fa78b9fab6472a6e8bebb9edfe03d414ea03d` |
+| modeling code SHA-256 | `7d8e5221095286eea991137760893fd7ba52727c0b4ebf48ec09e8bc56b45b9c` |
+| tokenizer SHA-256 | `41f3bf64213da8c012d8bd0871a58a1fdf70463e8f08f110ddbb1082f529f669` |
+| index SHA-256 | `d2cdb2f325f6682cf3ad1ad2526a9f979d857390b579380c0331d975136e0acf` |
+| shard 1 SHA-256 | `0d7e9f39bde40111a4c0f390b87497dce4565cf578d916395e6b2c7851f1e8da` |
+
+第一分片完整容纳 embedding、layer 0–6 和 layer 7 的一部分。为了不把缺参数的
+layer 7 冒充完整执行，本实验把证据边界钉在 layer 6。
+
+原始 checkpoint 分片不进入 Git；仓库只提交：
+
+- 可复现执行脚本；
+- 机器可读 trace；
+- 独立复跑比较结果；
+- 研究审计与网页可视化。
+
+### 1.2 模型配置
+
+| 对象 | 官方配置 |
+|---|---:|
+| reported total / active | 15.7B / 2.4B |
+| decoder layers | 27 |
+| hidden size | 2048 |
+| attention heads | 16 |
+| qk NoPE / RoPE head dim | 128 / 64 |
+| value head dim | 128 |
+| KV LoRA rank | 512 |
+| routed experts / active | 64 / 6 |
+| shared experts | 2 |
+| expert intermediate | 1408 |
+| dense prefix | 1 layer |
+| router | softmax + greedy top-k |
+| selected top-k normalization | false |
+
+`max_position_embeddings=163840` 与 YaRN 配置属于 checkpoint config；
+官方发布表中的上下文口径不应被本实验自行改写成同一个数字。
+
+### 1.3 软件与硬件
+
+| 对象 | 值 |
+|---|---|
+| GPU | NVIDIA GeForce RTX 5090，32,607 MiB |
+| driver | 595.84 |
+| Python | 3.10.14 |
+| PyTorch | 2.11.0+cu128 |
+| CUDA runtime | 12.8 |
+| Transformers | 4.41.2 |
+| safetensors | 0.8.0 |
+| TF32 matmul | disabled |
+
+官方 2024 remote code 在当前 Transformers 5.5 环境中会因已移除的
+`is_torch_fx_available` 入口而导入失败。因此实验使用隔离的 Transformers 4.41.2。
+模型源码保持只读、不打补丁；runner 只通过本地 package 方式导入官方
+`configuration_deepseek.py` 与 `modeling_deepseek.py`。
+
+这条版本边界很重要：**“模型代码公开”不等于“任意未来依赖版本都可直接运行”。**
+
+## 2. 实验设计
+
+### 2.1 为什么选择截断的精确 forward
+
+完整 15.7B BF16 权重与运行时状态虽然可能勉强接近单卡边界，但没有必要为了观察
+前六层真实路由而把“全模型能否高效生成”混进同一实验。本 runner：
+
+1. 用官方 tokenizer 一次编码 4 条 prompt；
+2. 从第一分片读取官方 embedding；
+3. 每次在 meta device 构造一个官方 decoder layer；
+4. 为这一层绑定真实 safetensors 参数并传到 GPU；
+5. 用上层输出作为下一层输入；
+6. 通过 hook 捕获实际 `kv_a_proj_with_mqa` 与实际 MoE gate 输出；
+7. 释放当前层，再加载下一层。
+
+因此得到的是**真实权重、真实层类、真实连续 hidden states**，而不是随机初始化、
+只读 config 或合成 router；但它仍不是完整 27 层生成。
+
+### 2.2 固定 prompt
+
+| ID | 标签 | token |
+|---|---|---:|
+| `zh_explanation` | 中文解释 | 26 |
+| `en_architecture` | English architecture | 18 |
+| `code` | Python code | 19 |
+| `math` | 数学推理 | 27 |
+
+batch padding 后形状为 `B=4, T=27`，有效 token 共 90。prompt 是用于建立可复现
+探针的人工样本，不是自然语料分布、benchmark 数据集或统计代表样本。
+
+## 3. MLA：一张算法状态账，一张实现物化账
+
+### 3.1 实际观测
+
+每个执行层的 `kv_a_proj_with_mqa` 输出都是：
+
+```text
+[batch, sequence, kv_lora_rank + qk_rope_head_dim]
+= [4, 27, 512 + 64]
+= [4, 27, 576]
+```
+
+这 576 个元素可以分为：
+
+- `c_t^KV`：512 维 joint latent；
+- `k_t^R`：64 维 decoupled RoPE key。
+
+后者不能因为体积小就从缓存公式中消失。
+
+### 3.2 HF eager 实际保存了什么
+
+本次 `use_cache=True` 的官方 Hugging Face eager 路径实际返回：
+
+```text
+key   [4, 16, 27, 192]
+value [4, 16, 27, 128]
+```
+
+所以每 token、每层的缓存元素是：
+
+```text
+expanded eager = 16 × (192 + 128) = 5,120 elements
+latent contract = 512 + 64 = 576 elements
+```
+
+在 BF16 下：
+
+```text
+eager bytes  = 5,120 × 2 = 10,240
+latent bytes =   576 × 2 =  1,152
+ratio        = 10,240 / 1,152 = 8.8889×
+reduction    = 1 - 1,152 / 10,240 = 88.75%
+```
+
+### 3.3 为什么这不是论文 93.3% 的复跑
+
+V2 报告的 `93.3%` 是完整 V2 相对 DeepSeek 67B、在论文指定配置和比较口径下的
+作者报告结果。本实验的 `88.75%` 是 V2-Lite 配置中：
+
+- HF eager 展开 K/V 元素；
+- 与 latent + RoPE key 理论存储元素；
+
+之间的逐 token、逐层 BF16 算术比较。模型、维度、基线和实现口径都不同，
+二者不能互相替代。
+
+进一步说，低秩架构只定义“可以保存什么最小状态”；生产系统是否真的只保存 latent，
+还取决于 attention kernel、权重吸收、RoPE 分叉、paged cache 和服务框架。
+
+## 4. MoE：真实路由显微镜
+
+### 4.1 总量守恒
+
+每个 MoE 层都满足：
+
+```text
+90 valid tokens × top-6 = 540 routed selections
+```
+
+6 个层合计 `3,240` 次。两条 shared expert 等价路径始终运行，但它们不属于 router
+输出的 0–63 routed expert ID，因此不写进 top-6 列表。
+
+### 4.2 六层聚合负载
+
+| layer | selections | used / 64 | zero | CV | Gini | effective experts | top expert |
+|---:|---:|---:|---:|---:|---:|---:|---|
+| 1 | 540 | 63 | 1 | 0.925 | 0.420 | 46.66 | E8 · 48 |
+| 2 | 540 | 64 | 0 | 0.549 | 0.303 | 55.08 | E62 · 23 |
+| 3 | 540 | 62 | 2 | 0.569 | 0.322 | 53.51 | E54 · 19 |
+| 4 | 540 | 62 | 2 | 0.866 | 0.417 | 47.03 | E48 · 40 |
+| 5 | 540 | 60 | 4 | 0.642 | 0.332 | 52.14 | E14 · 32 |
+| 6 | 540 | 61 | 3 | 0.765 | 0.378 | 49.48 | E25 · 40 |
+
+三个指标回答不同问题：
+
+- `used experts`：这批 token 是否至少触达某专家一次；
+- `CV = std / mean`：计数离散程度；
+- `effective experts = exp(entropy)`：若把当前熵换算成均匀分布，相当于多少个专家。
+
+因此 layer 1 触达 63 个专家，却只有 46.66 个 effective experts，且 E8 承担 48/540
+次选择。只报“63/64 被使用”会掩盖负载长尾。
+
+### 4.3 prompt 之间看到的差异
+
+同一层内，四条 prompt 的 used-expert set Jaccard 与 CV 会变化。例如：
+
+- layer 1：中文 prompt 使用 54 个专家、CV 0.898；英文架构 prompt 使用 40 个、
+  CV 1.360；代码 prompt 44 个、CV 1.263；数学 prompt 45 个、CV 1.184。
+- layer 4：聚合 top expert 是 E48；英文和代码各把 E48 路由 11 次，
+  中文 prompt 的 top expert 却是 E29、16 次。
+- layer 6：英文 prompt 使用 42 个专家、CV 1.439；中文为 49 个、1.031；
+  数学为 48 个、0.915。
+
+这些是**描述性 route traces**，说明 token 与上下文确实改变 gate 输出；它们不证明
+“E48 是代码专家”，更不允许以四条 prompt 推断训练语料上的领域分工。
+
+### 4.4 top-6 权重没有重新归一
+
+配置 `norm_topk_prob=false`。router 先对 64 个专家做 softmax，再选 top-6；
+被选中的 6 个概率不会重新缩放到和为 1。各层选中权重和的均值：
+
+| layer | mean selected top-6 weight sum |
+|---:|---:|
+| 1 | 0.3949 |
+| 2 | 0.4801 |
+| 3 | 0.4898 |
+| 4 | 0.4749 |
+| 5 | 0.4832 |
+| 6 | 0.4440 |
+
+所以网页显示某 token 的六个权重和小于 1 是真实配置行为，不是漏算。
+
+## 5. 可复现性闸门
+
+正式 trace：
+
+```text
+src/data/deepseek-v2-lite-trace.json
+SHA-256 81ff4ab34d20121fe42219c566dd4780f4c7f74252a31b596f90e5280e1f1aef
+captured 2026-07-29T05:55:35.599912+00:00
+```
+
+第二次独立执行：
+
+```text
+SHA-256 b5e2e7567e17765195313473499522a4529b9d37ea96963ba2b9455c3c0b86e6
+captured 2026-07-29T05:56:32.527770+00:00
+```
+
+比较结果：
+
+```text
+31 / 31 exact
+timing compared: false
+```
+
+精确检查覆盖：
+
+- provenance、configuration、prompt tokenization；
+- initial / final hidden hashes；
+- layer 0–6 hidden hashes 与 MLA shapes；
+- layer 1–6 aggregate loads；
+- layer 1–6 全部 token routes。
+
+计时会受首次 kernel、频率、温度、后台进程与同步位置影响，因此没有伪装成 deterministic
+evidence。本轮只把计时保留在原始 trace，网站不把它包装成性能 benchmark。
+
+## 6. 绝对禁止的外推
+
+| 本实验能够说 | 本实验不能说 |
+|---|---|
+| 固定 revision 的前 7 层产生了这些 hidden states | 完整模型会生成什么回答 |
+| 四条 prompt 在 6 层产生这些 top-6 routes | 训练集或线上请求的总体专家负载 |
+| 某层某 expert ID 被选中多少次 | 该专家具有什么稳定语义 |
+| HF eager 物化了展开 K/V | 所有 MLA serving kernel 都这样缓存 |
+| V2-Lite 配置可导出 576 元素 latent state | 完整 V2 的 93.3% 已被复跑 |
+| 两次相同环境执行 31/31 exact | 跨 GPU、依赖版本和 kernel 仍 bitwise identical |
+| layer 0–6 在第一分片内完整 | layer 7 或 layer 8–26 已执行 |
+
+跨层连线也被禁止：layer 1 的 E8 与 layer 2 的 E8 是不同参数对象。可视化只能在同一层内
+比较 expert load，不能画一条 “E8 专长” 纵贯 6 层。
+
+## 7. 复现入口
+
+环境与命令见：
+
+- `experiments/deepseek/README.md`
+- `experiments/deepseek/v2_lite_trace.py`
+- `experiments/deepseek/compare_v2_lite_traces.py`
+
+执行一次：
+
+```bash
+python experiments/deepseek/v2_lite_trace.py \
+  --artifact-dir /path/to/deepseek-v2-lite \
+  --output /path/to/trace.json
+```
+
+比较两次：
+
+```bash
+python experiments/deepseek/compare_v2_lite_traces.py \
+  --first /path/to/trace-1.json \
+  --second /path/to/trace-2.json \
+  --output /path/to/repro.json
+```
+
+## 8. 下一道研究闸门
+
+本轮已经把 DeepSeek 第三轮从“计划真实 trace”推进到真实 tokenizer、MLA state、
+MoE routes 与确定性复跑，但仍有四块未闭合：
+
+1. 用真正只保存 latent 的 MLA kernel 对照 HF eager materialization；
+2. 扩大公开语料样本，给 aggregate load 置信区间，并考察 batch / padding / sequence
+   对 expert parallel 的影响；
+3. 复跑 FP8、pipeline 或通信 trace，而不是继续用教学 schedule；
+4. 做可审计的 R1-like 小模型训练实验，记录 rollout、reward、KL、长度与梯度统计。
+
+在这些闸门完成之前，DeepSeek 专题保持“第三轮进行中”，不写成完整独立复现。
diff --git a/scripts/check-deepseek-browser.mjs b/scripts/check-deepseek-browser.mjs
index e80b70b..37a02d5 100644
--- a/scripts/check-deepseek-browser.mjs
+++ b/scripts/check-deepseek-browser.mjs
@@ -73,6 +73,9 @@ const overview = await evaluate(`(() => ({
   paperLinks: document.querySelectorAll("[data-deepseek-paper-chain] a").length,
   labTabs: document.querySelectorAll("[data-ds-tab]").length,
   labPanels: document.querySelectorAll("[data-ds-panel]").length,
+  artifactTabs: document.querySelectorAll("[data-artifact-tab]").length,
+  artifactPanels: document.querySelectorAll("[data-artifact-panel]").length,
+  artifactLayers: document.querySelectorAll(".layer-evidence > span").length,
   branches: document.querySelectorAll(".branch-grid > a").length,
   followups: document.querySelectorAll(".lineage-row.followup").length,
   navLinks: document.querySelectorAll(".top-nav a").length,
@@ -195,12 +198,111 @@ const rl = await evaluate(`(() => {
   };
 })()`);
 
+const artifactRoute = await evaluate(`(() => {
+  const root = document.querySelector("[data-dsv2-lab]");
+  const read = () => ({
+    panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
+    position: root.querySelector("[data-route-position]").textContent.trim(),
+    piece: root.querySelector("[data-route-piece]").textContent.trim(),
+    tokenId: root.querySelector("[data-route-token-id]").textContent.trim(),
+    weightSum: root.querySelector("[data-route-weight-sum]").textContent.trim(),
+    chosen: [...root.querySelectorAll("[data-route-experts] article")].map((node) => ({
+      expert: node.querySelector("b").textContent.trim(),
+      weight: node.querySelector("small").textContent.trim(),
+    })),
+    heatCells: root.querySelectorAll("[data-route-heatmap] > span").length,
+    selectedCells: root.querySelectorAll("[data-route-heatmap] > span.selected").length,
+    routes: root.querySelector("[data-route-count]").textContent.trim(),
+    used: root.querySelector("[data-route-used]").textContent.trim(),
+    cv: root.querySelector("[data-route-cv]").textContent.trim(),
+    effective: root.querySelector("[data-route-effective]").textContent.trim(),
+  });
+  const initial = read();
+  const layer = root.querySelector("[data-route-layer]");
+  const prompt = root.querySelector("[data-route-prompt]");
+  const token = root.querySelector("[data-route-token]");
+  layer.value = "4";
+  layer.dispatchEvent(new Event("change", { bubbles: true }));
+  prompt.value = "en_architecture";
+  prompt.dispatchEvent(new Event("change", { bubbles: true }));
+  token.value = String(token.options.length - 1);
+  token.dispatchEvent(new Event("change", { bubbles: true }));
+  const switched = read();
+  return { initial, switched, tokenOptions: token.options.length };
+})()`);
+
+const artifactLoad = await evaluate(`(() => {
+  const root = document.querySelector("[data-dsv2-lab]");
+  root.querySelector('[data-artifact-tab="load"]').click();
+  const read = () => ({
+    panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
+    used: root.querySelector("[data-load-used]").textContent.trim(),
+    zero: root.querySelector("[data-load-zero]").textContent.trim(),
+    cv: root.querySelector("[data-load-cv]").textContent.trim(),
+    gini: root.querySelector("[data-load-gini]").textContent.trim(),
+    effective: root.querySelector("[data-load-effective]").textContent.trim(),
+    rows: root.querySelectorAll("[data-load-rows] > div").length,
+    jaccards: root.querySelectorAll("[data-load-jaccard] > article").length,
+  });
+  const layer1 = read();
+  root.querySelector('[data-load-layer="2"]').click();
+  const layer2 = read();
+  root.querySelector('[data-load-layer="4"]').click();
+  const layer4 = read();
+  return { layer1, layer2, layer4 };
+})()`);
+
+const artifactCache = await evaluate(`(() => {
+  const root = document.querySelector("[data-dsv2-lab]");
+  root.querySelector('[data-artifact-tab="cache"]').click();
+  const read = () => ({
+    panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
+    latent: root.querySelector("[data-cache-latent-bytes]").textContent.trim(),
+    eager: root.querySelector("[data-cache-eager-bytes]").textContent.trim(),
+    ratio: root.querySelector("[data-cache-ratio]").textContent.trim(),
+    reduction: root.querySelector("[data-cache-reduction]").textContent.trim(),
+  });
+  const trace = read();
+  const context = root.querySelector("[data-cache-context]");
+  const batch = root.querySelector("[data-cache-batch]");
+  const layers = root.querySelector("[data-cache-layers]");
+  context.value = "1048576";
+  context.dispatchEvent(new Event("change", { bubbles: true }));
+  batch.value = "8";
+  batch.dispatchEvent(new Event("input", { bubbles: true }));
+  layers.value = "27";
+  layers.dispatchEvent(new Event("input", { bubbles: true }));
+  const million = read();
+  return { trace, million };
+})()`);
+
+const artifactEvidence = await evaluate(`(() => {
+  const root = document.querySelector("[data-dsv2-lab]");
+  root.querySelector('[data-artifact-tab="evidence"]').click();
+  const result = {
+    panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
+    layers: root.querySelectorAll(".layer-evidence > span").length,
+    executed: root.querySelectorAll(".layer-evidence > span.executed").length,
+    split: root.querySelectorAll(".layer-evidence > span.split").length,
+    unloaded: root.querySelectorAll(".layer-evidence > span.unloaded").length,
+    exact: root.querySelector(".repro-gate strong").textContent.trim(),
+    dependency: root.querySelector(".dependency-split").textContent.replaceAll(/\\s+/g, " ").trim(),
+    boundary: root.querySelector('[data-artifact-panel="evidence"] .artifact-boundary').textContent.replaceAll(/\\s+/g, " ").trim(),
+  };
+  const first = root.querySelector('[data-artifact-tab="route"]');
+  first.focus();
+  first.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowRight", bubbles: true }));
+  result.keyboardSelected = root.querySelector('[data-artifact-tab][aria-selected="true"]').dataset.artifactTab;
+  result.keyboardVisible = root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel;
+  return result;
+})()`);
+
 await evaluate(`(() => {
-  document.querySelector("[data-deepseek-lab]").scrollIntoView({ block: "start", behavior: "instant" });
+  document.querySelector("[data-dsv2-lab]").scrollIntoView({ block: "start", behavior: "instant" });
   window.scrollBy(0, -82);
 })()`);
 await pause(180);
-await screenshot("/tmp/llm-atlas-deepseek-lab-desktop.png");
+await screenshot("/tmp/llm-atlas-deepseek-artifact-desktop.png");
 
 await navigate("/");
 const home = await evaluate(`(() => ({
@@ -233,6 +335,7 @@ await command("Emulation.setDeviceMetricsOverride", {
 await navigate("/deepseek/");
 const mobile = await evaluate(`(() => {
   const root = document.querySelector("[data-deepseek-lab]");
+  const artifact = document.querySelector("[data-dsv2-lab]");
   root.scrollIntoView({ block: "start", behavior: "instant" });
   const toggle = document.querySelector("#menu-toggle");
   toggle?.click();
@@ -242,8 +345,10 @@ const mobile = await evaluate(`(() => {
     menuOpen: toggle.getAttribute("aria-expanded"),
     mobileLinks: document.querySelectorAll("#mobile-nav a").length,
     tabs: root.querySelectorAll("[data-ds-tab]").length,
+    artifactTabs: artifact.querySelectorAll("[data-artifact-tab]").length,
+    artifactHeatCells: artifact.querySelectorAll("[data-route-heatmap] > span").length,
     offenders: [...document.querySelectorAll("body *")]
-      .filter((node) => !node.closest(".paper-chain, .advantage-table, .precision-table, .mapping-table, [data-deepseek-lab]"))
+      .filter((node) => !node.closest(".paper-chain, .advantage-table, .precision-table, .mapping-table, [data-deepseek-lab], [data-dsv2-lab]"))
       .filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1)
       .slice(0, 12)
       .map((node) => ({
@@ -256,21 +361,23 @@ const mobile = await evaluate(`(() => {
 })()`);
 await evaluate(`(() => {
   document.querySelector("#menu-toggle")?.click();
+  document.querySelector("[data-dsv2-lab]").scrollIntoView({ block: "start", behavior: "instant" });
   window.scrollBy(0, -82);
 })()`);
 await pause(180);
-await screenshot("/tmp/llm-atlas-deepseek-mobile.png");
+await screenshot("/tmp/llm-atlas-deepseek-artifact-mobile.png");
 
-const report = { overview, capacity, cache, codesign, rl, home, papers, mobile, exceptions };
+const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactEvidence, home, papers, mobile, exceptions };
 console.log(JSON.stringify(report, null, 2));
 
 const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
 const failures = [];
 if (!overview.title.includes("为什么转向")) failures.push("专题标题异常");
-if (overview.sections !== 25 || overview.tocLinks !== 25) failures.push("二十四个编号专题加阅读链的目录结构异常");
+if (overview.sections !== 26 || overview.tocLinks !== 26) failures.push("二十五个编号专题加阅读链的目录结构异常");
 if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
 if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
 if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
+if (overview.artifactTabs !== 4 || overview.artifactPanels !== 4 || overview.artifactLayers !== 27) failures.push("真实权重四联实验结构异常");
 if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
 if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
 if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
@@ -287,9 +394,19 @@ if (rl.initial.signal !== "GROUP-RELATIVE SIGNAL" || rl.same.signal !== "ZERO GR
 if (!rl.dapo.provenance.includes("2503.14476") || !rl.dapo.algorithm.includes("FOLLOW-UP") || !rl.dr.provenance.includes("2503.20783")) failures.push("DAPO / Dr.GRPO 来源边界异常");
 if (!rl.r1.includes("cold start") || !rl.distill.includes("没有重演")) failures.push("R1 / distill 身份切换异常");
 if (rl.keyboardSelected !== "cache" || rl.keyboardVisible !== "cache") failures.push("实验键盘 tab 导航异常");
+if (artifactRoute.initial.panel !== "route" || artifactRoute.initial.chosen.length !== 6 || artifactRoute.initial.heatCells !== 64 || artifactRoute.initial.selectedCells !== 6) failures.push("真实 top-6 路由结构异常");
+if (artifactRoute.initial.routes !== "156" || artifactRoute.initial.used !== "54 / 64" || artifactRoute.initial.weightSum !== "0.4540") failures.push("Layer 1 中文 token 路由初值异常");
+if (artifactRoute.switched.chosen.length !== 6 || artifactRoute.switched.heatCells !== 64 || artifactRoute.tokenOptions !== 18 || numeric(artifactRoute.switched.weightSum) >= 1) failures.push("路由层 / prompt / token 切换异常");
+if (artifactLoad.layer1.panel !== "load" || artifactLoad.layer1.used !== "63" || artifactLoad.layer1.cv !== "0.925" || artifactLoad.layer1.rows !== 5 || artifactLoad.layer1.jaccards !== 6) failures.push("Layer 1 聚合负载账异常");
+if (artifactLoad.layer2.used !== "64" || artifactLoad.layer2.cv !== "0.549" || artifactLoad.layer4.used !== "62" || artifactLoad.layer4.gini !== "0.417") failures.push("跨层负载统计切换异常");
+if (artifactCache.trace.panel !== "cache" || artifactCache.trace.latent !== "850.50 KiB" || artifactCache.trace.eager !== "7.38 MiB" || artifactCache.trace.ratio !== "8.89×" || artifactCache.trace.reduction !== "88.75%") failures.push("V2-Lite trace 缓存实现账异常");
+if (!artifactCache.million.latent.includes("GiB") || !artifactCache.million.eager.includes("TiB")) failures.push("V2-Lite 百万 Token 缓存外推异常");
+if (artifactEvidence.panel !== "evidence" || artifactEvidence.layers !== 27 || artifactEvidence.executed !== 7 || artifactEvidence.split !== 1 || artifactEvidence.unloaded !== 19 || artifactEvidence.exact !== "31 / 31") failures.push("真实工件执行边界或复跑闸门异常");
+if (!artifactEvidence.dependency.includes("Transformers 5.5") || !artifactEvidence.dependency.includes("4.41.2") || !artifactEvidence.boundary.includes("完整 27 层生成")) failures.push("依赖版本或未覆盖边界异常");
+if (artifactEvidence.keyboardSelected !== "load" || artifactEvidence.keyboardVisible !== "load") failures.push("真实工件实验键盘 tab 导航异常");
 if (home.releaseCards !== 17 || !home.firstRelease.includes("47 页不再压成摘要") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 DeepSeek 首发入口或论文数异常");
 if (papers.total !== 486 || !papers.hasFilter || papers.visible < 20 || !papers.hasCoder || !papers.hasEngram) failures.push("论文库 DeepSeek 聚光异常");
-if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
+if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 4 || mobile.artifactHeatCells !== 64) failures.push("移动端导航或实验异常");
 if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
 if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
 
diff --git a/src/components/DeepSeekArtifactLab.astro b/src/components/DeepSeekArtifactLab.astro
new file mode 100644
index 0000000..23e018c
--- /dev/null
+++ b/src/components/DeepSeekArtifactLab.astro
@@ -0,0 +1,1084 @@
+---
+import rawTrace from "@/data/deepseek-v2-lite-trace.json";
+import rawRepro from "@/data/deepseek-v2-lite-trace-repro.json";
+
+const trace = rawTrace as any;
+const repro = rawRepro as any;
+const compact = {
+  prompts: trace.prompts.map((prompt: any) => ({
+    id: prompt.id,
+    label: prompt.label,
+    text: prompt.text,
+  })),
+  layers: trace.execution.layers.slice(1).map((layer: any) => ({
+    layer: layer.layer,
+    load: layer.routing.aggregate_load,
+    metrics: layer.routing.aggregate_metrics,
+    selectedWeight: layer.routing.selected_weight_sum,
+    topExperts: layer.routing.top_experts,
+    prompts: layer.routing.per_prompt.map((prompt: any) => ({
+      id: prompt.id,
+      label: prompt.label,
+      tokens: prompt.tokens,
+      load: prompt.load,
+      metrics: prompt.metrics,
+      topExperts: prompt.top_experts,
+      tokenRoutes: prompt.token_routes,
+    })),
+    jaccard: layer.routing.prompt_pair_jaccard,
+  })),
+  cache: trace.cache_accounting,
+  execution: {
+    batch: trace.execution.batch,
+    paddedSequence: trace.execution.padded_sequence,
+    validTokens: trace.execution.valid_tokens,
+  },
+};
+const compactJson = JSON.stringify(compact).replaceAll("<", "\\u003c");
+const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoint_tensor_bytes;
+---
+
+<figure class="artifact-lab" data-dsv2-lab>
+  <header class="artifact-head">
+    <div>
+      <p>REAL-WEIGHT LAB / DEEPSEEK-V2-LITE</p>
+      <h3>这一次不是调公式：让官方权重真的走过七层</h3>
+    </div>
+    <p>
+      固定官方 revision、tokenizer、模型代码和 BF16 第一分片；RTX 5090 连续执行 layer 0–6，
+      捕获真实 MLA 状态与 3,240 次 routed-expert 选择。所有结论都带证据身份与停止线。
+    </p>
+  </header>
+
+  <div class="artifact-status">
+    <span><i>X</i><b>本机执行</b>真实权重与 hidden states</span>
+    <span><i>O</i><b>官方工件</b>配置、代码与 checkpoint</span>
+    <span><i>D</i><b>确定推导</b>从 shape 计算 bytes</span>
+    <span><i>U</i><b>未覆盖</b>全模型、训练与生产 kernel</span>
+  </div>
+
+  <div class="artifact-tabs" role="tablist" aria-label="选择 DeepSeek V2-Lite 真实工件实验">
+    <button type="button" role="tab" data-artifact-tab="route" aria-selected="true">
+      <span>01</span><b>路由显微镜</b><small>token → top-6 experts</small>
+    </button>
+    <button type="button" role="tab" data-artifact-tab="load" aria-selected="false" tabindex="-1">
+      <span>02</span><b>覆盖 ≠ 均衡</b><small>CV · Gini · entropy</small>
+    </button>
+    <button type="button" role="tab" data-artifact-tab="cache" aria-selected="false" tabindex="-1">
+      <span>03</span><b>缓存实现账</b><small>latent vs HF eager</small>
+    </button>
+    <button type="button" role="tab" data-artifact-tab="evidence" aria-selected="false" tabindex="-1">
+      <span>04</span><b>证据断面</b><small>revision · shards · rerun</small>
+    </button>
+  </div>
+
+  <section class="artifact-panel" data-artifact-panel="route">
+    <div class="panel-lead">
+      <div><span>X / TOKEN-LEVEL ROUTES</span><h4>先选层，再选 prompt 和 token</h4></div>
+      <p>同一个 expert ID 只在当前层内有意义；跨层同号专家是不同参数，图中不会把它们连成“专长轨迹”。</p>
+    </div>
+
+    <div class="route-controls">
+      <label>
+        <span>decoder layer</span>
+        <select data-route-layer aria-label="选择路由层">
+          {[1, 2, 3, 4, 5, 6].map((layer) => <option value={layer}>layer {layer}</option>)}
+        </select>
+      </label>
+      <label>
+        <span>prompt</span>
+        <select data-route-prompt aria-label="选择 prompt">
+          {trace.prompts.map((prompt: any) => <option value={prompt.id}>{prompt.label} · {prompt.token_ids.length} tokens</option>)}
+        </select>
+      </label>
+      <label class="token-select">
+        <span>token position / decoded piece</span>
+        <select data-route-token aria-label="选择 token"></select>
+      </label>
+    </div>
+
+    <div class="token-context">
+      <span>PROMPT</span>
+      <p data-route-prompt-text></p>
+    </div>
+
+    <div class="route-stage">
+      <article class="token-card">
+        <span>TOKEN <b data-route-position>#0</b></span>
+        <strong data-route-piece></strong>
+        <small>ID <b data-route-token-id></b></small>
+      </article>
+      <i aria-hidden="true">→</i>
+      <div>
+        <span class="micro-title">ACTUAL TOP-6 / WEIGHT DESCENDING</span>
+        <div class="chosen-experts" data-route-experts></div>
+      </div>
+      <i aria-hidden="true">→</i>
+      <article class="sum-card">
+        <span>SELECTED WEIGHT SUM</span>
+        <strong data-route-weight-sum></strong>
+        <small>配置不把 top-6 重新归一到 1</small>
+      </article>
+    </div>
+
+    <div class="heat-head">
+      <div>
+        <span>64 ROUTED EXPERTS / CURRENT PROMPT</span>
+        <b data-route-heat-title></b>
+      </div>
+      <p>颜色表示这条 prompt 在当前层的选择次数；空白表示本小样本未触达，不表示专家失效。</p>
+    </div>
+    <div class="expert-heatmap" data-route-heatmap aria-label="64 个 routed expert 负载热图"></div>
+
+    <div class="route-metrics">
+      <article><span>ROUTES</span><b data-route-count></b><p>tokens × top-6</p></article>
+      <article><span>USED / 64</span><b data-route-used></b><p>至少被选一次</p></article>
+      <article><span>CV</span><b data-route-cv></b><p>std ÷ mean</p></article>
+      <article><span>EFFECTIVE</span><b data-route-effective></b><p>exp(route entropy)</p></article>
+    </div>
+
+    <div class="artifact-boundary">
+      <b>X / DESCRIPTIVE TRACE</b>
+      <p>这是 4 条固定 prompt、90 个有效 token 的前六个 MoE 层；不能据此命名专家、估计线上总体负载或判断训练均衡。</p>
+    </div>
+  </section>
+
+  <section class="artifact-panel" data-artifact-panel="load" hidden>
+    <div class="panel-lead">
+      <div><span>X + D / LOAD ACCOUNTING</span><h4>“64 个专家几乎都用到了”为什么仍可能不均衡</h4></div>
+      <p>覆盖只问是否出现过；CV、Gini 与 effective experts 才描述计数分布。这里同时展示 aggregate 与四条 prompt。</p>
+    </div>
+
+    <div class="layer-switch" role="group" aria-label="选择负载统计层">
+      {[1, 2, 3, 4, 5, 6].map((layer) => (
+        <button type="button" data-load-layer={layer} class={layer === 1 ? "active" : ""}>L{layer}</button>
+      ))}
+    </div>
+
+    <div class="aggregate-card">
+      <div>
+        <span>AGGREGATE / 90 TOKENS</span>
+        <strong><b data-load-used></b> / 64 used</strong>
+        <p><span data-load-zero></span> 个专家在本样本为零；总选择数固定为 540。</p>
+      </div>
+      <div class="load-dials">
+        <article><span>CV</span><b data-load-cv></b><i><em data-load-cv-bar></em></i></article>
+        <article><span>GINI</span><b data-load-gini></b><i><em data-load-gini-bar></em></i></article>
+        <article><span>EFFECTIVE</span><b data-load-effective></b><i><em data-load-effective-bar></em></i></article>
+      </div>
+    </div>
+
+    <div class="load-table" role="table" aria-label="按 prompt 的专家负载统计">
+      <div class="head" role="row">
+        <b role="columnheader">SCOPE</b><b role="columnheader">TOKENS</b><b role="columnheader">USED</b>
+        <b role="columnheader">CV</b><b role="columnheader">GINI</b><b role="columnheader">TOP EXPERT</b>
+      </div>
+      <div data-load-rows></div>
+    </div>
+
+    <div class="jaccard-block">
+      <div>
+        <span>USED-SET OVERLAP</span>
+        <h5>同层 prompt 两两 Jaccard</h5>
+        <p>只比较“用过哪些 ID”，不比较权重、次数或语义。</p>
+      </div>
+      <div data-load-jaccard></div>
+    </div>
+
+    <div class="load-lessons">
+      <article><span>FALSE SHORTCUT</span><b>63 / 64 used → balanced</b><p>Layer 1 虽触达 63 个专家，CV 仍为 0.925，E8 被选 48 次。</p></article>
+      <article><span>CORRECT READING</span><b>同看覆盖、离散度和熵</b><p>Layer 2 的 64/64、CV 0.549 与 effective 55.08 才构成较完整的描述。</p></article>
+      <article><span>HARD BOUNDARY</span><b>ID is not semantics</b><p>Layer 4 的 E48 只是该层参数索引；不能命名成“代码专家”。</p></article>
+    </div>
+  </section>
+
+  <section class="artifact-panel" data-artifact-panel="cache" hidden>
+    <div class="panel-lead">
+      <div><span>X + D / IMPLEMENTATION GAP</span><h4>MLA 定义了可压缩状态，不保证框架已经按压缩格式存</h4></div>
+      <p>上半区是本次真实 shape；下半区只把相同维度扩展到可调 batch、context、layers 和 dtype。</p>
+    </div>
+
+    <div class="observed-cache">
+      <article>
+        <span>X / ACTUAL PROJECTION</span>
+        <strong>[4, 27, <b>576</b>]</strong>
+        <p>512 KV latent + 64 decoupled RoPE key</p>
+      </article>
+      <i aria-hidden="true">≠</i>
+      <article class="expanded">
+        <span>X / HF EAGER CACHE</span>
+        <strong>K [4,16,27,<b>192</b>]</strong>
+        <strong>V [4,16,27,<b>128</b>]</strong>
+        <p>官方 eager 路径物化展开后的多头 K/V</p>
+      </article>
+    </div>
+
+    <div class="cache-controls">
+      <label><span>batch B <output data-cache-batch-label>4</output></span><input data-cache-batch type="range" min="1" max="32" value="4" /></label>
+      <label><span>context T</span><select data-cache-context><option value="27">27 / 本次 trace</option><option value="4096">4K</option><option value="32768">32K</option><option value="131072">128K</option><option value="1048576">1M</option></select></label>
+      <label><span>layers L <output data-cache-layers-label>7</output></span><input data-cache-layers type="range" min="1" max="27" value="7" /></label>
+      <label><span>storage</span><select data-cache-dtype><option value="1">1 byte</option><option value="2" selected>BF16 / 2 bytes</option><option value="4">FP32 / 4 bytes</option></select></label>
+    </div>
+
+    <div class="cache-ledger">
+      <article>
+        <header><span>D / LATENT CONTRACT</span><b>576 elements / token / layer</b></header>
+        <div class="cache-bar latent"><i></i></div>
+        <strong data-cache-latent-bytes></strong>
+        <p>512 latent + 64 RoPE key；需要配套吸收与 attention kernel 才能兑现。</p>
+      </article>
+      <article>
+        <header><span>X→D / HF EAGER FORMAT</span><b>5,120 elements / token / layer</b></header>
+        <div class="cache-bar eager"><i></i></div>
+        <strong data-cache-eager-bytes></strong>
+        <p>16 × (192 key + 128 value)；按真实 shape 外推，不是服务显存 benchmark。</p>
+      </article>
+    </div>
+
+    <div class="cache-ratio">
+      <article><span>EAGER / LATENT</span><b data-cache-ratio></b><p>相同 B/T/L/dtype 下的元素倍数</p></article>
+      <article><span>LATENT REDUCTION</span><b data-cache-reduction></b><p>相对本次 HF eager 物化口径</p></article>
+      <article class="warning"><span>NOT REPRODUCED</span><b>93.3%</b><p>这是 V2 报告的另一模型 / 基线 / 配置口径</p></article>
+    </div>
+
+    <div class="artifact-boundary">
+      <b>IMPLEMENTATION CONTRACT</b>
+      <p>算法上“可以缓存 576 元素”与本次框架“实际缓存 5,120 元素”同时为真；生产吞吐还需要真实 latent-cache kernel 与 serving runtime。</p>
+    </div>
+  </section>
+
+  <section class="artifact-panel" data-artifact-panel="evidence" hidden>
+    <div class="panel-lead">
+      <div><span>O + X / EVIDENCE SLICE</span><h4>为什么执行到 layer 6 就停，而不是把“部分下载”写成“完整复现”</h4></div>
+      <p>第一分片完整包含 layer 0–6；layer 7 跨两个分片。停止线由 checkpoint index 决定，不由页面叙事决定。</p>
+    </div>
+
+    <div class="artifact-identity">
+      <article>
+        <span>OFFICIAL MODEL</span>
+        <b>deepseek-ai/DeepSeek-V2-Lite</b>
+        <code>604d5664…2482de0</code>
+      </article>
+      <article>
+        <span>CHECKPOINT</span>
+        <b>29.261 GiB · 4 shards</b>
+        <code>shard 1 · 8.005 GiB · {(shardFraction * 100).toFixed(1)}%</code>
+      </article>
+      <article>
+        <span>EXECUTION</span>
+        <b>RTX 5090 · BF16</b>
+        <code>torch 2.11.0+cu128</code>
+      </article>
+    </div>
+
+    <div class="layer-evidence" aria-label="27 层模型的实际执行边界">
+      {Array.from({ length: 27 }, (_, layer) => (
+        <span class={layer <= 6 ? "executed" : layer === 7 ? "split" : "unloaded"} title={`layer ${layer}`}>
+          <b>{layer}</b><small>{layer <= 6 ? "X" : layer === 7 ? "SPLIT" : "U"}</small>
+        </span>
+      ))}
+    </div>
+    <div class="layer-legend">
+      <span><i class="executed"></i><b>0–6</b> 完整参数 + 连续 forward</span>
+      <span><i class="split"></i><b>7</b> 横跨 shard 1 / 2，未执行</span>
+      <span><i class="unloaded"></i><b>8–26</b> 未下载 / 未执行</span>
+    </div>
+
+    <div class="repro-gate">
+      <div>
+        <span>INDEPENDENT RERUN</span>
+        <strong>{repro.exact_checks} / {repro.total_checks}</strong>
+        <b>EXACT</b>
+      </div>
+      <ul>
+        <li>provenance、configuration、tokenization</li>
+        <li>initial / final 与逐层 hidden hashes</li>
+        <li>7 层 MLA shapes</li>
+        <li>6 层 aggregate loads 与全部 token routes</li>
+      </ul>
+      <p><b>计时不比较：</b>kernel warm-up、频率与系统噪声不属于确定性证据。</p>
+    </div>
+
+    <div class="dependency-split">
+      <article>
+        <span>FAILED CURRENT STACK</span>
+        <b>Transformers 5.5</b>
+        <p>官方 2024 remote code 引用已移除的 <code>is_torch_fx_available</code>。</p>
+      </article>
+      <i aria-hidden="true">→</i>
+      <article>
+        <span>PINNED READ-ONLY STACK</span>
+        <b>Transformers 4.41.2</b>
+        <p>隔离依赖、本地 package 导入；官方模型源码不打补丁。</p>
+      </article>
+    </div>
+
+    <div class="checksum-grid">
+      <article><span>TRACE SHA-256</span><code>81ff4ab34d20…1f1aef</code></article>
+      <article><span>SHARD 1 SHA-256</span><code>0d7e9f39bde4…1f1e8da</code></article>
+      <article><span>MODEL CODE SHA-256</span><code>7d8e52210952…b45b9c</code></article>
+      <article><span>TOKENIZER SHA-256</span><code>41f3bf64213d…29f669</code></article>
+    </div>
+
+    <div class="evidence-links">
+      <a href="https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite" rel="noreferrer">官方模型与 model card ↗</a>
+      <a href="https://github.com/deepseek-ai/DeepSeek-V2" rel="noreferrer">官方 DeepSeek-V2 仓库 ↗</a>
+      <a href="https://arxiv.org/abs/2405.04434" rel="noreferrer">DeepSeek-V2 技术报告 ↗</a>
+    </div>
+
+    <div class="artifact-boundary">
+      <b>U / STILL OPEN</b>
+      <p>完整 27 层生成、真实 latent-cache kernel、生产服务、训练负载、FP8/pipeline 与 R1-like 训练 trace 仍未覆盖。</p>
+    </div>
+  </section>
+
+  <figcaption>
+    <span>可复现入口</span>
+    <code>experiments/deepseek/v2_lite_trace.py</code> ·
+    <code>src/data/deepseek-v2-lite-trace.json</code> ·
+    <code>research/DEEPSEEK_V2_LITE_TRACE.md</code>
+  </figcaption>
+
+  <script is:inline type="application/json" data-dsv2-trace set:html={compactJson}></script>
+</figure>
+
+<script>
+  document.querySelectorAll<HTMLElement>("[data-dsv2-lab]").forEach((root) => {
+    const one = <T extends Element>(selector: string) => root.querySelector<T>(selector);
+    const all = <T extends Element>(selector: string) => [...root.querySelectorAll<T>(selector)];
+    const set = (selector: string, value: string) => {
+      const node = one<HTMLElement>(selector);
+      if (node) node.textContent = value;
+    };
+    const payloadNode = one<HTMLScriptElement>("[data-dsv2-trace]");
+    if (!payloadNode?.textContent) return;
+    const data = JSON.parse(payloadNode.textContent);
+
+    const tabs = all<HTMLButtonElement>("[data-artifact-tab]");
+    const panels = all<HTMLElement>("[data-artifact-panel]");
+    const selectTab = (target: HTMLButtonElement) => {
+      tabs.forEach((tab) => {
+        const selected = tab === target;
+        tab.setAttribute("aria-selected", String(selected));
+        tab.tabIndex = selected ? 0 : -1;
+      });
+      panels.forEach((panel) => panel.hidden = panel.dataset.artifactPanel !== target.dataset.artifactTab);
+    };
+    tabs.forEach((tab, index) => {
+      tab.addEventListener("click", () => selectTab(tab));
+      tab.addEventListener("keydown", (event) => {
+        if (!["ArrowLeft", "ArrowRight", "Home", "End"].includes(event.key)) return;
+        event.preventDefault();
+        let next = index;
+        if (event.key === "ArrowRight") next = (index + 1) % tabs.length;
+        if (event.key === "ArrowLeft") next = (index - 1 + tabs.length) % tabs.length;
+        if (event.key === "Home") next = 0;
+        if (event.key === "End") next = tabs.length - 1;
+        tabs[next].focus();
+        selectTab(tabs[next]);
+      });
+    });
+
+    const layerById = (layer: number) => data.layers.find((item: any) => item.layer === layer);
+    const promptMeta = (id: string) => data.prompts.find((prompt: any) => prompt.id === id);
+    const formatPiece = (piece: string) => piece === "" ? "∅" : piece.replaceAll("\n", "↵").replaceAll(" ", "␠");
+
+    const routeLayer = one<HTMLSelectElement>("[data-route-layer]");
+    const routePrompt = one<HTMLSelectElement>("[data-route-prompt]");
+    const routeToken = one<HTMLSelectElement>("[data-route-token]");
+    const renderRoute = (resetTokens = false) => {
+      if (!routeLayer || !routePrompt || !routeToken) return;
+      const layer = layerById(Number(routeLayer.value));
+      const prompt = layer.prompts.find((item: any) => item.id === routePrompt.value);
+      const meta = promptMeta(prompt.id);
+      if (resetTokens || !routeToken.options.length || Number(routeToken.value) >= prompt.tokenRoutes.length) {
+        routeToken.replaceChildren(...prompt.tokenRoutes.map((token: any) => {
+          const option = document.createElement("option");
+          option.value = String(token.position);
+          option.textContent = `#${token.position} · ${formatPiece(token.decoded_piece)} · id ${token.token_id}`;
+          return option;
+        }));
+      }
+      const token = prompt.tokenRoutes[Number(routeToken.value) || 0];
+      set("[data-route-prompt-text]", meta.text);
+      set("[data-route-position]", `#${token.position}`);
+      set("[data-route-piece]", formatPiece(token.decoded_piece));
+      set("[data-route-token-id]", String(token.token_id));
+      set("[data-route-weight-sum]", token.selected_weight_sum.toFixed(4));
+      set("[data-route-heat-title]", `${prompt.label} · layer ${layer.layer}`);
+      set("[data-route-count]", prompt.metrics.routes.toLocaleString());
+      set("[data-route-used]", `${prompt.metrics.used_experts} / 64`);
+      set("[data-route-cv]", prompt.metrics.cv.toFixed(3));
+      set("[data-route-effective]", prompt.metrics.effective_experts.toFixed(2));
+
+      const experts = one<HTMLElement>("[data-route-experts]");
+      if (experts) {
+        experts.replaceChildren(...token.experts_by_weight.map((entry: any, index: number) => {
+          const node = document.createElement("article");
+          const rank = document.createElement("span");
+          const expert = document.createElement("b");
+          const weight = document.createElement("small");
+          rank.textContent = `#${index + 1}`;
+          expert.textContent = `E${entry.expert}`;
+          weight.textContent = entry.weight.toFixed(4);
+          node.append(rank, expert, weight);
+          return node;
+        }));
+      }
+
+      const heatmap = one<HTMLElement>("[data-route-heatmap]");
+      if (heatmap) {
+        const max = Math.max(...prompt.load, 1);
+        heatmap.replaceChildren(...prompt.load.map((count: number, expert: number) => {
+          const cell = document.createElement("span");
+          const label = document.createElement("b");
+          const value = document.createElement("small");
+          cell.style.setProperty("--load", String(count / max));
+          cell.classList.toggle("selected", token.experts_by_weight.some((entry: any) => entry.expert === expert));
+          cell.title = `layer ${layer.layer} · E${expert} · ${count} routes`;
+          label.textContent = `E${expert}`;
+          value.textContent = String(count);
+          cell.append(label, value);
+          return cell;
+        }));
+      }
+    };
+    routeLayer?.addEventListener("change", () => renderRoute(true));
+    routePrompt?.addEventListener("change", () => renderRoute(true));
+    routeToken?.addEventListener("change", () => renderRoute(false));
+    renderRoute(true);
+
+    const promptNames = Object.fromEntries(data.prompts.map((prompt: any) => [prompt.id, prompt.label]));
+    const renderLoad = (layerNumber: number) => {
+      const layer = layerById(layerNumber);
+      all<HTMLButtonElement>("[data-load-layer]").forEach((button) => {
+        button.classList.toggle("active", Number(button.dataset.loadLayer) === layerNumber);
+      });
+      set("[data-load-used]", String(layer.metrics.used_experts));
+      set("[data-load-zero]", String(layer.metrics.zero_experts));
+      set("[data-load-cv]", layer.metrics.cv.toFixed(3));
+      set("[data-load-gini]", layer.metrics.gini.toFixed(3));
+      set("[data-load-effective]", layer.metrics.effective_experts.toFixed(2));
+      one<HTMLElement>("[data-load-cv-bar]")?.style.setProperty("--fraction", String(Math.min(layer.metrics.cv / 1.5, 1)));
+      one<HTMLElement>("[data-load-gini-bar]")?.style.setProperty("--fraction", String(layer.metrics.gini));
+      one<HTMLElement>("[data-load-effective-bar]")?.style.setProperty("--fraction", String(layer.metrics.effective_experts / 64));
+
+      const rows = one<HTMLElement>("[data-load-rows]");
+      if (rows) {
+        const aggregate = {
+          label: "全部 prompt",
+          tokens: data.execution.validTokens,
+          metrics: layer.metrics,
+          topExperts: layer.topExperts,
+        };
+        rows.replaceChildren(...[aggregate, ...layer.prompts].map((prompt: any, index: number) => {
+          const row = document.createElement("div");
+          if (index === 0) row.className = "aggregate";
+          [
+            prompt.label,
+            String(prompt.tokens),
+            `${prompt.metrics.used_experts}/64`,
+            prompt.metrics.cv.toFixed(3),
+            prompt.metrics.gini.toFixed(3),
+            `E${prompt.topExperts[0].expert} · ${prompt.topExperts[0].routes}`,
+          ].forEach((value) => {
+            const cell = document.createElement(index === 0 ? "b" : "span");
+            cell.textContent = value;
+            row.append(cell);
+          });
+          return row;
+        }));
+      }
+
+      const jaccard = one<HTMLElement>("[data-load-jaccard]");
+      if (jaccard) {
+        jaccard.replaceChildren(...layer.jaccard.map((pair: any) => {
+          const row = document.createElement("article");
+          const labels = document.createElement("span");
+          const track = document.createElement("i");
+          const fill = document.createElement("em");
+          const value = document.createElement("b");
+          labels.textContent = `${promptNames[pair.left]} ↔ ${promptNames[pair.right]}`;
+          fill.style.setProperty("--fraction", String(pair.used_expert_jaccard));
+          track.append(fill);
+          value.textContent = pair.used_expert_jaccard.toFixed(3);
+          row.append(labels, track, value);
+          return row;
+        }));
+      }
+    };
+    all<HTMLButtonElement>("[data-load-layer]").forEach((button) => {
+      button.addEventListener("click", () => renderLoad(Number(button.dataset.loadLayer)));
+    });
+    renderLoad(1);
+
+    const compactBytes = (bytes: number) => {
+      if (bytes >= 1024 ** 4) return `${(bytes / 1024 ** 4).toFixed(2)} TiB`;
+      if (bytes >= 1024 ** 3) return `${(bytes / 1024 ** 3).toFixed(2)} GiB`;
+      if (bytes >= 1024 ** 2) return `${(bytes / 1024 ** 2).toFixed(2)} MiB`;
+      if (bytes >= 1024) return `${(bytes / 1024).toFixed(2)} KiB`;
+      return `${bytes.toLocaleString()} B`;
+    };
+    const renderCache = () => {
+      const batch = Number(one<HTMLInputElement>("[data-cache-batch]")?.value ?? 4);
+      const context = Number(one<HTMLSelectElement>("[data-cache-context]")?.value ?? 27);
+      const layers = Number(one<HTMLInputElement>("[data-cache-layers]")?.value ?? 7);
+      const bytes = Number(one<HTMLSelectElement>("[data-cache-dtype]")?.value ?? 2);
+      const latent = batch * context * layers * data.cache.latent_elements_per_token_layer * bytes;
+      const eager = batch * context * layers * data.cache.hf_eager_elements_per_token_layer * bytes;
+      set("[data-cache-batch-label]", String(batch));
+      set("[data-cache-layers-label]", String(layers));
+      set("[data-cache-latent-bytes]", compactBytes(latent));
+      set("[data-cache-eager-bytes]", compactBytes(eager));
+      set("[data-cache-ratio]", `${data.cache.eager_over_latent_ratio.toFixed(2)}×`);
+      set("[data-cache-reduction]", `${(data.cache.latent_reduction_vs_eager * 100).toFixed(2)}%`);
+    };
+    all<HTMLInputElement | HTMLSelectElement>("[data-cache-batch], [data-cache-context], [data-cache-layers], [data-cache-dtype]")
+      .forEach((control) => {
+        control.addEventListener("input", renderCache);
+        control.addEventListener("change", renderCache);
+      });
+    renderCache();
+  });
+</script>
+
+<style>
+  .artifact-lab {
+    --ink: #202027;
+    --paper: #f4f0e7;
+    --blue: #62699b;
+    --teal: #39786e;
+    --amber: #ba762c;
+    --red: #a14d4d;
+    margin: 2rem 0 0;
+    border: 1px solid rgba(32, 32, 39, .22);
+    background: var(--paper);
+    color: var(--ink);
+    box-shadow: 10px 12px 0 rgba(32, 32, 39, .07);
+    overflow: hidden;
+  }
+  .artifact-head {
+    display: grid;
+    grid-template-columns: minmax(0, 1.05fr) minmax(18rem, .95fr);
+    gap: 2rem;
+    padding: 2rem;
+    background:
+      linear-gradient(115deg, rgba(98, 105, 155, .17), transparent 45%),
+      repeating-linear-gradient(90deg, transparent 0 39px, rgba(32,32,39,.035) 39px 40px);
+    border-bottom: 1px solid rgba(32, 32, 39, .18);
+  }
+  .artifact-head > div > p,
+  .panel-lead span,
+  .micro-title,
+  .heat-head span,
+  .aggregate-card span,
+  .jaccard-block span,
+  .observed-cache span,
+  .artifact-identity span,
+  .repro-gate span,
+  .dependency-split span,
+  .checksum-grid span {
+    margin: 0;
+    color: var(--blue);
+    font: 700 .69rem/1.3 var(--font-mono);
+    letter-spacing: .1em;
+  }
+  .artifact-head h3 {
+    margin: .55rem 0 0;
+    max-width: 14ch;
+    font: 720 clamp(1.65rem, 3.1vw, 2.8rem)/.99 var(--font-display);
+    letter-spacing: -.045em;
+  }
+  .artifact-head > p {
+    align-self: end;
+    margin: 0;
+    color: rgba(32, 32, 39, .72);
+    font-size: .94rem;
+    line-height: 1.75;
+  }
+  .artifact-status {
+    display: grid;
+    grid-template-columns: repeat(4, 1fr);
+    border-bottom: 1px solid rgba(32, 32, 39, .16);
+    background: #ece6da;
+  }
+  .artifact-status span {
+    display: grid;
+    grid-template-columns: 1.75rem auto;
+    align-items: center;
+    gap: .05rem .55rem;
+    padding: .8rem 1rem;
+    border-right: 1px solid rgba(32, 32, 39, .12);
+    color: rgba(32,32,39,.65);
+    font-size: .69rem;
+  }
+  .artifact-status span:last-child { border-right: 0; }
+  .artifact-status i {
+    grid-row: 1 / 3;
+    display: grid;
+    place-items: center;
+    width: 1.65rem;
+    height: 1.65rem;
+    border-radius: 50%;
+    background: var(--ink);
+    color: var(--paper);
+    font: 800 .7rem/1 var(--font-mono);
+  }
+  .artifact-status span:nth-child(1) i { background: var(--teal); }
+  .artifact-status span:nth-child(2) i { background: var(--blue); }
+  .artifact-status span:nth-child(3) i { background: var(--amber); }
+  .artifact-status span:nth-child(4) i { background: var(--red); }
+  .artifact-status b { color: var(--ink); font-size: .72rem; }
+  .artifact-tabs {
+    display: grid;
+    grid-template-columns: repeat(4, 1fr);
+    background: var(--ink);
+  }
+  .artifact-tabs button {
+    display: grid;
+    grid-template-columns: 2rem 1fr;
+    gap: .08rem .65rem;
+    padding: 1rem;
+    border: 0;
+    border-right: 1px solid rgba(255,255,255,.13);
+    background: transparent;
+    color: rgba(255,255,255,.58);
+    text-align: left;
+    cursor: pointer;
+  }
+  .artifact-tabs button:last-child { border-right: 0; }
+  .artifact-tabs button[aria-selected="true"] { background: var(--blue); color: white; }
+  .artifact-tabs button:focus-visible { outline: 3px solid #fff; outline-offset: -4px; }
+  .artifact-tabs span {
+    grid-row: 1 / 3;
+    color: currentColor;
+    font: 700 .7rem/1 var(--font-mono);
+  }
+  .artifact-tabs b { font-size: .83rem; }
+  .artifact-tabs small { font: .62rem/1.2 var(--font-mono); }
+  .artifact-panel { padding: 1.6rem; }
+  .artifact-panel[hidden] { display: none; }
+  .panel-lead {
+    display: grid;
+    grid-template-columns: minmax(0, 1fr) minmax(18rem, .9fr);
+    gap: 2rem;
+    align-items: end;
+    margin-bottom: 1.35rem;
+  }
+  .panel-lead h4 {
+    margin: .35rem 0 0;
+    font: 700 clamp(1.25rem, 2vw, 1.7rem)/1.1 var(--font-display);
+    letter-spacing: -.025em;
+  }
+  .panel-lead p { margin: 0; color: rgba(32,32,39,.68); font-size: .83rem; line-height: 1.65; }
+  .route-controls,
+  .cache-controls {
+    display: grid;
+    grid-template-columns: .7fr 1fr 1.45fr;
+    border: 1px solid rgba(32,32,39,.17);
+    background: #fffdf8;
+  }
+  .cache-controls { grid-template-columns: repeat(4, 1fr); }
+  .route-controls label,
+  .cache-controls label {
+    display: grid;
+    gap: .45rem;
+    padding: .8rem;
+    border-right: 1px solid rgba(32,32,39,.13);
+  }
+  .route-controls label:last-child,
+  .cache-controls label:last-child { border-right: 0; }
+  .route-controls label span,
+  .cache-controls label span {
+    color: rgba(32,32,39,.58);
+    font: 650 .65rem/1 var(--font-mono);
+    text-transform: uppercase;
+  }
+  .route-controls select,
+  .cache-controls select {
+    min-width: 0;
+    width: 100%;
+    border: 0;
+    border-bottom: 1px solid rgba(32,32,39,.28);
+    background: transparent;
+    color: var(--ink);
+    font: 650 .78rem/1.3 var(--font-mono);
+  }
+  .cache-controls input { width: 100%; accent-color: var(--blue); }
+  .token-context {
+    display: grid;
+    grid-template-columns: 5rem 1fr;
+    gap: .75rem;
+    margin-top: .65rem;
+    padding: .7rem .85rem;
+    background: rgba(98,105,155,.08);
+  }
+  .token-context span { color: var(--blue); font: 700 .65rem/1.5 var(--font-mono); }
+  .token-context p { margin: 0; font-size: .78rem; line-height: 1.5; }
+  .route-stage {
+    display: grid;
+    grid-template-columns: .7fr auto 2fr auto .9fr;
+    gap: .8rem;
+    align-items: center;
+    margin-top: 1rem;
+    padding: 1rem;
+    border: 1px solid rgba(32,32,39,.15);
+    background:
+      radial-gradient(circle at 35% 50%, rgba(98,105,155,.12), transparent 32%),
+      #e8e2d7;
+  }
+  .route-stage > i { color: rgba(32,32,39,.35); font: 400 1.35rem/1 var(--font-display); }
+  .token-card,
+  .sum-card {
+    display: grid;
+    gap: .3rem;
+    min-width: 0;
+    padding: .9rem;
+    background: var(--ink);
+    color: white;
+  }
+  .token-card span,
+  .sum-card span { color: rgba(255,255,255,.55); font: 650 .61rem/1 var(--font-mono); }
+  .token-card strong {
+    overflow: hidden;
+    text-overflow: ellipsis;
+    font: 750 1.35rem/1.2 var(--font-display);
+  }
+  .token-card small,
+  .sum-card small { color: rgba(255,255,255,.58); font: .62rem/1.35 var(--font-mono); }
+  .sum-card { background: var(--teal); }
+  .sum-card strong { font: 750 1.8rem/1 var(--font-display); }
+  .micro-title { display: block; margin-bottom: .45rem; }
+  .chosen-experts { display: grid; grid-template-columns: repeat(6, 1fr); gap: .35rem; }
+  .chosen-experts :global(article) {
+    display: grid;
+    justify-items: center;
+    gap: .2rem;
+    padding: .55rem .2rem;
+    border: 1px solid rgba(32,32,39,.17);
+    background: #fffdf8;
+  }
+  .chosen-experts :global(span) { color: rgba(32,32,39,.44); font: .56rem/1 var(--font-mono); }
+  .chosen-experts :global(b) { font: 750 .92rem/1 var(--font-mono); }
+  .chosen-experts :global(small) { color: var(--blue); font: 650 .57rem/1 var(--font-mono); }
+  .heat-head {
+    display: grid;
+    grid-template-columns: 1fr 1.2fr;
+    gap: 1rem;
+    align-items: end;
+    margin: 1.2rem 0 .55rem;
+  }
+  .heat-head div { display: grid; gap: .2rem; }
+  .heat-head b { font-size: .8rem; }
+  .heat-head p { margin: 0; color: rgba(32,32,39,.6); font-size: .7rem; line-height: 1.5; text-align: right; }
+  .expert-heatmap {
+    display: grid;
+    grid-template-columns: repeat(16, 1fr);
+    gap: 3px;
+  }
+  .expert-heatmap > :global(span) {
+    display: grid;
+    justify-items: center;
+    gap: .15rem;
+    padding: .45rem .1rem;
+    background: color-mix(in srgb, var(--blue) calc(var(--load) * 78%), #e6dfd3);
+    color: color-mix(in srgb, white calc(var(--load) * 100%), var(--ink));
+    outline: 0 solid var(--amber);
+    transition: transform .15s ease;
+  }
+  .expert-heatmap > :global(span.selected) { outline-width: 2px; z-index: 1; transform: scale(1.05); }
+  .expert-heatmap :global(b) { font: 700 .58rem/1 var(--font-mono); }
+  .expert-heatmap :global(small) { font: .55rem/1 var(--font-mono); }
+  .route-metrics,
+  .cache-ratio,
+  .load-lessons,
+  .artifact-identity,
+  .checksum-grid {
+    display: grid;
+    grid-template-columns: repeat(4, 1fr);
+    margin-top: .8rem;
+    border: 1px solid rgba(32,32,39,.14);
+  }
+  .route-metrics article,
+  .cache-ratio article,
+  .load-lessons article,
+  .artifact-identity article,
+  .checksum-grid article {
+    padding: .8rem;
+    border-right: 1px solid rgba(32,32,39,.12);
+  }
+  .route-metrics article:last-child,
+  .cache-ratio article:last-child,
+  .load-lessons article:last-child,
+  .artifact-identity article:last-child,
+  .checksum-grid article:last-child { border-right: 0; }
+  .route-metrics span,
+  .cache-ratio span,
+  .load-lessons span {
+    display: block;
+    color: var(--blue);
+    font: 700 .61rem/1 var(--font-mono);
+  }
+  .route-metrics b,
+  .cache-ratio b { display: block; margin-top: .35rem; font: 750 1.25rem/1 var(--font-display); }
+  .route-metrics p,
+  .cache-ratio p,
+  .load-lessons p { margin: .3rem 0 0; color: rgba(32,32,39,.56); font-size: .65rem; line-height: 1.45; }
+  .artifact-boundary {
+    display: grid;
+    grid-template-columns: 11rem 1fr;
+    gap: 1rem;
+    margin-top: 1rem;
+    padding: .85rem 1rem;
+    border-left: 4px solid var(--red);
+    background: rgba(161,77,77,.08);
+  }
+  .artifact-boundary b { color: var(--red); font: 750 .68rem/1.4 var(--font-mono); }
+  .artifact-boundary p { margin: 0; font-size: .74rem; line-height: 1.55; }
+  .layer-switch { display: flex; gap: .35rem; margin-bottom: .7rem; }
+  .layer-switch button {
+    width: 3rem;
+    padding: .55rem;
+    border: 1px solid rgba(32,32,39,.22);
+    background: #fffdf8;
+    color: var(--ink);
+    font: 700 .72rem/1 var(--font-mono);
+    cursor: pointer;
+  }
+  .layer-switch button.active { border-color: var(--blue); background: var(--blue); color: white; }
+  .aggregate-card {
+    display: grid;
+    grid-template-columns: .8fr 1.2fr;
+    gap: 1rem;
+    padding: 1rem;
+    background: var(--ink);
+    color: white;
+  }
+  .aggregate-card strong { display: block; margin-top: .4rem; font: 720 1.4rem/1 var(--font-display); }
+  .aggregate-card strong,
+  .aggregate-card strong b { color: white; }
+  .aggregate-card p { margin: .5rem 0 0; color: rgba(255,255,255,.55); font-size: .7rem; }
+  .load-dials { display: grid; grid-template-columns: repeat(3, 1fr); gap: .8rem; }
+  .load-dials article { display: grid; gap: .35rem; }
+  .load-dials b { font: 700 1rem/1 var(--font-mono); }
+  .load-dials i,
+  .jaccard-block i {
+    display: block;
+    height: 5px;
+    background: rgba(255,255,255,.16);
+    overflow: hidden;
+  }
+  .load-dials em,
+  .jaccard-block em {
+    display: block;
+    width: calc(var(--fraction) * 100%);
+    height: 100%;
+    background: #cbc5ef;
+  }
+  .load-table { margin-top: .8rem; border: 1px solid rgba(32,32,39,.16); overflow-x: auto; }
+  .load-table .head,
+  .load-table :global([data-load-rows] > div) {
+    display: grid;
+    grid-template-columns: 1.5fr repeat(4, .62fr) .85fr;
+    min-width: 650px;
+  }
+  .load-table .head { background: #dfd8cb; }
+  .load-table .head b,
+  .load-table :global([data-load-rows] span),
+  .load-table :global([data-load-rows] b) {
+    padding: .6rem;
+    border-right: 1px solid rgba(32,32,39,.1);
+    border-bottom: 1px solid rgba(32,32,39,.1);
+    font: 650 .66rem/1.3 var(--font-mono);
+  }
+  .load-table :global([data-load-rows] > div.aggregate) { background: rgba(98,105,155,.09); }
+  .jaccard-block {
+    display: grid;
+    grid-template-columns: .75fr 1.25fr;
+    gap: 1.2rem;
+    margin-top: .8rem;
+    padding: 1rem;
+    border: 1px solid rgba(32,32,39,.14);
+    background: #ebe5da;
+  }
+  .jaccard-block h5 { margin: .35rem 0; font: 700 1rem/1.1 var(--font-display); }
+  .jaccard-block p { margin: 0; color: rgba(32,32,39,.58); font-size: .68rem; line-height: 1.45; }
+  .jaccard-block :global([data-load-jaccard]) { display: grid; gap: .45rem; }
+  .jaccard-block :global([data-load-jaccard] article) { display: grid; grid-template-columns: 1.5fr 1fr 2.5rem; gap: .5rem; align-items: center; }
+  .jaccard-block :global([data-load-jaccard] article span) { color: var(--ink); letter-spacing: 0; font-size: .59rem; }
+  .jaccard-block :global([data-load-jaccard] i) {
+    display: block;
+    height: 5px;
+    overflow: hidden;
+    background: rgba(32,32,39,.12);
+  }
+  .jaccard-block :global([data-load-jaccard] em) {
+    display: block;
+    width: calc(var(--fraction) * 100%);
+    height: 100%;
+    background: var(--blue);
+  }
+  .jaccard-block :global([data-load-jaccard] article b) { font: 700 .62rem/1 var(--font-mono); text-align: right; }
+  .load-lessons { grid-template-columns: repeat(3, 1fr); }
+  .load-lessons b { display: block; margin-top: .35rem; font-size: .77rem; }
+  .observed-cache {
+    display: grid;
+    grid-template-columns: 1fr auto 1.25fr;
+    gap: 1rem;
+    align-items: center;
+    margin-bottom: .8rem;
+  }
+  .observed-cache > i { color: var(--red); font: 700 1.4rem/1 var(--font-display); }
+  .observed-cache article {
+    display: grid;
+    gap: .35rem;
+    padding: 1rem;
+    border: 1px solid rgba(32,32,39,.16);
+    background: rgba(57,120,110,.08);
+  }
+  .observed-cache article.expanded { background: rgba(161,77,77,.07); }
+  .observed-cache strong { font: 700 .93rem/1.2 var(--font-mono); }
+  .observed-cache p { margin: 0; color: rgba(32,32,39,.58); font-size: .68rem; }
+  .cache-ledger { display: grid; grid-template-columns: 1fr 1fr; gap: .8rem; margin-top: .8rem; }
+  .cache-ledger article { padding: 1rem; border: 1px solid rgba(32,32,39,.14); background: #fffdf8; }
+  .cache-ledger header { display: flex; justify-content: space-between; gap: 1rem; font: 650 .62rem/1.3 var(--font-mono); }
+  .cache-ledger header span { color: var(--blue); }
+  .cache-bar { height: 1.1rem; margin: .75rem 0; background: #ddd6cb; }
+  .cache-bar i { display: block; width: 11.25%; height: 100%; background: var(--teal); }
+  .cache-bar.eager i { width: 100%; background: var(--red); }
+  .cache-ledger strong { font: 750 1.35rem/1 var(--font-display); }
+  .cache-ledger p { margin: .35rem 0 0; color: rgba(32,32,39,.58); font-size: .68rem; line-height: 1.45; }
+  .cache-ratio { grid-template-columns: repeat(3, 1fr); }
+  .cache-ratio .warning { background: rgba(186,118,44,.09); }
+  .artifact-identity { grid-template-columns: repeat(3, 1fr); }
+  .artifact-identity article { display: grid; gap: .35rem; }
+  .artifact-identity b { font-size: .78rem; }
+  .artifact-identity code,
+  .checksum-grid code { font-size: .62rem; overflow-wrap: anywhere; }
+  .layer-evidence {
+    display: grid;
+    grid-template-columns: repeat(27, 1fr);
+    gap: 3px;
+    margin-top: 1rem;
+  }
+  .layer-evidence > span {
+    display: grid;
+    justify-items: center;
+    gap: .25rem;
+    min-width: 0;
+    padding: .55rem .05rem;
+    background: #ded8cc;
+  }
+  .layer-evidence > span.executed { background: var(--teal); color: white; }
+  .layer-evidence > span.split { background: var(--amber); color: white; }
+  .layer-evidence b { font: 700 .58rem/1 var(--font-mono); }
+  .layer-evidence small { font: 650 .43rem/1 var(--font-mono); }
+  .layer-legend { display: flex; flex-wrap: wrap; gap: .8rem 1.3rem; margin-top: .65rem; }
+  .layer-legend span { display: flex; align-items: center; gap: .35rem; color: rgba(32,32,39,.62); font-size: .68rem; }
+  .layer-legend i { width: .7rem; height: .7rem; background: #ded8cc; }
+  .layer-legend i.executed { background: var(--teal); }
+  .layer-legend i.split { background: var(--amber); }
+  .repro-gate {
+    display: grid;
+    grid-template-columns: .55fr 1fr .8fr;
+    gap: 1rem;
+    align-items: center;
+    margin-top: 1rem;
+    padding: 1rem;
+    background: var(--ink);
+    color: white;
+  }
+  .repro-gate > div { display: grid; gap: .3rem; }
+  .repro-gate strong { font: 760 2rem/1 var(--font-display); }
+  .repro-gate > div > b { color: #91cfc1; font: 750 .72rem/1 var(--font-mono); }
+  .repro-gate ul { margin: 0; padding-left: 1rem; color: rgba(255,255,255,.72); font-size: .68rem; line-height: 1.65; }
+  .repro-gate > p { margin: 0; padding-left: 1rem; border-left: 1px solid rgba(255,255,255,.18); color: rgba(255,255,255,.58); font-size: .68rem; line-height: 1.5; }
+  .dependency-split {
+    display: grid;
+    grid-template-columns: 1fr auto 1fr;
+    gap: .8rem;
+    align-items: center;
+    margin-top: .8rem;
+  }
+  .dependency-split > i { color: rgba(32,32,39,.38); font-style: normal; }
+  .dependency-split article { padding: .9rem; border: 1px solid rgba(32,32,39,.14); }
+  .dependency-split b { display: block; margin: .35rem 0; font-size: .8rem; }
+  .dependency-split p { margin: 0; color: rgba(32,32,39,.6); font-size: .68rem; line-height: 1.45; }
+  .checksum-grid article { display: grid; gap: .35rem; }
+  .evidence-links { display: flex; flex-wrap: wrap; gap: .5rem; margin-top: .8rem; }
+  .evidence-links a {
+    padding: .55rem .7rem;
+    border: 1px solid rgba(32,32,39,.18);
+    color: var(--ink);
+    font: 650 .66rem/1 var(--font-mono);
+    text-decoration: none;
+  }
+  .artifact-lab figcaption {
+    display: flex;
+    flex-wrap: wrap;
+    gap: .35rem;
+    padding: .85rem 1.6rem;
+    border-top: 1px solid rgba(32,32,39,.16);
+    background: #e7e0d4;
+    color: rgba(32,32,39,.58);
+    font-size: .64rem;
+  }
+  .artifact-lab figcaption span { color: var(--ink); font-weight: 750; }
+  .artifact-lab figcaption code { font-size: .61rem; }
+  @media (max-width: 860px) {
+    .artifact-head,
+    .panel-lead,
+    .heat-head,
+    .aggregate-card,
+    .jaccard-block { grid-template-columns: 1fr; }
+    .artifact-status { grid-template-columns: 1fr 1fr; }
+    .artifact-tabs { grid-template-columns: 1fr 1fr; }
+    .route-controls { grid-template-columns: 1fr 1fr; }
+    .route-controls .token-select { grid-column: 1 / -1; border-top: 1px solid rgba(32,32,39,.13); }
+    .route-stage { grid-template-columns: 1fr; }
+    .route-stage > i { transform: rotate(90deg); justify-self: center; }
+    .cache-controls { grid-template-columns: 1fr 1fr; }
+    .route-metrics,
+    .checksum-grid { grid-template-columns: 1fr 1fr; }
+    .layer-evidence { grid-template-columns: repeat(9, 1fr); }
+    .repro-gate { grid-template-columns: 1fr 1fr; }
+    .repro-gate > p { grid-column: 1 / -1; padding: .8rem 0 0; border-left: 0; border-top: 1px solid rgba(255,255,255,.18); }
+  }
+  @media (max-width: 560px) {
+    .artifact-head,
+    .artifact-panel { padding: 1rem; }
+    .artifact-head { gap: 1rem; }
+    .artifact-status { grid-template-columns: 1fr; }
+    .artifact-status span { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.1); }
+    .artifact-tabs { display: flex; overflow-x: auto; }
+    .artifact-tabs button { min-width: 10.5rem; flex: 1 0 auto; }
+    .route-controls,
+    .cache-controls { grid-template-columns: 1fr; }
+    .route-controls .token-select { grid-column: auto; }
+    .route-controls label,
+    .cache-controls label { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
+    .chosen-experts { grid-template-columns: repeat(3, 1fr); }
+    .expert-heatmap { grid-template-columns: repeat(8, 1fr); }
+    .route-metrics,
+    .cache-ratio,
+    .load-lessons,
+    .artifact-identity,
+    .checksum-grid,
+    .cache-ledger { grid-template-columns: 1fr; }
+    .route-metrics article,
+    .cache-ratio article,
+    .load-lessons article,
+    .artifact-identity article,
+    .checksum-grid article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
+    .artifact-boundary { grid-template-columns: 1fr; }
+    .load-dials { grid-template-columns: 1fr; }
+    .observed-cache,
+    .dependency-split { grid-template-columns: 1fr; }
+    .observed-cache > i,
+    .dependency-split > i { transform: rotate(90deg); justify-self: center; }
+    .layer-evidence { grid-template-columns: repeat(7, 1fr); }
+    .repro-gate { grid-template-columns: 1fr; }
+    .repro-gate > p { grid-column: auto; }
+    .artifact-lab figcaption { padding: .8rem 1rem; }
+  }
+</style>
diff --git a/src/data/deepseek-v2-lite-trace-repro.json b/src/data/deepseek-v2-lite-trace-repro.json
new file mode 100644
index 0000000..5e94b26
--- /dev/null
+++ b/src/data/deepseek-v2-lite-trace-repro.json
@@ -0,0 +1,142 @@
+{
+  "schema_version": 1,
+  "captured_at": "2026-07-29T05:57:27.848243+00:00",
+  "first": {
+    "captured_at": "2026-07-29T05:55:35.599912+00:00",
+    "sha256": "81ff4ab34d20121fe42219c566dd4780f4c7f74252a31b596f90e5280e1f1aef"
+  },
+  "second": {
+    "captured_at": "2026-07-29T05:56:32.527770+00:00",
+    "sha256": "b5e2e7567e17765195313473499522a4529b9d37ea96963ba2b9455c3c0b86e6"
+  },
+  "timing_compared": false,
+  "checks": [
+    {
+      "name": "provenance",
+      "exact": true
+    },
+    {
+      "name": "configuration",
+      "exact": true
+    },
+    {
+      "name": "prompt_tokenization",
+      "exact": true
+    },
+    {
+      "name": "initial_hidden_hash",
+      "exact": true
+    },
+    {
+      "name": "final_hidden_hash",
+      "exact": true
+    },
+    {
+      "name": "layer_0_hidden_hash",
+      "exact": true
+    },
+    {
+      "name": "layer_0_mla_shapes",
+      "exact": true
+    },
+    {
+      "name": "layer_1_hidden_hash",
+      "exact": true
+    },
+    {
+      "name": "layer_1_mla_shapes",
+      "exact": true
+    },
+    {
+      "name": "layer_1_aggregate_load",
+      "exact": true
+    },
+    {
+      "name": "layer_1_token_routes",
+      "exact": true
+    },
+    {
+      "name": "layer_2_hidden_hash",
+      "exact": true
+    },
+    {
+      "name": "layer_2_mla_shapes",
+      "exact": true
+    },
+    {
+      "name": "layer_2_aggregate_load",
+      "exact": true
+    },
+    {
+      "name": "layer_2_token_routes",
+      "exact": true
+    },
+    {
+      "name": "layer_3_hidden_hash",
+      "exact": true
+    },
+    {
+      "name": "layer_3_mla_shapes",
+      "exact": true
+    },
+    {
+      "name": "layer_3_aggregate_load",
+      "exact": true
+    },
+    {
+      "name": "layer_3_token_routes",
+      "exact": true
+    },
+    {
+      "name": "layer_4_hidden_hash",
+      "exact": true
+    },
+    {
+      "name": "layer_4_mla_shapes",
+      "exact": true
+    },
+    {
+      "name": "layer_4_aggregate_load",
+      "exact": true
+    },
+    {
+      "name": "layer_4_token_routes",
+      "exact": true
+    },
+    {
+      "name": "layer_5_hidden_hash",
+      "exact": true
+    },
+    {
+      "name": "layer_5_mla_shapes",
+      "exact": true
+    },
+    {
+      "name": "layer_5_aggregate_load",
+      "exact": true
+    },
+    {
+      "name": "layer_5_token_routes",
+      "exact": true
+    },
+    {
+      "name": "layer_6_hidden_hash",
+      "exact": true
+    },
+    {
+      "name": "layer_6_mla_shapes",
+      "exact": true
+    },
+    {
+      "name": "layer_6_aggregate_load",
+      "exact": true
+    },
+    {
+      "name": "layer_6_token_routes",
+      "exact": true
+    }
+  ],
+  "exact_checks": 31,
+  "total_checks": 31,
+  "all_exact": true
+}
diff --git a/src/data/deepseek-v2-lite-trace.json b/src/data/deepseek-v2-lite-trace.json
new file mode 100644
index 0000000..64eceb7
--- /dev/null
+++ b/src/data/deepseek-v2-lite-trace.json
@@ -0,0 +1,23332 @@
+{
+  "schema_version": 1,
+  "captured_at": "2026-07-29T05:55:35.599912+00:00",
+  "evidence_identity": "X / official BF16 weights and tokenizer, local truncated forward",
+  "boundary": {
+    "model": "DeepSeek-V2-Lite base",
+    "executed_layers": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6
+    ],
+    "total_model_layers": 27,
+    "full_model_generation": false,
+    "training_or_global_expert_load": false,
+    "expert_semantics_inferred": false,
+    "prompt_sample": "four authored prompts; descriptive trace, not population estimate",
+    "cache_note": "HF eager materializes expanded K/V; latent-cache arithmetic is derived from official dimensions"
+  },
+  "provenance": {
+    "huggingface_model": "deepseek-ai/DeepSeek-V2-Lite",
+    "huggingface_revision": "604d5664dddd88a0433dbae533b7fe9472482de0",
+    "sha256": {
+      "config": "f346286b0f1c8b044252fd54cb4fa78b9fab6472a6e8bebb9edfe03d414ea03d",
+      "modeling_code": "7d8e5221095286eea991137760893fd7ba52727c0b4ebf48ec09e8bc56b45b9c",
+      "tokenizer": "41f3bf64213da8c012d8bd0871a58a1fdf70463e8f08f110ddbb1082f529f669",
+      "index": "d2cdb2f325f6682cf3ad1ad2526a9f979d857390b579380c0331d975136e0acf",
+      "shard_1": "0d7e9f39bde40111a4c0f390b87497dce4565cf578d916395e6b2c7851f1e8da"
+    },
+    "checkpoint_tensor_bytes": 31412968448,
+    "shard_1_bytes": 8594887408
+  },
+  "environment": {
+    "python": "3.10.14",
+    "platform": "Linux-7.0.0-28-generic-x86_64-with-glibc2.43",
+    "libc": [
+      "glibc",
+      "2.43"
+    ],
+    "torch": "2.11.0+cu128",
+    "torch_cuda": "12.8",
+    "transformers": "4.41.2",
+    "safetensors": "0.8.0",
+    "device": "cuda",
+    "nvidia_smi": {
+      "name": "NVIDIA GeForce RTX 5090",
+      "driver_version": "595.84",
+      "memory.total": "32607",
+      "power.limit": "600.00",
+      "clocks.max.sm": "3105"
+    },
+    "matmul_allow_tf32": false
+  },
+  "configuration": {
+    "total_parameters_reported": "15.7B",
+    "activated_parameters_reported": "2.4B",
+    "layers": 27,
+    "hidden": 2048,
+    "attention_heads": 16,
+    "qk_nope_head_dim": 128,
+    "qk_rope_head_dim": 64,
+    "v_head_dim": 128,
+    "kv_lora_rank": 512,
+    "routed_experts": 64,
+    "active_routed_experts": 6,
+    "shared_experts": 2,
+    "expert_intermediate": 1408,
+    "first_dense_layers": 1,
+    "router_scoring": "softmax",
+    "router_topk_method": "greedy",
+    "normalize_selected_weights": false
+  },
+  "prompts": [
+    {
+      "id": "zh_explanation",
+      "label": "中文解释",
+      "text": "用通俗的语言解释，为什么稀疏专家模型可以拥有很多参数，但每个 token 只使用其中一小部分。",
+      "token_ids": [
+        100000,
+        1082,
+        83983,
+        91476,
+        17882,
+        19304,
+        10285,
+        21844,
+        20923,
+        10900,
+        39322,
+        1876,
+        9476,
+        3599,
+        21273,
+        19304,
+        1705,
+        13712,
+        10728,
+        207,
+        2141,
+        5118,
+        6802,
+        51905,
+        5781,
+        398
+      ],
+      "tokens": [
+        "<｜begin▁of▁sentence｜>",
+        "çĶ¨",
+        "éĢļä¿Ĺ",
+        "çļĦè¯Ńè¨Ģ",
+        "解éĩĬ",
+        "ï¼Į",
+        "为ä»Ģä¹Ī",
+        "ç¨Ģ",
+        "çĸı",
+        "ä¸ĵ家",
+        "模åŀĭ",
+        "åı¯ä»¥",
+        "æĭ¥æľī",
+        "å¾Īå¤ļ",
+        "åıĤæķ°",
+        "ï¼Į",
+        "ä½Ĩ",
+        "æ¯ı个",
+        "Ġtoken",
+        "Ġ",
+        "åıª",
+        "使çĶ¨",
+        "åħ¶ä¸Ń",
+        "ä¸Ģå°ı",
+        "éĥ¨åĪĨ",
+        "ãĢĤ"
+      ]
+    },
+    {
+      "id": "en_architecture",
+      "label": "English architecture",
+      "text": "Explain how a compressed key-value latent changes the memory cost of autoregressive decoding.",
+      "token_ids": [
+        100000,
+        99297,
+        946,
+        245,
+        29640,
+        2131,
+        12,
+        2834,
+        28937,
+        4183,
+        254,
+        4887,
+        2295,
+        280,
+        77939,
+        79184,
+        39671,
+        13
+      ],
+      "tokens": [
+        "<｜begin▁of▁sentence｜>",
+        "Explain",
+        "Ġhow",
+        "Ġa",
+        "Ġcompressed",
+        "Ġkey",
+        "-",
+        "value",
+        "Ġlatent",
+        "Ġchanges",
+        "Ġthe",
+        "Ġmemory",
+        "Ġcost",
+        "Ġof",
+        "Ġautore",
+        "gressive",
+        "Ġdecoding",
+        "."
+      ]
+    },
+    {
+      "id": "code",
+      "label": "Python code",
+      "text": "Write a Python function that returns the first repeated element in a list and explain its complexity.",
+      "token_ids": [
+        100000,
+        9081,
+        245,
+        12974,
+        1157,
+        344,
+        7578,
+        254,
+        1022,
+        9842,
+        3699,
+        279,
+        245,
+        1525,
+        285,
+        6716,
+        895,
+        13913,
+        13
+      ],
+      "tokens": [
+        "<｜begin▁of▁sentence｜>",
+        "Write",
+        "Ġa",
+        "ĠPython",
+        "Ġfunction",
+        "Ġthat",
+        "Ġreturns",
+        "Ġthe",
+        "Ġfirst",
+        "Ġrepeated",
+        "Ġelement",
+        "Ġin",
+        "Ġa",
+        "Ġlist",
+        "Ġand",
+        "Ġexplain",
+        "Ġits",
+        "Ġcomplexity",
+        "."
+      ]
+    },
+    {
+      "id": "math",
+      "label": "数学推理",
+      "text": "若正数 x 满足 x 加上它的倒数等于 3，求 x 的平方加上倒数的平方。",
+      "token_ids": [
+        100000,
+        8246,
+        2206,
+        2266,
+        1376,
+        207,
+        13211,
+        1376,
+        207,
+        14855,
+        12985,
+        8270,
+        2266,
+        37360,
+        207,
+        18,
+        19304,
+        2989,
+        1376,
+        207,
+        337,
+        18718,
+        14855,
+        8270,
+        51971,
+        18718,
+        398
+      ],
+      "tokens": [
+        "<｜begin▁of▁sentence｜>",
+        "èĭ¥",
+        "æŃ£",
+        "æķ°",
+        "Ġx",
+        "Ġ",
+        "满足",
+        "Ġx",
+        "Ġ",
+        "åĬłä¸Ĭ",
+        "å®ĥçļĦ",
+        "åĢĴ",
+        "æķ°",
+        "çŃīäºİ",
+        "Ġ",
+        "3",
+        "ï¼Į",
+        "æ±Ĥ",
+        "Ġx",
+        "Ġ",
+        "çļĦ",
+        "å¹³æĸ¹",
+        "åĬłä¸Ĭ",
+        "åĢĴ",
+        "æķ°çļĦ",
+        "å¹³æĸ¹",
+        "ãĢĤ"
+      ]
+    }
+  ],
+  "execution": {
+    "batch": 4,
+    "padded_sequence": 27,
+    "valid_tokens": 90,
+    "embedding_ms": 88.24645899585448,
+    "initial_hidden": {
+      "valid_tokens": 90,
+      "hidden_rms": {
+        "mean": 0.09717819839715958,
+        "p10": 0.06574996560811996,
+        "p50": 0.09712168574333191,
+        "p90": 0.13157525658607483
+      },
+      "l2_norm": {
+        "mean": 4.397783279418945,
+        "p10": 2.97550368309021,
+        "p50": 4.395225524902344,
+        "p90": 5.954416275024414
+      },
+      "finite": true,
+      "sha256_fp32": "837950812a7e04233f5a10439c8e5b2017241797652a1f06cac94c122e41b8e5"
+    },
+    "layers": [
+      {
+        "layer": 0,
+        "ffn": "dense",
+        "state_tensors": 10,
+        "state_numel": 81007104,
+        "state_bytes": 162014208,
+        "timing_ms": {
+          "load_cpu": 506.36647900682874,
+          "transfer_to_device": 24.61088399286382,
+          "forward": 125.80606798292138
+        },
+        "peak_device_allocated_mib": 205.18603515625,
+        "hidden_before": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 0.09717819839715958,
+            "p10": 0.06574996560811996,
+            "p50": 0.09712168574333191,
+            "p90": 0.13157525658607483
+          },
+          "l2_norm": {
+            "mean": 4.397783279418945,
+            "p10": 2.97550368309021,
+            "p50": 4.395225524902344,
+            "p90": 5.954416275024414
+          },
+          "finite": true,
+          "sha256_fp32": "837950812a7e04233f5a10439c8e5b2017241797652a1f06cac94c122e41b8e5"
+        },
+        "hidden_after": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 0.10381482541561127,
+            "p10": 0.07184165716171265,
+            "p50": 0.10495559871196747,
+            "p90": 0.13329632580280304
+          },
+          "l2_norm": {
+            "mean": 4.698122978210449,
+            "p10": 3.2511825561523438,
+            "p50": 4.749748229980469,
+            "p90": 6.032303333282471
+          },
+          "finite": true,
+          "sha256_fp32": "7c907ba1a1d9c0e9c47523d9c1267f2bef2c8fde8a7cc317d6be36ff41394e7e"
+        },
+        "mla": {
+          "compressed_projection_shape": [
+            4,
+            27,
+            576
+          ],
+          "latent_shape": [
+            4,
+            27,
+            512
+          ],
+          "rope_key_shape": [
+            4,
+            27,
+            64
+          ],
+          "latent_rms": 1.287002682685852,
+          "rope_key_rms": 2.812988758087158,
+          "eager_key_cache_shape": [
+            4,
+            16,
+            27,
+            192
+          ],
+          "eager_value_cache_shape": [
+            4,
+            16,
+            27,
+            128
+          ],
+          "eager_cache_bytes": 1105920
+        }
+      },
+      {
+        "layer": 1,
+        "ffn": "moe",
+        "state_tensors": 203,
+        "state_numel": 584847872,
+        "state_bytes": 1169695744,
+        "timing_ms": {
+          "load_cpu": 24.967125995317474,
+          "transfer_to_device": 128.2450080034323,
+          "forward": 89.96552301687188
+        },
+        "peak_device_allocated_mib": 1171.78759765625,
+        "hidden_before": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 0.10381482541561127,
+            "p10": 0.07184165716171265,
+            "p50": 0.10495559871196747,
+            "p90": 0.13329632580280304
+          },
+          "l2_norm": {
+            "mean": 4.698122978210449,
+            "p10": 3.2511825561523438,
+            "p50": 4.749748229980469,
+            "p90": 6.032303333282471
+          },
+          "finite": true,
+          "sha256_fp32": "7c907ba1a1d9c0e9c47523d9c1267f2bef2c8fde8a7cc317d6be36ff41394e7e"
+        },
+        "hidden_after": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 0.12936507165431976,
+            "p10": 0.08545936644077301,
+            "p50": 0.12583057582378387,
+            "p90": 0.1601526141166687
+          },
+          "l2_norm": {
+            "mean": 5.854394435882568,
+            "p10": 3.8674492835998535,
+            "p50": 5.694441795349121,
+            "p90": 7.247680187225342
+          },
+          "finite": true,
+          "sha256_fp32": "bc8c6a2a556f09b333c171b50788905902a2aa18e0bbac784837a7604d931fad"
+        },
+        "mla": {
+          "compressed_projection_shape": [
+            4,
+            27,
+            576
+          ],
+          "latent_shape": [
+            4,
+            27,
+            512
+          ],
+          "rope_key_shape": [
+            4,
+            27,
+            64
+          ],
+          "latent_rms": 2.390777587890625,
+          "rope_key_rms": 3.4520363807678223,
+          "eager_key_cache_shape": [
+            4,
+            16,
+            27,
+            192
+          ],
+          "eager_value_cache_shape": [
+            4,
+            16,
+            27,
+            128
+          ],
+          "eager_cache_bytes": 1105920
+        },
+        "routing": {
+          "aggregate_load": [
+            7,
+            6,
+            9,
+            6,
+            7,
+            8,
+            22,
+            5,
+            48,
+            2,
+            5,
+            17,
+            0,
+            6,
+            5,
+            2,
+            1,
+            29,
+            4,
+            1,
+            3,
+            8,
+            6,
+            4,
+            25,
+            4,
+            13,
+            7,
+            4,
+            9,
+            5,
+            3,
+            23,
+            20,
+            9,
+            5,
+            4,
+            8,
+            7,
+            10,
+            3,
+            10,
+            11,
+            15,
+            7,
+            11,
+            1,
+            6,
+            8,
+            7,
+            5,
+            8,
+            5,
+            10,
+            3,
+            3,
+            12,
+            16,
+            3,
+            12,
+            5,
+            2,
+            4,
+            6
+          ],
+          "aggregate_metrics": {
+            "routes": 540,
+            "used_experts": 63,
+            "zero_experts": 1,
+            "mean": 8.4375,
+            "std": 7.800844669342041,
+            "cv": 0.9245445728302002,
+            "gini": 0.42042824625968933,
+            "entropy_nats": 3.842954635620117,
+            "effective_experts": 46.66314354467963,
+            "min": 0.0,
+            "p50": 6.0,
+            "p90": 16.699996948242188,
+            "max": 48.0
+          },
+          "top_experts": [
+            {
+              "expert": 8,
+              "routes": 48
+            },
+            {
+              "expert": 17,
+              "routes": 29
+            },
+            {
+              "expert": 24,
+              "routes": 25
+            },
+            {
+              "expert": 32,
+              "routes": 23
+            },
+            {
+              "expert": 6,
+              "routes": 22
+            },
+            {
+              "expert": 33,
+              "routes": 20
+            },
+            {
+              "expert": 11,
+              "routes": 17
+            },
+            {
+              "expert": 57,
+              "routes": 16
+            },
+            {
+              "expert": 43,
+              "routes": 15
+            },
+            {
+              "expert": 26,
+              "routes": 13
+            },
+            {
+              "expert": 56,
+              "routes": 12
+            },
+            {
+              "expert": 59,
+              "routes": 12
+            }
+          ],
+          "selected_weight_sum": {
+            "mean": 0.3948917090892792,
+            "p10": 0.31607890129089355,
+            "p50": 0.3835059404373169,
+            "p90": 0.48514068126678467
+          },
+          "top1_top2_margin": {
+            "mean": 0.04313327372074127,
+            "p50": 0.024309832602739334,
+            "p90": 0.1049143448472023
+          },
+          "per_prompt": [
+            {
+              "id": "zh_explanation",
+              "label": "中文解释",
+              "tokens": 26,
+              "load": [
+                2,
+                2,
+                3,
+                0,
+                3,
+                2,
+                3,
+                2,
+                13,
+                1,
+                3,
+                6,
+                0,
+                1,
+                1,
+                0,
+                1,
+                3,
+                1,
+                0,
+                1,
+                1,
+                2,
+                1,
+                6,
+                3,
+                3,
+                3,
+                2,
+                2,
+                2,
+                1,
+                6,
+                3,
+                4,
+                1,
+                0,
+                4,
+                1,
+                2,
+                2,
+                6,
+                4,
+                5,
+                4,
+                4,
+                1,
+                3,
+                4,
+                0,
+                1,
+                3,
+                0,
+                3,
+                0,
+                0,
+                5,
+                5,
+                0,
+                3,
+                1,
+                1,
+                1,
+                5
+              ],
+              "metrics": {
+                "routes": 156,
+                "used_experts": 54,
+                "zero_experts": 10,
+                "mean": 2.4375,
+                "std": 2.1885342597961426,
+                "cv": 0.8978602290153503,
+                "gini": 0.4509214758872986,
+                "entropy_nats": 3.77732253074646,
+                "effective_experts": 43.69888254607027,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 5.0,
+                "max": 13.0
+              },
+              "top_experts": [
+                {
+                  "expert": 8,
+                  "routes": 13
+                },
+                {
+                  "expert": 11,
+                  "routes": 6
+                },
+                {
+                  "expert": 24,
+                  "routes": 6
+                },
+                {
+                  "expert": 32,
+                  "routes": 6
+                },
+                {
+                  "expert": 41,
+                  "routes": 6
+                },
+                {
+                  "expert": 43,
+                  "routes": 5
+                },
+                {
+                  "expert": 56,
+                  "routes": 5
+                },
+                {
+                  "expert": 57,
+                  "routes": 5
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.19667254388332367
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.09833430498838425
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04889551177620888
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04098263755440712
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03667396679520607
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03248872235417366
+                    }
+                  ],
+                  "selected_weight_sum": 0.45404768735170364,
+                  "top1_top2_margin": 0.09833823889493942
+                },
+                {
+                  "position": 1,
+                  "token_id": 1082,
+                  "token": "çĶ¨",
+                  "decoded_piece": "用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 41,
+                      "weight": 0.06996385753154755
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.06454664468765259
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.052981361746788025
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.04313444346189499
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.03537776321172714
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.03254995867609978
+                    }
+                  ],
+                  "selected_weight_sum": 0.29855402931571007,
+                  "top1_top2_margin": 0.0054172128438949585
+                },
+                {
+                  "position": 2,
+                  "token_id": 83983,
+                  "token": "éĢļä¿Ĺ",
+                  "decoded_piece": "通俗",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.10692810267210007
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.10059474408626556
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.05983731523156166
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.04131235554814339
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.03579572215676308
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.03260975703597069
+                    }
+                  ],
+                  "selected_weight_sum": 0.37707799673080444,
+                  "top1_top2_margin": 0.006333358585834503
+                },
+                {
+                  "position": 3,
+                  "token_id": 91476,
+                  "token": "çļĦè¯Ńè¨Ģ",
+                  "decoded_piece": "的语言",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.1252635419368744
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.10231700539588928
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.08574634790420532
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.035869430750608444
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.03414113447070122
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.03363116458058357
+                    }
+                  ],
+                  "selected_weight_sum": 0.41696862503886223,
+                  "top1_top2_margin": 0.022946536540985107
+                },
+                {
+                  "position": 4,
+                  "token_id": 17882,
+                  "token": "解éĩĬ",
+                  "decoded_piece": "解释",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.11453470587730408
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.10259787738323212
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.0590984970331192
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.051855940371751785
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.038944680243730545
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.036984872072935104
+                    }
+                  ],
+                  "selected_weight_sum": 0.40401657298207283,
+                  "top1_top2_margin": 0.01193682849407196
+                },
+                {
+                  "position": 5,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.17547623813152313
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.1141868308186531
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.03966223821043968
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.0280931256711483
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.026378853246569633
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.026011040434241295
+                    }
+                  ],
+                  "selected_weight_sum": 0.40980832651257515,
+                  "top1_top2_margin": 0.061289407312870026
+                },
+                {
+                  "position": 6,
+                  "token_id": 10285,
+                  "token": "为ä»Ģä¹Ī",
+                  "decoded_piece": "为什么",
+                  "experts_by_weight": [
+                    {
+                      "expert": 5,
+                      "weight": 0.08941874653100967
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.08715063333511353
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.08338002115488052
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.05639657378196716
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.044801801443099976
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.03526010364294052
+                    }
+                  ],
+                  "selected_weight_sum": 0.3964078798890114,
+                  "top1_top2_margin": 0.0022681131958961487
+                },
+                {
+                  "position": 7,
+                  "token_id": 21844,
+                  "token": "ç¨Ģ",
+                  "decoded_piece": "稀",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.1019749641418457
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.09891179949045181
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.06054722145199776
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.0462147556245327
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.03941645100712776
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.03171882405877113
+                    }
+                  ],
+                  "selected_weight_sum": 0.37878401577472687,
+                  "top1_top2_margin": 0.0030631646513938904
+                },
+                {
+                  "position": 8,
+                  "token_id": 20923,
+                  "token": "çĸı",
+                  "decoded_piece": "疏",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.19530633091926575
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.08023042231798172
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.06899507343769073
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.06873473525047302
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.058919522911310196
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.048458561301231384
+                    }
+                  ],
+                  "selected_weight_sum": 0.5206446461379528,
+                  "top1_top2_margin": 0.11507590860128403
+                },
+                {
+                  "position": 9,
+                  "token_id": 10900,
+                  "token": "ä¸ĵ家",
+                  "decoded_piece": "专家",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.09651511907577515
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.09186359494924545
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.0887473002076149
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.0633588582277298
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.05882406234741211
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04215756803750992
+                    }
+                  ],
+                  "selected_weight_sum": 0.4414665028452873,
+                  "top1_top2_margin": 0.004651524126529694
+                },
+                {
+                  "position": 10,
+                  "token_id": 39322,
+                  "token": "模åŀĭ",
+                  "decoded_piece": "模型",
+                  "experts_by_weight": [
+                    {
+                      "expert": 2,
+                      "weight": 0.1100841760635376
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.10468249022960663
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.06450875848531723
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.054316893219947815
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.049031782895326614
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.0398552268743515
+                    }
+                  ],
+                  "selected_weight_sum": 0.4224793277680874,
+                  "top1_top2_margin": 0.005401685833930969
+                },
+                {
+                  "position": 11,
+                  "token_id": 1876,
+                  "token": "åı¯ä»¥",
+                  "decoded_piece": "可以",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.08753079921007156
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.07256003469228745
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.06117047369480133
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.055068738758563995
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.02229606918990612
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.022259222343564034
+                    }
+                  ],
+                  "selected_weight_sum": 0.3208853378891945,
+                  "top1_top2_margin": 0.014970764517784119
+                },
+                {
+                  "position": 12,
+                  "token_id": 9476,
+                  "token": "æĭ¥æľī",
+                  "decoded_piece": "拥有",
+                  "experts_by_weight": [
+                    {
+                      "expert": 32,
+                      "weight": 0.06885721534490585
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.058269087225198746
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.05805779621005058
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.05642109364271164
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.045869726687669754
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.03334243223071098
+                    }
+                  ],
+                  "selected_weight_sum": 0.32081735134124756,
+                  "top1_top2_margin": 0.010588128119707108
+                },
+                {
+                  "position": 13,
+                  "token_id": 3599,
+                  "token": "å¾Īå¤ļ",
+                  "decoded_piece": "很多",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.10499579459428787
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.09095507115125656
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.06519477069377899
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.05123397707939148
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.027353990823030472
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.026716141030192375
+                    }
+                  ],
+                  "selected_weight_sum": 0.36644974537193775,
+                  "top1_top2_margin": 0.014040723443031311
+                },
+                {
+                  "position": 14,
+                  "token_id": 21273,
+                  "token": "åıĤæķ°",
+                  "decoded_piece": "参数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 2,
+                      "weight": 0.09879104048013687
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.09676965326070786
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.06424219161272049
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.04395608976483345
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.04187783598899841
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.04052548110485077
+                    }
+                  ],
+                  "selected_weight_sum": 0.38616229221224785,
+                  "top1_top2_margin": 0.002021387219429016
+                },
+                {
+                  "position": 15,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 51,
+                      "weight": 0.116199791431427
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.1023869439959526
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04513482749462128
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.028082912787795067
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.026392823085188866
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.022527199238538742
+                    }
+                  ],
+                  "selected_weight_sum": 0.34072449803352356,
+                  "top1_top2_margin": 0.013812847435474396
+                },
+                {
+                  "position": 16,
+                  "token_id": 1705,
+                  "token": "ä½Ĩ",
+                  "decoded_piece": "但",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.07215771824121475
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.06972325593233109
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.05969814583659172
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.0547492578625679
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.025664115324616432
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.02441459707915783
+                    }
+                  ],
+                  "selected_weight_sum": 0.3064070902764797,
+                  "top1_top2_margin": 0.002434462308883667
+                },
+                {
+                  "position": 17,
+                  "token_id": 13712,
+                  "token": "æ¯ı个",
+                  "decoded_piece": "每个",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.09334097057580948
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.08088334649801254
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.06753778457641602
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.033187661319971085
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.028868872672319412
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.028245503082871437
+                    }
+                  ],
+                  "selected_weight_sum": 0.33206413872539997,
+                  "top1_top2_margin": 0.012457624077796936
+                },
+                {
+                  "position": 18,
+                  "token_id": 10728,
+                  "token": "Ġtoken",
+                  "decoded_piece": " token",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.21899603307247162
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.10515932738780975
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.08899100124835968
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.05321361497044563
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.04883015528321266
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04874962940812111
+                    }
+                  ],
+                  "selected_weight_sum": 0.5639397613704205,
+                  "top1_top2_margin": 0.11383670568466187
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.09224173426628113
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.08393412828445435
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.06542550027370453
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.042790114879608154
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.03504375368356705
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.02880442701280117
+                    }
+                  ],
+                  "selected_weight_sum": 0.3482396584004164,
+                  "top1_top2_margin": 0.008307605981826782
+                },
+                {
+                  "position": 20,
+                  "token_id": 2141,
+                  "token": "åıª",
+                  "decoded_piece": "只",
+                  "experts_by_weight": [
+                    {
+                      "expert": 42,
+                      "weight": 0.08134366571903229
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.06718657910823822
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.06551655381917953
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.0429602786898613
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.035062018781900406
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.032046057283878326
+                    }
+                  ],
+                  "selected_weight_sum": 0.3241151534020901,
+                  "top1_top2_margin": 0.014157086610794067
+                },
+                {
+                  "position": 21,
+                  "token_id": 5118,
+                  "token": "使çĶ¨",
+                  "decoded_piece": "使用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 56,
+                      "weight": 0.16159342229366302
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.055032674223184586
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.05452797934412956
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.03965990990400314
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.037650369107723236
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.03701045364141464
+                    }
+                  ],
+                  "selected_weight_sum": 0.3854748085141182,
+                  "top1_top2_margin": 0.10656074807047844
+                },
+                {
+                  "position": 22,
+                  "token_id": 6802,
+                  "token": "åħ¶ä¸Ń",
+                  "decoded_piece": "其中",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.07383304089307785
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.0634578987956047
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.06159329041838646
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.05860455334186554
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.0438292920589447
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.02614068053662777
+                    }
+                  ],
+                  "selected_weight_sum": 0.327458756044507,
+                  "top1_top2_margin": 0.010375142097473145
+                },
+                {
+                  "position": 23,
+                  "token_id": 51905,
+                  "token": "ä¸Ģå°ı",
+                  "decoded_piece": "一小",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.10564204305410385
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.10040134191513062
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.07950114458799362
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.046730685979127884
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.04516390711069107
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.04194054380059242
+                    }
+                  ],
+                  "selected_weight_sum": 0.41937966644763947,
+                  "top1_top2_margin": 0.005240701138973236
+                },
+                {
+                  "position": 24,
+                  "token_id": 5781,
+                  "token": "éĥ¨åĪĨ",
+                  "decoded_piece": "部分",
+                  "experts_by_weight": [
+                    {
+                      "expert": 47,
+                      "weight": 0.25300896167755127
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.05960262939333916
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.057047683745622635
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.04516291618347168
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.03590259701013565
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03268502280116081
+                    }
+                  ],
+                  "selected_weight_sum": 0.4834098108112812,
+                  "top1_top2_margin": 0.1934063322842121
+                },
+                {
+                  "position": 25,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.08622618764638901
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.08423644304275513
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.05857071280479431
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.0584087148308754
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.023649701848626137
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.023337118327617645
+                    }
+                  ],
+                  "selected_weight_sum": 0.3344288785010576,
+                  "top1_top2_margin": 0.0019897446036338806
+                }
+              ]
+            },
+            {
+              "id": "en_architecture",
+              "label": "English architecture",
+              "tokens": 18,
+              "load": [
+                0,
+                0,
+                3,
+                2,
+                1,
+                1,
+                7,
+                0,
+                11,
+                0,
+                0,
+                4,
+                0,
+                0,
+                2,
+                0,
+                0,
+                7,
+                3,
+                1,
+                0,
+                3,
+                0,
+                0,
+                1,
+                0,
+                2,
+                0,
+                1,
+                1,
+                1,
+                2,
+                9,
+                8,
+                1,
+                2,
+                0,
+                0,
+                1,
+                3,
+                0,
+                1,
+                0,
+                3,
+                2,
+                0,
+                0,
+                0,
+                3,
+                2,
+                2,
+                1,
+                3,
+                0,
+                2,
+                1,
+                2,
+                3,
+                0,
+                2,
+                2,
+                1,
+                0,
+                1
+              ],
+              "metrics": {
+                "routes": 108,
+                "used_experts": 40,
+                "zero_experts": 24,
+                "mean": 1.6875,
+                "std": 2.2947490215301514,
+                "cv": 1.3598512411117554,
+                "gini": 0.6241319179534912,
+                "entropy_nats": 3.399005651473999,
+                "effective_experts": 29.93432009696054,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 3.0,
+                "max": 11.0
+              },
+              "top_experts": [
+                {
+                  "expert": 8,
+                  "routes": 11
+                },
+                {
+                  "expert": 32,
+                  "routes": 9
+                },
+                {
+                  "expert": 33,
+                  "routes": 8
+                },
+                {
+                  "expert": 6,
+                  "routes": 7
+                },
+                {
+                  "expert": 17,
+                  "routes": 7
+                },
+                {
+                  "expert": 11,
+                  "routes": 4
+                },
+                {
+                  "expert": 2,
+                  "routes": 3
+                },
+                {
+                  "expert": 18,
+                  "routes": 3
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.19667254388332367
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.09833430498838425
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04889551177620888
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04098263755440712
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03667396679520607
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03248872235417366
+                    }
+                  ],
+                  "selected_weight_sum": 0.45404768735170364,
+                  "top1_top2_margin": 0.09833823889493942
+                },
+                {
+                  "position": 1,
+                  "token_id": 99297,
+                  "token": "Explain",
+                  "decoded_piece": "Explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.12304119020700455
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.07146582752466202
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.04816020652651787
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.046724919229745865
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04111320152878761
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.04073270410299301
+                    }
+                  ],
+                  "selected_weight_sum": 0.3712380491197109,
+                  "top1_top2_margin": 0.05157536268234253
+                },
+                {
+                  "position": 2,
+                  "token_id": 946,
+                  "token": "Ġhow",
+                  "decoded_piece": " how",
+                  "experts_by_weight": [
+                    {
+                      "expert": 43,
+                      "weight": 0.11726353317499161
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.10638604313135147
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.10123571753501892
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.09169016033411026
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.05507644638419151
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.026508130133152008
+                    }
+                  ],
+                  "selected_weight_sum": 0.4981600306928158,
+                  "top1_top2_margin": 0.010877490043640137
+                },
+                {
+                  "position": 3,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.12597927451133728
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.08409375697374344
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.07757996022701263
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.07270722091197968
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.052257198840379715
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.02372206375002861
+                    }
+                  ],
+                  "selected_weight_sum": 0.43633947521448135,
+                  "top1_top2_margin": 0.04188551753759384
+                },
+                {
+                  "position": 4,
+                  "token_id": 29640,
+                  "token": "Ġcompressed",
+                  "decoded_piece": " compressed",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.10477275401353836
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.07370883226394653
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.06914308667182922
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.048680469393730164
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.04684734344482422
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.04026190936565399
+                    }
+                  ],
+                  "selected_weight_sum": 0.3834143951535225,
+                  "top1_top2_margin": 0.031063921749591827
+                },
+                {
+                  "position": 5,
+                  "token_id": 2131,
+                  "token": "Ġkey",
+                  "decoded_piece": " key",
+                  "experts_by_weight": [
+                    {
+                      "expert": 2,
+                      "weight": 0.09092872589826584
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.06525559723377228
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.05327434092760086
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.0475538969039917
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.043563321232795715
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.03889021649956703
+                    }
+                  ],
+                  "selected_weight_sum": 0.3394660986959934,
+                  "top1_top2_margin": 0.02567312866449356
+                },
+                {
+                  "position": 6,
+                  "token_id": 12,
+                  "token": "-",
+                  "decoded_piece": "-",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.09165064245462418
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.08469503372907639
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.07083211094141006
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.050766441971063614
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.038136690855026245
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.02105005271732807
+                    }
+                  ],
+                  "selected_weight_sum": 0.35713097266852856,
+                  "top1_top2_margin": 0.0069556087255477905
+                },
+                {
+                  "position": 7,
+                  "token_id": 2834,
+                  "token": "value",
+                  "decoded_piece": "value",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.11641892790794373
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.09496383368968964
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.04481128975749016
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.042960286140441895
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04264334961771965
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.035825781524181366
+                    }
+                  ],
+                  "selected_weight_sum": 0.37762346863746643,
+                  "top1_top2_margin": 0.02145509421825409
+                },
+                {
+                  "position": 8,
+                  "token_id": 28937,
+                  "token": "Ġlatent",
+                  "decoded_piece": " latent",
+                  "experts_by_weight": [
+                    {
+                      "expert": 6,
+                      "weight": 0.06952936947345734
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.06664305180311203
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.06500348448753357
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04819929972290993
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.04369500279426575
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.04272771626710892
+                    }
+                  ],
+                  "selected_weight_sum": 0.3357979245483875,
+                  "top1_top2_margin": 0.0028863176703453064
+                },
+                {
+                  "position": 9,
+                  "token_id": 4183,
+                  "token": "Ġchanges",
+                  "decoded_piece": " changes",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.0739561915397644
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.05987396091222763
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.05065435543656349
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.04441293329000473
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.04319792985916138
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.041305456310510635
+                    }
+                  ],
+                  "selected_weight_sum": 0.31340082734823227,
+                  "top1_top2_margin": 0.014082230627536774
+                },
+                {
+                  "position": 10,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 50,
+                      "weight": 0.1116655096411705
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.09895200282335281
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.0893455520272255
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.02281578816473484
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.022700324654579163
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.021918442100286484
+                    }
+                  ],
+                  "selected_weight_sum": 0.3673976194113493,
+                  "top1_top2_margin": 0.012713506817817688
+                },
+                {
+                  "position": 11,
+                  "token_id": 4887,
+                  "token": "Ġmemory",
+                  "decoded_piece": " memory",
+                  "experts_by_weight": [
+                    {
+                      "expert": 33,
+                      "weight": 0.09330408275127411
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.08162695169448853
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.06619293242692947
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.060682181268930435
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04300126060843468
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.03936149552464485
+                    }
+                  ],
+                  "selected_weight_sum": 0.38416890427470207,
+                  "top1_top2_margin": 0.011677131056785583
+                },
+                {
+                  "position": 12,
+                  "token_id": 2295,
+                  "token": "Ġcost",
+                  "decoded_piece": " cost",
+                  "experts_by_weight": [
+                    {
+                      "expert": 33,
+                      "weight": 0.215052992105484
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.061788905411958694
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.054121699184179306
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.04691912233829498
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.04069184884428978
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.03785664215683937
+                    }
+                  ],
+                  "selected_weight_sum": 0.45643121004104614,
+                  "top1_top2_margin": 0.15326408669352531
+                },
+                {
+                  "position": 13,
+                  "token_id": 280,
+                  "token": "Ġof",
+                  "decoded_piece": " of",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.084696464240551
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.05794258788228035
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.045855727046728134
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.044749513268470764
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.04419646039605141
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.0432145893573761
+                    }
+                  ],
+                  "selected_weight_sum": 0.32065534219145775,
+                  "top1_top2_margin": 0.026753876358270645
+                },
+                {
+                  "position": 14,
+                  "token_id": 77939,
+                  "token": "Ġautore",
+                  "decoded_piece": " autore",
+                  "experts_by_weight": [
+                    {
+                      "expert": 30,
+                      "weight": 0.09902932494878769
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.07717865705490112
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.0736161321401596
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.05943404510617256
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.050912320613861084
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.04211832955479622
+                    }
+                  ],
+                  "selected_weight_sum": 0.4022888094186783,
+                  "top1_top2_margin": 0.021850667893886566
+                },
+                {
+                  "position": 15,
+                  "token_id": 79184,
+                  "token": "gressive",
+                  "decoded_piece": "gressive",
+                  "experts_by_weight": [
+                    {
+                      "expert": 33,
+                      "weight": 0.1954519897699356
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.10430377721786499
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05457714945077896
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.052528709173202515
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.051252055913209915
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.03450470045208931
+                    }
+                  ],
+                  "selected_weight_sum": 0.4926183819770813,
+                  "top1_top2_margin": 0.09114821255207062
+                },
+                {
+                  "position": 16,
+                  "token_id": 39671,
+                  "token": "Ġdecoding",
+                  "decoded_piece": " decoding",
+                  "experts_by_weight": [
+                    {
+                      "expert": 33,
+                      "weight": 0.10569825768470764
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.049603477120399475
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.044465046375989914
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.040463220328092575
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.039218440651893616
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.03692803531885147
+                    }
+                  ],
+                  "selected_weight_sum": 0.3163764774799347,
+                  "top1_top2_margin": 0.056094780564308167
+                },
+                {
+                  "position": 17,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.10390158742666245
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.07251034677028656
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.06104576960206032
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.04427094757556915
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.02940911054611206
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.029390856623649597
+                    }
+                  ],
+                  "selected_weight_sum": 0.34052861854434013,
+                  "top1_top2_margin": 0.031391240656375885
+                }
+              ]
+            },
+            {
+              "id": "code",
+              "label": "Python code",
+              "tokens": 19,
+              "load": [
+                1,
+                2,
+                3,
+                2,
+                1,
+                1,
+                7,
+                1,
+                12,
+                1,
+                1,
+                3,
+                0,
+                0,
+                0,
+                0,
+                0,
+                4,
+                0,
+                0,
+                1,
+                4,
+                3,
+                1,
+                7,
+                0,
+                3,
+                0,
+                1,
+                3,
+                1,
+                0,
+                8,
+                5,
+                0,
+                1,
+                4,
+                1,
+                1,
+                2,
+                0,
+                0,
+                1,
+                3,
+                0,
+                0,
+                0,
+                3,
+                1,
+                5,
+                2,
+                2,
+                0,
+                2,
+                1,
+                1,
+                2,
+                2,
+                0,
+                1,
+                1,
+                0,
+                2,
+                0
+              ],
+              "metrics": {
+                "routes": 114,
+                "used_experts": 44,
+                "zero_experts": 20,
+                "mean": 1.78125,
+                "std": 2.25,
+                "cv": 1.263157844543457,
+                "gini": 0.5923793911933899,
+                "entropy_nats": 3.49080753326416,
+                "effective_experts": 32.812434142501516,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 4.0,
+                "max": 12.0
+              },
+              "top_experts": [
+                {
+                  "expert": 8,
+                  "routes": 12
+                },
+                {
+                  "expert": 32,
+                  "routes": 8
+                },
+                {
+                  "expert": 6,
+                  "routes": 7
+                },
+                {
+                  "expert": 24,
+                  "routes": 7
+                },
+                {
+                  "expert": 33,
+                  "routes": 5
+                },
+                {
+                  "expert": 49,
+                  "routes": 5
+                },
+                {
+                  "expert": 17,
+                  "routes": 4
+                },
+                {
+                  "expert": 21,
+                  "routes": 4
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.19667254388332367
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.09833430498838425
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04889551177620888
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04098263755440712
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03667396679520607
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03248872235417366
+                    }
+                  ],
+                  "selected_weight_sum": 0.45404768735170364,
+                  "top1_top2_margin": 0.09833823889493942
+                },
+                {
+                  "position": 1,
+                  "token_id": 9081,
+                  "token": "Write",
+                  "decoded_piece": "Write",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.09549997746944427
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.08619722723960876
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.08242639154195786
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.07498472929000854
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.0488131120800972
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.032170750200748444
+                    }
+                  ],
+                  "selected_weight_sum": 0.4200921878218651,
+                  "top1_top2_margin": 0.00930275022983551
+                },
+                {
+                  "position": 2,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 49,
+                      "weight": 0.14923442900180817
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.13033752143383026
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.08116056770086288
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.061656370759010315
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.057049237191677094
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.0237296000123024
+                    }
+                  ],
+                  "selected_weight_sum": 0.5031677260994911,
+                  "top1_top2_margin": 0.018896907567977905
+                },
+                {
+                  "position": 3,
+                  "token_id": 12974,
+                  "token": "ĠPython",
+                  "decoded_piece": " Python",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.10350564122200012
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.059758465737104416
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.05859985575079918
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.05378400906920433
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.0489177405834198
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.04772283136844635
+                    }
+                  ],
+                  "selected_weight_sum": 0.3722885437309742,
+                  "top1_top2_margin": 0.043747175484895706
+                },
+                {
+                  "position": 4,
+                  "token_id": 1157,
+                  "token": "Ġfunction",
+                  "decoded_piece": " function",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.1556571125984192
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.11224940419197083
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.08262474089860916
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.061063818633556366
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.0245226863771677
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.023945074528455734
+                    }
+                  ],
+                  "selected_weight_sum": 0.460062837228179,
+                  "top1_top2_margin": 0.043407708406448364
+                },
+                {
+                  "position": 5,
+                  "token_id": 344,
+                  "token": "Ġthat",
+                  "decoded_piece": " that",
+                  "experts_by_weight": [
+                    {
+                      "expert": 43,
+                      "weight": 0.1358845978975296
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.12254359573125839
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.06546720862388611
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.0648581013083458
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.02445332705974579
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.022628234699368477
+                    }
+                  ],
+                  "selected_weight_sum": 0.43583506532013416,
+                  "top1_top2_margin": 0.01334100216627121
+                },
+                {
+                  "position": 6,
+                  "token_id": 7578,
+                  "token": "Ġreturns",
+                  "decoded_piece": " returns",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.13557755947113037
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.09798909723758698
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04169568419456482
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.03891124948859215
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.0314752459526062
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.02870868146419525
+                    }
+                  ],
+                  "selected_weight_sum": 0.37435751780867577,
+                  "top1_top2_margin": 0.037588462233543396
+                },
+                {
+                  "position": 7,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 21,
+                      "weight": 0.14209777116775513
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.06827035546302795
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.04493793100118637
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.03718099743127823
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.031235428526997566
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.02947692945599556
+                    }
+                  ],
+                  "selected_weight_sum": 0.3531994130462408,
+                  "top1_top2_margin": 0.07382741570472717
+                },
+                {
+                  "position": 8,
+                  "token_id": 1022,
+                  "token": "Ġfirst",
+                  "decoded_piece": " first",
+                  "experts_by_weight": [
+                    {
+                      "expert": 3,
+                      "weight": 0.11077141761779785
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.0720604807138443
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.06530284881591797
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.05896109715104103
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.041370950639247894
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.03513072803616524
+                    }
+                  ],
+                  "selected_weight_sum": 0.3835975229740143,
+                  "top1_top2_margin": 0.03871093690395355
+                },
+                {
+                  "position": 9,
+                  "token_id": 9842,
+                  "token": "Ġrepeated",
+                  "decoded_piece": " repeated",
+                  "experts_by_weight": [
+                    {
+                      "expert": 32,
+                      "weight": 0.18357861042022705
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.11116449534893036
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.05987098067998886
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.05746008828282356
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.043144918978214264
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.029252201318740845
+                    }
+                  ],
+                  "selected_weight_sum": 0.48447129502892494,
+                  "top1_top2_margin": 0.07241411507129669
+                },
+                {
+                  "position": 10,
+                  "token_id": 3699,
+                  "token": "Ġelement",
+                  "decoded_piece": " element",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.11392679065465927
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.055260807275772095
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.05243296921253204
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.04332876577973366
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.035943593829870224
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.03192679211497307
+                    }
+                  ],
+                  "selected_weight_sum": 0.33281971886754036,
+                  "top1_top2_margin": 0.058665983378887177
+                },
+                {
+                  "position": 11,
+                  "token_id": 279,
+                  "token": "Ġin",
+                  "decoded_piece": " in",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.08486376702785492
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.06781504303216934
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.06028468906879425
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.03633364662528038
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.03177472949028015
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03020312637090683
+                    }
+                  ],
+                  "selected_weight_sum": 0.3112750016152859,
+                  "top1_top2_margin": 0.017048723995685577
+                },
+                {
+                  "position": 12,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.10673930495977402
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.10209112614393234
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.06482338160276413
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.03735273331403732
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.03692882880568504
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.029585134238004684
+                    }
+                  ],
+                  "selected_weight_sum": 0.37752050906419754,
+                  "top1_top2_margin": 0.004648178815841675
+                },
+                {
+                  "position": 13,
+                  "token_id": 1525,
+                  "token": "Ġlist",
+                  "decoded_piece": " list",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.10538846254348755
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.06087666004896164
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.059287186712026596
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.04374944046139717
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.04220329970121384
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.028590604662895203
+                    }
+                  ],
+                  "selected_weight_sum": 0.340095654129982,
+                  "top1_top2_margin": 0.04451180249452591
+                },
+                {
+                  "position": 14,
+                  "token_id": 285,
+                  "token": "Ġand",
+                  "decoded_piece": " and",
+                  "experts_by_weight": [
+                    {
+                      "expert": 51,
+                      "weight": 0.1155504509806633
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.05517224967479706
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.03504233807325363
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.026940295472741127
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.025131845846772194
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.02402118220925331
+                    }
+                  ],
+                  "selected_weight_sum": 0.2818583622574806,
+                  "top1_top2_margin": 0.06037820130586624
+                },
+                {
+                  "position": 15,
+                  "token_id": 6716,
+                  "token": "Ġexplain",
+                  "decoded_piece": " explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 33,
+                      "weight": 0.14533041417598724
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.11240343749523163
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.07190889120101929
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.0551474392414093
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.03574245423078537
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.034908734261989594
+                    }
+                  ],
+                  "selected_weight_sum": 0.4554413706064224,
+                  "top1_top2_margin": 0.032926976680755615
+                },
+                {
+                  "position": 16,
+                  "token_id": 895,
+                  "token": "Ġits",
+                  "decoded_piece": " its",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.08592283725738525
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.08066461980342865
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.06557497382164001
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.053587011992931366
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.045721616595983505
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.024450529366731644
+                    }
+                  ],
+                  "selected_weight_sum": 0.35592158883810043,
+                  "top1_top2_margin": 0.005258217453956604
+                },
+                {
+                  "position": 17,
+                  "token_id": 13913,
+                  "token": "Ġcomplexity",
+                  "decoded_piece": " complexity",
+                  "experts_by_weight": [
+                    {
+                      "expert": 32,
+                      "weight": 0.11456479132175446
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.07901880145072937
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.07786156237125397
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.06110609322786331
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.044746607542037964
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.041917551308870316
+                    }
+                  ],
+                  "selected_weight_sum": 0.4192154072225094,
+                  "top1_top2_margin": 0.035545989871025085
+                },
+                {
+                  "position": 18,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.10835051536560059
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.07074465602636337
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.056041520088911057
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.04474358260631561
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03019094280898571
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.027098262682557106
+                    }
+                  ],
+                  "selected_weight_sum": 0.33716947957873344,
+                  "top1_top2_margin": 0.03760585933923721
+                }
+              ]
+            },
+            {
+              "id": "math",
+              "label": "数学推理",
+              "tokens": 27,
+              "load": [
+                4,
+                2,
+                0,
+                2,
+                2,
+                4,
+                5,
+                2,
+                12,
+                0,
+                1,
+                4,
+                0,
+                5,
+                2,
+                2,
+                0,
+                15,
+                0,
+                0,
+                1,
+                0,
+                1,
+                2,
+                11,
+                1,
+                5,
+                4,
+                0,
+                3,
+                1,
+                0,
+                0,
+                4,
+                4,
+                1,
+                0,
+                3,
+                4,
+                3,
+                1,
+                3,
+                6,
+                4,
+                1,
+                7,
+                0,
+                0,
+                0,
+                0,
+                0,
+                2,
+                2,
+                5,
+                0,
+                1,
+                3,
+                6,
+                3,
+                6,
+                1,
+                0,
+                1,
+                0
+              ],
+              "metrics": {
+                "routes": 162,
+                "used_experts": 45,
+                "zero_experts": 19,
+                "mean": 2.53125,
+                "std": 2.997187852859497,
+                "cv": 1.1840741634368896,
+                "gini": 0.5760030746459961,
+                "entropy_nats": 3.53518009185791,
+                "effective_experts": 34.301191523325606,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 5.6999969482421875,
+                "max": 15.0
+              },
+              "top_experts": [
+                {
+                  "expert": 17,
+                  "routes": 15
+                },
+                {
+                  "expert": 8,
+                  "routes": 12
+                },
+                {
+                  "expert": 24,
+                  "routes": 11
+                },
+                {
+                  "expert": 45,
+                  "routes": 7
+                },
+                {
+                  "expert": 42,
+                  "routes": 6
+                },
+                {
+                  "expert": 57,
+                  "routes": 6
+                },
+                {
+                  "expert": 59,
+                  "routes": 6
+                },
+                {
+                  "expert": 6,
+                  "routes": 5
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.19667254388332367
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.09833430498838425
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04889551177620888
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04098263755440712
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03667396679520607
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03248872235417366
+                    }
+                  ],
+                  "selected_weight_sum": 0.45404768735170364,
+                  "top1_top2_margin": 0.09833823889493942
+                },
+                {
+                  "position": 1,
+                  "token_id": 8246,
+                  "token": "èĭ¥",
+                  "decoded_piece": "若",
+                  "experts_by_weight": [
+                    {
+                      "expert": 43,
+                      "weight": 0.1394088715314865
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.06345965713262558
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.042812079191207886
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.04217054322361946
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.04117071256041527
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.029628943651914597
+                    }
+                  ],
+                  "selected_weight_sum": 0.3586508072912693,
+                  "top1_top2_margin": 0.07594921439886093
+                },
+                {
+                  "position": 2,
+                  "token_id": 2206,
+                  "token": "æŃ£",
+                  "decoded_piece": "正",
+                  "experts_by_weight": [
+                    {
+                      "expert": 58,
+                      "weight": 0.10335297882556915
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.0671335905790329
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.04563439264893532
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.037449806928634644
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.03589992597699165
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.034529875963926315
+                    }
+                  ],
+                  "selected_weight_sum": 0.32400057092309,
+                  "top1_top2_margin": 0.036219388246536255
+                },
+                {
+                  "position": 3,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.08278294652700424
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.06642577052116394
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.053992412984371185
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.0396278016269207
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.03648533672094345
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.03367812559008598
+                    }
+                  ],
+                  "selected_weight_sum": 0.3129923939704895,
+                  "top1_top2_margin": 0.0163571760058403
+                },
+                {
+                  "position": 4,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.21853145956993103
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.15359973907470703
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.07538707554340363
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.06006627157330513
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.04238240048289299
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.022744085639715195
+                    }
+                  ],
+                  "selected_weight_sum": 0.572711031883955,
+                  "top1_top2_margin": 0.064931720495224
+                },
+                {
+                  "position": 5,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.12307935208082199
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.11029956489801407
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.04811596870422363
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.04188108816742897
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.041408829391002655
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.035461392253637314
+                    }
+                  ],
+                  "selected_weight_sum": 0.40024619549512863,
+                  "top1_top2_margin": 0.012779787182807922
+                },
+                {
+                  "position": 6,
+                  "token_id": 13211,
+                  "token": "满足",
+                  "decoded_piece": "满足",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.2006656676530838
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.09593425691127777
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.051401205360889435
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.043837904930114746
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.03912341967225075
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.036207932978868484
+                    }
+                  ],
+                  "selected_weight_sum": 0.467170387506485,
+                  "top1_top2_margin": 0.10473141074180603
+                },
+                {
+                  "position": 7,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.16054129600524902
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.14097601175308228
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.04995854198932648
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.047472260892391205
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.031187430024147034
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.0243584793061018
+                    }
+                  ],
+                  "selected_weight_sum": 0.4544940199702978,
+                  "top1_top2_margin": 0.019565284252166748
+                },
+                {
+                  "position": 8,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.10879295319318771
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.10352718830108643
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.04323926195502281
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.037018291652202606
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.034377746284008026
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.028740879148244858
+                    }
+                  ],
+                  "selected_weight_sum": 0.35569632053375244,
+                  "top1_top2_margin": 0.005265764892101288
+                },
+                {
+                  "position": 9,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.1886313408613205
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.10988873988389969
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.07265309244394302
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.05920105054974556
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.03127415105700493
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.02951677516102791
+                    }
+                  ],
+                  "selected_weight_sum": 0.4911651499569416,
+                  "top1_top2_margin": 0.0787426009774208
+                },
+                {
+                  "position": 10,
+                  "token_id": 12985,
+                  "token": "å®ĥçļĦ",
+                  "decoded_piece": "它的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.17231044173240662
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.11317699402570724
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.06843745708465576
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.06510318070650101
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.03418071195483208
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.027257278561592102
+                    }
+                  ],
+                  "selected_weight_sum": 0.4804660640656948,
+                  "top1_top2_margin": 0.05913344770669937
+                },
+                {
+                  "position": 11,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.32550475001335144
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.0801854208111763
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.07861076295375824
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.05385090410709381
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.046686459332704544
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.04537269100546837
+                    }
+                  ],
+                  "selected_weight_sum": 0.6302109882235527,
+                  "top1_top2_margin": 0.24531932920217514
+                },
+                {
+                  "position": 12,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.203230619430542
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.061504725366830826
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.05149833485484123
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.04692880064249039
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.042546626180410385
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.038524165749549866
+                    }
+                  ],
+                  "selected_weight_sum": 0.4442332722246647,
+                  "top1_top2_margin": 0.14172589406371117
+                },
+                {
+                  "position": 13,
+                  "token_id": 37360,
+                  "token": "çŃīäºİ",
+                  "decoded_piece": "等于",
+                  "experts_by_weight": [
+                    {
+                      "expert": 43,
+                      "weight": 0.08651761710643768
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.08569129556417465
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.08219180256128311
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.07018200308084488
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.03885370492935181
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.036339085549116135
+                    }
+                  ],
+                  "selected_weight_sum": 0.39977550879120827,
+                  "top1_top2_margin": 0.000826321542263031
+                },
+                {
+                  "position": 14,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.14133651554584503
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.08895187824964523
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.06798499077558517
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.031565990298986435
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.028622372075915337
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.024421988055109978
+                    }
+                  ],
+                  "selected_weight_sum": 0.3828837350010872,
+                  "top1_top2_margin": 0.0523846372961998
+                },
+                {
+                  "position": 15,
+                  "token_id": 18,
+                  "token": "3",
+                  "decoded_piece": "3",
+                  "experts_by_weight": [
+                    {
+                      "expert": 13,
+                      "weight": 0.062128130346536636
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.06102198734879494
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.036797475069761276
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.032095663249492645
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.030832836404442787
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.029817776754498482
+                    }
+                  ],
+                  "selected_weight_sum": 0.25269386917352676,
+                  "top1_top2_margin": 0.0011061429977416992
+                },
+                {
+                  "position": 16,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 51,
+                      "weight": 0.1388871967792511
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.08805988729000092
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.05442273989319801
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.02597079984843731
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.02435963973402977
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.0232080165296793
+                    }
+                  ],
+                  "selected_weight_sum": 0.3549082800745964,
+                  "top1_top2_margin": 0.05082730948925018
+                },
+                {
+                  "position": 17,
+                  "token_id": 2989,
+                  "token": "æ±Ĥ",
+                  "decoded_piece": "求",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.13024798035621643
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.12015853077173233
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.04987603798508644
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.049373287707567215
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.04563625529408455
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.040090806782245636
+                    }
+                  ],
+                  "selected_weight_sum": 0.4353828988969326,
+                  "top1_top2_margin": 0.0100894495844841
+                },
+                {
+                  "position": 18,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.14243966341018677
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.06603427976369858
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.06250512599945068
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.04692105948925018
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.033770523965358734
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.026090526953339577
+                    }
+                  ],
+                  "selected_weight_sum": 0.3777611795812845,
+                  "top1_top2_margin": 0.07640538364648819
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.08014163374900818
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.061893001198768616
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.0616314522922039
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04130232706665993
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.03659188747406006
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.024267001077532768
+                    }
+                  ],
+                  "selected_weight_sum": 0.30582730285823345,
+                  "top1_top2_margin": 0.018248632550239563
+                },
+                {
+                  "position": 20,
+                  "token_id": 337,
+                  "token": "çļĦ",
+                  "decoded_piece": "的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.054228734225034714
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.052332378923892975
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.04236849769949913
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.03419002890586853
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.032592903822660446
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.0297713503241539
+                    }
+                  ],
+                  "selected_weight_sum": 0.2454838939011097,
+                  "top1_top2_margin": 0.001896355301141739
+                },
+                {
+                  "position": 21,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.13682566583156586
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.12559771537780762
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.05148334428668022
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.043595705181360245
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.042157530784606934
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.028853029012680054
+                    }
+                  ],
+                  "selected_weight_sum": 0.4285129904747009,
+                  "top1_top2_margin": 0.01122795045375824
+                },
+                {
+                  "position": 22,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.1608467400074005
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.07378645241260529
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.06305347383022308
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.03693674877285957
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03622017800807953
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.033631253987550735
+                    }
+                  ],
+                  "selected_weight_sum": 0.4044748470187187,
+                  "top1_top2_margin": 0.08706028759479523
+                },
+                {
+                  "position": 23,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.22711610794067383
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.08942194283008575
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.06970909237861633
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.05923660099506378
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.05873405933380127
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.04915972426533699
+                    }
+                  ],
+                  "selected_weight_sum": 0.553377527743578,
+                  "top1_top2_margin": 0.13769416511058807
+                },
+                {
+                  "position": 24,
+                  "token_id": 51971,
+                  "token": "æķ°çļĦ",
+                  "decoded_piece": "数的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.17918023467063904
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.05802371725440025
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.05564861372113228
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.04281933978199959
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.042292799800634384
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.04076436161994934
+                    }
+                  ],
+                  "selected_weight_sum": 0.4187290668487549,
+                  "top1_top2_margin": 0.12115651741623878
+                },
+                {
+                  "position": 25,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.12711961567401886
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.12048730254173279
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.05697174370288849
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.045253265649080276
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.03940247744321823
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.030314432457089424
+                    }
+                  ],
+                  "selected_weight_sum": 0.41954883746802807,
+                  "top1_top2_margin": 0.006632313132286072
+                },
+                {
+                  "position": 26,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.09412042051553726
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.06351388990879059
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.06225387006998062
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.06197313964366913
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.027688290923833847
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.025522006675601006
+                    }
+                  ],
+                  "selected_weight_sum": 0.33507161773741245,
+                  "top1_top2_margin": 0.030606530606746674
+                }
+              ]
+            }
+          ],
+          "prompt_pair_jaccard": [
+            {
+              "left": "zh_explanation",
+              "right": "en_architecture",
+              "used_expert_jaccard": 0.5666666666666667,
+              "shared_experts": 34,
+              "union_experts": 60
+            },
+            {
+              "left": "zh_explanation",
+              "right": "code",
+              "used_expert_jaccard": 0.6610169491525424,
+              "shared_experts": 39,
+              "union_experts": 59
+            },
+            {
+              "left": "zh_explanation",
+              "right": "math",
+              "used_expert_jaccard": 0.6779661016949152,
+              "shared_experts": 40,
+              "union_experts": 59
+            },
+            {
+              "left": "en_architecture",
+              "right": "code",
+              "used_expert_jaccard": 0.5555555555555556,
+              "shared_experts": 30,
+              "union_experts": 54
+            },
+            {
+              "left": "en_architecture",
+              "right": "math",
+              "used_expert_jaccard": 0.46551724137931033,
+              "shared_experts": 27,
+              "union_experts": 58
+            },
+            {
+              "left": "code",
+              "right": "math",
+              "used_expert_jaccard": 0.5892857142857143,
+              "shared_experts": 33,
+              "union_experts": 56
+            }
+          ]
+        }
+      },
+      {
+        "layer": 2,
+        "ffn": "moe",
+        "state_tensors": 203,
+        "state_numel": 584847872,
+        "state_bytes": 1169695744,
+        "timing_ms": {
+          "load_cpu": 24.617862014565617,
+          "transfer_to_device": 123.9008959964849,
+          "forward": 4.285834002075717
+        },
+        "peak_device_allocated_mib": 1173.2724609375,
+        "hidden_before": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 0.12936507165431976,
+            "p10": 0.08545936644077301,
+            "p50": 0.12583057582378387,
+            "p90": 0.1601526141166687
+          },
+          "l2_norm": {
+            "mean": 5.854394435882568,
+            "p10": 3.8674492835998535,
+            "p50": 5.694441795349121,
+            "p90": 7.247680187225342
+          },
+          "finite": true,
+          "sha256_fp32": "bc8c6a2a556f09b333c171b50788905902a2aa18e0bbac784837a7604d931fad"
+        },
+        "hidden_after": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 0.22556711733341217,
+            "p10": 0.10850249230861664,
+            "p50": 0.1559184491634369,
+            "p90": 0.19672976434230804
+          },
+          "l2_norm": {
+            "mean": 10.208003044128418,
+            "p10": 4.910262107849121,
+            "p50": 7.056063175201416,
+            "p90": 8.902973175048828
+          },
+          "finite": true,
+          "sha256_fp32": "76c1ba59ea5ef19f16162513c7c36e6e8d17a9af9fa3fdedbf657e80882fa0a3"
+        },
+        "mla": {
+          "compressed_projection_shape": [
+            4,
+            27,
+            576
+          ],
+          "latent_shape": [
+            4,
+            27,
+            512
+          ],
+          "rope_key_shape": [
+            4,
+            27,
+            64
+          ],
+          "latent_rms": 2.27437686920166,
+          "rope_key_rms": 3.476457118988037,
+          "eager_key_cache_shape": [
+            4,
+            16,
+            27,
+            192
+          ],
+          "eager_value_cache_shape": [
+            4,
+            16,
+            27,
+            128
+          ],
+          "eager_cache_bytes": 1105920
+        },
+        "routing": {
+          "aggregate_load": [
+            13,
+            3,
+            1,
+            13,
+            5,
+            8,
+            6,
+            9,
+            9,
+            10,
+            1,
+            6,
+            7,
+            6,
+            4,
+            12,
+            10,
+            13,
+            11,
+            4,
+            10,
+            2,
+            11,
+            5,
+            8,
+            3,
+            13,
+            12,
+            5,
+            4,
+            14,
+            4,
+            14,
+            5,
+            14,
+            18,
+            8,
+            3,
+            7,
+            7,
+            10,
+            4,
+            7,
+            9,
+            5,
+            13,
+            9,
+            19,
+            8,
+            8,
+            4,
+            5,
+            13,
+            2,
+            3,
+            13,
+            5,
+            11,
+            10,
+            4,
+            17,
+            9,
+            23,
+            11
+          ],
+          "aggregate_metrics": {
+            "routes": 540,
+            "used_experts": 64,
+            "zero_experts": 0,
+            "mean": 8.4375,
+            "std": 4.6321001052856445,
+            "cv": 0.5489896535873413,
+            "gini": 0.30329862236976624,
+            "entropy_nats": 4.008847713470459,
+            "effective_experts": 55.08336216433062,
+            "min": 1.0,
+            "p50": 8.0,
+            "p90": 13.699996948242188,
+            "max": 23.0
+          },
+          "top_experts": [
+            {
+              "expert": 62,
+              "routes": 23
+            },
+            {
+              "expert": 47,
+              "routes": 19
+            },
+            {
+              "expert": 35,
+              "routes": 18
+            },
+            {
+              "expert": 60,
+              "routes": 17
+            },
+            {
+              "expert": 30,
+              "routes": 14
+            },
+            {
+              "expert": 32,
+              "routes": 14
+            },
+            {
+              "expert": 34,
+              "routes": 14
+            },
+            {
+              "expert": 0,
+              "routes": 13
+            },
+            {
+              "expert": 3,
+              "routes": 13
+            },
+            {
+              "expert": 17,
+              "routes": 13
+            },
+            {
+              "expert": 26,
+              "routes": 13
+            },
+            {
+              "expert": 45,
+              "routes": 13
+            }
+          ],
+          "selected_weight_sum": {
+            "mean": 0.48013466596603394,
+            "p10": 0.35627180337905884,
+            "p50": 0.4594435691833496,
+            "p90": 0.587585985660553
+          },
+          "top1_top2_margin": {
+            "mean": 0.10077717900276184,
+            "p50": 0.0558306984603405,
+            "p90": 0.1981295347213745
+          },
+          "per_prompt": [
+            {
+              "id": "zh_explanation",
+              "label": "中文解释",
+              "tokens": 26,
+              "load": [
+                3,
+                1,
+                0,
+                4,
+                0,
+                2,
+                1,
+                4,
+                3,
+                3,
+                0,
+                3,
+                2,
+                1,
+                2,
+                4,
+                2,
+                5,
+                3,
+                1,
+                4,
+                2,
+                2,
+                2,
+                4,
+                1,
+                3,
+                4,
+                2,
+                3,
+                4,
+                2,
+                4,
+                0,
+                3,
+                4,
+                2,
+                1,
+                1,
+                3,
+                0,
+                1,
+                2,
+                1,
+                3,
+                5,
+                3,
+                6,
+                3,
+                3,
+                0,
+                2,
+                2,
+                0,
+                2,
+                2,
+                2,
+                2,
+                4,
+                2,
+                3,
+                4,
+                6,
+                3
+              ],
+              "metrics": {
+                "routes": 156,
+                "used_experts": 57,
+                "zero_experts": 7,
+                "mean": 2.4375,
+                "std": 1.457057237625122,
+                "cv": 0.5977670550346375,
+                "gini": 0.32912659645080566,
+                "entropy_nats": 3.9409890174865723,
+                "effective_experts": 51.469480352336895,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 4.0,
+                "max": 6.0
+              },
+              "top_experts": [
+                {
+                  "expert": 47,
+                  "routes": 6
+                },
+                {
+                  "expert": 62,
+                  "routes": 6
+                },
+                {
+                  "expert": 17,
+                  "routes": 5
+                },
+                {
+                  "expert": 45,
+                  "routes": 5
+                },
+                {
+                  "expert": 3,
+                  "routes": 4
+                },
+                {
+                  "expert": 7,
+                  "routes": 4
+                },
+                {
+                  "expert": 15,
+                  "routes": 4
+                },
+                {
+                  "expert": 20,
+                  "routes": 4
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 3,
+                      "weight": 0.7993167042732239
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.018388060852885246
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.010282253846526146
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.009434774518013
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.009029458276927471
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.008924801833927631
+                    }
+                  ],
+                  "selected_weight_sum": 0.8553760536015034,
+                  "top1_top2_margin": 0.7809286434203386
+                },
+                {
+                  "position": 1,
+                  "token_id": 1082,
+                  "token": "çĶ¨",
+                  "decoded_piece": "用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.20668703317642212
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.17426803708076477
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.07382696866989136
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.053661659359931946
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03640357777476311
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.027232389897108078
+                    }
+                  ],
+                  "selected_weight_sum": 0.5720796659588814,
+                  "top1_top2_margin": 0.03241899609565735
+                },
+                {
+                  "position": 2,
+                  "token_id": 83983,
+                  "token": "éĢļä¿Ĺ",
+                  "decoded_piece": "通俗",
+                  "experts_by_weight": [
+                    {
+                      "expert": 18,
+                      "weight": 0.2451930046081543
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.08248794078826904
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.060274556279182434
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.042924247682094574
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.03560378775000572
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.035133421421051025
+                    }
+                  ],
+                  "selected_weight_sum": 0.5016169585287571,
+                  "top1_top2_margin": 0.16270506381988525
+                },
+                {
+                  "position": 3,
+                  "token_id": 91476,
+                  "token": "çļĦè¯Ńè¨Ģ",
+                  "decoded_piece": "的语言",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.23428620398044586
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.1792253851890564
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.08116163313388824
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.06720003485679626
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.03637100011110306
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.023322662338614464
+                    }
+                  ],
+                  "selected_weight_sum": 0.6215669196099043,
+                  "top1_top2_margin": 0.055060818791389465
+                },
+                {
+                  "position": 4,
+                  "token_id": 17882,
+                  "token": "解éĩĬ",
+                  "decoded_piece": "解释",
+                  "experts_by_weight": [
+                    {
+                      "expert": 9,
+                      "weight": 0.133388951420784
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.10470598191022873
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.09774415194988251
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.0421929694712162
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.03626151382923126
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.02716749534010887
+                    }
+                  ],
+                  "selected_weight_sum": 0.44146106392145157,
+                  "top1_top2_margin": 0.028682969510555267
+                },
+                {
+                  "position": 5,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.13226112723350525
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.12259065359830856
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.07745791226625443
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.04579097405076027
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.03493405878543854
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.03442758321762085
+                    }
+                  ],
+                  "selected_weight_sum": 0.4474623091518879,
+                  "top1_top2_margin": 0.009670473635196686
+                },
+                {
+                  "position": 6,
+                  "token_id": 10285,
+                  "token": "为ä»Ģä¹Ī",
+                  "decoded_piece": "为什么",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.12769539654254913
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.08412055671215057
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.046987637877464294
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.04405679926276207
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.03699975833296776
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.026510879397392273
+                    }
+                  ],
+                  "selected_weight_sum": 0.3663710281252861,
+                  "top1_top2_margin": 0.04357483983039856
+                },
+                {
+                  "position": 7,
+                  "token_id": 21844,
+                  "token": "ç¨Ģ",
+                  "decoded_piece": "稀",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.16432742774486542
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.0821835920214653
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.054694924503564835
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.037129003554582596
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.036085568368434906
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.03494443744421005
+                    }
+                  ],
+                  "selected_weight_sum": 0.4093649536371231,
+                  "top1_top2_margin": 0.08214383572340012
+                },
+                {
+                  "position": 8,
+                  "token_id": 20923,
+                  "token": "çĸı",
+                  "decoded_piece": "疏",
+                  "experts_by_weight": [
+                    {
+                      "expert": 49,
+                      "weight": 0.2578503489494324
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.12817910313606262
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.07802081108093262
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03407106548547745
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.026755359023809433
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.022954754531383514
+                    }
+                  ],
+                  "selected_weight_sum": 0.547831442207098,
+                  "top1_top2_margin": 0.12967124581336975
+                },
+                {
+                  "position": 9,
+                  "token_id": 10900,
+                  "token": "ä¸ĵ家",
+                  "decoded_piece": "专家",
+                  "experts_by_weight": [
+                    {
+                      "expert": 20,
+                      "weight": 0.17112381756305695
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.0664329081773758
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05617460608482361
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.05288673937320709
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.051459167152643204
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.038296010345220566
+                    }
+                  ],
+                  "selected_weight_sum": 0.4363732486963272,
+                  "top1_top2_margin": 0.10469090938568115
+                },
+                {
+                  "position": 10,
+                  "token_id": 39322,
+                  "token": "模åŀĭ",
+                  "decoded_piece": "模型",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.17979325354099274
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.10962751507759094
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.0830262154340744
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.0828174576163292
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.04263872653245926
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.03515750914812088
+                    }
+                  ],
+                  "selected_weight_sum": 0.5330606773495674,
+                  "top1_top2_margin": 0.0701657384634018
+                },
+                {
+                  "position": 11,
+                  "token_id": 1876,
+                  "token": "åı¯ä»¥",
+                  "decoded_piece": "可以",
+                  "experts_by_weight": [
+                    {
+                      "expert": 15,
+                      "weight": 0.1157049909234047
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.11091289669275284
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.050560325384140015
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.0373777411878109
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.032718632370233536
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.025750335305929184
+                    }
+                  ],
+                  "selected_weight_sum": 0.37302492186427116,
+                  "top1_top2_margin": 0.0047920942306518555
+                },
+                {
+                  "position": 12,
+                  "token_id": 9476,
+                  "token": "æĭ¥æľī",
+                  "decoded_piece": "拥有",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.17254002392292023
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.08969754725694656
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.07656068354845047
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.038461942225694656
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03243434801697731
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.02500019408762455
+                    }
+                  ],
+                  "selected_weight_sum": 0.4346947390586138,
+                  "top1_top2_margin": 0.08284247666597366
+                },
+                {
+                  "position": 13,
+                  "token_id": 3599,
+                  "token": "å¾Īå¤ļ",
+                  "decoded_piece": "很多",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.11956122517585754
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.11295492947101593
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.06735346466302872
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.061125051230192184
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.042656976729631424
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.027563519775867462
+                    }
+                  ],
+                  "selected_weight_sum": 0.43121516704559326,
+                  "top1_top2_margin": 0.006606295704841614
+                },
+                {
+                  "position": 14,
+                  "token_id": 21273,
+                  "token": "åıĤæķ°",
+                  "decoded_piece": "参数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 45,
+                      "weight": 0.09754917025566101
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.07696481049060822
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.04299183189868927
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04288526996970177
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.036022163927555084
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.029702916741371155
+                    }
+                  ],
+                  "selected_weight_sum": 0.3261161632835865,
+                  "top1_top2_margin": 0.020584359765052795
+                },
+                {
+                  "position": 15,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.10675214231014252
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.10003915429115295
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.07917536050081253
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.03613792732357979
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.029818829149007797
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.0273029413074255
+                    }
+                  ],
+                  "selected_weight_sum": 0.3792263548821211,
+                  "top1_top2_margin": 0.006712988018989563
+                },
+                {
+                  "position": 16,
+                  "token_id": 1705,
+                  "token": "ä½Ĩ",
+                  "decoded_piece": "但",
+                  "experts_by_weight": [
+                    {
+                      "expert": 44,
+                      "weight": 0.0980173721909523
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.08442847430706024
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.08428655564785004
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.036362823098897934
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.031637802720069885
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03130587935447693
+                    }
+                  ],
+                  "selected_weight_sum": 0.3660389073193073,
+                  "top1_top2_margin": 0.01358889788389206
+                },
+                {
+                  "position": 17,
+                  "token_id": 13712,
+                  "token": "æ¯ı个",
+                  "decoded_piece": "每个",
+                  "experts_by_weight": [
+                    {
+                      "expert": 61,
+                      "weight": 0.11883475631475449
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.11768975853919983
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.04394533485174179
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.03521232306957245
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.02718379534780979
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.025340577587485313
+                    }
+                  ],
+                  "selected_weight_sum": 0.36820654571056366,
+                  "top1_top2_margin": 0.001144997775554657
+                },
+                {
+                  "position": 18,
+                  "token_id": 10728,
+                  "token": "Ġtoken",
+                  "decoded_piece": " token",
+                  "experts_by_weight": [
+                    {
+                      "expert": 45,
+                      "weight": 0.14425362646579742
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.12026972323656082
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.07732635736465454
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.06987600773572922
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.030445804819464684
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.03043285384774208
+                    }
+                  ],
+                  "selected_weight_sum": 0.47260437346994877,
+                  "top1_top2_margin": 0.023983903229236603
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.16929486393928528
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.09825362265110016
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.04029633849859238
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.032645758241415024
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.026754116639494896
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.022297902032732964
+                    }
+                  ],
+                  "selected_weight_sum": 0.3895426020026207,
+                  "top1_top2_margin": 0.07104124128818512
+                },
+                {
+                  "position": 20,
+                  "token_id": 2141,
+                  "token": "åıª",
+                  "decoded_piece": "只",
+                  "experts_by_weight": [
+                    {
+                      "expert": 5,
+                      "weight": 0.1217007115483284
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.06380197405815125
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.039825357496738434
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.03771115466952324
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.036976952105760574
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.033029451966285706
+                    }
+                  ],
+                  "selected_weight_sum": 0.3330456018447876,
+                  "top1_top2_margin": 0.057898737490177155
+                },
+                {
+                  "position": 21,
+                  "token_id": 5118,
+                  "token": "使çĶ¨",
+                  "decoded_piece": "使用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 9,
+                      "weight": 0.1564585566520691
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.15290194749832153
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.09143153578042984
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.045543499290943146
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.03213364630937576
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.030746884644031525
+                    }
+                  ],
+                  "selected_weight_sum": 0.5092160701751709,
+                  "top1_top2_margin": 0.0035566091537475586
+                },
+                {
+                  "position": 22,
+                  "token_id": 6802,
+                  "token": "åħ¶ä¸Ń",
+                  "decoded_piece": "其中",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.18916168808937073
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.14847588539123535
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.06800840049982071
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.03237553685903549
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.029806064441800117
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.029624512419104576
+                    }
+                  ],
+                  "selected_weight_sum": 0.497452087700367,
+                  "top1_top2_margin": 0.040685802698135376
+                },
+                {
+                  "position": 23,
+                  "token_id": 51905,
+                  "token": "ä¸Ģå°ı",
+                  "decoded_piece": "一小",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.2096928209066391
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.18414442241191864
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.06322769820690155
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.03188531473278999
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.026658499613404274
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.023066453635692596
+                    }
+                  ],
+                  "selected_weight_sum": 0.5386752095073462,
+                  "top1_top2_margin": 0.02554839849472046
+                },
+                {
+                  "position": 24,
+                  "token_id": 5781,
+                  "token": "éĥ¨åĪĨ",
+                  "decoded_piece": "部分",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.19247254729270935
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.09449118375778198
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.08716513961553574
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.050964564085006714
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.04497525840997696
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.02731938287615776
+                    }
+                  ],
+                  "selected_weight_sum": 0.4973880760371685,
+                  "top1_top2_margin": 0.09798136353492737
+                },
+                {
+                  "position": 25,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.13403064012527466
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.1253703534603119
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.07578480988740921
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.05519406497478485
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.027481039986014366
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.02377881109714508
+                    }
+                  ],
+                  "selected_weight_sum": 0.44163971953094006,
+                  "top1_top2_margin": 0.008660286664962769
+                }
+              ]
+            },
+            {
+              "id": "en_architecture",
+              "label": "English architecture",
+              "tokens": 18,
+              "load": [
+                2,
+                1,
+                1,
+                2,
+                3,
+                2,
+                3,
+                1,
+                1,
+                0,
+                0,
+                1,
+                1,
+                2,
+                0,
+                2,
+                1,
+                2,
+                2,
+                1,
+                0,
+                0,
+                3,
+                0,
+                1,
+                0,
+                6,
+                4,
+                2,
+                0,
+                3,
+                1,
+                3,
+                0,
+                2,
+                5,
+                2,
+                0,
+                1,
+                0,
+                6,
+                1,
+                1,
+                4,
+                0,
+                3,
+                1,
+                1,
+                1,
+                1,
+                2,
+                1,
+                1,
+                1,
+                1,
+                5,
+                0,
+                3,
+                0,
+                0,
+                4,
+                2,
+                4,
+                4
+              ],
+              "metrics": {
+                "routes": 108,
+                "used_experts": 49,
+                "zero_experts": 15,
+                "mean": 1.6875,
+                "std": 1.5520083904266357,
+                "cv": 0.9197086691856384,
+                "gini": 0.48755785822868347,
+                "entropy_nats": 3.7096519470214844,
+                "effective_experts": 40.83958971219074,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 4.0,
+                "max": 6.0
+              },
+              "top_experts": [
+                {
+                  "expert": 26,
+                  "routes": 6
+                },
+                {
+                  "expert": 40,
+                  "routes": 6
+                },
+                {
+                  "expert": 35,
+                  "routes": 5
+                },
+                {
+                  "expert": 55,
+                  "routes": 5
+                },
+                {
+                  "expert": 27,
+                  "routes": 4
+                },
+                {
+                  "expert": 43,
+                  "routes": 4
+                },
+                {
+                  "expert": 60,
+                  "routes": 4
+                },
+                {
+                  "expert": 62,
+                  "routes": 4
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 3,
+                      "weight": 0.7993167042732239
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.018388060852885246
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.010282253846526146
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.009434774518013
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.009029458276927471
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.008924801833927631
+                    }
+                  ],
+                  "selected_weight_sum": 0.8553760536015034,
+                  "top1_top2_margin": 0.7809286434203386
+                },
+                {
+                  "position": 1,
+                  "token_id": 99297,
+                  "token": "Explain",
+                  "decoded_piece": "Explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.1933971792459488
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.1331486701965332
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.07104431837797165
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.06700316816568375
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.061555638909339905
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.047345709055662155
+                    }
+                  ],
+                  "selected_weight_sum": 0.5734946839511395,
+                  "top1_top2_margin": 0.06024850904941559
+                },
+                {
+                  "position": 2,
+                  "token_id": 946,
+                  "token": "Ġhow",
+                  "decoded_piece": " how",
+                  "experts_by_weight": [
+                    {
+                      "expert": 5,
+                      "weight": 0.11682333797216415
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.09969403594732285
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.07592637836933136
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.05120022967457771
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.03427540510892868
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.029783086851239204
+                    }
+                  ],
+                  "selected_weight_sum": 0.40770247392356396,
+                  "top1_top2_margin": 0.01712930202484131
+                },
+                {
+                  "position": 3,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.24449819326400757
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.07239027321338654
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.05962592363357544
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05310329422354698
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.04907490685582161
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.047838661819696426
+                    }
+                  ],
+                  "selected_weight_sum": 0.5265312530100346,
+                  "top1_top2_margin": 0.17210792005062103
+                },
+                {
+                  "position": 4,
+                  "token_id": 29640,
+                  "token": "Ġcompressed",
+                  "decoded_piece": " compressed",
+                  "experts_by_weight": [
+                    {
+                      "expert": 18,
+                      "weight": 0.11223237961530685
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.09656558930873871
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.04231010004878044
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.03794880956411362
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.034158121794462204
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.03112885169684887
+                    }
+                  ],
+                  "selected_weight_sum": 0.3543438520282507,
+                  "top1_top2_margin": 0.015666790306568146
+                },
+                {
+                  "position": 5,
+                  "token_id": 2131,
+                  "token": "Ġkey",
+                  "decoded_piece": " key",
+                  "experts_by_weight": [
+                    {
+                      "expert": 30,
+                      "weight": 0.17843440175056458
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.11469165980815887
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.06070755422115326
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.05988939851522446
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.05858447402715683
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.05206485837697983
+                    }
+                  ],
+                  "selected_weight_sum": 0.5243723466992378,
+                  "top1_top2_margin": 0.0637427419424057
+                },
+                {
+                  "position": 6,
+                  "token_id": 12,
+                  "token": "-",
+                  "decoded_piece": "-",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.15919552743434906
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.10320810228586197
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.09070276468992233
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.042428430169820786
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04033466428518295
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.03656792640686035
+                    }
+                  ],
+                  "selected_weight_sum": 0.47243741527199745,
+                  "top1_top2_margin": 0.05598742514848709
+                },
+                {
+                  "position": 7,
+                  "token_id": 2834,
+                  "token": "value",
+                  "decoded_piece": "value",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.14338308572769165
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.1320078819990158
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.13028068840503693
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.04761816933751106
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04696355387568474
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.043221261352300644
+                    }
+                  ],
+                  "selected_weight_sum": 0.5434746406972408,
+                  "top1_top2_margin": 0.011375203728675842
+                },
+                {
+                  "position": 8,
+                  "token_id": 28937,
+                  "token": "Ġlatent",
+                  "decoded_piece": " latent",
+                  "experts_by_weight": [
+                    {
+                      "expert": 55,
+                      "weight": 0.11452669650316238
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.10020633041858673
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.058247704058885574
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.05524905398488045
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.038477592170238495
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.0318424291908741
+                    }
+                  ],
+                  "selected_weight_sum": 0.39854980632662773,
+                  "top1_top2_margin": 0.014320366084575653
+                },
+                {
+                  "position": 9,
+                  "token_id": 4183,
+                  "token": "Ġchanges",
+                  "decoded_piece": " changes",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.10610464960336685
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.08787750452756882
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.08229339122772217
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.06280581653118134
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.047021713107824326
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.04179764166474342
+                    }
+                  ],
+                  "selected_weight_sum": 0.4279007166624069,
+                  "top1_top2_margin": 0.018227145075798035
+                },
+                {
+                  "position": 10,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.10047860443592072
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.09288082271814346
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.08753325045108795
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.03979819640517235
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.033622682094573975
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03130070120096207
+                    }
+                  ],
+                  "selected_weight_sum": 0.3856142573058605,
+                  "top1_top2_margin": 0.007597781717777252
+                },
+                {
+                  "position": 11,
+                  "token_id": 4887,
+                  "token": "Ġmemory",
+                  "decoded_piece": " memory",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.10118205845355988
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.08818523585796356
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.06513181328773499
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.03309033811092377
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03305121511220932
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.02862332947552204
+                    }
+                  ],
+                  "selected_weight_sum": 0.34926399029791355,
+                  "top1_top2_margin": 0.012996822595596313
+                },
+                {
+                  "position": 12,
+                  "token_id": 2295,
+                  "token": "Ġcost",
+                  "decoded_piece": " cost",
+                  "experts_by_weight": [
+                    {
+                      "expert": 40,
+                      "weight": 0.2086726278066635
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.1317712515592575
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.08160752803087234
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.06133318692445755
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.04480573907494545
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.040977973490953445
+                    }
+                  ],
+                  "selected_weight_sum": 0.5691683068871498,
+                  "top1_top2_margin": 0.076901376247406
+                },
+                {
+                  "position": 13,
+                  "token_id": 280,
+                  "token": "Ġof",
+                  "decoded_piece": " of",
+                  "experts_by_weight": [
+                    {
+                      "expert": 15,
+                      "weight": 0.14979112148284912
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.11800755560398102
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.1067730262875557
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.0327712818980217
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.02830866351723671
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.02398701384663582
+                    }
+                  ],
+                  "selected_weight_sum": 0.45963866263628006,
+                  "top1_top2_margin": 0.0317835658788681
+                },
+                {
+                  "position": 14,
+                  "token_id": 77939,
+                  "token": "Ġautore",
+                  "decoded_piece": " autore",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.10697006434202194
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.05533495172858238
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.05411042273044586
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.03916148468852043
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.03792927414178848
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.03450075909495354
+                    }
+                  ],
+                  "selected_weight_sum": 0.32800695672631264,
+                  "top1_top2_margin": 0.05163511261343956
+                },
+                {
+                  "position": 15,
+                  "token_id": 79184,
+                  "token": "gressive",
+                  "decoded_piece": "gressive",
+                  "experts_by_weight": [
+                    {
+                      "expert": 28,
+                      "weight": 0.2267676740884781
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.13144974410533905
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.06287088990211487
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.05779385194182396
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.05445825681090355
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.033917136490345
+                    }
+                  ],
+                  "selected_weight_sum": 0.5672575533390045,
+                  "top1_top2_margin": 0.09531792998313904
+                },
+                {
+                  "position": 16,
+                  "token_id": 39671,
+                  "token": "Ġdecoding",
+                  "decoded_piece": " decoding",
+                  "experts_by_weight": [
+                    {
+                      "expert": 55,
+                      "weight": 0.11397071182727814
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.08910854160785675
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.07447721809148788
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.05076982080936432
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.046305228024721146
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.04167233407497406
+                    }
+                  ],
+                  "selected_weight_sum": 0.4163038544356823,
+                  "top1_top2_margin": 0.024862170219421387
+                },
+                {
+                  "position": 17,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.17542369663715363
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.10671209543943405
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.07208603620529175
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.0618680976331234
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.05220293253660202
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.040070801973342896
+                    }
+                  ],
+                  "selected_weight_sum": 0.5083636604249477,
+                  "top1_top2_margin": 0.06871160119771957
+                }
+              ]
+            },
+            {
+              "id": "code",
+              "label": "Python code",
+              "tokens": 19,
+              "load": [
+                3,
+                0,
+                0,
+                5,
+                2,
+                1,
+                0,
+                2,
+                1,
+                3,
+                0,
+                2,
+                2,
+                1,
+                0,
+                3,
+                2,
+                4,
+                3,
+                1,
+                3,
+                0,
+                2,
+                1,
+                0,
+                1,
+                4,
+                0,
+                1,
+                1,
+                5,
+                0,
+                2,
+                2,
+                2,
+                3,
+                2,
+                2,
+                1,
+                3,
+                2,
+                2,
+                1,
+                4,
+                1,
+                4,
+                2,
+                3,
+                2,
+                1,
+                0,
+                1,
+                4,
+                0,
+                0,
+                3,
+                1,
+                2,
+                1,
+                1,
+                3,
+                1,
+                3,
+                2
+              ],
+              "metrics": {
+                "routes": 114,
+                "used_experts": 52,
+                "zero_experts": 12,
+                "mean": 1.78125,
+                "std": 1.3269938230514526,
+                "cv": 0.7449789643287659,
+                "gini": 0.4098135828971863,
+                "entropy_nats": 3.8270628452301025,
+                "effective_experts": 45.9274439238954,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 3.6999969482421875,
+                "max": 5.0
+              },
+              "top_experts": [
+                {
+                  "expert": 3,
+                  "routes": 5
+                },
+                {
+                  "expert": 30,
+                  "routes": 5
+                },
+                {
+                  "expert": 17,
+                  "routes": 4
+                },
+                {
+                  "expert": 26,
+                  "routes": 4
+                },
+                {
+                  "expert": 43,
+                  "routes": 4
+                },
+                {
+                  "expert": 45,
+                  "routes": 4
+                },
+                {
+                  "expert": 52,
+                  "routes": 4
+                },
+                {
+                  "expert": 0,
+                  "routes": 3
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 3,
+                      "weight": 0.7993167042732239
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.018388060852885246
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.010282253846526146
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.009434774518013
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.009029458276927471
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.008924801833927631
+                    }
+                  ],
+                  "selected_weight_sum": 0.8553760536015034,
+                  "top1_top2_margin": 0.7809286434203386
+                },
+                {
+                  "position": 1,
+                  "token_id": 9081,
+                  "token": "Write",
+                  "decoded_piece": "Write",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.22308281064033508
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.10125719755887985
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.07881859689950943
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.07489147037267685
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.06411762535572052
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.03780627250671387
+                    }
+                  ],
+                  "selected_weight_sum": 0.5799739733338356,
+                  "top1_top2_margin": 0.12182561308145523
+                },
+                {
+                  "position": 2,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.11058355867862701
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.09710631519556046
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.09567485749721527
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.07556460052728653
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.040634334087371826
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.03968478739261627
+                    }
+                  ],
+                  "selected_weight_sum": 0.45924845337867737,
+                  "top1_top2_margin": 0.013477243483066559
+                },
+                {
+                  "position": 3,
+                  "token_id": 12974,
+                  "token": "ĠPython",
+                  "decoded_piece": " Python",
+                  "experts_by_weight": [
+                    {
+                      "expert": 43,
+                      "weight": 0.3106133043766022
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.10412774980068207
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.07865095138549805
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.03615188226103783
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.03441538289189339
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.03303976729512215
+                    }
+                  ],
+                  "selected_weight_sum": 0.5969990380108356,
+                  "top1_top2_margin": 0.2064855545759201
+                },
+                {
+                  "position": 4,
+                  "token_id": 1157,
+                  "token": "Ġfunction",
+                  "decoded_piece": " function",
+                  "experts_by_weight": [
+                    {
+                      "expert": 12,
+                      "weight": 0.2627643644809723
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.20828509330749512
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.07966912537813187
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.04565059021115303
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.04058252274990082
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.032550934702157974
+                    }
+                  ],
+                  "selected_weight_sum": 0.6695026308298111,
+                  "top1_top2_margin": 0.05447927117347717
+                },
+                {
+                  "position": 5,
+                  "token_id": 344,
+                  "token": "Ġthat",
+                  "decoded_piece": " that",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.18105126917362213
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.12537729740142822
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.06263353675603867
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.04046915844082832
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03560767322778702
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.035325679928064346
+                    }
+                  ],
+                  "selected_weight_sum": 0.4804646149277687,
+                  "top1_top2_margin": 0.05567397177219391
+                },
+                {
+                  "position": 6,
+                  "token_id": 7578,
+                  "token": "Ġreturns",
+                  "decoded_piece": " returns",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.07301884144544601
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.06769263744354248
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.057351455092430115
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.057337403297424316
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.05577944219112396
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.0453062430024147
+                    }
+                  ],
+                  "selected_weight_sum": 0.3564860224723816,
+                  "top1_top2_margin": 0.005326204001903534
+                },
+                {
+                  "position": 7,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.1945454478263855
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.0892791748046875
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.08212577551603317
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.03960113599896431
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.038635849952697754
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.03487828001379967
+                    }
+                  ],
+                  "selected_weight_sum": 0.4790656641125679,
+                  "top1_top2_margin": 0.105266273021698
+                },
+                {
+                  "position": 8,
+                  "token_id": 1022,
+                  "token": "Ġfirst",
+                  "decoded_piece": " first",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.14187313616275787
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05546113848686218
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.038507647812366486
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.03536592423915863
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.030138704925775528
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.029435299336910248
+                    }
+                  ],
+                  "selected_weight_sum": 0.33078185096383095,
+                  "top1_top2_margin": 0.08641199767589569
+                },
+                {
+                  "position": 9,
+                  "token_id": 9842,
+                  "token": "Ġrepeated",
+                  "decoded_piece": " repeated",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.1300642341375351
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.09043307602405548
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.07752960920333862
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.06140988692641258
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.03912041708827019
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.03831544145941734
+                    }
+                  ],
+                  "selected_weight_sum": 0.4368726648390293,
+                  "top1_top2_margin": 0.039631158113479614
+                },
+                {
+                  "position": 10,
+                  "token_id": 3699,
+                  "token": "Ġelement",
+                  "decoded_piece": " element",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.13783960044384003
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.12804515659809113
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.09556166082620621
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.09431081265211105
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.06392165273427963
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.04370862618088722
+                    }
+                  ],
+                  "selected_weight_sum": 0.5633875094354153,
+                  "top1_top2_margin": 0.009794443845748901
+                },
+                {
+                  "position": 11,
+                  "token_id": 279,
+                  "token": "Ġin",
+                  "decoded_piece": " in",
+                  "experts_by_weight": [
+                    {
+                      "expert": 9,
+                      "weight": 0.12035012990236282
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.11592075228691101
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.05035826563835144
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.04238616302609444
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.030062539502978325
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.029798714444041252
+                    }
+                  ],
+                  "selected_weight_sum": 0.3888765648007393,
+                  "top1_top2_margin": 0.004429377615451813
+                },
+                {
+                  "position": 12,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.1413310319185257
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.06695874035358429
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.06325921416282654
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.035412270575761795
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.02976227179169655
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.029165809974074364
+                    }
+                  ],
+                  "selected_weight_sum": 0.36588933877646923,
+                  "top1_top2_margin": 0.0743722915649414
+                },
+                {
+                  "position": 13,
+                  "token_id": 1525,
+                  "token": "Ġlist",
+                  "decoded_piece": " list",
+                  "experts_by_weight": [
+                    {
+                      "expert": 30,
+                      "weight": 0.1110549345612526
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.09031160920858383
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.07663130760192871
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.06359056383371353
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.049891188740730286
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.04514267295598984
+                    }
+                  ],
+                  "selected_weight_sum": 0.4366222769021988,
+                  "top1_top2_margin": 0.020743325352668762
+                },
+                {
+                  "position": 14,
+                  "token_id": 285,
+                  "token": "Ġand",
+                  "decoded_piece": " and",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.11731312423944473
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.07047951221466064
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.05180724337697029
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.044003911316394806
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.03136441856622696
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.03092145547270775
+                    }
+                  ],
+                  "selected_weight_sum": 0.3458896651864052,
+                  "top1_top2_margin": 0.04683361202478409
+                },
+                {
+                  "position": 15,
+                  "token_id": 6716,
+                  "token": "Ġexplain",
+                  "decoded_piece": " explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 44,
+                      "weight": 0.15685030817985535
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.11473170667886734
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.07455103099346161
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.0551246777176857
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.034474410116672516
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.028114324435591698
+                    }
+                  ],
+                  "selected_weight_sum": 0.4638464581221342,
+                  "top1_top2_margin": 0.04211860150098801
+                },
+                {
+                  "position": 16,
+                  "token_id": 895,
+                  "token": "Ġits",
+                  "decoded_piece": " its",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.1131797656416893
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.10879005491733551
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.061025477945804596
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.057621393352746964
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.03110535442829132
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.02715742588043213
+                    }
+                  ],
+                  "selected_weight_sum": 0.3988794721662998,
+                  "top1_top2_margin": 0.00438971072435379
+                },
+                {
+                  "position": 17,
+                  "token_id": 13913,
+                  "token": "Ġcomplexity",
+                  "decoded_piece": " complexity",
+                  "experts_by_weight": [
+                    {
+                      "expert": 33,
+                      "weight": 0.08646200597286224
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.08283549547195435
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.06919881701469421
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.060465697199106216
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.05625320225954056
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.04574577510356903
+                    }
+                  ],
+                  "selected_weight_sum": 0.4009609930217266,
+                  "top1_top2_margin": 0.003626510500907898
+                },
+                {
+                  "position": 18,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.19963113963603973
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.1043989434838295
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.07004683464765549
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.04874078184366226
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.03005041368305683
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.02090730145573616
+                    }
+                  ],
+                  "selected_weight_sum": 0.47377541474998,
+                  "top1_top2_margin": 0.09523219615221024
+                }
+              ]
+            },
+            {
+              "id": "math",
+              "label": "数学推理",
+              "tokens": 27,
+              "load": [
+                5,
+                1,
+                0,
+                2,
+                0,
+                3,
+                2,
+                2,
+                4,
+                4,
+                1,
+                0,
+                2,
+                2,
+                2,
+                3,
+                5,
+                2,
+                3,
+                1,
+                3,
+                0,
+                4,
+                2,
+                3,
+                1,
+                0,
+                4,
+                0,
+                0,
+                2,
+                1,
+                5,
+                3,
+                7,
+                6,
+                2,
+                0,
+                4,
+                1,
+                2,
+                0,
+                3,
+                0,
+                1,
+                1,
+                3,
+                9,
+                2,
+                3,
+                2,
+                1,
+                6,
+                1,
+                0,
+                3,
+                2,
+                4,
+                5,
+                1,
+                7,
+                2,
+                10,
+                2
+              ],
+              "metrics": {
+                "routes": 162,
+                "used_experts": 53,
+                "zero_experts": 11,
+                "mean": 2.53125,
+                "std": 2.1892142295837402,
+                "cv": 0.8648747801780701,
+                "gini": 0.4535107910633087,
+                "entropy_nats": 3.7780678272247314,
+                "effective_experts": 43.73146330899206,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 5.0,
+                "max": 10.0
+              },
+              "top_experts": [
+                {
+                  "expert": 62,
+                  "routes": 10
+                },
+                {
+                  "expert": 47,
+                  "routes": 9
+                },
+                {
+                  "expert": 34,
+                  "routes": 7
+                },
+                {
+                  "expert": 60,
+                  "routes": 7
+                },
+                {
+                  "expert": 35,
+                  "routes": 6
+                },
+                {
+                  "expert": 52,
+                  "routes": 6
+                },
+                {
+                  "expert": 0,
+                  "routes": 5
+                },
+                {
+                  "expert": 16,
+                  "routes": 5
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 3,
+                      "weight": 0.7993167042732239
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.018388060852885246
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.010282253846526146
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.009434774518013
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.009029458276927471
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.008924801833927631
+                    }
+                  ],
+                  "selected_weight_sum": 0.8553760536015034,
+                  "top1_top2_margin": 0.7809286434203386
+                },
+                {
+                  "position": 1,
+                  "token_id": 8246,
+                  "token": "èĭ¥",
+                  "decoded_piece": "若",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.13331177830696106
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.12700548768043518
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.10285352915525436
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.07681424915790558
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.06962653994560242
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.06532615423202515
+                    }
+                  ],
+                  "selected_weight_sum": 0.5749377384781837,
+                  "top1_top2_margin": 0.006306290626525879
+                },
+                {
+                  "position": 2,
+                  "token_id": 2206,
+                  "token": "æŃ£",
+                  "decoded_piece": "正",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.2814550995826721
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.08446836471557617
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.07371900975704193
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.06035252660512924
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05827755853533745
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.02826753258705139
+                    }
+                  ],
+                  "selected_weight_sum": 0.5865400917828083,
+                  "top1_top2_margin": 0.19698673486709595
+                },
+                {
+                  "position": 3,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 49,
+                      "weight": 0.25039201974868774
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.22088326513767242
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.20361556112766266
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.02990611642599106
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.019317051395773888
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.018254980444908142
+                    }
+                  ],
+                  "selected_weight_sum": 0.7423689942806959,
+                  "top1_top2_margin": 0.02950875461101532
+                },
+                {
+                  "position": 4,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.3537174165248871
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.08283980935811996
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.0720146968960762
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.0697723850607872
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.052806876599788666
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.04913831129670143
+                    }
+                  ],
+                  "selected_weight_sum": 0.6802894957363605,
+                  "top1_top2_margin": 0.2708776071667671
+                },
+                {
+                  "position": 5,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.1735713630914688
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.07046853005886078
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.037892214953899384
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.028465814888477325
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.02768941968679428
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.0260683111846447
+                    }
+                  ],
+                  "selected_weight_sum": 0.3641556538641453,
+                  "top1_top2_margin": 0.10310283303260803
+                },
+                {
+                  "position": 6,
+                  "token_id": 13211,
+                  "token": "满足",
+                  "decoded_piece": "满足",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.2298489511013031
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.0791824534535408
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.05738123506307602
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.055221591144800186
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.04613054171204567
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.037003934383392334
+                    }
+                  ],
+                  "selected_weight_sum": 0.5047687068581581,
+                  "top1_top2_margin": 0.1506664976477623
+                },
+                {
+                  "position": 7,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.296394020318985
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.08370600640773773
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.037053510546684265
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03541581705212593
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.02703707478940487
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.02383062243461609
+                    }
+                  ],
+                  "selected_weight_sum": 0.5034370515495539,
+                  "top1_top2_margin": 0.21268801391124725
+                },
+                {
+                  "position": 8,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.16342000663280487
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.08331643044948578
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.04279105365276337
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.037890754640102386
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.03473474085330963
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.02922108583152294
+                    }
+                  ],
+                  "selected_weight_sum": 0.391374072059989,
+                  "top1_top2_margin": 0.08010357618331909
+                },
+                {
+                  "position": 9,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.318281352519989
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.04051169380545616
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.040131304413080215
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.03766006976366043
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03394027799367905
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.02833012118935585
+                    }
+                  ],
+                  "selected_weight_sum": 0.4988548196852207,
+                  "top1_top2_margin": 0.27776965871453285
+                },
+                {
+                  "position": 10,
+                  "token_id": 12985,
+                  "token": "å®ĥçļĦ",
+                  "decoded_piece": "它的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 20,
+                      "weight": 0.10639920830726624
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.08816708624362946
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.08003807067871094
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.05453759431838989
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.046933963894844055
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.04441061615943909
+                    }
+                  ],
+                  "selected_weight_sum": 0.42048653960227966,
+                  "top1_top2_margin": 0.01823212206363678
+                },
+                {
+                  "position": 11,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.16316896677017212
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.060158684849739075
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.05992881581187248
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.037876423448324203
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.03393488749861717
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.03238135948777199
+                    }
+                  ],
+                  "selected_weight_sum": 0.38744913786649704,
+                  "top1_top2_margin": 0.10301028192043304
+                },
+                {
+                  "position": 12,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 49,
+                      "weight": 0.18513305485248566
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.1456661969423294
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.11328830569982529
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.05659753084182739
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.04169565439224243
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.03690284490585327
+                    }
+                  ],
+                  "selected_weight_sum": 0.5792835876345634,
+                  "top1_top2_margin": 0.03946685791015625
+                },
+                {
+                  "position": 13,
+                  "token_id": 37360,
+                  "token": "çŃīäºİ",
+                  "decoded_piece": "等于",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.2411312758922577
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.10274491459131241
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.05373065173625946
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.03419595956802368
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.02804616279900074
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.0268552228808403
+                    }
+                  ],
+                  "selected_weight_sum": 0.4867041874676943,
+                  "top1_top2_margin": 0.13838636130094528
+                },
+                {
+                  "position": 14,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.16179583966732025
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.07096794247627258
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.06854509562253952
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04157784581184387
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.03384833782911301
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.0296489167958498
+                    }
+                  ],
+                  "selected_weight_sum": 0.40638397820293903,
+                  "top1_top2_margin": 0.09082789719104767
+                },
+                {
+                  "position": 15,
+                  "token_id": 18,
+                  "token": "3",
+                  "decoded_piece": "3",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.16428224742412567
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.10361717641353607
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.08202363550662994
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.04421326518058777
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.02995552122592926
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.02580142579972744
+                    }
+                  ],
+                  "selected_weight_sum": 0.44989327155053616,
+                  "top1_top2_margin": 0.0606650710105896
+                },
+                {
+                  "position": 16,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.13362829387187958
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.093042753636837
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.07785298675298691
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.071451835334301
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.06596096605062485
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.029706455767154694
+                    }
+                  ],
+                  "selected_weight_sum": 0.471643291413784,
+                  "top1_top2_margin": 0.04058554023504257
+                },
+                {
+                  "position": 17,
+                  "token_id": 2989,
+                  "token": "æ±Ĥ",
+                  "decoded_piece": "求",
+                  "experts_by_weight": [
+                    {
+                      "expert": 61,
+                      "weight": 0.23183642327785492
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.16105559468269348
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.05859864875674248
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.04714159667491913
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.03799108788371086
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.03278881683945656
+                    }
+                  ],
+                  "selected_weight_sum": 0.5694121681153774,
+                  "top1_top2_margin": 0.07078082859516144
+                },
+                {
+                  "position": 18,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.2791516184806824
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.08195051550865173
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.03374714404344559
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.03366595879197121
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.026014486327767372
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.02444468066096306
+                    }
+                  ],
+                  "selected_weight_sum": 0.47897440381348133,
+                  "top1_top2_margin": 0.19720110297203064
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.1385689079761505
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.04262688755989075
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.041786808520555496
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03194677457213402
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.028856562450528145
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.02796977385878563
+                    }
+                  ],
+                  "selected_weight_sum": 0.31175571493804455,
+                  "top1_top2_margin": 0.09594202041625977
+                },
+                {
+                  "position": 20,
+                  "token_id": 337,
+                  "token": "çļĦ",
+                  "decoded_piece": "的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.13563965260982513
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.057844046503305435
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.05458466708660126
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04892478883266449
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.03986985981464386
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.031419653445482254
+                    }
+                  ],
+                  "selected_weight_sum": 0.36828266829252243,
+                  "top1_top2_margin": 0.0777956061065197
+                },
+                {
+                  "position": 21,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.3164118230342865
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.04724707081913948
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.044920627027750015
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.029304083436727524
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.025028377771377563
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.024010613560676575
+                    }
+                  ],
+                  "selected_weight_sum": 0.48692259564995766,
+                  "top1_top2_margin": 0.269164752215147
+                },
+                {
+                  "position": 22,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.20498308539390564
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.06528150290250778
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.046608418226242065
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.04563548043370247
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.04310647398233414
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.03512641042470932
+                    }
+                  ],
+                  "selected_weight_sum": 0.4407413713634014,
+                  "top1_top2_margin": 0.13970158249139786
+                },
+                {
+                  "position": 23,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 1,
+                      "weight": 0.07203083485364914
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.07109834998846054
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.0562785267829895
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.05611051246523857
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.042118486016988754
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.04034203663468361
+                    }
+                  ],
+                  "selected_weight_sum": 0.3379787467420101,
+                  "top1_top2_margin": 0.0009324848651885986
+                },
+                {
+                  "position": 24,
+                  "token_id": 51971,
+                  "token": "æķ°çļĦ",
+                  "decoded_piece": "数的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 49,
+                      "weight": 0.19896872341632843
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.11013466864824295
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.09039244800806046
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.05881365388631821
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.05270054563879967
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.05187920853495598
+                    }
+                  ],
+                  "selected_weight_sum": 0.5628892481327057,
+                  "top1_top2_margin": 0.08883405476808548
+                },
+                {
+                  "position": 25,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.2541465163230896
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.06222054362297058
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.048966165632009506
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03323732689023018
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.031393036246299744
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.02776632457971573
+                    }
+                  ],
+                  "selected_weight_sum": 0.45772991329431534,
+                  "top1_top2_margin": 0.19192597270011902
+                },
+                {
+                  "position": 26,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.14508213102817535
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.12345562130212784
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.06189163029193878
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.056270845234394073
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.03983943909406662
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.02559591457247734
+                    }
+                  ],
+                  "selected_weight_sum": 0.45213558152318,
+                  "top1_top2_margin": 0.021626509726047516
+                }
+              ]
+            }
+          ],
+          "prompt_pair_jaccard": [
+            {
+              "left": "zh_explanation",
+              "right": "en_architecture",
+              "used_expert_jaccard": 0.7096774193548387,
+              "shared_experts": 44,
+              "union_experts": 62
+            },
+            {
+              "left": "zh_explanation",
+              "right": "code",
+              "used_expert_jaccard": 0.8166666666666667,
+              "shared_experts": 49,
+              "union_experts": 60
+            },
+            {
+              "left": "zh_explanation",
+              "right": "math",
+              "used_expert_jaccard": 0.7741935483870968,
+              "shared_experts": 48,
+              "union_experts": 62
+            },
+            {
+              "left": "en_architecture",
+              "right": "code",
+              "used_expert_jaccard": 0.6557377049180327,
+              "shared_experts": 40,
+              "union_experts": 61
+            },
+            {
+              "left": "en_architecture",
+              "right": "math",
+              "used_expert_jaccard": 0.6721311475409836,
+              "shared_experts": 41,
+              "union_experts": 61
+            },
+            {
+              "left": "code",
+              "right": "math",
+              "used_expert_jaccard": 0.7213114754098361,
+              "shared_experts": 44,
+              "union_experts": 61
+            }
+          ]
+        }
+      },
+      {
+        "layer": 3,
+        "ffn": "moe",
+        "state_tensors": 203,
+        "state_numel": 584847872,
+        "state_bytes": 1169695744,
+        "timing_ms": {
+          "load_cpu": 25.33966102055274,
+          "transfer_to_device": 121.24854500871152,
+          "forward": 3.839854005491361
+        },
+        "peak_device_allocated_mib": 1174.7490234375,
+        "hidden_before": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 0.22556711733341217,
+            "p10": 0.10850249230861664,
+            "p50": 0.1559184491634369,
+            "p90": 0.19672976434230804
+          },
+          "l2_norm": {
+            "mean": 10.208003044128418,
+            "p10": 4.910262107849121,
+            "p50": 7.056063175201416,
+            "p90": 8.902973175048828
+          },
+          "finite": true,
+          "sha256_fp32": "76c1ba59ea5ef19f16162513c7c36e6e8d17a9af9fa3fdedbf657e80882fa0a3"
+        },
+        "hidden_after": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 2.651595115661621,
+            "p10": 0.14781513810157776,
+            "p50": 0.1792253851890564,
+            "p90": 0.22313819825649261
+          },
+          "l2_norm": {
+            "mean": 119.99750518798828,
+            "p10": 6.68934965133667,
+            "p50": 8.110815048217773,
+            "p90": 10.098081588745117
+          },
+          "finite": true,
+          "sha256_fp32": "aa223453a9b51ecdb244412002e27c70edbcaeeb29a6e1e064f340622ff4044c"
+        },
+        "mla": {
+          "compressed_projection_shape": [
+            4,
+            27,
+            576
+          ],
+          "latent_shape": [
+            4,
+            27,
+            512
+          ],
+          "rope_key_shape": [
+            4,
+            27,
+            64
+          ],
+          "latent_rms": 4.113536357879639,
+          "rope_key_rms": 2.1909468173980713,
+          "eager_key_cache_shape": [
+            4,
+            16,
+            27,
+            192
+          ],
+          "eager_value_cache_shape": [
+            4,
+            16,
+            27,
+            128
+          ],
+          "eager_cache_bytes": 1105920
+        },
+        "routing": {
+          "aggregate_load": [
+            14,
+            5,
+            8,
+            9,
+            17,
+            15,
+            16,
+            6,
+            2,
+            10,
+            2,
+            9,
+            16,
+            3,
+            9,
+            3,
+            14,
+            1,
+            7,
+            9,
+            16,
+            5,
+            8,
+            0,
+            8,
+            9,
+            16,
+            8,
+            8,
+            12,
+            6,
+            4,
+            3,
+            16,
+            0,
+            8,
+            12,
+            15,
+            1,
+            16,
+            7,
+            5,
+            4,
+            12,
+            7,
+            5,
+            11,
+            7,
+            11,
+            12,
+            2,
+            6,
+            3,
+            6,
+            19,
+            9,
+            8,
+            3,
+            10,
+            6,
+            11,
+            12,
+            13,
+            5
+          ],
+          "aggregate_metrics": {
+            "routes": 540,
+            "used_experts": 62,
+            "zero_experts": 2,
+            "mean": 8.4375,
+            "std": 4.797072410583496,
+            "cv": 0.568541944026947,
+            "gini": 0.3218750059604645,
+            "entropy_nats": 3.979837417602539,
+            "effective_experts": 53.50833400703197,
+            "min": 0.0,
+            "p50": 8.0,
+            "p90": 16.0,
+            "max": 19.0
+          },
+          "top_experts": [
+            {
+              "expert": 54,
+              "routes": 19
+            },
+            {
+              "expert": 4,
+              "routes": 17
+            },
+            {
+              "expert": 6,
+              "routes": 16
+            },
+            {
+              "expert": 12,
+              "routes": 16
+            },
+            {
+              "expert": 20,
+              "routes": 16
+            },
+            {
+              "expert": 26,
+              "routes": 16
+            },
+            {
+              "expert": 33,
+              "routes": 16
+            },
+            {
+              "expert": 39,
+              "routes": 16
+            },
+            {
+              "expert": 5,
+              "routes": 15
+            },
+            {
+              "expert": 37,
+              "routes": 15
+            },
+            {
+              "expert": 0,
+              "routes": 14
+            },
+            {
+              "expert": 16,
+              "routes": 14
+            }
+          ],
+          "selected_weight_sum": {
+            "mean": 0.4898033142089844,
+            "p10": 0.37064844369888306,
+            "p50": 0.46657082438468933,
+            "p90": 0.5860897302627563
+          },
+          "top1_top2_margin": {
+            "mean": 0.10427366942167282,
+            "p50": 0.036837637424468994,
+            "p90": 0.19226056337356567
+          },
+          "per_prompt": [
+            {
+              "id": "zh_explanation",
+              "label": "中文解释",
+              "tokens": 26,
+              "load": [
+                6,
+                0,
+                4,
+                1,
+                5,
+                6,
+                9,
+                3,
+                1,
+                3,
+                2,
+                2,
+                5,
+                1,
+                2,
+                0,
+                5,
+                0,
+                2,
+                5,
+                5,
+                1,
+                3,
+                0,
+                2,
+                1,
+                5,
+                1,
+                3,
+                3,
+                3,
+                2,
+                2,
+                5,
+                0,
+                2,
+                2,
+                6,
+                0,
+                1,
+                1,
+                3,
+                2,
+                2,
+                4,
+                1,
+                2,
+                0,
+                1,
+                2,
+                0,
+                1,
+                3,
+                2,
+                3,
+                3,
+                4,
+                1,
+                2,
+                3,
+                2,
+                2,
+                2,
+                1
+              ],
+              "metrics": {
+                "routes": 156,
+                "used_experts": 56,
+                "zero_experts": 8,
+                "mean": 2.4375,
+                "std": 1.8505682945251465,
+                "cv": 0.7592074871063232,
+                "gini": 0.40444710850715637,
+                "entropy_nats": 3.8562657833099365,
+                "effective_experts": 47.28843598733856,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 5.0,
+                "max": 9.0
+              },
+              "top_experts": [
+                {
+                  "expert": 6,
+                  "routes": 9
+                },
+                {
+                  "expert": 0,
+                  "routes": 6
+                },
+                {
+                  "expert": 5,
+                  "routes": 6
+                },
+                {
+                  "expert": 37,
+                  "routes": 6
+                },
+                {
+                  "expert": 4,
+                  "routes": 5
+                },
+                {
+                  "expert": 12,
+                  "routes": 5
+                },
+                {
+                  "expert": 16,
+                  "routes": 5
+                },
+                {
+                  "expert": 19,
+                  "routes": 5
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.9711988568305969
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.0014021513052284718
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.001076161745004356
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.0008088500471785665
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.0007386832148768008
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.000706107763107866
+                    }
+                  ],
+                  "selected_weight_sum": 0.975930810905993,
+                  "top1_top2_margin": 0.9697967055253685
+                },
+                {
+                  "position": 1,
+                  "token_id": 1082,
+                  "token": "çĶ¨",
+                  "decoded_piece": "用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 6,
+                      "weight": 0.16575224697589874
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.05748995393514633
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.04968932643532753
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.03696934133768082
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.03696615993976593
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.03662595525383949
+                    }
+                  ],
+                  "selected_weight_sum": 0.38349298387765884,
+                  "top1_top2_margin": 0.10826229304075241
+                },
+                {
+                  "position": 2,
+                  "token_id": 83983,
+                  "token": "éĢļä¿Ĺ",
+                  "decoded_piece": "通俗",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.09529686719179153
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.07287721335887909
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.05606567859649658
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.05045276880264282
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.03255721554160118
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.030352246016263962
+                    }
+                  ],
+                  "selected_weight_sum": 0.33760198950767517,
+                  "top1_top2_margin": 0.022419653832912445
+                },
+                {
+                  "position": 3,
+                  "token_id": 91476,
+                  "token": "çļĦè¯Ńè¨Ģ",
+                  "decoded_piece": "的语言",
+                  "experts_by_weight": [
+                    {
+                      "expert": 37,
+                      "weight": 0.1348549872636795
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.1058119535446167
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.06803137063980103
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.056730255484580994
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.056706808507442474
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.036494504660367966
+                    }
+                  ],
+                  "selected_weight_sum": 0.45862988010048866,
+                  "top1_top2_margin": 0.029043033719062805
+                },
+                {
+                  "position": 4,
+                  "token_id": 17882,
+                  "token": "解éĩĬ",
+                  "decoded_piece": "解释",
+                  "experts_by_weight": [
+                    {
+                      "expert": 37,
+                      "weight": 0.23425081372261047
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.11718924343585968
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.053890570998191833
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.04609973728656769
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.03850342333316803
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.033879783004522324
+                    }
+                  ],
+                  "selected_weight_sum": 0.52381357178092,
+                  "top1_top2_margin": 0.1170615702867508
+                },
+                {
+                  "position": 5,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.19864459335803986
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.15572796761989594
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.0822681188583374
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.0573173351585865
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.056325022131204605
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.03425804525613785
+                    }
+                  ],
+                  "selected_weight_sum": 0.5845410823822021,
+                  "top1_top2_margin": 0.04291662573814392
+                },
+                {
+                  "position": 6,
+                  "token_id": 10285,
+                  "token": "为ä»Ģä¹Ī",
+                  "decoded_piece": "为什么",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.2489544302225113
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.06371285766363144
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.04518381506204605
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.03755857050418854
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.03596162050962448
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.0341949537396431
+                    }
+                  ],
+                  "selected_weight_sum": 0.4655662477016449,
+                  "top1_top2_margin": 0.18524157255887985
+                },
+                {
+                  "position": 7,
+                  "token_id": 21844,
+                  "token": "ç¨Ģ",
+                  "decoded_piece": "稀",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.13185125589370728
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.0922761932015419
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.07850739359855652
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.05424924194812775
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.041256386786699295
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.03371671214699745
+                    }
+                  ],
+                  "selected_weight_sum": 0.4318571835756302,
+                  "top1_top2_margin": 0.039575062692165375
+                },
+                {
+                  "position": 8,
+                  "token_id": 20923,
+                  "token": "çĸı",
+                  "decoded_piece": "疏",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.10015559196472168
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.08910446614027023
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.08510752767324448
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.06459127366542816
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.031923577189445496
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03015722520649433
+                    }
+                  ],
+                  "selected_weight_sum": 0.4010396618396044,
+                  "top1_top2_margin": 0.011051125824451447
+                },
+                {
+                  "position": 9,
+                  "token_id": 10900,
+                  "token": "ä¸ĵ家",
+                  "decoded_piece": "专家",
+                  "experts_by_weight": [
+                    {
+                      "expert": 60,
+                      "weight": 0.08528561145067215
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.0805189460515976
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.07657847553491592
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.06688879430294037
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.04128864407539368
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.03574351966381073
+                    }
+                  ],
+                  "selected_weight_sum": 0.38630399107933044,
+                  "top1_top2_margin": 0.0047666653990745544
+                },
+                {
+                  "position": 10,
+                  "token_id": 39322,
+                  "token": "模åŀĭ",
+                  "decoded_piece": "模型",
+                  "experts_by_weight": [
+                    {
+                      "expert": 20,
+                      "weight": 0.20124658942222595
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.14611957967281342
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.060534000396728516
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.05685422196984291
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.04330537095665932
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.0407714769244194
+                    }
+                  ],
+                  "selected_weight_sum": 0.5488312393426895,
+                  "top1_top2_margin": 0.05512700974941254
+                },
+                {
+                  "position": 11,
+                  "token_id": 1876,
+                  "token": "åı¯ä»¥",
+                  "decoded_piece": "可以",
+                  "experts_by_weight": [
+                    {
+                      "expert": 30,
+                      "weight": 0.10151458531618118
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.0651351809501648
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.05840770900249481
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.05432477220892906
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.05122113600373268
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.043339185416698456
+                    }
+                  ],
+                  "selected_weight_sum": 0.373942568898201,
+                  "top1_top2_margin": 0.03637940436601639
+                },
+                {
+                  "position": 12,
+                  "token_id": 9476,
+                  "token": "æĭ¥æľī",
+                  "decoded_piece": "拥有",
+                  "experts_by_weight": [
+                    {
+                      "expert": 6,
+                      "weight": 0.12864404916763306
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.07182051241397858
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.0615667849779129
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.05789991840720177
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.03908246010541916
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.034220051020383835
+                    }
+                  ],
+                  "selected_weight_sum": 0.3932337760925293,
+                  "top1_top2_margin": 0.05682353675365448
+                },
+                {
+                  "position": 13,
+                  "token_id": 3599,
+                  "token": "å¾Īå¤ļ",
+                  "decoded_piece": "很多",
+                  "experts_by_weight": [
+                    {
+                      "expert": 6,
+                      "weight": 0.12923511862754822
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.08376506716012955
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.0825040265917778
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05150343105196953
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03739800676703453
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.027514372020959854
+                    }
+                  ],
+                  "selected_weight_sum": 0.4119200222194195,
+                  "top1_top2_margin": 0.04547005146741867
+                },
+                {
+                  "position": 14,
+                  "token_id": 21273,
+                  "token": "åıĤæķ°",
+                  "decoded_piece": "参数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 20,
+                      "weight": 0.15276853740215302
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.11104021966457367
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.06659099459648132
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.04421878978610039
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.04282462224364281
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.042411528527736664
+                    }
+                  ],
+                  "selected_weight_sum": 0.45985469222068787,
+                  "top1_top2_margin": 0.041728317737579346
+                },
+                {
+                  "position": 15,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.10195695608854294
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.09532091021537781
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.09049556404352188
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.08353115618228912
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.06399564445018768
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.04496682062745094
+                    }
+                  ],
+                  "selected_weight_sum": 0.4802670516073704,
+                  "top1_top2_margin": 0.006636045873165131
+                },
+                {
+                  "position": 16,
+                  "token_id": 1705,
+                  "token": "ä½Ĩ",
+                  "decoded_piece": "但",
+                  "experts_by_weight": [
+                    {
+                      "expert": 44,
+                      "weight": 0.11888333410024643
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.11220570653676987
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.05584236606955528
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.03397982940077782
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.02580212987959385
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.025049369782209396
+                    }
+                  ],
+                  "selected_weight_sum": 0.37176273576915264,
+                  "top1_top2_margin": 0.0066776275634765625
+                },
+                {
+                  "position": 17,
+                  "token_id": 13712,
+                  "token": "æ¯ı个",
+                  "decoded_piece": "每个",
+                  "experts_by_weight": [
+                    {
+                      "expert": 6,
+                      "weight": 0.10931649059057236
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.07729648053646088
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.037208620458841324
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.03507556393742561
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.03295507654547691
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.030445924028754234
+                    }
+                  ],
+                  "selected_weight_sum": 0.3222981560975313,
+                  "top1_top2_margin": 0.03202001005411148
+                },
+                {
+                  "position": 18,
+                  "token_id": 10728,
+                  "token": "Ġtoken",
+                  "decoded_piece": " token",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.2605063021183014
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.10610145330429077
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.092587411403656
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.055407967418432236
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.05411171168088913
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.03317439556121826
+                    }
+                  ],
+                  "selected_weight_sum": 0.6018892414867878,
+                  "top1_top2_margin": 0.15440484881401062
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.5464491248130798
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.10585428774356842
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.04159308969974518
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.037740062922239304
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.030981898307800293
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.028572754934430122
+                    }
+                  ],
+                  "selected_weight_sum": 0.7911912184208632,
+                  "top1_top2_margin": 0.4405948370695114
+                },
+                {
+                  "position": 20,
+                  "token_id": 2141,
+                  "token": "åıª",
+                  "decoded_piece": "只",
+                  "experts_by_weight": [
+                    {
+                      "expert": 19,
+                      "weight": 0.10716968029737473
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.09130243957042694
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.06434521079063416
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05338294059038162
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.0471247173845768
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.04004571959376335
+                    }
+                  ],
+                  "selected_weight_sum": 0.4033707082271576,
+                  "top1_top2_margin": 0.015867240726947784
+                },
+                {
+                  "position": 21,
+                  "token_id": 5118,
+                  "token": "使çĶ¨",
+                  "decoded_piece": "使用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 6,
+                      "weight": 0.10279721766710281
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.07755965739488602
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.06039987877011299
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.04671044647693634
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.04322323203086853
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.03838641941547394
+                    }
+                  ],
+                  "selected_weight_sum": 0.36907685175538063,
+                  "top1_top2_margin": 0.025237560272216797
+                },
+                {
+                  "position": 22,
+                  "token_id": 6802,
+                  "token": "åħ¶ä¸Ń",
+                  "decoded_piece": "其中",
+                  "experts_by_weight": [
+                    {
+                      "expert": 11,
+                      "weight": 0.11038965731859207
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.09876789152622223
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.08621250838041306
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.07344488799571991
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.06631779670715332
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.06579618155956268
+                    }
+                  ],
+                  "selected_weight_sum": 0.5009289234876633,
+                  "top1_top2_margin": 0.011621765792369843
+                },
+                {
+                  "position": 23,
+                  "token_id": 51905,
+                  "token": "ä¸Ģå°ı",
+                  "decoded_piece": "一小",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.09538868069648743
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.0859629362821579
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.08134215325117111
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.07922398298978806
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.07351940125226974
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.0516715943813324
+                    }
+                  ],
+                  "selected_weight_sum": 0.46710874885320663,
+                  "top1_top2_margin": 0.009425744414329529
+                },
+                {
+                  "position": 24,
+                  "token_id": 5781,
+                  "token": "éĥ¨åĪĨ",
+                  "decoded_piece": "部分",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.17455773055553436
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.07393549382686615
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.07050646096467972
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.06601278483867645
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.05656343698501587
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.024452252313494682
+                    }
+                  ],
+                  "selected_weight_sum": 0.46602815948426723,
+                  "top1_top2_margin": 0.10062223672866821
+                },
+                {
+                  "position": 25,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.11224387586116791
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.11114215850830078
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.09781858325004578
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.08452824503183365
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.07711736857891083
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.041843514889478683
+                    }
+                  ],
+                  "selected_weight_sum": 0.5246937461197376,
+                  "top1_top2_margin": 0.0011017173528671265
+                }
+              ]
+            },
+            {
+              "id": "en_architecture",
+              "label": "English architecture",
+              "tokens": 18,
+              "load": [
+                3,
+                0,
+                2,
+                2,
+                4,
+                3,
+                1,
+                1,
+                0,
+                3,
+                0,
+                3,
+                3,
+                1,
+                2,
+                0,
+                0,
+                0,
+                1,
+                0,
+                6,
+                4,
+                1,
+                0,
+                3,
+                1,
+                0,
+                0,
+                1,
+                2,
+                0,
+                2,
+                0,
+                1,
+                0,
+                4,
+                6,
+                1,
+                0,
+                5,
+                5,
+                0,
+                1,
+                6,
+                0,
+                1,
+                1,
+                1,
+                2,
+                2,
+                0,
+                0,
+                0,
+                1,
+                5,
+                1,
+                3,
+                2,
+                2,
+                1,
+                1,
+                4,
+                0,
+                3
+              ],
+              "metrics": {
+                "routes": 108,
+                "used_experts": 44,
+                "zero_experts": 20,
+                "mean": 1.6875,
+                "std": 1.735484004020691,
+                "cv": 1.0284349918365479,
+                "gini": 0.5465856194496155,
+                "entropy_nats": 3.5949339866638184,
+                "effective_experts": 36.41329614537717,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 4.0,
+                "max": 6.0
+              },
+              "top_experts": [
+                {
+                  "expert": 20,
+                  "routes": 6
+                },
+                {
+                  "expert": 36,
+                  "routes": 6
+                },
+                {
+                  "expert": 43,
+                  "routes": 6
+                },
+                {
+                  "expert": 39,
+                  "routes": 5
+                },
+                {
+                  "expert": 40,
+                  "routes": 5
+                },
+                {
+                  "expert": 54,
+                  "routes": 5
+                },
+                {
+                  "expert": 4,
+                  "routes": 4
+                },
+                {
+                  "expert": 21,
+                  "routes": 4
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.9711988568305969
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.0014021513052284718
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.001076161745004356
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.0008088500471785665
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.0007386832148768008
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.000706107763107866
+                    }
+                  ],
+                  "selected_weight_sum": 0.975930810905993,
+                  "top1_top2_margin": 0.9697967055253685
+                },
+                {
+                  "position": 1,
+                  "token_id": 99297,
+                  "token": "Explain",
+                  "decoded_piece": "Explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.23815517127513885
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.09801939874887466
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.067540742456913
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.05258971080183983
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.049972981214523315
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.03837980329990387
+                    }
+                  ],
+                  "selected_weight_sum": 0.5446578077971935,
+                  "top1_top2_margin": 0.1401357725262642
+                },
+                {
+                  "position": 2,
+                  "token_id": 946,
+                  "token": "Ġhow",
+                  "decoded_piece": " how",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.16547216475009918
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.10329186171293259
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.10013425350189209
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.07191096246242523
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.04293400049209595
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.030487103387713432
+                    }
+                  ],
+                  "selected_weight_sum": 0.5142303463071585,
+                  "top1_top2_margin": 0.062180303037166595
+                },
+                {
+                  "position": 3,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.1549440622329712
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.1339392364025116
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.08629074692726135
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03606937825679779
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.030499035492539406
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.026507094502449036
+                    }
+                  ],
+                  "selected_weight_sum": 0.4682495538145304,
+                  "top1_top2_margin": 0.021004825830459595
+                },
+                {
+                  "position": 4,
+                  "token_id": 29640,
+                  "token": "Ġcompressed",
+                  "decoded_piece": " compressed",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.09938042610883713
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.07479962706565857
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.07033173739910126
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.05891317501664162
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.04771549254655838
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.024970492348074913
+                    }
+                  ],
+                  "selected_weight_sum": 0.37611095048487186,
+                  "top1_top2_margin": 0.02458079904317856
+                },
+                {
+                  "position": 5,
+                  "token_id": 2131,
+                  "token": "Ġkey",
+                  "decoded_piece": " key",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.2045895755290985
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.1725901961326599
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.09196025133132935
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.08283847570419312
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.052055127918720245
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.023450862616300583
+                    }
+                  ],
+                  "selected_weight_sum": 0.6274844892323017,
+                  "top1_top2_margin": 0.0319993793964386
+                },
+                {
+                  "position": 6,
+                  "token_id": 12,
+                  "token": "-",
+                  "decoded_piece": "-",
+                  "experts_by_weight": [
+                    {
+                      "expert": 18,
+                      "weight": 0.21182844042778015
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.08291414380073547
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.05877301096916199
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.058278877288103104
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.05514194443821907
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03826766088604927
+                    }
+                  ],
+                  "selected_weight_sum": 0.5052040778100491,
+                  "top1_top2_margin": 0.12891429662704468
+                },
+                {
+                  "position": 7,
+                  "token_id": 2834,
+                  "token": "value",
+                  "decoded_piece": "value",
+                  "experts_by_weight": [
+                    {
+                      "expert": 20,
+                      "weight": 0.15702477097511292
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.12268862873315811
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.0899953618645668
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.0687999576330185
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.057560667395591736
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.03539532423019409
+                    }
+                  ],
+                  "selected_weight_sum": 0.5314647108316422,
+                  "top1_top2_margin": 0.0343361422419548
+                },
+                {
+                  "position": 8,
+                  "token_id": 28937,
+                  "token": "Ġlatent",
+                  "decoded_piece": " latent",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.0926990807056427
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.06688005477190018
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.06084948405623436
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.05863860249519348
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.056106846779584885
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.050941843539476395
+                    }
+                  ],
+                  "selected_weight_sum": 0.386115912348032,
+                  "top1_top2_margin": 0.025819025933742523
+                },
+                {
+                  "position": 9,
+                  "token_id": 4183,
+                  "token": "Ġchanges",
+                  "decoded_piece": " changes",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.1537322998046875
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.1268528699874878
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.08116747438907623
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.0690811425447464
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.049206629395484924
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.04574333503842354
+                    }
+                  ],
+                  "selected_weight_sum": 0.5257837511599064,
+                  "top1_top2_margin": 0.026879429817199707
+                },
+                {
+                  "position": 10,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.12472306936979294
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.10463881492614746
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.06110163778066635
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.04021468013525009
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.039409756660461426
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.03180879354476929
+                    }
+                  ],
+                  "selected_weight_sum": 0.40189675241708755,
+                  "top1_top2_margin": 0.020084254443645477
+                },
+                {
+                  "position": 11,
+                  "token_id": 4887,
+                  "token": "Ġmemory",
+                  "decoded_piece": " memory",
+                  "experts_by_weight": [
+                    {
+                      "expert": 43,
+                      "weight": 0.10257595032453537
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.09988170117139816
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.04727212339639664
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.0421602837741375
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.04022814705967903
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.03870483115315437
+                    }
+                  ],
+                  "selected_weight_sum": 0.37082303687930107,
+                  "top1_top2_margin": 0.002694249153137207
+                },
+                {
+                  "position": 12,
+                  "token_id": 2295,
+                  "token": "Ġcost",
+                  "decoded_piece": " cost",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.19642455875873566
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.16629761457443237
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.04809309169650078
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.04783359542489052
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.0408785454928875
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.034761298447847366
+                    }
+                  ],
+                  "selected_weight_sum": 0.5342887043952942,
+                  "top1_top2_margin": 0.030126944184303284
+                },
+                {
+                  "position": 13,
+                  "token_id": 280,
+                  "token": "Ġof",
+                  "decoded_piece": " of",
+                  "experts_by_weight": [
+                    {
+                      "expert": 56,
+                      "weight": 0.1839616447687149
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.04788229241967201
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.033062588423490524
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.03048531897366047
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.027693433687090874
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.026338256895542145
+                    }
+                  ],
+                  "selected_weight_sum": 0.34942353516817093,
+                  "top1_top2_margin": 0.1360793523490429
+                },
+                {
+                  "position": 14,
+                  "token_id": 77939,
+                  "token": "Ġautore",
+                  "decoded_piece": " autore",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.05693405494093895
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.05210784822702408
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.04772772267460823
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.036565009504556656
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.03430648520588875
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03330330550670624
+                    }
+                  ],
+                  "selected_weight_sum": 0.2609444260597229,
+                  "top1_top2_margin": 0.004826206713914871
+                },
+                {
+                  "position": 15,
+                  "token_id": 79184,
+                  "token": "gressive",
+                  "decoded_piece": "gressive",
+                  "experts_by_weight": [
+                    {
+                      "expert": 11,
+                      "weight": 0.23832271993160248
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.09413538128137589
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.03894481435418129
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.03155319020152092
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.03087766282260418
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.02754705585539341
+                    }
+                  ],
+                  "selected_weight_sum": 0.46138082444667816,
+                  "top1_top2_margin": 0.1441873386502266
+                },
+                {
+                  "position": 16,
+                  "token_id": 39671,
+                  "token": "Ġdecoding",
+                  "decoded_piece": " decoding",
+                  "experts_by_weight": [
+                    {
+                      "expert": 61,
+                      "weight": 0.10946545749902725
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.07026691734790802
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.06993477791547775
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.0477374792098999
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.04432488977909088
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.04071218520402908
+                    }
+                  ],
+                  "selected_weight_sum": 0.3824417069554329,
+                  "top1_top2_margin": 0.03919854015111923
+                },
+                {
+                  "position": 17,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.12724851071834564
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.11848980188369751
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.09815538674592972
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.06956201046705246
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.05623827129602432
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.04838328808546066
+                    }
+                  ],
+                  "selected_weight_sum": 0.5180772691965103,
+                  "top1_top2_margin": 0.008758708834648132
+                }
+              ]
+            },
+            {
+              "id": "code",
+              "label": "Python code",
+              "tokens": 19,
+              "load": [
+                4,
+                1,
+                1,
+                1,
+                4,
+                3,
+                3,
+                0,
+                1,
+                1,
+                0,
+                2,
+                6,
+                0,
+                1,
+                0,
+                3,
+                1,
+                0,
+                2,
+                4,
+                0,
+                1,
+                0,
+                1,
+                3,
+                3,
+                1,
+                3,
+                5,
+                1,
+                0,
+                1,
+                5,
+                0,
+                2,
+                3,
+                7,
+                0,
+                3,
+                0,
+                0,
+                0,
+                4,
+                0,
+                1,
+                0,
+                2,
+                2,
+                1,
+                2,
+                1,
+                0,
+                2,
+                8,
+                2,
+                0,
+                0,
+                3,
+                2,
+                2,
+                4,
+                0,
+                1
+              ],
+              "metrics": {
+                "routes": 114,
+                "used_experts": 45,
+                "zero_experts": 19,
+                "mean": 1.78125,
+                "std": 1.8384668827056885,
+                "cv": 1.0321217775344849,
+                "gini": 0.5386512875556946,
+                "entropy_nats": 3.610292911529541,
+                "effective_experts": 36.976882183839685,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 4.0,
+                "max": 8.0
+              },
+              "top_experts": [
+                {
+                  "expert": 54,
+                  "routes": 8
+                },
+                {
+                  "expert": 37,
+                  "routes": 7
+                },
+                {
+                  "expert": 12,
+                  "routes": 6
+                },
+                {
+                  "expert": 29,
+                  "routes": 5
+                },
+                {
+                  "expert": 33,
+                  "routes": 5
+                },
+                {
+                  "expert": 0,
+                  "routes": 4
+                },
+                {
+                  "expert": 4,
+                  "routes": 4
+                },
+                {
+                  "expert": 20,
+                  "routes": 4
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.9711988568305969
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.0014021513052284718
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.001076161745004356
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.0008088500471785665
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.0007386832148768008
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.000706107763107866
+                    }
+                  ],
+                  "selected_weight_sum": 0.975930810905993,
+                  "top1_top2_margin": 0.9697967055253685
+                },
+                {
+                  "position": 1,
+                  "token_id": 9081,
+                  "token": "Write",
+                  "decoded_piece": "Write",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.15111351013183594
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.11959145218133926
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.08922455459833145
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.05587117373943329
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.038532257080078125
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.034965526312589645
+                    }
+                  ],
+                  "selected_weight_sum": 0.4892984740436077,
+                  "top1_top2_margin": 0.031522057950496674
+                },
+                {
+                  "position": 2,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 58,
+                      "weight": 0.2080550640821457
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.14192862808704376
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.08557718247175217
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.07180862873792648
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.032067738473415375
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.030239181593060493
+                    }
+                  ],
+                  "selected_weight_sum": 0.569676423445344,
+                  "top1_top2_margin": 0.06612643599510193
+                },
+                {
+                  "position": 3,
+                  "token_id": 12974,
+                  "token": "ĠPython",
+                  "decoded_piece": " Python",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.1371878981590271
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.12900573015213013
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.09749352186918259
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.049715057015419006
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.046508997678756714
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.041462209075689316
+                    }
+                  ],
+                  "selected_weight_sum": 0.5013734139502048,
+                  "top1_top2_margin": 0.008182168006896973
+                },
+                {
+                  "position": 4,
+                  "token_id": 1157,
+                  "token": "Ġfunction",
+                  "decoded_piece": " function",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.20584429800510406
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.14856287837028503
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.0871359184384346
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.08152839541435242
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.051778245717287064
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.02517823502421379
+                    }
+                  ],
+                  "selected_weight_sum": 0.600027970969677,
+                  "top1_top2_margin": 0.05728141963481903
+                },
+                {
+                  "position": 5,
+                  "token_id": 344,
+                  "token": "Ġthat",
+                  "decoded_piece": " that",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.18090112507343292
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.11123941093683243
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.07971292734146118
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.02637108787894249
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.025111805647611618
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.02362849749624729
+                    }
+                  ],
+                  "selected_weight_sum": 0.44696485437452793,
+                  "top1_top2_margin": 0.0696617141366005
+                },
+                {
+                  "position": 6,
+                  "token_id": 7578,
+                  "token": "Ġreturns",
+                  "decoded_piece": " returns",
+                  "experts_by_weight": [
+                    {
+                      "expert": 19,
+                      "weight": 0.1616554856300354
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.11549834907054901
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.07323762774467468
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.05683310702443123
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.043762173503637314
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.03263093903660774
+                    }
+                  ],
+                  "selected_weight_sum": 0.4836176820099354,
+                  "top1_top2_margin": 0.04615713655948639
+                },
+                {
+                  "position": 7,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.22255076467990875
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.07750241458415985
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.05901200696825981
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.04646346718072891
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.03500976786017418
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.025494523346424103
+                    }
+                  ],
+                  "selected_weight_sum": 0.4660329446196556,
+                  "top1_top2_margin": 0.1450483500957489
+                },
+                {
+                  "position": 8,
+                  "token_id": 1022,
+                  "token": "Ġfirst",
+                  "decoded_piece": " first",
+                  "experts_by_weight": [
+                    {
+                      "expert": 33,
+                      "weight": 0.23225565254688263
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.10645926743745804
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.05182158946990967
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04605036601424217
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.03481481224298477
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.027712024748325348
+                    }
+                  ],
+                  "selected_weight_sum": 0.4991137124598026,
+                  "top1_top2_margin": 0.1257963851094246
+                },
+                {
+                  "position": 9,
+                  "token_id": 9842,
+                  "token": "Ġrepeated",
+                  "decoded_piece": " repeated",
+                  "experts_by_weight": [
+                    {
+                      "expert": 33,
+                      "weight": 0.18555673956871033
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.0932159349322319
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.0786258727312088
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.049028702080249786
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.030850498005747795
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.02684546820819378
+                    }
+                  ],
+                  "selected_weight_sum": 0.4641232155263424,
+                  "top1_top2_margin": 0.09234080463647842
+                },
+                {
+                  "position": 10,
+                  "token_id": 3699,
+                  "token": "Ġelement",
+                  "decoded_piece": " element",
+                  "experts_by_weight": [
+                    {
+                      "expert": 20,
+                      "weight": 0.13938002288341522
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.08577956259250641
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.07479023188352585
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.059141431003808975
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.05091153085231781
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.04373912513256073
+                    }
+                  ],
+                  "selected_weight_sum": 0.453741904348135,
+                  "top1_top2_margin": 0.053600460290908813
+                },
+                {
+                  "position": 11,
+                  "token_id": 279,
+                  "token": "Ġin",
+                  "decoded_piece": " in",
+                  "experts_by_weight": [
+                    {
+                      "expert": 28,
+                      "weight": 0.14614824950695038
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.11818400770425797
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.09291267395019531
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.05692411959171295
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05534731596708298
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.03634467348456383
+                    }
+                  ],
+                  "selected_weight_sum": 0.5058610402047634,
+                  "top1_top2_margin": 0.027964241802692413
+                },
+                {
+                  "position": 12,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 28,
+                      "weight": 0.09758827090263367
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.091375432908535
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.07514315843582153
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.053120844066143036
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04297527298331261
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.03407928720116615
+                    }
+                  ],
+                  "selected_weight_sum": 0.394282266497612,
+                  "top1_top2_margin": 0.006212837994098663
+                },
+                {
+                  "position": 13,
+                  "token_id": 1525,
+                  "token": "Ġlist",
+                  "decoded_piece": " list",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.15618914365768433
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.058529019355773926
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.05490716174244881
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.049499817192554474
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.039970483630895615
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.03016483783721924
+                    }
+                  ],
+                  "selected_weight_sum": 0.3892604634165764,
+                  "top1_top2_margin": 0.0976601243019104
+                },
+                {
+                  "position": 14,
+                  "token_id": 285,
+                  "token": "Ġand",
+                  "decoded_piece": " and",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.140435129404068
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.11578470468521118
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.08345910906791687
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.038944266736507416
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.037622421979904175
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.030940167605876923
+                    }
+                  ],
+                  "selected_weight_sum": 0.44718579947948456,
+                  "top1_top2_margin": 0.02465042471885681
+                },
+                {
+                  "position": 15,
+                  "token_id": 6716,
+                  "token": "Ġexplain",
+                  "decoded_piece": " explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 37,
+                      "weight": 0.12016983330249786
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.11260108649730682
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.07255109399557114
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.0641496405005455
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.0602947473526001
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.02975073829293251
+                    }
+                  ],
+                  "selected_weight_sum": 0.45951713994145393,
+                  "top1_top2_margin": 0.00756874680519104
+                },
+                {
+                  "position": 16,
+                  "token_id": 895,
+                  "token": "Ġits",
+                  "decoded_piece": " its",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.18681631982326508
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.07873330265283585
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.06885220855474472
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.0670696347951889
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.04396243393421173
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.029901383444666862
+                    }
+                  ],
+                  "selected_weight_sum": 0.47533528320491314,
+                  "top1_top2_margin": 0.10808301717042923
+                },
+                {
+                  "position": 17,
+                  "token_id": 13913,
+                  "token": "Ġcomplexity",
+                  "decoded_piece": " complexity",
+                  "experts_by_weight": [
+                    {
+                      "expert": 20,
+                      "weight": 0.07373698800802231
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.06951963156461716
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.05116838589310646
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.041290223598480225
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.03023359179496765
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.027252253144979477
+                    }
+                  ],
+                  "selected_weight_sum": 0.2932010740041733,
+                  "top1_top2_margin": 0.004217356443405151
+                },
+                {
+                  "position": 18,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.11643100529909134
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.11100540310144424
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.10169129818677902
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.058091938495635986
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.03980956971645355
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03621777892112732
+                    }
+                  ],
+                  "selected_weight_sum": 0.46324699372053146,
+                  "top1_top2_margin": 0.005425602197647095
+                }
+              ]
+            },
+            {
+              "id": "math",
+              "label": "数学推理",
+              "tokens": 27,
+              "load": [
+                1,
+                4,
+                1,
+                5,
+                4,
+                3,
+                3,
+                2,
+                0,
+                3,
+                0,
+                2,
+                2,
+                1,
+                4,
+                3,
+                6,
+                0,
+                4,
+                2,
+                1,
+                0,
+                3,
+                0,
+                2,
+                4,
+                8,
+                6,
+                1,
+                2,
+                2,
+                0,
+                0,
+                5,
+                0,
+                0,
+                1,
+                1,
+                1,
+                7,
+                1,
+                2,
+                1,
+                0,
+                3,
+                2,
+                8,
+                4,
+                6,
+                7,
+                0,
+                4,
+                0,
+                1,
+                3,
+                3,
+                1,
+                0,
+                3,
+                0,
+                6,
+                2,
+                11,
+                0
+              ],
+              "metrics": {
+                "routes": 162,
+                "used_experts": 49,
+                "zero_experts": 15,
+                "mean": 2.53125,
+                "std": 2.442797899246216,
+                "cv": 0.9650559425354004,
+                "gini": 0.5117669701576233,
+                "entropy_nats": 3.6781580448150635,
+                "effective_experts": 39.573434406292506,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 6.0,
+                "max": 11.0
+              },
+              "top_experts": [
+                {
+                  "expert": 62,
+                  "routes": 11
+                },
+                {
+                  "expert": 26,
+                  "routes": 8
+                },
+                {
+                  "expert": 46,
+                  "routes": 8
+                },
+                {
+                  "expert": 39,
+                  "routes": 7
+                },
+                {
+                  "expert": 49,
+                  "routes": 7
+                },
+                {
+                  "expert": 16,
+                  "routes": 6
+                },
+                {
+                  "expert": 27,
+                  "routes": 6
+                },
+                {
+                  "expert": 48,
+                  "routes": 6
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.9711988568305969
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.0014021513052284718
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.001076161745004356
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.0008088500471785665
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.0007386832148768008
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.000706107763107866
+                    }
+                  ],
+                  "selected_weight_sum": 0.975930810905993,
+                  "top1_top2_margin": 0.9697967055253685
+                },
+                {
+                  "position": 1,
+                  "token_id": 8246,
+                  "token": "èĭ¥",
+                  "decoded_piece": "若",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.14951738715171814
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.0980856642127037
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.08160083740949631
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.07467349618673325
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.06317417323589325
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.052538976073265076
+                    }
+                  ],
+                  "selected_weight_sum": 0.5195905342698097,
+                  "top1_top2_margin": 0.051431722939014435
+                },
+                {
+                  "position": 2,
+                  "token_id": 2206,
+                  "token": "æŃ£",
+                  "decoded_piece": "正",
+                  "experts_by_weight": [
+                    {
+                      "expert": 2,
+                      "weight": 0.10717478394508362
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.10145746916532516
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.0593704953789711
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.055084649473428726
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.05256998538970947
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.05031483620405197
+                    }
+                  ],
+                  "selected_weight_sum": 0.42597221955657005,
+                  "top1_top2_margin": 0.005717314779758453
+                },
+                {
+                  "position": 3,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.16861963272094727
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.11267384886741638
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.09004956483840942
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.07321920245885849
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.06758833676576614
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.06270019710063934
+                    }
+                  ],
+                  "selected_weight_sum": 0.574850782752037,
+                  "top1_top2_margin": 0.055945783853530884
+                },
+                {
+                  "position": 4,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.2162427455186844
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.13654249906539917
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.06265389919281006
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.054540734738111496
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.05205049738287926
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.045574069023132324
+                    }
+                  ],
+                  "selected_weight_sum": 0.5676044449210167,
+                  "top1_top2_margin": 0.07970024645328522
+                },
+                {
+                  "position": 5,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.486960232257843
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.12458961457014084
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.06354541331529617
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.04251669719815254
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.037103284150362015
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.03097430430352688
+                    }
+                  ],
+                  "selected_weight_sum": 0.7856895457953215,
+                  "top1_top2_margin": 0.3623706176877022
+                },
+                {
+                  "position": 6,
+                  "token_id": 13211,
+                  "token": "满足",
+                  "decoded_piece": "满足",
+                  "experts_by_weight": [
+                    {
+                      "expert": 6,
+                      "weight": 0.16329914331436157
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.1260579377412796
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.06854663044214249
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.06128653883934021
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.03816733509302139
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.03762584179639816
+                    }
+                  ],
+                  "selected_weight_sum": 0.4949834272265434,
+                  "top1_top2_margin": 0.03724120557308197
+                },
+                {
+                  "position": 7,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.1666949838399887
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.13348163664340973
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.08109060674905777
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.06654893606901169
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03289414197206497
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.030425915494561195
+                    }
+                  ],
+                  "selected_weight_sum": 0.5111362207680941,
+                  "top1_top2_margin": 0.03321334719657898
+                },
+                {
+                  "position": 8,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.30760839581489563
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05372292920947075
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.05369928479194641
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04193925857543945
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.041528962552547455
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.04065709561109543
+                    }
+                  ],
+                  "selected_weight_sum": 0.5391559265553951,
+                  "top1_top2_margin": 0.2538854666054249
+                },
+                {
+                  "position": 9,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.15855351090431213
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.1316342055797577
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.05368504300713539
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.04912524297833443
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.04827090725302696
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.036717962473630905
+                    }
+                  ],
+                  "selected_weight_sum": 0.4779868721961975,
+                  "top1_top2_margin": 0.026919305324554443
+                },
+                {
+                  "position": 10,
+                  "token_id": 12985,
+                  "token": "å®ĥçļĦ",
+                  "decoded_piece": "它的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.116221122443676
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.10054290294647217
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.07659035921096802
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.06001657620072365
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.05728625878691673
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.049379147589206696
+                    }
+                  ],
+                  "selected_weight_sum": 0.46003636717796326,
+                  "top1_top2_margin": 0.015678219497203827
+                },
+                {
+                  "position": 11,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.12973536550998688
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.07649380713701248
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.04296094924211502
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.041707731783390045
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.03551975265145302
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.03439413383603096
+                    }
+                  ],
+                  "selected_weight_sum": 0.3608117401599884,
+                  "top1_top2_margin": 0.053241558372974396
+                },
+                {
+                  "position": 12,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.1611926555633545
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.1397518813610077
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.09240048378705978
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.044278018176555634
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.04391821101307869
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03572217747569084
+                    }
+                  ],
+                  "selected_weight_sum": 0.5172634273767471,
+                  "top1_top2_margin": 0.021440774202346802
+                },
+                {
+                  "position": 13,
+                  "token_id": 37360,
+                  "token": "çŃīäºİ",
+                  "decoded_piece": "等于",
+                  "experts_by_weight": [
+                    {
+                      "expert": 19,
+                      "weight": 0.1503891944885254
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.10245323926210403
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.060541197657585144
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.05172496289014816
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.035423070192337036
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.02933596819639206
+                    }
+                  ],
+                  "selected_weight_sum": 0.4298676326870918,
+                  "top1_top2_margin": 0.047935955226421356
+                },
+                {
+                  "position": 14,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.24005234241485596
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.08549993485212326
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.07248266786336899
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05309956893324852
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.04152917489409447
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.03998137637972832
+                    }
+                  ],
+                  "selected_weight_sum": 0.5326450653374195,
+                  "top1_top2_margin": 0.1545524075627327
+                },
+                {
+                  "position": 15,
+                  "token_id": 18,
+                  "token": "3",
+                  "decoded_piece": "3",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.18014958500862122
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.1050652265548706
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.062478359788656235
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.04287721961736679
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.03764505684375763
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.024584602564573288
+                    }
+                  ],
+                  "selected_weight_sum": 0.45280005037784576,
+                  "top1_top2_margin": 0.07508435845375061
+                },
+                {
+                  "position": 16,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 1,
+                      "weight": 0.10770390182733536
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.10210539400577545
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.08225701749324799
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.07177606225013733
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.068714439868927
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.060516949743032455
+                    }
+                  ],
+                  "selected_weight_sum": 0.4930737651884556,
+                  "top1_top2_margin": 0.005598507821559906
+                },
+                {
+                  "position": 17,
+                  "token_id": 2989,
+                  "token": "æ±Ĥ",
+                  "decoded_piece": "求",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.12940554320812225
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.1288883239030838
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.06063942238688469
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.05899067223072052
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.03491080552339554
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.033715102821588516
+                    }
+                  ],
+                  "selected_weight_sum": 0.4465498700737953,
+                  "top1_top2_margin": 0.0005172193050384521
+                },
+                {
+                  "position": 18,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.13900531828403473
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.12066206336021423
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.07627727091312408
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.07498259097337723
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.037425458431243896
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03221821039915085
+                    }
+                  ],
+                  "selected_weight_sum": 0.480570912361145,
+                  "top1_top2_margin": 0.018343254923820496
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.29581424593925476
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.06302740424871445
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.044536203145980835
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03947015479207039
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.038034409284591675
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.031326256692409515
+                    }
+                  ],
+                  "selected_weight_sum": 0.5122086741030216,
+                  "top1_top2_margin": 0.2327868416905403
+                },
+                {
+                  "position": 20,
+                  "token_id": 337,
+                  "token": "çļĦ",
+                  "decoded_piece": "的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.10390287637710571
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.07279452681541443
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.06581421941518784
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.05750293657183647
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.035950709134340286
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.031022725626826286
+                    }
+                  ],
+                  "selected_weight_sum": 0.366987993940711,
+                  "top1_top2_margin": 0.031108349561691284
+                },
+                {
+                  "position": 21,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.25225111842155457
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05924615636467934
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.046080317348241806
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.03298409655690193
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.029271787032485008
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.028183098882436752
+                    }
+                  ],
+                  "selected_weight_sum": 0.4480165746062994,
+                  "top1_top2_margin": 0.19300496205687523
+                },
+                {
+                  "position": 22,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.14583228528499603
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.10939821600914001
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.05952773243188858
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.04561968520283699
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.034100957214832306
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.030454030260443687
+                    }
+                  ],
+                  "selected_weight_sum": 0.4249329064041376,
+                  "top1_top2_margin": 0.03643406927585602
+                },
+                {
+                  "position": 23,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.09405837953090668
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.060965847223997116
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.047729793936014175
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.04451792314648628
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04123474285006523
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.03994479775428772
+                    }
+                  ],
+                  "selected_weight_sum": 0.3284514844417572,
+                  "top1_top2_margin": 0.03309253230690956
+                },
+                {
+                  "position": 24,
+                  "token_id": 51971,
+                  "token": "æķ°çļĦ",
+                  "decoded_piece": "数的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.12274246662855148
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.11193068325519562
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.11071715503931046
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.05617574602365494
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.04331094026565552
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.037420324981212616
+                    }
+                  ],
+                  "selected_weight_sum": 0.4822973161935806,
+                  "top1_top2_margin": 0.010811783373355865
+                },
+                {
+                  "position": 25,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.24886038899421692
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.056682538241147995
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.04547945782542229
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.03942480683326721
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.027925729751586914
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.025453777983784676
+                    }
+                  ],
+                  "selected_weight_sum": 0.443826699629426,
+                  "top1_top2_margin": 0.19217785075306892
+                },
+                {
+                  "position": 26,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 45,
+                      "weight": 0.1179732158780098
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.1001434326171875
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.08874468505382538
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.06503507494926453
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.056857381016016006
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.052832894027233124
+                    }
+                  ],
+                  "selected_weight_sum": 0.48158668354153633,
+                  "top1_top2_margin": 0.017829783260822296
+                }
+              ]
+            }
+          ],
+          "prompt_pair_jaccard": [
+            {
+              "left": "zh_explanation",
+              "right": "en_architecture",
+              "used_expert_jaccard": 0.7543859649122807,
+              "shared_experts": 43,
+              "union_experts": 57
+            },
+            {
+              "left": "zh_explanation",
+              "right": "code",
+              "used_expert_jaccard": 0.6833333333333333,
+              "shared_experts": 41,
+              "union_experts": 60
+            },
+            {
+              "left": "zh_explanation",
+              "right": "math",
+              "used_expert_jaccard": 0.75,
+              "shared_experts": 45,
+              "union_experts": 60
+            },
+            {
+              "left": "en_architecture",
+              "right": "code",
+              "used_expert_jaccard": 0.6181818181818182,
+              "shared_experts": 34,
+              "union_experts": 55
+            },
+            {
+              "left": "en_architecture",
+              "right": "math",
+              "used_expert_jaccard": 0.6607142857142857,
+              "shared_experts": 37,
+              "union_experts": 56
+            },
+            {
+              "left": "code",
+              "right": "math",
+              "used_expert_jaccard": 0.6491228070175439,
+              "shared_experts": 37,
+              "union_experts": 57
+            }
+          ]
+        }
+      },
+      {
+        "layer": 4,
+        "ffn": "moe",
+        "state_tensors": 203,
+        "state_numel": 584847872,
+        "state_bytes": 1169695744,
+        "timing_ms": {
+          "load_cpu": 24.96491398778744,
+          "transfer_to_device": 123.27372501022182,
+          "forward": 3.8379190082196146
+        },
+        "peak_device_allocated_mib": 1176.2255859375,
+        "hidden_before": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 2.651595115661621,
+            "p10": 0.14781513810157776,
+            "p50": 0.1792253851890564,
+            "p90": 0.22313819825649261
+          },
+          "l2_norm": {
+            "mean": 119.99750518798828,
+            "p10": 6.68934965133667,
+            "p50": 8.110815048217773,
+            "p90": 10.098081588745117
+          },
+          "finite": true,
+          "sha256_fp32": "aa223453a9b51ecdb244412002e27c70edbcaeeb29a6e1e064f340622ff4044c"
+        },
+        "hidden_after": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 3.085932731628418,
+            "p10": 0.19619415700435638,
+            "p50": 0.23260724544525146,
+            "p90": 0.2969282567501068
+          },
+          "l2_norm": {
+            "mean": 139.65338134765625,
+            "p10": 8.878734588623047,
+            "p50": 10.526601791381836,
+            "p90": 13.43743896484375
+          },
+          "finite": true,
+          "sha256_fp32": "5d5d80c17f47a78f419f6ec9bf16668392a8246393ebfe2b0233841be4875b6e"
+        },
+        "mla": {
+          "compressed_projection_shape": [
+            4,
+            27,
+            576
+          ],
+          "latent_shape": [
+            4,
+            27,
+            512
+          ],
+          "rope_key_shape": [
+            4,
+            27,
+            64
+          ],
+          "latent_rms": 2.3880224227905273,
+          "rope_key_rms": 2.2479918003082275,
+          "eager_key_cache_shape": [
+            4,
+            16,
+            27,
+            192
+          ],
+          "eager_value_cache_shape": [
+            4,
+            16,
+            27,
+            128
+          ],
+          "eager_cache_bytes": 1105920
+        },
+        "routing": {
+          "aggregate_load": [
+            1,
+            7,
+            3,
+            8,
+            9,
+            3,
+            4,
+            20,
+            9,
+            18,
+            5,
+            11,
+            4,
+            0,
+            5,
+            2,
+            5,
+            12,
+            3,
+            4,
+            11,
+            7,
+            10,
+            12,
+            16,
+            5,
+            6,
+            6,
+            4,
+            37,
+            10,
+            17,
+            3,
+            5,
+            9,
+            11,
+            10,
+            6,
+            7,
+            9,
+            7,
+            13,
+            7,
+            1,
+            1,
+            3,
+            4,
+            18,
+            40,
+            8,
+            10,
+            1,
+            9,
+            1,
+            17,
+            15,
+            6,
+            11,
+            3,
+            3,
+            0,
+            13,
+            11,
+            4
+          ],
+          "aggregate_metrics": {
+            "routes": 540,
+            "used_experts": 62,
+            "zero_experts": 2,
+            "mean": 8.4375,
+            "std": 7.309213638305664,
+            "cv": 0.8662771582603455,
+            "gini": 0.4173611104488373,
+            "entropy_nats": 3.8508799076080322,
+            "effective_experts": 47.03443098267191,
+            "min": 0.0,
+            "p50": 7.0,
+            "p90": 16.699996948242188,
+            "max": 40.0
+          },
+          "top_experts": [
+            {
+              "expert": 48,
+              "routes": 40
+            },
+            {
+              "expert": 29,
+              "routes": 37
+            },
+            {
+              "expert": 7,
+              "routes": 20
+            },
+            {
+              "expert": 9,
+              "routes": 18
+            },
+            {
+              "expert": 47,
+              "routes": 18
+            },
+            {
+              "expert": 31,
+              "routes": 17
+            },
+            {
+              "expert": 54,
+              "routes": 17
+            },
+            {
+              "expert": 24,
+              "routes": 16
+            },
+            {
+              "expert": 55,
+              "routes": 15
+            },
+            {
+              "expert": 41,
+              "routes": 13
+            },
+            {
+              "expert": 61,
+              "routes": 13
+            },
+            {
+              "expert": 17,
+              "routes": 12
+            }
+          ],
+          "selected_weight_sum": {
+            "mean": 0.4748785197734833,
+            "p10": 0.36896076798439026,
+            "p50": 0.45237767696380615,
+            "p90": 0.5860763788223267
+          },
+          "top1_top2_margin": {
+            "mean": 0.08469092845916748,
+            "p50": 0.03858078271150589,
+            "p90": 0.18687288463115692
+          },
+          "per_prompt": [
+            {
+              "id": "zh_explanation",
+              "label": "中文解释",
+              "tokens": 26,
+              "load": [
+                0,
+                3,
+                1,
+                1,
+                5,
+                2,
+                1,
+                3,
+                5,
+                6,
+                0,
+                5,
+                0,
+                0,
+                1,
+                1,
+                2,
+                3,
+                0,
+                0,
+                3,
+                2,
+                3,
+                3,
+                4,
+                2,
+                1,
+                4,
+                2,
+                16,
+                3,
+                2,
+                0,
+                1,
+                3,
+                2,
+                2,
+                1,
+                3,
+                4,
+                0,
+                5,
+                1,
+                0,
+                1,
+                1,
+                0,
+                6,
+                11,
+                2,
+                1,
+                0,
+                4,
+                0,
+                6,
+                5,
+                2,
+                1,
+                0,
+                0,
+                0,
+                5,
+                4,
+                1
+              ],
+              "metrics": {
+                "routes": 156,
+                "used_experts": 49,
+                "zero_experts": 15,
+                "mean": 2.4375,
+                "std": 2.7306315898895264,
+                "cv": 1.120259165763855,
+                "gini": 0.5314503312110901,
+                "entropy_nats": 3.623417377471924,
+                "effective_experts": 37.46538265656308,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 5.0,
+                "max": 16.0
+              },
+              "top_experts": [
+                {
+                  "expert": 29,
+                  "routes": 16
+                },
+                {
+                  "expert": 48,
+                  "routes": 11
+                },
+                {
+                  "expert": 9,
+                  "routes": 6
+                },
+                {
+                  "expert": 47,
+                  "routes": 6
+                },
+                {
+                  "expert": 54,
+                  "routes": 6
+                },
+                {
+                  "expert": 4,
+                  "routes": 5
+                },
+                {
+                  "expert": 8,
+                  "routes": 5
+                },
+                {
+                  "expert": 11,
+                  "routes": 5
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 21,
+                      "weight": 0.2854343056678772
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.028558792546391487
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.026019850745797157
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.018602484837174416
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.01662612520158291
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.016511129215359688
+                    }
+                  ],
+                  "selected_weight_sum": 0.39175268821418285,
+                  "top1_top2_margin": 0.2568755131214857
+                },
+                {
+                  "position": 1,
+                  "token_id": 1082,
+                  "token": "çĶ¨",
+                  "decoded_piece": "用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.3073623776435852
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.058887239545583725
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.04573711007833481
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.036742717027664185
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.028751496225595474
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.024523386731743813
+                    }
+                  ],
+                  "selected_weight_sum": 0.5020043272525072,
+                  "top1_top2_margin": 0.24847513809800148
+                },
+                {
+                  "position": 2,
+                  "token_id": 83983,
+                  "token": "éĢļä¿Ĺ",
+                  "decoded_piece": "通俗",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.139014333486557
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.1303897500038147
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.07262816280126572
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.06052057072520256
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.04703924432396889
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.03525257483124733
+                    }
+                  ],
+                  "selected_weight_sum": 0.4848446361720562,
+                  "top1_top2_margin": 0.00862458348274231
+                },
+                {
+                  "position": 3,
+                  "token_id": 91476,
+                  "token": "çļĦè¯Ńè¨Ģ",
+                  "decoded_piece": "的语言",
+                  "experts_by_weight": [
+                    {
+                      "expert": 41,
+                      "weight": 0.19071631133556366
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.09350986033678055
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.082207590341568
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.06859856098890305
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.06743662059307098
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.059965796768665314
+                    }
+                  ],
+                  "selected_weight_sum": 0.5624347403645515,
+                  "top1_top2_margin": 0.09720645099878311
+                },
+                {
+                  "position": 4,
+                  "token_id": 17882,
+                  "token": "解éĩĬ",
+                  "decoded_piece": "解释",
+                  "experts_by_weight": [
+                    {
+                      "expert": 41,
+                      "weight": 0.16050276160240173
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.1405077874660492
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.10335756838321686
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.06643447279930115
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.057550542056560516
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.040715765208005905
+                    }
+                  ],
+                  "selected_weight_sum": 0.5690688975155354,
+                  "top1_top2_margin": 0.01999497413635254
+                },
+                {
+                  "position": 5,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.17840619385242462
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.13315673172473907
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.13139554858207703
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.1226491779088974
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.06122437119483948
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.035768091678619385
+                    }
+                  ],
+                  "selected_weight_sum": 0.662600114941597,
+                  "top1_top2_margin": 0.04524946212768555
+                },
+                {
+                  "position": 6,
+                  "token_id": 10285,
+                  "token": "为ä»Ģä¹Ī",
+                  "decoded_piece": "为什么",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.20878379046916962
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.11653995513916016
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.06885909289121628
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.06782285124063492
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.04838939756155014
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.026638401672244072
+                    }
+                  ],
+                  "selected_weight_sum": 0.5370334889739752,
+                  "top1_top2_margin": 0.09224383533000946
+                },
+                {
+                  "position": 7,
+                  "token_id": 21844,
+                  "token": "ç¨Ģ",
+                  "decoded_piece": "稀",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.07411029189825058
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.061866600066423416
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.060064975172281265
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.047518208622932434
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.04366019740700722
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.0434911884367466
+                    }
+                  ],
+                  "selected_weight_sum": 0.3307114616036415,
+                  "top1_top2_margin": 0.012243691831827164
+                },
+                {
+                  "position": 8,
+                  "token_id": 20923,
+                  "token": "çĸı",
+                  "decoded_piece": "疏",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.09887184202671051
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.0959555059671402
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.08672784268856049
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.07467734813690186
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.05776987597346306
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.04089703410863876
+                    }
+                  ],
+                  "selected_weight_sum": 0.45489944890141487,
+                  "top1_top2_margin": 0.0029163360595703125
+                },
+                {
+                  "position": 9,
+                  "token_id": 10900,
+                  "token": "ä¸ĵ家",
+                  "decoded_piece": "专家",
+                  "experts_by_weight": [
+                    {
+                      "expert": 62,
+                      "weight": 0.13037250936031342
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.10683903843164444
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.10298178344964981
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.06250814348459244
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.05680561065673828
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.03991430625319481
+                    }
+                  ],
+                  "selected_weight_sum": 0.4994213916361332,
+                  "top1_top2_margin": 0.023533470928668976
+                },
+                {
+                  "position": 10,
+                  "token_id": 39322,
+                  "token": "模åŀĭ",
+                  "decoded_piece": "模型",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.13188892602920532
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.104781374335289
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.09326699376106262
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.08980964124202728
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.06314408034086227
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.04349623620510101
+                    }
+                  ],
+                  "selected_weight_sum": 0.5263872519135475,
+                  "top1_top2_margin": 0.02710755169391632
+                },
+                {
+                  "position": 11,
+                  "token_id": 1876,
+                  "token": "åı¯ä»¥",
+                  "decoded_piece": "可以",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.09696580469608307
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.09488947689533234
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.07881587743759155
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.049439214169979095
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04129276052117348
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.04105379432439804
+                    }
+                  ],
+                  "selected_weight_sum": 0.40245692804455757,
+                  "top1_top2_margin": 0.0020763278007507324
+                },
+                {
+                  "position": 12,
+                  "token_id": 9476,
+                  "token": "æĭ¥æľī",
+                  "decoded_piece": "拥有",
+                  "experts_by_weight": [
+                    {
+                      "expert": 23,
+                      "weight": 0.10077859461307526
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.09732674062252045
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.07639233767986298
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.06467738747596741
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.058186598122119904
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.04759038984775543
+                    }
+                  ],
+                  "selected_weight_sum": 0.4449520483613014,
+                  "top1_top2_margin": 0.0034518539905548096
+                },
+                {
+                  "position": 13,
+                  "token_id": 3599,
+                  "token": "å¾Īå¤ļ",
+                  "decoded_piece": "很多",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.08523216098546982
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.07970038056373596
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.06723999232053757
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.05728509649634361
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.05710846558213234
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.05230521410703659
+                    }
+                  ],
+                  "selected_weight_sum": 0.3988713100552559,
+                  "top1_top2_margin": 0.005531780421733856
+                },
+                {
+                  "position": 14,
+                  "token_id": 21273,
+                  "token": "åıĤæķ°",
+                  "decoded_piece": "参数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.13637880980968475
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.13288584351539612
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.09305819869041443
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.08986103534698486
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.06073031201958656
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.046965353190898895
+                    }
+                  ],
+                  "selected_weight_sum": 0.5598795525729656,
+                  "top1_top2_margin": 0.0034929662942886353
+                },
+                {
+                  "position": 15,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.1541302353143692
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.13401253521442413
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.12823665142059326
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.10574101656675339
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.06683357059955597
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.06087338924407959
+                    }
+                  ],
+                  "selected_weight_sum": 0.6498273983597755,
+                  "top1_top2_margin": 0.02011770009994507
+                },
+                {
+                  "position": 16,
+                  "token_id": 1705,
+                  "token": "ä½Ĩ",
+                  "decoded_piece": "但",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.10471916198730469
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.08865265548229218
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.08270280063152313
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.06964780390262604
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.053717441856861115
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.031225744634866714
+                    }
+                  ],
+                  "selected_weight_sum": 0.43066560849547386,
+                  "top1_top2_margin": 0.016066506505012512
+                },
+                {
+                  "position": 17,
+                  "token_id": 13712,
+                  "token": "æ¯ı个",
+                  "decoded_piece": "每个",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.11605170369148254
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.09560476988554001
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.0483170710504055
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.045952338725328445
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04325094819068909
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.03830089792609215
+                    }
+                  ],
+                  "selected_weight_sum": 0.38747772946953773,
+                  "top1_top2_margin": 0.020446933805942535
+                },
+                {
+                  "position": 18,
+                  "token_id": 10728,
+                  "token": "Ġtoken",
+                  "decoded_piece": " token",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.15618282556533813
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.05938476696610451
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.057606328278779984
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.056335169821977615
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.050878867506980896
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.04674487188458443
+                    }
+                  ],
+                  "selected_weight_sum": 0.42713283002376556,
+                  "top1_top2_margin": 0.09679805859923363
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 38,
+                      "weight": 0.9936936497688293
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.0005532832001335919
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.00046768385800532997
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.0003529697423800826
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.0003311999316792935
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.0002939346304628998
+                    }
+                  ],
+                  "selected_weight_sum": 0.9956927211314905,
+                  "top1_top2_margin": 0.9931403665686958
+                },
+                {
+                  "position": 20,
+                  "token_id": 2141,
+                  "token": "åıª",
+                  "decoded_piece": "只",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.12353485822677612
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.09975308179855347
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.03714262694120407
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03439706191420555
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.03421425074338913
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.031277887523174286
+                    }
+                  ],
+                  "selected_weight_sum": 0.3603197671473026,
+                  "top1_top2_margin": 0.023781776428222656
+                },
+                {
+                  "position": 21,
+                  "token_id": 5118,
+                  "token": "使çĶ¨",
+                  "decoded_piece": "使用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 61,
+                      "weight": 0.1071867048740387
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.1065455824136734
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.05777136981487274
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04117731377482414
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.0401187427341938
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.03573920205235481
+                    }
+                  ],
+                  "selected_weight_sum": 0.3885389156639576,
+                  "top1_top2_margin": 0.0006411224603652954
+                },
+                {
+                  "position": 22,
+                  "token_id": 6802,
+                  "token": "åħ¶ä¸Ń",
+                  "decoded_piece": "其中",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.16480860114097595
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.07635561376810074
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.04757159948348999
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.044671352952718735
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.041422437876462936
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.041200462728738785
+                    }
+                  ],
+                  "selected_weight_sum": 0.41603006795048714,
+                  "top1_top2_margin": 0.08845298737287521
+                },
+                {
+                  "position": 23,
+                  "token_id": 51905,
+                  "token": "ä¸Ģå°ı",
+                  "decoded_piece": "一小",
+                  "experts_by_weight": [
+                    {
+                      "expert": 47,
+                      "weight": 0.14340093731880188
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.0659688413143158
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.06596291810274124
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.05866997316479683
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.046418167650699615
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04551021382212639
+                    }
+                  ],
+                  "selected_weight_sum": 0.42593105137348175,
+                  "top1_top2_margin": 0.07743209600448608
+                },
+                {
+                  "position": 24,
+                  "token_id": 5781,
+                  "token": "éĥ¨åĪĨ",
+                  "decoded_piece": "部分",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.128562793135643
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.09527810662984848
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.07127159088850021
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.060349851846694946
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.04831140115857124
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.044088590890169144
+                    }
+                  ],
+                  "selected_weight_sum": 0.44786233454942703,
+                  "top1_top2_margin": 0.033284686505794525
+                },
+                {
+                  "position": 25,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.15205636620521545
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.136318176984787
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.095779187977314
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.09394431859254837
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.08958881348371506
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.07937369495630264
+                    }
+                  ],
+                  "selected_weight_sum": 0.6470605581998825,
+                  "top1_top2_margin": 0.015738189220428467
+                }
+              ]
+            },
+            {
+              "id": "en_architecture",
+              "label": "English architecture",
+              "tokens": 18,
+              "load": [
+                0,
+                0,
+                2,
+                1,
+                1,
+                0,
+                0,
+                7,
+                1,
+                3,
+                0,
+                0,
+                4,
+                0,
+                1,
+                1,
+                0,
+                3,
+                1,
+                0,
+                1,
+                1,
+                1,
+                1,
+                4,
+                1,
+                1,
+                0,
+                1,
+                5,
+                2,
+                6,
+                2,
+                1,
+                2,
+                2,
+                6,
+                1,
+                1,
+                1,
+                0,
+                2,
+                1,
+                1,
+                0,
+                0,
+                0,
+                4,
+                11,
+                1,
+                6,
+                0,
+                1,
+                1,
+                3,
+                2,
+                2,
+                1,
+                0,
+                0,
+                0,
+                1,
+                6,
+                0
+              ],
+              "metrics": {
+                "routes": 108,
+                "used_experts": 45,
+                "zero_experts": 19,
+                "mean": 1.6875,
+                "std": 2.1371617317199707,
+                "cv": 1.2664662599563599,
+                "gini": 0.5908564925193787,
+                "entropy_nats": 3.490825891494751,
+                "effective_experts": 32.81303652626308,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 4.6999969482421875,
+                "max": 11.0
+              },
+              "top_experts": [
+                {
+                  "expert": 48,
+                  "routes": 11
+                },
+                {
+                  "expert": 7,
+                  "routes": 7
+                },
+                {
+                  "expert": 31,
+                  "routes": 6
+                },
+                {
+                  "expert": 36,
+                  "routes": 6
+                },
+                {
+                  "expert": 50,
+                  "routes": 6
+                },
+                {
+                  "expert": 62,
+                  "routes": 6
+                },
+                {
+                  "expert": 29,
+                  "routes": 5
+                },
+                {
+                  "expert": 12,
+                  "routes": 4
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 21,
+                      "weight": 0.2854343056678772
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.028558792546391487
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.026019850745797157
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.018602484837174416
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.01662612520158291
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.016511129215359688
+                    }
+                  ],
+                  "selected_weight_sum": 0.39175268821418285,
+                  "top1_top2_margin": 0.2568755131214857
+                },
+                {
+                  "position": 1,
+                  "token_id": 99297,
+                  "token": "Explain",
+                  "decoded_piece": "Explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.16727285087108612
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.1338152289390564
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.07673134654760361
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.05953230708837509
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.05908035859465599
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.04080839455127716
+                    }
+                  ],
+                  "selected_weight_sum": 0.5372404865920544,
+                  "top1_top2_margin": 0.033457621932029724
+                },
+                {
+                  "position": 2,
+                  "token_id": 946,
+                  "token": "Ġhow",
+                  "decoded_piece": " how",
+                  "experts_by_weight": [
+                    {
+                      "expert": 9,
+                      "weight": 0.21537113189697266
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.13318757712841034
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.08081577718257904
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.05205032229423523
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.03154856711626053
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.030927589163184166
+                    }
+                  ],
+                  "selected_weight_sum": 0.543900964781642,
+                  "top1_top2_margin": 0.08218355476856232
+                },
+                {
+                  "position": 3,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 47,
+                      "weight": 0.1722368448972702
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.13884633779525757
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.11268851161003113
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.043849557638168335
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.023122970014810562
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.020444709807634354
+                    }
+                  ],
+                  "selected_weight_sum": 0.5111889317631721,
+                  "top1_top2_margin": 0.033390507102012634
+                },
+                {
+                  "position": 4,
+                  "token_id": 29640,
+                  "token": "Ġcompressed",
+                  "decoded_piece": " compressed",
+                  "experts_by_weight": [
+                    {
+                      "expert": 47,
+                      "weight": 0.08341158926486969
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.07845339179039001
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.0757032185792923
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.07314742356538773
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.06734482198953629
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.044416025280952454
+                    }
+                  ],
+                  "selected_weight_sum": 0.42247647047042847,
+                  "top1_top2_margin": 0.004958197474479675
+                },
+                {
+                  "position": 5,
+                  "token_id": 2131,
+                  "token": "Ġkey",
+                  "decoded_piece": " key",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.10751821100711823
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.10609526187181473
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.07796686142683029
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.05935220420360565
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.051578737795352936
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.04501781985163689
+                    }
+                  ],
+                  "selected_weight_sum": 0.4475290961563587,
+                  "top1_top2_margin": 0.0014229491353034973
+                },
+                {
+                  "position": 6,
+                  "token_id": 12,
+                  "token": "-",
+                  "decoded_piece": "-",
+                  "experts_by_weight": [
+                    {
+                      "expert": 42,
+                      "weight": 0.126819908618927
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.10091755539178848
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.0688774585723877
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.058631282299757004
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.04395686089992523
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.04179288074374199
+                    }
+                  ],
+                  "selected_weight_sum": 0.4409959465265274,
+                  "top1_top2_margin": 0.02590235322713852
+                },
+                {
+                  "position": 7,
+                  "token_id": 2834,
+                  "token": "value",
+                  "decoded_piece": "value",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.2559155821800232
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.11273658275604248
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.09117651730775833
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.08211402595043182
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.036896999925374985
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.033982954919338226
+                    }
+                  ],
+                  "selected_weight_sum": 0.612822663038969,
+                  "top1_top2_margin": 0.1431789994239807
+                },
+                {
+                  "position": 8,
+                  "token_id": 28937,
+                  "token": "Ġlatent",
+                  "decoded_piece": " latent",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.17185746133327484
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.08515983074903488
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.06257638335227966
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.04313451051712036
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.040846388787031174
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.040019016712903976
+                    }
+                  ],
+                  "selected_weight_sum": 0.4435935914516449,
+                  "top1_top2_margin": 0.08669763058423996
+                },
+                {
+                  "position": 9,
+                  "token_id": 4183,
+                  "token": "Ġchanges",
+                  "decoded_piece": " changes",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.09715737402439117
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.09069112688302994
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.056275662034749985
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.0479937382042408
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.04470612481236458
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.04298648610711098
+                    }
+                  ],
+                  "selected_weight_sum": 0.37981051206588745,
+                  "top1_top2_margin": 0.006466247141361237
+                },
+                {
+                  "position": 10,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 61,
+                      "weight": 0.08081479370594025
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.06830991804599762
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.06264370679855347
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.053321268409490585
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.043203629553318024
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.04058090224862099
+                    }
+                  ],
+                  "selected_weight_sum": 0.34887421876192093,
+                  "top1_top2_margin": 0.012504875659942627
+                },
+                {
+                  "position": 11,
+                  "token_id": 4887,
+                  "token": "Ġmemory",
+                  "decoded_piece": " memory",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.12293228507041931
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.0906774029135704
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.08118465542793274
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.07018449157476425
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.06667269021272659
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.04963458329439163
+                    }
+                  ],
+                  "selected_weight_sum": 0.48128610849380493,
+                  "top1_top2_margin": 0.03225488215684891
+                },
+                {
+                  "position": 12,
+                  "token_id": 2295,
+                  "token": "Ġcost",
+                  "decoded_piece": " cost",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.16012635827064514
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.1397150307893753
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.08156415820121765
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.04813092201948166
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.0403243824839592
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.03788750618696213
+                    }
+                  ],
+                  "selected_weight_sum": 0.5077483579516411,
+                  "top1_top2_margin": 0.020411327481269836
+                },
+                {
+                  "position": 13,
+                  "token_id": 280,
+                  "token": "Ġof",
+                  "decoded_piece": " of",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.11296866834163666
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.10245229303836823
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.09608335793018341
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.05060618370771408
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.03964516520500183
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03191471844911575
+                    }
+                  ],
+                  "selected_weight_sum": 0.43367038667201996,
+                  "top1_top2_margin": 0.010516375303268433
+                },
+                {
+                  "position": 14,
+                  "token_id": 77939,
+                  "token": "Ġautore",
+                  "decoded_piece": " autore",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.11619260907173157
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.06598556041717529
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.05050069838762283
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.04955204576253891
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.043280262500047684
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.03925475850701332
+                    }
+                  ],
+                  "selected_weight_sum": 0.3647659346461296,
+                  "top1_top2_margin": 0.050207048654556274
+                },
+                {
+                  "position": 15,
+                  "token_id": 79184,
+                  "token": "gressive",
+                  "decoded_piece": "gressive",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.15186402201652527
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.06799350678920746
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.043805334717035294
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.04156063124537468
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.03523245453834534
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.03416448459029198
+                    }
+                  ],
+                  "selected_weight_sum": 0.37462043389678,
+                  "top1_top2_margin": 0.08387051522731781
+                },
+                {
+                  "position": 16,
+                  "token_id": 39671,
+                  "token": "Ġdecoding",
+                  "decoded_piece": " decoding",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.14867179095745087
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.1063278466463089
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.1001177579164505
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.05450600013136864
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.05367346480488777
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.0394548736512661
+                    }
+                  ],
+                  "selected_weight_sum": 0.5027517341077328,
+                  "top1_top2_margin": 0.04234394431114197
+                },
+                {
+                  "position": 17,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.14864285290241241
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.12844759225845337
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.082750603556633
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.08111844211816788
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.07854798436164856
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.0698799341917038
+                    }
+                  ],
+                  "selected_weight_sum": 0.589387409389019,
+                  "top1_top2_margin": 0.020195260643959045
+                }
+              ]
+            },
+            {
+              "id": "code",
+              "label": "Python code",
+              "tokens": 19,
+              "load": [
+                0,
+                2,
+                0,
+                1,
+                1,
+                0,
+                2,
+                2,
+                1,
+                8,
+                1,
+                0,
+                0,
+                0,
+                1,
+                0,
+                1,
+                3,
+                0,
+                2,
+                3,
+                1,
+                2,
+                4,
+                5,
+                1,
+                0,
+                1,
+                1,
+                7,
+                2,
+                2,
+                1,
+                1,
+                2,
+                4,
+                1,
+                1,
+                1,
+                2,
+                1,
+                2,
+                0,
+                0,
+                0,
+                0,
+                1,
+                6,
+                11,
+                1,
+                3,
+                0,
+                2,
+                0,
+                3,
+                4,
+                2,
+                5,
+                2,
+                0,
+                0,
+                3,
+                1,
+                0
+              ],
+              "metrics": {
+                "routes": 114,
+                "used_experts": 46,
+                "zero_experts": 18,
+                "mean": 1.78125,
+                "std": 2.1117115020751953,
+                "cv": 1.185522198677063,
+                "gini": 0.562774121761322,
+                "entropy_nats": 3.5553629398345947,
+                "effective_experts": 35.00052074595525,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 4.0,
+                "max": 11.0
+              },
+              "top_experts": [
+                {
+                  "expert": 48,
+                  "routes": 11
+                },
+                {
+                  "expert": 9,
+                  "routes": 8
+                },
+                {
+                  "expert": 29,
+                  "routes": 7
+                },
+                {
+                  "expert": 47,
+                  "routes": 6
+                },
+                {
+                  "expert": 24,
+                  "routes": 5
+                },
+                {
+                  "expert": 57,
+                  "routes": 5
+                },
+                {
+                  "expert": 23,
+                  "routes": 4
+                },
+                {
+                  "expert": 35,
+                  "routes": 4
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 21,
+                      "weight": 0.2854343056678772
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.028558792546391487
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.026019850745797157
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.018602484837174416
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.01662612520158291
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.016511129215359688
+                    }
+                  ],
+                  "selected_weight_sum": 0.39175268821418285,
+                  "top1_top2_margin": 0.2568755131214857
+                },
+                {
+                  "position": 1,
+                  "token_id": 9081,
+                  "token": "Write",
+                  "decoded_piece": "Write",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.1973877102136612
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.08133529126644135
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.07558465003967285
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.07121656835079193
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.0485970601439476
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.04800603911280632
+                    }
+                  ],
+                  "selected_weight_sum": 0.5221273191273212,
+                  "top1_top2_margin": 0.11605241894721985
+                },
+                {
+                  "position": 2,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 47,
+                      "weight": 0.2108405977487564
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.12334326654672623
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.1129695475101471
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.06092284992337227
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.03568430244922638
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.02485974319279194
+                    }
+                  ],
+                  "selected_weight_sum": 0.5686203073710203,
+                  "top1_top2_margin": 0.08749733120203018
+                },
+                {
+                  "position": 3,
+                  "token_id": 12974,
+                  "token": "ĠPython",
+                  "decoded_piece": " Python",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.09828471392393112
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.08841812610626221
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.06621435284614563
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.05853234604001045
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.0506991446018219
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.049311794340610504
+                    }
+                  ],
+                  "selected_weight_sum": 0.4114604778587818,
+                  "top1_top2_margin": 0.009866587817668915
+                },
+                {
+                  "position": 4,
+                  "token_id": 1157,
+                  "token": "Ġfunction",
+                  "decoded_piece": " function",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.18939025700092316
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.06420006603002548
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.05571218580007553
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.045223336666822433
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.039364468306303024
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.03576372563838959
+                    }
+                  ],
+                  "selected_weight_sum": 0.4296540394425392,
+                  "top1_top2_margin": 0.12519019097089767
+                },
+                {
+                  "position": 5,
+                  "token_id": 344,
+                  "token": "Ġthat",
+                  "decoded_piece": " that",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.11677073687314987
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.10798279941082001
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.09661772847175598
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.0699259340763092
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.06776062399148941
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.06706580519676208
+                    }
+                  ],
+                  "selected_weight_sum": 0.5261236280202866,
+                  "top1_top2_margin": 0.008787937462329865
+                },
+                {
+                  "position": 6,
+                  "token_id": 7578,
+                  "token": "Ġreturns",
+                  "decoded_piece": " returns",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.08541617542505264
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.08021987229585648
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.06539089232683182
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.05553075298666954
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03810269385576248
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.03640105947852135
+                    }
+                  ],
+                  "selected_weight_sum": 0.3610614463686943,
+                  "top1_top2_margin": 0.005196303129196167
+                },
+                {
+                  "position": 7,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 47,
+                      "weight": 0.11266642808914185
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.07434440404176712
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.06673765182495117
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.06281394511461258
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.0450727753341198
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.039807505905628204
+                    }
+                  ],
+                  "selected_weight_sum": 0.4014427103102207,
+                  "top1_top2_margin": 0.038322024047374725
+                },
+                {
+                  "position": 8,
+                  "token_id": 1022,
+                  "token": "Ġfirst",
+                  "decoded_piece": " first",
+                  "experts_by_weight": [
+                    {
+                      "expert": 47,
+                      "weight": 0.1453605592250824
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.12199264764785767
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.054375119507312775
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.052063606679439545
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.045551251620054245
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.04497215151786804
+                    }
+                  ],
+                  "selected_weight_sum": 0.46431533619761467,
+                  "top1_top2_margin": 0.02336791157722473
+                },
+                {
+                  "position": 9,
+                  "token_id": 9842,
+                  "token": "Ġrepeated",
+                  "decoded_piece": " repeated",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.14767202734947205
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.092751644551754
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.08925823867321014
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.058891624212265015
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.04904596880078316
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.040675900876522064
+                    }
+                  ],
+                  "selected_weight_sum": 0.4782954044640064,
+                  "top1_top2_margin": 0.05492038279771805
+                },
+                {
+                  "position": 10,
+                  "token_id": 3699,
+                  "token": "Ġelement",
+                  "decoded_piece": " element",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.16412603855133057
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.09820627421140671
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.04715190455317497
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.04653613641858101
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.044597916305065155
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.04324425011873245
+                    }
+                  ],
+                  "selected_weight_sum": 0.44386252015829086,
+                  "top1_top2_margin": 0.06591976433992386
+                },
+                {
+                  "position": 11,
+                  "token_id": 279,
+                  "token": "Ġin",
+                  "decoded_piece": " in",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.1437619924545288
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.12850943207740784
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.082257479429245
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.06908990442752838
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.06730543076992035
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.03207061067223549
+                    }
+                  ],
+                  "selected_weight_sum": 0.5229948498308659,
+                  "top1_top2_margin": 0.015252560377120972
+                },
+                {
+                  "position": 12,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 47,
+                      "weight": 0.1672954112291336
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.11855437606573105
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.05668004974722862
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.03611814230680466
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03381320834159851
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03321889415383339
+                    }
+                  ],
+                  "selected_weight_sum": 0.44568008184432983,
+                  "top1_top2_margin": 0.04874103516340256
+                },
+                {
+                  "position": 13,
+                  "token_id": 1525,
+                  "token": "Ġlist",
+                  "decoded_piece": " list",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.11332352459430695
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.11224807053804398
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.07935135066509247
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.07445082813501358
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.03989847004413605
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.030583657324314117
+                    }
+                  ],
+                  "selected_weight_sum": 0.44985590130090714,
+                  "top1_top2_margin": 0.00107545405626297
+                },
+                {
+                  "position": 14,
+                  "token_id": 285,
+                  "token": "Ġand",
+                  "decoded_piece": " and",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.15972866117954254
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.11525166779756546
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.08245518058538437
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.06731696426868439
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.05268394947052002
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.04680750146508217
+                    }
+                  ],
+                  "selected_weight_sum": 0.524243924766779,
+                  "top1_top2_margin": 0.04447699338197708
+                },
+                {
+                  "position": 15,
+                  "token_id": 6716,
+                  "token": "Ġexplain",
+                  "decoded_piece": " explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 9,
+                      "weight": 0.16870331764221191
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.1038370430469513
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.07737299054861069
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.052502091974020004
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.043238840997219086
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.04061140865087509
+                    }
+                  ],
+                  "selected_weight_sum": 0.4862656928598881,
+                  "top1_top2_margin": 0.06486627459526062
+                },
+                {
+                  "position": 16,
+                  "token_id": 895,
+                  "token": "Ġits",
+                  "decoded_piece": " its",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.16002975404262543
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.13303902745246887
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.060442566871643066
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.05143991485238075
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.0367891788482666
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.030291805043816566
+                    }
+                  ],
+                  "selected_weight_sum": 0.4720322471112013,
+                  "top1_top2_margin": 0.026990726590156555
+                },
+                {
+                  "position": 17,
+                  "token_id": 13913,
+                  "token": "Ġcomplexity",
+                  "decoded_piece": " complexity",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.12802357971668243
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.08918403834104538
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.08422041684389114
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.06852987408638
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.06726609915494919
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.04570431262254715
+                    }
+                  ],
+                  "selected_weight_sum": 0.4829283207654953,
+                  "top1_top2_margin": 0.038839541375637054
+                },
+                {
+                  "position": 18,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.13413840532302856
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.10561732202768326
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.07496768981218338
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.07266590744256973
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.07079854607582092
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.0666135922074318
+                    }
+                  ],
+                  "selected_weight_sum": 0.5248014628887177,
+                  "top1_top2_margin": 0.028521083295345306
+                }
+              ]
+            },
+            {
+              "id": "math",
+              "label": "数学推理",
+              "tokens": 27,
+              "load": [
+                1,
+                2,
+                0,
+                5,
+                2,
+                1,
+                1,
+                8,
+                2,
+                1,
+                4,
+                6,
+                0,
+                0,
+                2,
+                0,
+                2,
+                3,
+                2,
+                2,
+                4,
+                3,
+                4,
+                4,
+                3,
+                1,
+                4,
+                1,
+                0,
+                9,
+                3,
+                7,
+                0,
+                2,
+                2,
+                3,
+                1,
+                3,
+                2,
+                2,
+                6,
+                4,
+                5,
+                0,
+                0,
+                2,
+                3,
+                2,
+                7,
+                4,
+                0,
+                1,
+                2,
+                0,
+                5,
+                4,
+                0,
+                4,
+                1,
+                3,
+                0,
+                4,
+                0,
+                3
+              ],
+              "metrics": {
+                "routes": 162,
+                "used_experts": 51,
+                "zero_experts": 13,
+                "mean": 2.53125,
+                "std": 2.130420207977295,
+                "cv": 0.8416475057601929,
+                "gini": 0.4552469253540039,
+                "entropy_nats": 3.7677295207977295,
+                "effective_experts": 43.281683030034465,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 5.0,
+                "max": 9.0
+              },
+              "top_experts": [
+                {
+                  "expert": 29,
+                  "routes": 9
+                },
+                {
+                  "expert": 7,
+                  "routes": 8
+                },
+                {
+                  "expert": 31,
+                  "routes": 7
+                },
+                {
+                  "expert": 48,
+                  "routes": 7
+                },
+                {
+                  "expert": 11,
+                  "routes": 6
+                },
+                {
+                  "expert": 40,
+                  "routes": 6
+                },
+                {
+                  "expert": 3,
+                  "routes": 5
+                },
+                {
+                  "expert": 42,
+                  "routes": 5
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 21,
+                      "weight": 0.2854343056678772
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.028558792546391487
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.026019850745797157
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.018602484837174416
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.01662612520158291
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.016511129215359688
+                    }
+                  ],
+                  "selected_weight_sum": 0.39175268821418285,
+                  "top1_top2_margin": 0.2568755131214857
+                },
+                {
+                  "position": 1,
+                  "token_id": 8246,
+                  "token": "èĭ¥",
+                  "decoded_piece": "若",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.32871076464653015
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.08848361670970917
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.0565631277859211
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.05635221675038338
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.0430927611887455
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.028256550431251526
+                    }
+                  ],
+                  "selected_weight_sum": 0.6014590375125408,
+                  "top1_top2_margin": 0.24022714793682098
+                },
+                {
+                  "position": 2,
+                  "token_id": 2206,
+                  "token": "æŃ£",
+                  "decoded_piece": "正",
+                  "experts_by_weight": [
+                    {
+                      "expert": 33,
+                      "weight": 0.08287578076124191
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.0791545957326889
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.07821590453386307
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.06597484648227692
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.05521147698163986
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.05128243565559387
+                    }
+                  ],
+                  "selected_weight_sum": 0.41271504014730453,
+                  "top1_top2_margin": 0.003721185028553009
+                },
+                {
+                  "position": 3,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.2757400870323181
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.07084589451551437
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.05401374399662018
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.04900094121694565
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.04531301185488701
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.03455324470996857
+                    }
+                  ],
+                  "selected_weight_sum": 0.5294669233262539,
+                  "top1_top2_margin": 0.20489419251680374
+                },
+                {
+                  "position": 4,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 40,
+                      "weight": 0.18691080808639526
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.06510350108146667
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.06282351911067963
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.061096590012311935
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.058895643800497055
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04298863559961319
+                    }
+                  ],
+                  "selected_weight_sum": 0.47781869769096375,
+                  "top1_top2_margin": 0.12180730700492859
+                },
+                {
+                  "position": 5,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 38,
+                      "weight": 0.9915505647659302
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.0008498544339090586
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.0008058093371801078
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.0007542146486230195
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.0006039484869688749
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.0004561223613563925
+                    }
+                  ],
+                  "selected_weight_sum": 0.9950205140339676,
+                  "top1_top2_margin": 0.9907007103320211
+                },
+                {
+                  "position": 6,
+                  "token_id": 13211,
+                  "token": "满足",
+                  "decoded_piece": "满足",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.11553625762462616
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.084024578332901
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.06242275983095169
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.056678250432014465
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.040774986147880554
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.039831630885601044
+                    }
+                  ],
+                  "selected_weight_sum": 0.3992684632539749,
+                  "top1_top2_margin": 0.03151167929172516
+                },
+                {
+                  "position": 7,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 40,
+                      "weight": 0.1632770299911499
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.06992333382368088
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.046842269599437714
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.043937139213085175
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.037441857159137726
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.03594948351383209
+                    }
+                  ],
+                  "selected_weight_sum": 0.3973711133003235,
+                  "top1_top2_margin": 0.09335369616746902
+                },
+                {
+                  "position": 8,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.19643105566501617
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.09501662850379944
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.08256134390830994
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.06957186758518219
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.06131204590201378
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.05051179975271225
+                    }
+                  ],
+                  "selected_weight_sum": 0.5554047413170338,
+                  "top1_top2_margin": 0.10141442716121674
+                },
+                {
+                  "position": 9,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 42,
+                      "weight": 0.24526570737361908
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.060395147651433945
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.0489298477768898
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.043506477028131485
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.03604341298341751
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.035008423030376434
+                    }
+                  ],
+                  "selected_weight_sum": 0.46914901584386826,
+                  "top1_top2_margin": 0.18487055972218513
+                },
+                {
+                  "position": 10,
+                  "token_id": 12985,
+                  "token": "å®ĥçļĦ",
+                  "decoded_piece": "它的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.18621590733528137
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.07778092473745346
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.045964907854795456
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.03565226122736931
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.03342017903923988
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.02921866998076439
+                    }
+                  ],
+                  "selected_weight_sum": 0.40825285017490387,
+                  "top1_top2_margin": 0.10843498259782791
+                },
+                {
+                  "position": 11,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.09323352575302124
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.09294472634792328
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.07912302017211914
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.044920433312654495
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.041864652186632156
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.035855911672115326
+                    }
+                  ],
+                  "selected_weight_sum": 0.38794226944446564,
+                  "top1_top2_margin": 0.0002887994050979614
+                },
+                {
+                  "position": 12,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.20481090247631073
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.0889652669429779
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.07102048397064209
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.055527132004499435
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.048422686755657196
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.04809562489390373
+                    }
+                  ],
+                  "selected_weight_sum": 0.5168420970439911,
+                  "top1_top2_margin": 0.11584563553333282
+                },
+                {
+                  "position": 13,
+                  "token_id": 37360,
+                  "token": "çŃīäºİ",
+                  "decoded_piece": "等于",
+                  "experts_by_weight": [
+                    {
+                      "expert": 42,
+                      "weight": 0.15381541848182678
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.11250929534435272
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.042643386870622635
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.0421457402408123
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.038802195340394974
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.033083997666835785
+                    }
+                  ],
+                  "selected_weight_sum": 0.4230000339448452,
+                  "top1_top2_margin": 0.04130612313747406
+                },
+                {
+                  "position": 14,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 3,
+                      "weight": 0.133352592587471
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.12619337439537048
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.1030726283788681
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.07036352902650833
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05759592354297638
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.055169083178043365
+                    }
+                  ],
+                  "selected_weight_sum": 0.5457471311092377,
+                  "top1_top2_margin": 0.007159218192100525
+                },
+                {
+                  "position": 15,
+                  "token_id": 18,
+                  "token": "3",
+                  "decoded_piece": "3",
+                  "experts_by_weight": [
+                    {
+                      "expert": 40,
+                      "weight": 0.23435364663600922
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.11713644117116928
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.03138922527432442
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.0267313364893198
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.026693860068917274
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.020750872790813446
+                    }
+                  ],
+                  "selected_weight_sum": 0.45705538243055344,
+                  "top1_top2_margin": 0.11721720546483994
+                },
+                {
+                  "position": 16,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.14716902375221252
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.14222267270088196
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.10041153430938721
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.09503325819969177
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05392856523394585
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.04694347456097603
+                    }
+                  ],
+                  "selected_weight_sum": 0.5857085287570953,
+                  "top1_top2_margin": 0.004946351051330566
+                },
+                {
+                  "position": 17,
+                  "token_id": 2989,
+                  "token": "æ±Ĥ",
+                  "decoded_piece": "求",
+                  "experts_by_weight": [
+                    {
+                      "expert": 9,
+                      "weight": 0.14854924380779266
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.0715351402759552
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.05261973291635513
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.04394381120800972
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.03829782083630562
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.03392321988940239
+                    }
+                  ],
+                  "selected_weight_sum": 0.3888689689338207,
+                  "top1_top2_margin": 0.07701410353183746
+                },
+                {
+                  "position": 18,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 40,
+                      "weight": 0.13323214650154114
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.05016558617353439
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.043240778148174286
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04313679784536362
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.0390276238322258
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.03554553538560867
+                    }
+                  ],
+                  "selected_weight_sum": 0.3443484678864479,
+                  "top1_top2_margin": 0.08306656032800674
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.14701107144355774
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.08602465689182281
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.08064025640487671
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.07543258368968964
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.07340933382511139
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.04117288440465927
+                    }
+                  ],
+                  "selected_weight_sum": 0.5036907866597176,
+                  "top1_top2_margin": 0.060986414551734924
+                },
+                {
+                  "position": 20,
+                  "token_id": 337,
+                  "token": "çļĦ",
+                  "decoded_piece": "的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.16039715707302094
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.05364707112312317
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.05302438884973526
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.03915613889694214
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.032873403280973434
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.03032870404422283
+                    }
+                  ],
+                  "selected_weight_sum": 0.36942686326801777,
+                  "top1_top2_margin": 0.10675008594989777
+                },
+                {
+                  "position": 21,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 40,
+                      "weight": 0.10204068571329117
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.05241134762763977
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.051519397646188736
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.04217739403247833
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.030644165351986885
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.030136745423078537
+                    }
+                  ],
+                  "selected_weight_sum": 0.30892973579466343,
+                  "top1_top2_margin": 0.0496293380856514
+                },
+                {
+                  "position": 22,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 42,
+                      "weight": 0.21430017054080963
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.04553015157580376
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.04181747883558273
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.04179762303829193
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.03857774659991264
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.03725086897611618
+                    }
+                  ],
+                  "selected_weight_sum": 0.4192740395665169,
+                  "top1_top2_margin": 0.16877001896500587
+                },
+                {
+                  "position": 23,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 54,
+                      "weight": 0.07403554767370224
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.0634426474571228
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.05088704824447632
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.05061543732881546
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.038594309240579605
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.029449019581079483
+                    }
+                  ],
+                  "selected_weight_sum": 0.3070240095257759,
+                  "top1_top2_margin": 0.010592900216579437
+                },
+                {
+                  "position": 24,
+                  "token_id": 51971,
+                  "token": "æķ°çļĦ",
+                  "decoded_piece": "数的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.20678895711898804
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.06979629397392273
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.06282166391611099
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.06192285567522049
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.048118963837623596
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.04148270562291145
+                    }
+                  ],
+                  "selected_weight_sum": 0.4909314401447773,
+                  "top1_top2_margin": 0.1369926631450653
+                },
+                {
+                  "position": 25,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 40,
+                      "weight": 0.08547789603471756
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.05890219286084175
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.058209121227264404
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.042358338832855225
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.032743848860263824
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.032649461179971695
+                    }
+                  ],
+                  "selected_weight_sum": 0.31034085899591446,
+                  "top1_top2_margin": 0.02657570317387581
+                },
+                {
+                  "position": 26,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.14948272705078125
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.11477416008710861
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.0903598740696907
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.08863123506307602
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.08091817051172256
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.07229965180158615
+                    }
+                  ],
+                  "selected_weight_sum": 0.5964658185839653,
+                  "top1_top2_margin": 0.03470856696367264
+                }
+              ]
+            }
+          ],
+          "prompt_pair_jaccard": [
+            {
+              "left": "zh_explanation",
+              "right": "en_architecture",
+              "used_expert_jaccard": 0.7407407407407407,
+              "shared_experts": 40,
+              "union_experts": 54
+            },
+            {
+              "left": "zh_explanation",
+              "right": "code",
+              "used_expert_jaccard": 0.7272727272727273,
+              "shared_experts": 40,
+              "union_experts": 55
+            },
+            {
+              "left": "zh_explanation",
+              "right": "math",
+              "used_expert_jaccard": 0.7241379310344828,
+              "shared_experts": 42,
+              "union_experts": 58
+            },
+            {
+              "left": "en_architecture",
+              "right": "code",
+              "used_expert_jaccard": 0.6851851851851852,
+              "shared_experts": 37,
+              "union_experts": 54
+            },
+            {
+              "left": "en_architecture",
+              "right": "math",
+              "used_expert_jaccard": 0.5737704918032787,
+              "shared_experts": 35,
+              "union_experts": 61
+            },
+            {
+              "left": "code",
+              "right": "math",
+              "used_expert_jaccard": 0.7321428571428571,
+              "shared_experts": 41,
+              "union_experts": 56
+            }
+          ]
+        }
+      },
+      {
+        "layer": 5,
+        "ffn": "moe",
+        "state_tensors": 203,
+        "state_numel": 584847872,
+        "state_bytes": 1169695744,
+        "timing_ms": {
+          "load_cpu": 24.87539401045069,
+          "transfer_to_device": 122.59778601583093,
+          "forward": 4.010354983620346
+        },
+        "peak_device_allocated_mib": 1177.7021484375,
+        "hidden_before": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 3.085932731628418,
+            "p10": 0.19619415700435638,
+            "p50": 0.23260724544525146,
+            "p90": 0.2969282567501068
+          },
+          "l2_norm": {
+            "mean": 139.65338134765625,
+            "p10": 8.878734588623047,
+            "p50": 10.526601791381836,
+            "p90": 13.43743896484375
+          },
+          "finite": true,
+          "sha256_fp32": "5d5d80c17f47a78f419f6ec9bf16668392a8246393ebfe2b0233841be4875b6e"
+        },
+        "hidden_after": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 3.2100954055786133,
+            "p10": 0.22690242528915405,
+            "p50": 0.26399680972099304,
+            "p90": 0.33678531646728516
+          },
+          "l2_norm": {
+            "mean": 145.2723388671875,
+            "p10": 10.268431663513184,
+            "p50": 11.947131156921387,
+            "p90": 15.24116325378418
+          },
+          "finite": true,
+          "sha256_fp32": "88f158b68ecb980c55b1b77eebfeb2276d7820443732aa10effca9c85c653481"
+        },
+        "mla": {
+          "compressed_projection_shape": [
+            4,
+            27,
+            576
+          ],
+          "latent_shape": [
+            4,
+            27,
+            512
+          ],
+          "rope_key_shape": [
+            4,
+            27,
+            64
+          ],
+          "latent_rms": 2.0517733097076416,
+          "rope_key_rms": 2.469125509262085,
+          "eager_key_cache_shape": [
+            4,
+            16,
+            27,
+            192
+          ],
+          "eager_value_cache_shape": [
+            4,
+            16,
+            27,
+            128
+          ],
+          "eager_cache_bytes": 1105920
+        },
+        "routing": {
+          "aggregate_load": [
+            6,
+            12,
+            10,
+            13,
+            13,
+            0,
+            10,
+            19,
+            8,
+            0,
+            7,
+            15,
+            9,
+            3,
+            32,
+            6,
+            6,
+            4,
+            9,
+            14,
+            7,
+            3,
+            8,
+            15,
+            7,
+            7,
+            4,
+            9,
+            11,
+            13,
+            4,
+            0,
+            3,
+            5,
+            22,
+            10,
+            12,
+            3,
+            9,
+            5,
+            4,
+            6,
+            11,
+            9,
+            11,
+            4,
+            11,
+            4,
+            11,
+            14,
+            0,
+            9,
+            7,
+            9,
+            3,
+            10,
+            10,
+            7,
+            3,
+            15,
+            9,
+            3,
+            9,
+            8
+          ],
+          "aggregate_metrics": {
+            "routes": 540,
+            "used_experts": 60,
+            "zero_experts": 4,
+            "mean": 8.4375,
+            "std": 5.415659427642822,
+            "cv": 0.6418559551239014,
+            "gini": 0.3323495388031006,
+            "entropy_nats": 3.9539036750793457,
+            "effective_experts": 52.13850186069128,
+            "min": 0.0,
+            "p50": 8.5,
+            "p90": 14.0,
+            "max": 32.0
+          },
+          "top_experts": [
+            {
+              "expert": 14,
+              "routes": 32
+            },
+            {
+              "expert": 34,
+              "routes": 22
+            },
+            {
+              "expert": 7,
+              "routes": 19
+            },
+            {
+              "expert": 11,
+              "routes": 15
+            },
+            {
+              "expert": 23,
+              "routes": 15
+            },
+            {
+              "expert": 59,
+              "routes": 15
+            },
+            {
+              "expert": 19,
+              "routes": 14
+            },
+            {
+              "expert": 49,
+              "routes": 14
+            },
+            {
+              "expert": 3,
+              "routes": 13
+            },
+            {
+              "expert": 4,
+              "routes": 13
+            },
+            {
+              "expert": 29,
+              "routes": 13
+            },
+            {
+              "expert": 1,
+              "routes": 12
+            }
+          ],
+          "selected_weight_sum": {
+            "mean": 0.4831911623477936,
+            "p10": 0.37563803791999817,
+            "p50": 0.4518430829048157,
+            "p90": 0.6178299784660339
+          },
+          "top1_top2_margin": {
+            "mean": 0.09345383197069168,
+            "p50": 0.027145538479089737,
+            "p90": 0.16611436009407043
+          },
+          "per_prompt": [
+            {
+              "id": "zh_explanation",
+              "label": "中文解释",
+              "tokens": 26,
+              "load": [
+                3,
+                2,
+                2,
+                2,
+                1,
+                0,
+                4,
+                6,
+                3,
+                0,
+                0,
+                2,
+                3,
+                0,
+                7,
+                1,
+                1,
+                3,
+                3,
+                4,
+                2,
+                2,
+                2,
+                4,
+                2,
+                0,
+                3,
+                4,
+                1,
+                5,
+                2,
+                0,
+                2,
+                2,
+                4,
+                5,
+                6,
+                1,
+                3,
+                1,
+                2,
+                1,
+                3,
+                4,
+                2,
+                1,
+                4,
+                1,
+                3,
+                5,
+                0,
+                4,
+                2,
+                0,
+                1,
+                1,
+                2,
+                5,
+                0,
+                9,
+                1,
+                1,
+                5,
+                1
+              ],
+              "metrics": {
+                "routes": 156,
+                "used_experts": 55,
+                "zero_experts": 9,
+                "mean": 2.4375,
+                "std": 1.9096662998199463,
+                "cv": 0.7834528684616089,
+                "gini": 0.4216746687889099,
+                "entropy_nats": 3.832216262817383,
+                "effective_experts": 46.16473813406393,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 5.0,
+                "max": 9.0
+              },
+              "top_experts": [
+                {
+                  "expert": 59,
+                  "routes": 9
+                },
+                {
+                  "expert": 14,
+                  "routes": 7
+                },
+                {
+                  "expert": 7,
+                  "routes": 6
+                },
+                {
+                  "expert": 36,
+                  "routes": 6
+                },
+                {
+                  "expert": 29,
+                  "routes": 5
+                },
+                {
+                  "expert": 35,
+                  "routes": 5
+                },
+                {
+                  "expert": 49,
+                  "routes": 5
+                },
+                {
+                  "expert": 57,
+                  "routes": 5
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.6668257713317871
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.015490565448999405
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.012896407395601273
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.01244935393333435
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.012159599922597408
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.011999384500086308
+                    }
+                  ],
+                  "selected_weight_sum": 0.7318210825324059,
+                  "top1_top2_margin": 0.6513352058827877
+                },
+                {
+                  "position": 1,
+                  "token_id": 1082,
+                  "token": "çĶ¨",
+                  "decoded_piece": "用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.26780611276626587
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.17193788290023804
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.054537083953619
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.034212712198495865
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.03277919813990593
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.02963159792125225
+                    }
+                  ],
+                  "selected_weight_sum": 0.590904587879777,
+                  "top1_top2_margin": 0.09586822986602783
+                },
+                {
+                  "position": 2,
+                  "token_id": 83983,
+                  "token": "éĢļä¿Ĺ",
+                  "decoded_piece": "通俗",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.13310463726520538
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.10224568098783493
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.08373391628265381
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.07357697933912277
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.06322353333234787
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.056075744330883026
+                    }
+                  ],
+                  "selected_weight_sum": 0.5119604915380478,
+                  "top1_top2_margin": 0.030858956277370453
+                },
+                {
+                  "position": 3,
+                  "token_id": 91476,
+                  "token": "çļĦè¯Ńè¨Ģ",
+                  "decoded_piece": "的语言",
+                  "experts_by_weight": [
+                    {
+                      "expert": 19,
+                      "weight": 0.237516850233078
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.16881954669952393
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.1347130984067917
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.12524957954883575
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.051083795726299286
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.022646525874733925
+                    }
+                  ],
+                  "selected_weight_sum": 0.7400293964892626,
+                  "top1_top2_margin": 0.06869730353355408
+                },
+                {
+                  "position": 4,
+                  "token_id": 17882,
+                  "token": "解éĩĬ",
+                  "decoded_piece": "解释",
+                  "experts_by_weight": [
+                    {
+                      "expert": 6,
+                      "weight": 0.1273176521062851
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.10280857235193253
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.07830130308866501
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.07459865510463715
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.06277203559875488
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.05869143456220627
+                    }
+                  ],
+                  "selected_weight_sum": 0.5044896528124809,
+                  "top1_top2_margin": 0.02450907975435257
+                },
+                {
+                  "position": 5,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 27,
+                      "weight": 0.1488560438156128
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.08283150941133499
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.08140339702367783
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.07671284675598145
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.058525532484054565
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.044425301253795624
+                    }
+                  ],
+                  "selected_weight_sum": 0.49275463074445724,
+                  "top1_top2_margin": 0.0660245344042778
+                },
+                {
+                  "position": 6,
+                  "token_id": 10285,
+                  "token": "为ä»Ģä¹Ī",
+                  "decoded_piece": "为什么",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.08674228936433792
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.07026661932468414
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.06282687932252884
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.057485852390527725
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.047155700623989105
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.04366069287061691
+                    }
+                  ],
+                  "selected_weight_sum": 0.36813803389668465,
+                  "top1_top2_margin": 0.016475670039653778
+                },
+                {
+                  "position": 7,
+                  "token_id": 21844,
+                  "token": "ç¨Ģ",
+                  "decoded_piece": "稀",
+                  "experts_by_weight": [
+                    {
+                      "expert": 11,
+                      "weight": 0.10328876972198486
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.08392632752656937
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.0741342306137085
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.050005458295345306
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.03275921195745468
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.028308894485235214
+                    }
+                  ],
+                  "selected_weight_sum": 0.37242289260029793,
+                  "top1_top2_margin": 0.019362442195415497
+                },
+                {
+                  "position": 8,
+                  "token_id": 20923,
+                  "token": "çĸı",
+                  "decoded_piece": "疏",
+                  "experts_by_weight": [
+                    {
+                      "expert": 11,
+                      "weight": 0.21587729454040527
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.09414123743772507
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.04408562555909157
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.03777015209197998
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.032802462577819824
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.031176554039120674
+                    }
+                  ],
+                  "selected_weight_sum": 0.4558533262461424,
+                  "top1_top2_margin": 0.1217360571026802
+                },
+                {
+                  "position": 9,
+                  "token_id": 10900,
+                  "token": "ä¸ĵ家",
+                  "decoded_piece": "专家",
+                  "experts_by_weight": [
+                    {
+                      "expert": 2,
+                      "weight": 0.08871850371360779
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.07161764800548553
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.06334264576435089
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.059080276638269424
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.05416818708181381
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.05293413996696472
+                    }
+                  ],
+                  "selected_weight_sum": 0.38986140117049217,
+                  "top1_top2_margin": 0.017100855708122253
+                },
+                {
+                  "position": 10,
+                  "token_id": 39322,
+                  "token": "模åŀĭ",
+                  "decoded_piece": "模型",
+                  "experts_by_weight": [
+                    {
+                      "expert": 2,
+                      "weight": 0.17880289256572723
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.1402723491191864
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.13747455179691315
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.09143506735563278
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.08162415027618408
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.037828776985406876
+                    }
+                  ],
+                  "selected_weight_sum": 0.6674377880990505,
+                  "top1_top2_margin": 0.03853054344654083
+                },
+                {
+                  "position": 11,
+                  "token_id": 1876,
+                  "token": "åı¯ä»¥",
+                  "decoded_piece": "可以",
+                  "experts_by_weight": [
+                    {
+                      "expert": 56,
+                      "weight": 0.12182933837175369
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.10752717405557632
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.08328860998153687
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.0359572097659111
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.03254731744527817
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.03201925754547119
+                    }
+                  ],
+                  "selected_weight_sum": 0.41316890716552734,
+                  "top1_top2_margin": 0.014302164316177368
+                },
+                {
+                  "position": 12,
+                  "token_id": 9476,
+                  "token": "æĭ¥æľī",
+                  "decoded_piece": "拥有",
+                  "experts_by_weight": [
+                    {
+                      "expert": 51,
+                      "weight": 0.1414400339126587
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.10390999913215637
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.05466355383396149
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.05069989338517189
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.043549034744501114
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.03670540824532509
+                    }
+                  ],
+                  "selected_weight_sum": 0.43096792325377464,
+                  "top1_top2_margin": 0.03753003478050232
+                },
+                {
+                  "position": 13,
+                  "token_id": 3599,
+                  "token": "å¾Īå¤ļ",
+                  "decoded_piece": "很多",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.10780058056116104
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.09324204176664352
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.08552185446023941
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.04821997880935669
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.03283743932843208
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.03034812957048416
+                    }
+                  ],
+                  "selected_weight_sum": 0.3979700244963169,
+                  "top1_top2_margin": 0.014558538794517517
+                },
+                {
+                  "position": 14,
+                  "token_id": 21273,
+                  "token": "åıĤæķ°",
+                  "decoded_piece": "参数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.10011549293994904
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.09130356460809708
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.08848028630018234
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.07407595962285995
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.04830920696258545
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03818826377391815
+                    }
+                  ],
+                  "selected_weight_sum": 0.440472774207592,
+                  "top1_top2_margin": 0.00881192833185196
+                },
+                {
+                  "position": 15,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 43,
+                      "weight": 0.12919080257415771
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.10803435742855072
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.0981711596250534
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.05666274577379227
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.055799245834350586
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.05242020636796951
+                    }
+                  ],
+                  "selected_weight_sum": 0.5002785176038742,
+                  "top1_top2_margin": 0.021156445145606995
+                },
+                {
+                  "position": 16,
+                  "token_id": 1705,
+                  "token": "ä½Ĩ",
+                  "decoded_piece": "但",
+                  "experts_by_weight": [
+                    {
+                      "expert": 43,
+                      "weight": 0.0960824117064476
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.0716608390212059
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.06996157020330429
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.06649202108383179
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.06584745645523071
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.04214527830481529
+                    }
+                  ],
+                  "selected_weight_sum": 0.4121895767748356,
+                  "top1_top2_margin": 0.0244215726852417
+                },
+                {
+                  "position": 17,
+                  "token_id": 13712,
+                  "token": "æ¯ı个",
+                  "decoded_piece": "每个",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.06841839849948883
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.06105462834239006
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.052708983421325684
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.04746530205011368
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.04617984592914581
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.037067096680402756
+                    }
+                  ],
+                  "selected_weight_sum": 0.3128942549228668,
+                  "top1_top2_margin": 0.00736377015709877
+                },
+                {
+                  "position": 18,
+                  "token_id": 10728,
+                  "token": "Ġtoken",
+                  "decoded_piece": " token",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.11643253266811371
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.10292921215295792
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.06338663399219513
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.04246857762336731
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.04223647341132164
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.03386823087930679
+                    }
+                  ],
+                  "selected_weight_sum": 0.4013216607272625,
+                  "top1_top2_margin": 0.013503320515155792
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.9029439091682434
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.004053776618093252
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.003351924940943718
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.0033151512034237385
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.0029632176738232374
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.0028055852744728327
+                    }
+                  ],
+                  "selected_weight_sum": 0.9194335648790002,
+                  "top1_top2_margin": 0.8988901325501502
+                },
+                {
+                  "position": 20,
+                  "token_id": 2141,
+                  "token": "åıª",
+                  "decoded_piece": "只",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.09061706066131592
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.07269977033138275
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.06583838164806366
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.05064796283841133
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.0499817356467247
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.04805685207247734
+                    }
+                  ],
+                  "selected_weight_sum": 0.3778417631983757,
+                  "top1_top2_margin": 0.017917290329933167
+                },
+                {
+                  "position": 21,
+                  "token_id": 5118,
+                  "token": "使çĶ¨",
+                  "decoded_piece": "使用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 23,
+                      "weight": 0.09294749051332474
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.09201069921255112
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.0672682672739029
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.06171053647994995
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.052574750036001205
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.04288235679268837
+                    }
+                  ],
+                  "selected_weight_sum": 0.4093941003084183,
+                  "top1_top2_margin": 0.0009367913007736206
+                },
+                {
+                  "position": 22,
+                  "token_id": 6802,
+                  "token": "åħ¶ä¸Ń",
+                  "decoded_piece": "其中",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.07986949384212494
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.06633445620536804
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.05573779717087746
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.0536210760474205
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.05267202854156494
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.0491383858025074
+                    }
+                  ],
+                  "selected_weight_sum": 0.3573732376098633,
+                  "top1_top2_margin": 0.013535037636756897
+                },
+                {
+                  "position": 23,
+                  "token_id": 51905,
+                  "token": "ä¸Ģå°ı",
+                  "decoded_piece": "一小",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.1389620155096054
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.07460430264472961
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05769191309809685
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.048479143530130386
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.03461878374218941
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03341403976082802
+                    }
+                  ],
+                  "selected_weight_sum": 0.3877701982855797,
+                  "top1_top2_margin": 0.0643577128648758
+                },
+                {
+                  "position": 24,
+                  "token_id": 5781,
+                  "token": "éĥ¨åĪĨ",
+                  "decoded_piece": "部分",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.09375614672899246
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.08696839958429337
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.06338794529438019
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.059497132897377014
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.05874354764819145
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.053002435714006424
+                    }
+                  ],
+                  "selected_weight_sum": 0.4153556078672409,
+                  "top1_top2_margin": 0.006787747144699097
+                },
+                {
+                  "position": 25,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 12,
+                      "weight": 0.14567047357559204
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.11958866566419601
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.10240183025598526
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.07829892635345459
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.05260668322443962
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.05196297913789749
+                    }
+                  ],
+                  "selected_weight_sum": 0.550529558211565,
+                  "top1_top2_margin": 0.026081807911396027
+                }
+              ]
+            },
+            {
+              "id": "en_architecture",
+              "label": "English architecture",
+              "tokens": 18,
+              "load": [
+                0,
+                0,
+                6,
+                6,
+                5,
+                0,
+                2,
+                8,
+                1,
+                0,
+                1,
+                6,
+                1,
+                0,
+                1,
+                0,
+                1,
+                0,
+                1,
+                1,
+                1,
+                0,
+                2,
+                2,
+                0,
+                1,
+                0,
+                1,
+                3,
+                3,
+                0,
+                0,
+                0,
+                1,
+                7,
+                1,
+                1,
+                1,
+                2,
+                1,
+                0,
+                2,
+                1,
+                1,
+                5,
+                2,
+                6,
+                1,
+                4,
+                1,
+                0,
+                1,
+                2,
+                0,
+                0,
+                2,
+                2,
+                2,
+                0,
+                1,
+                3,
+                2,
+                2,
+                1
+              ],
+              "metrics": {
+                "routes": 108,
+                "used_experts": 46,
+                "zero_experts": 18,
+                "mean": 1.6875,
+                "std": 1.9426295757293701,
+                "cv": 1.1511878967285156,
+                "gini": 0.5627893805503845,
+                "entropy_nats": 3.557856321334839,
+                "effective_experts": 35.087899285630286,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 5.0,
+                "max": 8.0
+              },
+              "top_experts": [
+                {
+                  "expert": 7,
+                  "routes": 8
+                },
+                {
+                  "expert": 34,
+                  "routes": 7
+                },
+                {
+                  "expert": 2,
+                  "routes": 6
+                },
+                {
+                  "expert": 3,
+                  "routes": 6
+                },
+                {
+                  "expert": 11,
+                  "routes": 6
+                },
+                {
+                  "expert": 46,
+                  "routes": 6
+                },
+                {
+                  "expert": 4,
+                  "routes": 5
+                },
+                {
+                  "expert": 44,
+                  "routes": 5
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.6668257713317871
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.015490565448999405
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.012896407395601273
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.01244935393333435
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.012159599922597408
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.011999384500086308
+                    }
+                  ],
+                  "selected_weight_sum": 0.7318210825324059,
+                  "top1_top2_margin": 0.6513352058827877
+                },
+                {
+                  "position": 1,
+                  "token_id": 99297,
+                  "token": "Explain",
+                  "decoded_piece": "Explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.17686711251735687
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.137218177318573
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.11987470090389252
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.057543590664863586
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.04869488254189491
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.04298888146877289
+                    }
+                  ],
+                  "selected_weight_sum": 0.5831873454153538,
+                  "top1_top2_margin": 0.039648935198783875
+                },
+                {
+                  "position": 2,
+                  "token_id": 946,
+                  "token": "Ġhow",
+                  "decoded_piece": " how",
+                  "experts_by_weight": [
+                    {
+                      "expert": 60,
+                      "weight": 0.11648083478212357
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.10566136986017227
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.06881820410490036
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.059229787439107895
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.048904579132795334
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.03643996641039848
+                    }
+                  ],
+                  "selected_weight_sum": 0.4355347417294979,
+                  "top1_top2_margin": 0.010819464921951294
+                },
+                {
+                  "position": 3,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.23619119822978973
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.07677339762449265
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.03644350543618202
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.03643375635147095
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.034974295645952225
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.03322301805019379
+                    }
+                  ],
+                  "selected_weight_sum": 0.45403917133808136,
+                  "top1_top2_margin": 0.1594178006052971
+                },
+                {
+                  "position": 4,
+                  "token_id": 29640,
+                  "token": "Ġcompressed",
+                  "decoded_piece": " compressed",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.14481888711452484
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.12344990670681
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.056090760976076126
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.04771822690963745
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.040006957948207855
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.037562284618616104
+                    }
+                  ],
+                  "selected_weight_sum": 0.4496470242738724,
+                  "top1_top2_margin": 0.021368980407714844
+                },
+                {
+                  "position": 5,
+                  "token_id": 2131,
+                  "token": "Ġkey",
+                  "decoded_piece": " key",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.15713807940483093
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.12892881035804749
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.0764535441994667
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.06165243312716484
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04296897351741791
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.042255908250808716
+                    }
+                  ],
+                  "selected_weight_sum": 0.5093977488577366,
+                  "top1_top2_margin": 0.028209269046783447
+                },
+                {
+                  "position": 6,
+                  "token_id": 12,
+                  "token": "-",
+                  "decoded_piece": "-",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.22240972518920898
+                    },
+                    {
+                      "expert": 61,
+                      "weight": 0.04973777011036873
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.03715226426720619
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.03665049001574516
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.033358726650476456
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.03200354427099228
+                    }
+                  ],
+                  "selected_weight_sum": 0.4113125205039978,
+                  "top1_top2_margin": 0.17267195507884026
+                },
+                {
+                  "position": 7,
+                  "token_id": 2834,
+                  "token": "value",
+                  "decoded_piece": "value",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.15645183622837067
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.09938643872737885
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.0829453244805336
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.052371591329574585
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.050064388662576675
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.036176759749650955
+                    }
+                  ],
+                  "selected_weight_sum": 0.4773963391780853,
+                  "top1_top2_margin": 0.05706539750099182
+                },
+                {
+                  "position": 8,
+                  "token_id": 28937,
+                  "token": "Ġlatent",
+                  "decoded_piece": " latent",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.16698865592479706
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.09657170623540878
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.09212639927864075
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.06852614879608154
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.06448085606098175
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.060119882225990295
+                    }
+                  ],
+                  "selected_weight_sum": 0.5488136485219002,
+                  "top1_top2_margin": 0.07041694968938828
+                },
+                {
+                  "position": 9,
+                  "token_id": 4183,
+                  "token": "Ġchanges",
+                  "decoded_piece": " changes",
+                  "experts_by_weight": [
+                    {
+                      "expert": 45,
+                      "weight": 0.09613549709320068
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.07573243230581284
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.06894052773714066
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.06686387956142426
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.05422895401716232
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.04179750010371208
+                    }
+                  ],
+                  "selected_weight_sum": 0.40369879081845284,
+                  "top1_top2_margin": 0.020403064787387848
+                },
+                {
+                  "position": 10,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.1237715482711792
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.0828896015882492
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.06915713101625443
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.05710538104176521
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.04857157543301582
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.04292594641447067
+                    }
+                  ],
+                  "selected_weight_sum": 0.42442118376493454,
+                  "top1_top2_margin": 0.04088194668292999
+                },
+                {
+                  "position": 11,
+                  "token_id": 4887,
+                  "token": "Ġmemory",
+                  "decoded_piece": " memory",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.10212866216897964
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.08982594311237335
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.05629178136587143
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.045767851173877716
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.045069701969623566
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.040722765028476715
+                    }
+                  ],
+                  "selected_weight_sum": 0.3798067048192024,
+                  "top1_top2_margin": 0.012302719056606293
+                },
+                {
+                  "position": 12,
+                  "token_id": 2295,
+                  "token": "Ġcost",
+                  "decoded_piece": " cost",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.1462714523077011
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.08453454077243805
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.08272839337587357
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.06534243375062943
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.05198255553841591
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.04178442433476448
+                    }
+                  ],
+                  "selected_weight_sum": 0.47264380007982254,
+                  "top1_top2_margin": 0.06173691153526306
+                },
+                {
+                  "position": 13,
+                  "token_id": 280,
+                  "token": "Ġof",
+                  "decoded_piece": " of",
+                  "experts_by_weight": [
+                    {
+                      "expert": 60,
+                      "weight": 0.14627379179000854
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.09714733064174652
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.09307599067687988
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.061775561422109604
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.03882799297571182
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.037830740213394165
+                    }
+                  ],
+                  "selected_weight_sum": 0.47493140771985054,
+                  "top1_top2_margin": 0.049126461148262024
+                },
+                {
+                  "position": 14,
+                  "token_id": 77939,
+                  "token": "Ġautore",
+                  "decoded_piece": " autore",
+                  "experts_by_weight": [
+                    {
+                      "expert": 44,
+                      "weight": 0.0866532251238823
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.08156659454107285
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.0580621175467968
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.05063927173614502
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.04073431342840195
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.03586898744106293
+                    }
+                  ],
+                  "selected_weight_sum": 0.35352450981736183,
+                  "top1_top2_margin": 0.005086630582809448
+                },
+                {
+                  "position": 15,
+                  "token_id": 79184,
+                  "token": "gressive",
+                  "decoded_piece": "gressive",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.20286299288272858
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.060117512941360474
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.059362683445215225
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.044468119740486145
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.044006701558828354
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.0434880256652832
+                    }
+                  ],
+                  "selected_weight_sum": 0.454306036233902,
+                  "top1_top2_margin": 0.1427454799413681
+                },
+                {
+                  "position": 16,
+                  "token_id": 39671,
+                  "token": "Ġdecoding",
+                  "decoded_piece": " decoding",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.22519521415233612
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.10506051033735275
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.10295503586530685
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.0685448870062828
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.06022454798221588
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.04068287834525108
+                    }
+                  ],
+                  "selected_weight_sum": 0.6026630736887455,
+                  "top1_top2_margin": 0.12013470381498337
+                },
+                {
+                  "position": 17,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 12,
+                      "weight": 0.10836317390203476
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.09578173607587814
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.08955655246973038
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.08248470723628998
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.07762836664915085
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.06929574906826019
+                    }
+                  ],
+                  "selected_weight_sum": 0.5231102854013443,
+                  "top1_top2_margin": 0.012581437826156616
+                }
+              ]
+            },
+            {
+              "id": "code",
+              "label": "Python code",
+              "tokens": 19,
+              "load": [
+                1,
+                1,
+                2,
+                4,
+                3,
+                0,
+                4,
+                2,
+                0,
+                0,
+                0,
+                4,
+                2,
+                1,
+                5,
+                1,
+                1,
+                0,
+                2,
+                5,
+                2,
+                0,
+                3,
+                5,
+                3,
+                0,
+                0,
+                2,
+                3,
+                5,
+                2,
+                0,
+                1,
+                0,
+                4,
+                2,
+                1,
+                1,
+                3,
+                1,
+                0,
+                2,
+                2,
+                2,
+                2,
+                1,
+                1,
+                1,
+                3,
+                6,
+                0,
+                2,
+                1,
+                1,
+                0,
+                2,
+                3,
+                0,
+                3,
+                1,
+                5,
+                0,
+                0,
+                0
+              ],
+              "metrics": {
+                "routes": 114,
+                "used_experts": 47,
+                "zero_experts": 17,
+                "mean": 1.78125,
+                "std": 1.618040680885315,
+                "cv": 0.9083737134933472,
+                "gini": 0.4936951696872711,
+                "entropy_nats": 3.69284725189209,
+                "effective_experts": 40.15902719636842,
+                "min": 0.0,
+                "p50": 1.5,
+                "p90": 4.0,
+                "max": 6.0
+              },
+              "top_experts": [
+                {
+                  "expert": 49,
+                  "routes": 6
+                },
+                {
+                  "expert": 14,
+                  "routes": 5
+                },
+                {
+                  "expert": 19,
+                  "routes": 5
+                },
+                {
+                  "expert": 23,
+                  "routes": 5
+                },
+                {
+                  "expert": 29,
+                  "routes": 5
+                },
+                {
+                  "expert": 60,
+                  "routes": 5
+                },
+                {
+                  "expert": 3,
+                  "routes": 4
+                },
+                {
+                  "expert": 6,
+                  "routes": 4
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.6668257713317871
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.015490565448999405
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.012896407395601273
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.01244935393333435
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.012159599922597408
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.011999384500086308
+                    }
+                  ],
+                  "selected_weight_sum": 0.7318210825324059,
+                  "top1_top2_margin": 0.6513352058827877
+                },
+                {
+                  "position": 1,
+                  "token_id": 9081,
+                  "token": "Write",
+                  "decoded_piece": "Write",
+                  "experts_by_weight": [
+                    {
+                      "expert": 38,
+                      "weight": 0.2189263552427292
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.1511840671300888
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.05854736641049385
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05501477047801018
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.052550721913576126
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.04810547083616257
+                    }
+                  ],
+                  "selected_weight_sum": 0.5843287520110607,
+                  "top1_top2_margin": 0.06774228811264038
+                },
+                {
+                  "position": 2,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 38,
+                      "weight": 0.2076040804386139
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.10240605473518372
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.08799075335264206
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.04287009313702583
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.040391046553850174
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.03840652480721474
+                    }
+                  ],
+                  "selected_weight_sum": 0.5196685530245304,
+                  "top1_top2_margin": 0.10519802570343018
+                },
+                {
+                  "position": 3,
+                  "token_id": 12974,
+                  "token": "ĠPython",
+                  "decoded_piece": " Python",
+                  "experts_by_weight": [
+                    {
+                      "expert": 49,
+                      "weight": 0.20095491409301758
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.12679241597652435
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.056457847356796265
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.05235559493303299
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.04632683843374252
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.04559082165360451
+                    }
+                  ],
+                  "selected_weight_sum": 0.5284784324467182,
+                  "top1_top2_margin": 0.07416249811649323
+                },
+                {
+                  "position": 4,
+                  "token_id": 1157,
+                  "token": "Ġfunction",
+                  "decoded_piece": " function",
+                  "experts_by_weight": [
+                    {
+                      "expert": 49,
+                      "weight": 0.3816191852092743
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.07868051528930664
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.05729304254055023
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.043323855847120285
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.04145321995019913
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.026589658111333847
+                    }
+                  ],
+                  "selected_weight_sum": 0.6289594769477844,
+                  "top1_top2_margin": 0.30293866991996765
+                },
+                {
+                  "position": 5,
+                  "token_id": 344,
+                  "token": "Ġthat",
+                  "decoded_piece": " that",
+                  "experts_by_weight": [
+                    {
+                      "expert": 60,
+                      "weight": 0.2202168107032776
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.11744972318410873
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.09239882975816727
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03130319342017174
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.031124742701649666
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.02125764638185501
+                    }
+                  ],
+                  "selected_weight_sum": 0.51375094614923,
+                  "top1_top2_margin": 0.10276708751916885
+                },
+                {
+                  "position": 6,
+                  "token_id": 7578,
+                  "token": "Ġreturns",
+                  "decoded_piece": " returns",
+                  "experts_by_weight": [
+                    {
+                      "expert": 3,
+                      "weight": 0.12485012412071228
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.10983602702617645
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.09923596680164337
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.07021182030439377
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.0490649938583374
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.042169809341430664
+                    }
+                  ],
+                  "selected_weight_sum": 0.49536874145269394,
+                  "top1_top2_margin": 0.015014097094535828
+                },
+                {
+                  "position": 7,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 3,
+                      "weight": 0.14295123517513275
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.13394472002983093
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.04427902773022652
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.0420200414955616
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.03992121294140816
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.03541431576013565
+                    }
+                  ],
+                  "selected_weight_sum": 0.4385305531322956,
+                  "top1_top2_margin": 0.009006515145301819
+                },
+                {
+                  "position": 8,
+                  "token_id": 1022,
+                  "token": "Ġfirst",
+                  "decoded_piece": " first",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.10442701727151871
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.08264623582363129
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.049039799720048904
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.046509917825460434
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.029447149485349655
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.027899539098143578
+                    }
+                  ],
+                  "selected_weight_sum": 0.33996965922415257,
+                  "top1_top2_margin": 0.02178078144788742
+                },
+                {
+                  "position": 9,
+                  "token_id": 9842,
+                  "token": "Ġrepeated",
+                  "decoded_piece": " repeated",
+                  "experts_by_weight": [
+                    {
+                      "expert": 11,
+                      "weight": 0.23813141882419586
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.16236260533332825
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.07772964239120483
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.04792751371860504
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.04581587761640549
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04462626948952675
+                    }
+                  ],
+                  "selected_weight_sum": 0.6165933273732662,
+                  "top1_top2_margin": 0.07576881349086761
+                },
+                {
+                  "position": 10,
+                  "token_id": 3699,
+                  "token": "Ġelement",
+                  "decoded_piece": " element",
+                  "experts_by_weight": [
+                    {
+                      "expert": 28,
+                      "weight": 0.10746581852436066
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.081493079662323
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.07607961446046829
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.07128653675317764
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.05328736826777458
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.05184241756796837
+                    }
+                  ],
+                  "selected_weight_sum": 0.44145483523607254,
+                  "top1_top2_margin": 0.02597273886203766
+                },
+                {
+                  "position": 11,
+                  "token_id": 279,
+                  "token": "Ġin",
+                  "decoded_piece": " in",
+                  "experts_by_weight": [
+                    {
+                      "expert": 60,
+                      "weight": 0.1786193996667862
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.17806856334209442
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.09450752288103104
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.07455217838287354
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.02587931603193283
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.025224294513463974
+                    }
+                  ],
+                  "selected_weight_sum": 0.576851274818182,
+                  "top1_top2_margin": 0.0005508363246917725
+                },
+                {
+                  "position": 12,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 60,
+                      "weight": 0.1739369034767151
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.11336424946784973
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.07465675473213196
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.06730871647596359
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.052558671683073044
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.0453951470553875
+                    }
+                  ],
+                  "selected_weight_sum": 0.5272204428911209,
+                  "top1_top2_margin": 0.060572654008865356
+                },
+                {
+                  "position": 13,
+                  "token_id": 1525,
+                  "token": "Ġlist",
+                  "decoded_piece": " list",
+                  "experts_by_weight": [
+                    {
+                      "expert": 29,
+                      "weight": 0.12000925838947296
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.11664143949747086
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04033638536930084
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.038349755108356476
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.03568904474377632
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.031670890748500824
+                    }
+                  ],
+                  "selected_weight_sum": 0.3826967738568783,
+                  "top1_top2_margin": 0.0033678188920021057
+                },
+                {
+                  "position": 14,
+                  "token_id": 285,
+                  "token": "Ġand",
+                  "decoded_piece": " and",
+                  "experts_by_weight": [
+                    {
+                      "expert": 24,
+                      "weight": 0.09494847059249878
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.07503247261047363
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.07015768438577652
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.06664929538965225
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.06461586058139801
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.05943237245082855
+                    }
+                  ],
+                  "selected_weight_sum": 0.43083615601062775,
+                  "top1_top2_margin": 0.019915997982025146
+                },
+                {
+                  "position": 15,
+                  "token_id": 6716,
+                  "token": "Ġexplain",
+                  "decoded_piece": " explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 3,
+                      "weight": 0.1462646871805191
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.12913043797016144
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.06748204678297043
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.062301188707351685
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.04884091764688492
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.045381322503089905
+                    }
+                  ],
+                  "selected_weight_sum": 0.4994006007909775,
+                  "top1_top2_margin": 0.017134249210357666
+                },
+                {
+                  "position": 16,
+                  "token_id": 895,
+                  "token": "Ġits",
+                  "decoded_piece": " its",
+                  "experts_by_weight": [
+                    {
+                      "expert": 48,
+                      "weight": 0.10237778723239899
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.08468203991651535
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.07570596784353256
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.0703137144446373
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.038703132420778275
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.0378028005361557
+                    }
+                  ],
+                  "selected_weight_sum": 0.4095854423940182,
+                  "top1_top2_margin": 0.017695747315883636
+                },
+                {
+                  "position": 17,
+                  "token_id": 13913,
+                  "token": "Ġcomplexity",
+                  "decoded_piece": " complexity",
+                  "experts_by_weight": [
+                    {
+                      "expert": 37,
+                      "weight": 0.1097944900393486
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.0862223356962204
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.07457230985164642
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.05728883668780327
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.05647817999124527
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.037518858909606934
+                    }
+                  ],
+                  "selected_weight_sum": 0.4218750111758709,
+                  "top1_top2_margin": 0.023572154343128204
+                },
+                {
+                  "position": 18,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 12,
+                      "weight": 0.12260974943637848
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.11430303007364273
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.09481319785118103
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.0880577489733696
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.07388219237327576
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05371580645442009
+                    }
+                  ],
+                  "selected_weight_sum": 0.5473817251622677,
+                  "top1_top2_margin": 0.008306719362735748
+                }
+              ]
+            },
+            {
+              "id": "math",
+              "label": "数学推理",
+              "tokens": 27,
+              "load": [
+                2,
+                9,
+                0,
+                1,
+                4,
+                0,
+                0,
+                3,
+                4,
+                0,
+                6,
+                3,
+                3,
+                2,
+                19,
+                4,
+                3,
+                1,
+                3,
+                4,
+                2,
+                1,
+                1,
+                4,
+                2,
+                6,
+                1,
+                2,
+                4,
+                0,
+                0,
+                0,
+                0,
+                2,
+                7,
+                2,
+                4,
+                0,
+                1,
+                2,
+                2,
+                1,
+                5,
+                2,
+                2,
+                0,
+                0,
+                1,
+                1,
+                2,
+                0,
+                2,
+                2,
+                8,
+                2,
+                5,
+                3,
+                0,
+                0,
+                4,
+                0,
+                0,
+                2,
+                6
+              ],
+              "metrics": {
+                "routes": 162,
+                "used_experts": 48,
+                "zero_experts": 16,
+                "mean": 2.53125,
+                "std": 2.975928544998169,
+                "cv": 1.1756755113601685,
+                "gini": 0.5358796119689941,
+                "entropy_nats": 3.6021676063537598,
+                "effective_experts": 36.67765105006978,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 5.6999969482421875,
+                "max": 19.0
+              },
+              "top_experts": [
+                {
+                  "expert": 14,
+                  "routes": 19
+                },
+                {
+                  "expert": 1,
+                  "routes": 9
+                },
+                {
+                  "expert": 53,
+                  "routes": 8
+                },
+                {
+                  "expert": 34,
+                  "routes": 7
+                },
+                {
+                  "expert": 10,
+                  "routes": 6
+                },
+                {
+                  "expert": 25,
+                  "routes": 6
+                },
+                {
+                  "expert": 63,
+                  "routes": 6
+                },
+                {
+                  "expert": 42,
+                  "routes": 5
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.6668257713317871
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.015490565448999405
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.012896407395601273
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.01244935393333435
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.012159599922597408
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.011999384500086308
+                    }
+                  ],
+                  "selected_weight_sum": 0.7318210825324059,
+                  "top1_top2_margin": 0.6513352058827877
+                },
+                {
+                  "position": 1,
+                  "token_id": 8246,
+                  "token": "èĭ¥",
+                  "decoded_piece": "若",
+                  "experts_by_weight": [
+                    {
+                      "expert": 35,
+                      "weight": 0.2157912403345108
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.20943520963191986
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.05833398550748825
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.043179191648960114
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.03460429981350899
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.03122684173285961
+                    }
+                  ],
+                  "selected_weight_sum": 0.5925707686692476,
+                  "top1_top2_margin": 0.006356030702590942
+                },
+                {
+                  "position": 2,
+                  "token_id": 2206,
+                  "token": "æŃ£",
+                  "decoded_piece": "正",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.2386857569217682
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.07329999655485153
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.07202289253473282
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.038597580045461655
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.026481110602617264
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.022812768816947937
+                    }
+                  ],
+                  "selected_weight_sum": 0.4719001054763794,
+                  "top1_top2_margin": 0.16538576036691666
+                },
+                {
+                  "position": 3,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.09990332275629044
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.09463457763195038
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.07485338300466537
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.06988027691841125
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.04555794224143028
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.044183291494846344
+                    }
+                  ],
+                  "selected_weight_sum": 0.42901279404759407,
+                  "top1_top2_margin": 0.005268745124340057
+                },
+                {
+                  "position": 4,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.10638757795095444
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.09196751564741135
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.051636818796396255
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.05064014717936516
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.04116741195321083
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.03419582173228264
+                    }
+                  ],
+                  "selected_weight_sum": 0.37599529325962067,
+                  "top1_top2_margin": 0.01442006230354309
+                },
+                {
+                  "position": 5,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.9029526710510254
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.004224724601954222
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.0033379215747117996
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.003321886295452714
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.0029146261513233185
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.0027777589857578278
+                    }
+                  ],
+                  "selected_weight_sum": 0.9195295886602253,
+                  "top1_top2_margin": 0.8987279464490712
+                },
+                {
+                  "position": 6,
+                  "token_id": 13211,
+                  "token": "满足",
+                  "decoded_piece": "满足",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.28209275007247925
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.07677952200174332
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.061268191784620285
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.04783352091908455
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.0477428138256073
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.03472767025232315
+                    }
+                  ],
+                  "selected_weight_sum": 0.5504444688558578,
+                  "top1_top2_margin": 0.20531322807073593
+                },
+                {
+                  "position": 7,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 42,
+                      "weight": 0.15821927785873413
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.049939896911382675
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.0359053835272789
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.035540301352739334
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.03167761489748955
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.029953069984912872
+                    }
+                  ],
+                  "selected_weight_sum": 0.34123554453253746,
+                  "top1_top2_margin": 0.10827938094735146
+                },
+                {
+                  "position": 8,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 18,
+                      "weight": 0.13474272191524506
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.08470243215560913
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.06641152501106262
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.049692537635564804
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.04948724806308746
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.04123137518763542
+                    }
+                  ],
+                  "selected_weight_sum": 0.4262678399682045,
+                  "top1_top2_margin": 0.050040289759635925
+                },
+                {
+                  "position": 9,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.15503190457820892
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.11396767944097519
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.0618465431034565
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.03268251195549965
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.03129611164331436
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.02996181882917881
+                    }
+                  ],
+                  "selected_weight_sum": 0.42478656955063343,
+                  "top1_top2_margin": 0.041064225137233734
+                },
+                {
+                  "position": 10,
+                  "token_id": 12985,
+                  "token": "å®ĥçļĦ",
+                  "decoded_piece": "它的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.12337606400251389
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.10133391618728638
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.07631479203701019
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.06342805176973343
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.06302573531866074
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.046065572649240494
+                    }
+                  ],
+                  "selected_weight_sum": 0.4735441319644451,
+                  "top1_top2_margin": 0.02204214781522751
+                },
+                {
+                  "position": 11,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.1234094426035881
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.0897468701004982
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.06823419779539108
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.06477421522140503
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.04059724509716034
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.03771129250526428
+                    }
+                  ],
+                  "selected_weight_sum": 0.42447326332330704,
+                  "top1_top2_margin": 0.033662572503089905
+                },
+                {
+                  "position": 12,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 19,
+                      "weight": 0.09373485296964645
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.08521629124879837
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.079887256026268
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.07751046121120453
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.07490264624357224
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.05471431463956833
+                    }
+                  ],
+                  "selected_weight_sum": 0.4659658223390579,
+                  "top1_top2_margin": 0.008518561720848083
+                },
+                {
+                  "position": 13,
+                  "token_id": 37360,
+                  "token": "çŃīäºİ",
+                  "decoded_piece": "等于",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.180343896150589
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.11404763162136078
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.06184849888086319
+                    },
+                    {
+                      "expert": 62,
+                      "weight": 0.06134380027651787
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.04147454723715782
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.029968442395329475
+                    }
+                  ],
+                  "selected_weight_sum": 0.4890268165618181,
+                  "top1_top2_margin": 0.06629626452922821
+                },
+                {
+                  "position": 14,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 18,
+                      "weight": 0.14621861279010773
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.13439302146434784
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.07689664512872696
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.05354950577020645
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.05216725543141365
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.0497673936188221
+                    }
+                  ],
+                  "selected_weight_sum": 0.5129924342036247,
+                  "top1_top2_margin": 0.011825591325759888
+                },
+                {
+                  "position": 15,
+                  "token_id": 18,
+                  "token": "3",
+                  "decoded_piece": "3",
+                  "experts_by_weight": [
+                    {
+                      "expert": 1,
+                      "weight": 0.12946932017803192
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.10507825016975403
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.07482244819402695
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.042674217373132706
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.02928566187620163
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.027979303151369095
+                    }
+                  ],
+                  "selected_weight_sum": 0.4093092009425163,
+                  "top1_top2_margin": 0.024391070008277893
+                },
+                {
+                  "position": 16,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.17965057492256165
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.10295615345239639
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.088253453373909
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.07324301451444626
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.058590613305568695
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.044678375124931335
+                    }
+                  ],
+                  "selected_weight_sum": 0.5473721846938133,
+                  "top1_top2_margin": 0.07669442147016525
+                },
+                {
+                  "position": 17,
+                  "token_id": 2989,
+                  "token": "æ±Ĥ",
+                  "decoded_piece": "求",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.15167519450187683
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.07489567995071411
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.05545367673039436
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.05460234358906746
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.05284403637051582
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.04110542684793472
+                    }
+                  ],
+                  "selected_weight_sum": 0.4305763579905033,
+                  "top1_top2_margin": 0.07677951455116272
+                },
+                {
+                  "position": 18,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 42,
+                      "weight": 0.1560184359550476
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.039979103952646255
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.039141543209552765
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.03823362663388252
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.0362531840801239
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.034050628542900085
+                    }
+                  ],
+                  "selected_weight_sum": 0.34367652237415314,
+                  "top1_top2_margin": 0.11603933200240135
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 18,
+                      "weight": 0.08407339453697205
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.08105730265378952
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.07007438689470291
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.06364663690328598
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.045173339545726776
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.037717703729867935
+                    }
+                  ],
+                  "selected_weight_sum": 0.38174276426434517,
+                  "top1_top2_margin": 0.0030160918831825256
+                },
+                {
+                  "position": 20,
+                  "token_id": 337,
+                  "token": "çļĦ",
+                  "decoded_piece": "的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.1197085976600647
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.0795862078666687
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.06330224871635437
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.057824112474918365
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.03476481884717941
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.030369188636541367
+                    }
+                  ],
+                  "selected_weight_sum": 0.3855551742017269,
+                  "top1_top2_margin": 0.040122389793395996
+                },
+                {
+                  "position": 21,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 53,
+                      "weight": 0.11117389798164368
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.07823505997657776
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.06898786872625351
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.049189817160367966
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.0485650971531868
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.036493945866823196
+                    }
+                  ],
+                  "selected_weight_sum": 0.3926456868648529,
+                  "top1_top2_margin": 0.03293883800506592
+                },
+                {
+                  "position": 22,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.1784549057483673
+                    },
+                    {
+                      "expert": 10,
+                      "weight": 0.06671992689371109
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.05699232220649719
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.04226744547486305
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.03548656776547432
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.03004826046526432
+                    }
+                  ],
+                  "selected_weight_sum": 0.4099694285541773,
+                  "top1_top2_margin": 0.11173497885465622
+                },
+                {
+                  "position": 23,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 14,
+                      "weight": 0.07740753144025803
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.06655178219079971
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.0630197674036026
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.05319470912218094
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.05203474313020706
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.03889422491192818
+                    }
+                  ],
+                  "selected_weight_sum": 0.3511027581989765,
+                  "top1_top2_margin": 0.010855749249458313
+                },
+                {
+                  "position": 24,
+                  "token_id": 51971,
+                  "token": "æķ°çļĦ",
+                  "decoded_piece": "数的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 1,
+                      "weight": 0.08456672728061676
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.08250632137060165
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.06846188753843307
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.06562276184558868
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.062368445098400116
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04836544767022133
+                    }
+                  ],
+                  "selected_weight_sum": 0.4118915908038616,
+                  "top1_top2_margin": 0.002060405910015106
+                },
+                {
+                  "position": 25,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 1,
+                      "weight": 0.10623366385698318
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.08463753014802933
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.05453135445713997
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.051909394562244415
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.04595055803656578
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.040663786232471466
+                    }
+                  ],
+                  "selected_weight_sum": 0.38392628729343414,
+                  "top1_top2_margin": 0.021596133708953857
+                },
+                {
+                  "position": 26,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 12,
+                      "weight": 0.12392057478427887
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.1172504797577858
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.0955967903137207
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.08561467379331589
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.05499466508626938
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.05483439192175865
+                    }
+                  ],
+                  "selected_weight_sum": 0.5322115756571293,
+                  "top1_top2_margin": 0.0066700950264930725
+                }
+              ]
+            }
+          ],
+          "prompt_pair_jaccard": [
+            {
+              "left": "zh_explanation",
+              "right": "en_architecture",
+              "used_expert_jaccard": 0.7719298245614035,
+              "shared_experts": 44,
+              "union_experts": 57
+            },
+            {
+              "left": "zh_explanation",
+              "right": "code",
+              "used_expert_jaccard": 0.7586206896551724,
+              "shared_experts": 44,
+              "union_experts": 58
+            },
+            {
+              "left": "zh_explanation",
+              "right": "math",
+              "used_expert_jaccard": 0.7457627118644068,
+              "shared_experts": 44,
+              "union_experts": 59
+            },
+            {
+              "left": "en_architecture",
+              "right": "code",
+              "used_expert_jaccard": 0.6909090909090909,
+              "shared_experts": 38,
+              "union_experts": 55
+            },
+            {
+              "left": "en_architecture",
+              "right": "math",
+              "used_expert_jaccard": 0.6491228070175439,
+              "shared_experts": 37,
+              "union_experts": 57
+            },
+            {
+              "left": "code",
+              "right": "math",
+              "used_expert_jaccard": 0.6379310344827587,
+              "shared_experts": 37,
+              "union_experts": 58
+            }
+          ]
+        }
+      },
+      {
+        "layer": 6,
+        "ffn": "moe",
+        "state_tensors": 203,
+        "state_numel": 584847872,
+        "state_bytes": 1169695744,
+        "timing_ms": {
+          "load_cpu": 24.91675701458007,
+          "transfer_to_device": 123.45660501159728,
+          "forward": 3.9505639870185405
+        },
+        "peak_device_allocated_mib": 1179.1787109375,
+        "hidden_before": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 3.2100954055786133,
+            "p10": 0.22690242528915405,
+            "p50": 0.26399680972099304,
+            "p90": 0.33678531646728516
+          },
+          "l2_norm": {
+            "mean": 145.2723388671875,
+            "p10": 10.268431663513184,
+            "p50": 11.947131156921387,
+            "p90": 15.24116325378418
+          },
+          "finite": true,
+          "sha256_fp32": "88f158b68ecb980c55b1b77eebfeb2276d7820443732aa10effca9c85c653481"
+        },
+        "hidden_after": {
+          "valid_tokens": 90,
+          "hidden_rms": {
+            "mean": 3.3090813159942627,
+            "p10": 0.2696530222892761,
+            "p50": 0.31665489077568054,
+            "p90": 0.3847275674343109
+          },
+          "l2_norm": {
+            "mean": 149.75193786621094,
+            "p10": 12.203103065490723,
+            "p50": 14.330163955688477,
+            "p90": 17.410781860351562
+          },
+          "finite": true,
+          "sha256_fp32": "1ffd41521b5a32af38695b767a5ecc556ddeae1e1efe5339743e5c8c7644030e"
+        },
+        "mla": {
+          "compressed_projection_shape": [
+            4,
+            27,
+            576
+          ],
+          "latent_shape": [
+            4,
+            27,
+            512
+          ],
+          "rope_key_shape": [
+            4,
+            27,
+            64
+          ],
+          "latent_rms": 2.3470914363861084,
+          "rope_key_rms": 2.1612274646759033,
+          "eager_key_cache_shape": [
+            4,
+            16,
+            27,
+            192
+          ],
+          "eager_value_cache_shape": [
+            4,
+            16,
+            27,
+            128
+          ],
+          "eager_cache_bytes": 1105920
+        },
+        "routing": {
+          "aggregate_load": [
+            16,
+            3,
+            14,
+            4,
+            25,
+            9,
+            7,
+            14,
+            11,
+            5,
+            0,
+            3,
+            9,
+            5,
+            2,
+            6,
+            10,
+            17,
+            6,
+            10,
+            4,
+            7,
+            6,
+            5,
+            2,
+            40,
+            7,
+            6,
+            16,
+            10,
+            11,
+            9,
+            7,
+            2,
+            6,
+            1,
+            19,
+            16,
+            3,
+            8,
+            10,
+            9,
+            10,
+            14,
+            8,
+            6,
+            10,
+            8,
+            2,
+            5,
+            7,
+            2,
+            11,
+            5,
+            2,
+            6,
+            17,
+            14,
+            11,
+            8,
+            7,
+            0,
+            0,
+            7
+          ],
+          "aggregate_metrics": {
+            "routes": 540,
+            "used_experts": 61,
+            "zero_experts": 3,
+            "mean": 8.4375,
+            "std": 6.458352565765381,
+            "cv": 0.7654343843460083,
+            "gini": 0.3778356611728668,
+            "entropy_nats": 3.9015467166900635,
+            "effective_experts": 49.47891982211325,
+            "min": 0.0,
+            "p50": 7.0,
+            "p90": 16.0,
+            "max": 40.0
+          },
+          "top_experts": [
+            {
+              "expert": 25,
+              "routes": 40
+            },
+            {
+              "expert": 4,
+              "routes": 25
+            },
+            {
+              "expert": 36,
+              "routes": 19
+            },
+            {
+              "expert": 17,
+              "routes": 17
+            },
+            {
+              "expert": 56,
+              "routes": 17
+            },
+            {
+              "expert": 0,
+              "routes": 16
+            },
+            {
+              "expert": 28,
+              "routes": 16
+            },
+            {
+              "expert": 37,
+              "routes": 16
+            },
+            {
+              "expert": 2,
+              "routes": 14
+            },
+            {
+              "expert": 7,
+              "routes": 14
+            },
+            {
+              "expert": 43,
+              "routes": 14
+            },
+            {
+              "expert": 57,
+              "routes": 14
+            }
+          ],
+          "selected_weight_sum": {
+            "mean": 0.44403982162475586,
+            "p10": 0.332468181848526,
+            "p50": 0.425670325756073,
+            "p90": 0.571208655834198
+          },
+          "top1_top2_margin": {
+            "mean": 0.053601086139678955,
+            "p50": 0.026227694004774094,
+            "p90": 0.07640297710895538
+          },
+          "per_prompt": [
+            {
+              "id": "zh_explanation",
+              "label": "中文解释",
+              "tokens": 26,
+              "load": [
+                7,
+                1,
+                6,
+                3,
+                1,
+                3,
+                1,
+                3,
+                6,
+                0,
+                0,
+                0,
+                4,
+                1,
+                0,
+                1,
+                3,
+                4,
+                3,
+                2,
+                0,
+                2,
+                2,
+                1,
+                1,
+                13,
+                2,
+                1,
+                7,
+                5,
+                1,
+                1,
+                1,
+                1,
+                1,
+                0,
+                5,
+                3,
+                0,
+                2,
+                2,
+                2,
+                3,
+                8,
+                2,
+                2,
+                4,
+                4,
+                0,
+                0,
+                1,
+                0,
+                5,
+                3,
+                1,
+                0,
+                6,
+                5,
+                4,
+                0,
+                6,
+                0,
+                0,
+                0
+              ],
+              "metrics": {
+                "routes": 156,
+                "used_experts": 49,
+                "zero_experts": 15,
+                "mean": 2.4375,
+                "std": 2.5126662254333496,
+                "cv": 1.0308374166488647,
+                "gini": 0.5288461446762085,
+                "entropy_nats": 3.646120071411133,
+                "effective_experts": 38.325676313953316,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 6.0,
+                "max": 13.0
+              },
+              "top_experts": [
+                {
+                  "expert": 25,
+                  "routes": 13
+                },
+                {
+                  "expert": 43,
+                  "routes": 8
+                },
+                {
+                  "expert": 0,
+                  "routes": 7
+                },
+                {
+                  "expert": 28,
+                  "routes": 7
+                },
+                {
+                  "expert": 2,
+                  "routes": 6
+                },
+                {
+                  "expert": 8,
+                  "routes": 6
+                },
+                {
+                  "expert": 56,
+                  "routes": 6
+                },
+                {
+                  "expert": 60,
+                  "routes": 6
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.5437576770782471
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.02887026220560074
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.022947292774915695
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.020739872008562088
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.019217059016227722
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.01740080676972866
+                    }
+                  ],
+                  "selected_weight_sum": 0.652932969853282,
+                  "top1_top2_margin": 0.5148874148726463
+                },
+                {
+                  "position": 1,
+                  "token_id": 1082,
+                  "token": "çĶ¨",
+                  "decoded_piece": "用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.19940678775310516
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.08581104874610901
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.07319452613592148
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.05708487331867218
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.04857153818011284
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.03971230611205101
+                    }
+                  ],
+                  "selected_weight_sum": 0.5037810802459717,
+                  "top1_top2_margin": 0.11359573900699615
+                },
+                {
+                  "position": 2,
+                  "token_id": 83983,
+                  "token": "éĢļä¿Ĺ",
+                  "decoded_piece": "通俗",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.0797773152589798
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.07066472619771957
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.05039288476109505
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.047804661095142365
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.04701150208711624
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.03956286236643791
+                    }
+                  ],
+                  "selected_weight_sum": 0.33521395176649094,
+                  "top1_top2_margin": 0.009112589061260223
+                },
+                {
+                  "position": 3,
+                  "token_id": 91476,
+                  "token": "çļĦè¯Ńè¨Ģ",
+                  "decoded_piece": "的语言",
+                  "experts_by_weight": [
+                    {
+                      "expert": 2,
+                      "weight": 0.20149613916873932
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.06121385097503662
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.045187439769506454
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.04054460674524307
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.04029814898967743
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.038398370146751404
+                    }
+                  ],
+                  "selected_weight_sum": 0.4271385557949543,
+                  "top1_top2_margin": 0.1402822881937027
+                },
+                {
+                  "position": 4,
+                  "token_id": 17882,
+                  "token": "解éĩĬ",
+                  "decoded_piece": "解释",
+                  "experts_by_weight": [
+                    {
+                      "expert": 44,
+                      "weight": 0.08854366093873978
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.07951278984546661
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.06590777635574341
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.06553405523300171
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.055855643004179
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.038224026560783386
+                    }
+                  ],
+                  "selected_weight_sum": 0.3935779519379139,
+                  "top1_top2_margin": 0.009030871093273163
+                },
+                {
+                  "position": 5,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.13611505925655365
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.1061541959643364
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.10146880894899368
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.0781978890299797
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.07308310270309448
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.058526765555143356
+                    }
+                  ],
+                  "selected_weight_sum": 0.5535458214581013,
+                  "top1_top2_margin": 0.029960863292217255
+                },
+                {
+                  "position": 6,
+                  "token_id": 10285,
+                  "token": "为ä»Ģä¹Ī",
+                  "decoded_piece": "为什么",
+                  "experts_by_weight": [
+                    {
+                      "expert": 5,
+                      "weight": 0.12138903886079788
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.0896969884634018
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.07434116303920746
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.043124958872795105
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.042676858603954315
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.04177377000451088
+                    }
+                  ],
+                  "selected_weight_sum": 0.41300277784466743,
+                  "top1_top2_margin": 0.03169205039739609
+                },
+                {
+                  "position": 7,
+                  "token_id": 21844,
+                  "token": "ç¨Ģ",
+                  "decoded_piece": "稀",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.06200406327843666
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05429893359541893
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.04909799247980118
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.047943904995918274
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.03833471238613129
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03801535442471504
+                    }
+                  ],
+                  "selected_weight_sum": 0.28969496116042137,
+                  "top1_top2_margin": 0.007705129683017731
+                },
+                {
+                  "position": 8,
+                  "token_id": 20923,
+                  "token": "çĸı",
+                  "decoded_piece": "疏",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.17407171428203583
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.1118895411491394
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.09100628644227982
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.08462925255298615
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.04508446529507637
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.03972841799259186
+                    }
+                  ],
+                  "selected_weight_sum": 0.5464096777141094,
+                  "top1_top2_margin": 0.06218217313289642
+                },
+                {
+                  "position": 9,
+                  "token_id": 10900,
+                  "token": "ä¸ĵ家",
+                  "decoded_piece": "专家",
+                  "experts_by_weight": [
+                    {
+                      "expert": 47,
+                      "weight": 0.10310626029968262
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.08589243143796921
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.05800008028745651
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.05487441271543503
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.052628058940172195
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.04682231321930885
+                    }
+                  ],
+                  "selected_weight_sum": 0.4013235569000244,
+                  "top1_top2_margin": 0.01721382886171341
+                },
+                {
+                  "position": 10,
+                  "token_id": 39322,
+                  "token": "模åŀĭ",
+                  "decoded_piece": "模型",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.14891529083251953
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.12191086262464523
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.0684787854552269
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.06209414824843407
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.0608627125620842
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.05374256893992424
+                    }
+                  ],
+                  "selected_weight_sum": 0.5160043686628342,
+                  "top1_top2_margin": 0.027004428207874298
+                },
+                {
+                  "position": 11,
+                  "token_id": 1876,
+                  "token": "åı¯ä»¥",
+                  "decoded_piece": "可以",
+                  "experts_by_weight": [
+                    {
+                      "expert": 43,
+                      "weight": 0.09292622655630112
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.06740861386060715
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.06254689395427704
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.04910411313176155
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03792039304971695
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.03512315824627876
+                    }
+                  ],
+                  "selected_weight_sum": 0.34502939879894257,
+                  "top1_top2_margin": 0.02551761269569397
+                },
+                {
+                  "position": 12,
+                  "token_id": 9476,
+                  "token": "æĭ¥æľī",
+                  "decoded_piece": "拥有",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.10773254930973053
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.09062737226486206
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.08368217200040817
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.07159384340047836
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.04937493056058884
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.024813424795866013
+                    }
+                  ],
+                  "selected_weight_sum": 0.427824292331934,
+                  "top1_top2_margin": 0.01710517704486847
+                },
+                {
+                  "position": 13,
+                  "token_id": 3599,
+                  "token": "å¾Īå¤ļ",
+                  "decoded_piece": "很多",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.1371069699525833
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.08427796512842178
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.08417855203151703
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.04595799744129181
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.045302487909793854
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.041003432124853134
+                    }
+                  ],
+                  "selected_weight_sum": 0.4378274045884609,
+                  "top1_top2_margin": 0.05282900482416153
+                },
+                {
+                  "position": 14,
+                  "token_id": 21273,
+                  "token": "åıĤæķ°",
+                  "decoded_piece": "参数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.13282836973667145
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.09014931321144104
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.0724765732884407
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.0615108460187912
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.05170028656721115
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.03531194105744362
+                    }
+                  ],
+                  "selected_weight_sum": 0.44397732987999916,
+                  "top1_top2_margin": 0.04267905652523041
+                },
+                {
+                  "position": 15,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.11967949569225311
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.10622544586658478
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.08323302865028381
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.07838869094848633
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.06008601188659668
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.05834601819515228
+                    }
+                  ],
+                  "selected_weight_sum": 0.505958691239357,
+                  "top1_top2_margin": 0.013454049825668335
+                },
+                {
+                  "position": 16,
+                  "token_id": 1705,
+                  "token": "ä½Ĩ",
+                  "decoded_piece": "但",
+                  "experts_by_weight": [
+                    {
+                      "expert": 18,
+                      "weight": 0.09913211315870285
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.09904447197914124
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.054669421166181564
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.054521091282367706
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.04755903407931328
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.03531811386346817
+                    }
+                  ],
+                  "selected_weight_sum": 0.3902442455291748,
+                  "top1_top2_margin": 8.764117956161499e-05
+                },
+                {
+                  "position": 17,
+                  "token_id": 13712,
+                  "token": "æ¯ı个",
+                  "decoded_piece": "每个",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.08446930348873138
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.07750283181667328
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.06822194904088974
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.0491325743496418
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.037747886031866074
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.0376180075109005
+                    }
+                  ],
+                  "selected_weight_sum": 0.3546925522387028,
+                  "top1_top2_margin": 0.0069664716720581055
+                },
+                {
+                  "position": 18,
+                  "token_id": 10728,
+                  "token": "Ġtoken",
+                  "decoded_piece": " token",
+                  "experts_by_weight": [
+                    {
+                      "expert": 28,
+                      "weight": 0.1324920803308487
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.1233772411942482
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.06866113841533661
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.047023892402648926
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.044400230050086975
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.033736228942871094
+                    }
+                  ],
+                  "selected_weight_sum": 0.4496908113360405,
+                  "top1_top2_margin": 0.009114839136600494
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.048708073794841766
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.038140591233968735
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.03512806445360184
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.03305252268910408
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.029746539890766144
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.029309971258044243
+                    }
+                  ],
+                  "selected_weight_sum": 0.2140857633203268,
+                  "top1_top2_margin": 0.010567482560873032
+                },
+                {
+                  "position": 20,
+                  "token_id": 2141,
+                  "token": "åıª",
+                  "decoded_piece": "只",
+                  "experts_by_weight": [
+                    {
+                      "expert": 28,
+                      "weight": 0.09421364217996597
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.08210399001836777
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.0750744491815567
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.05757107213139534
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.0511380210518837
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.031326521188020706
+                    }
+                  ],
+                  "selected_weight_sum": 0.3914276957511902,
+                  "top1_top2_margin": 0.012109652161598206
+                },
+                {
+                  "position": 21,
+                  "token_id": 5118,
+                  "token": "使çĶ¨",
+                  "decoded_piece": "使用",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.13225959241390228
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.08616293966770172
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.05643099546432495
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.0435882993042469
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.03772665560245514
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.03494388610124588
+                    }
+                  ],
+                  "selected_weight_sum": 0.3911123685538769,
+                  "top1_top2_margin": 0.04609665274620056
+                },
+                {
+                  "position": 22,
+                  "token_id": 6802,
+                  "token": "åħ¶ä¸Ń",
+                  "decoded_piece": "其中",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.09671742469072342
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.08142291009426117
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.08033161610364914
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.06336305290460587
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.05350400134921074
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.04034680873155594
+                    }
+                  ],
+                  "selected_weight_sum": 0.41568581387400627,
+                  "top1_top2_margin": 0.01529451459646225
+                },
+                {
+                  "position": 23,
+                  "token_id": 51905,
+                  "token": "ä¸Ģå°ı",
+                  "decoded_piece": "一小",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.1243104413151741
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.09060382097959518
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.07431431114673615
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.06538720428943634
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.04744257777929306
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.032568901777267456
+                    }
+                  ],
+                  "selected_weight_sum": 0.4346272572875023,
+                  "top1_top2_margin": 0.03370662033557892
+                },
+                {
+                  "position": 24,
+                  "token_id": 5781,
+                  "token": "éĥ¨åĪĨ",
+                  "decoded_piece": "部分",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.139021098613739
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.0859280452132225
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.07490529119968414
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.07370275259017944
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.04788759723305702
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.029776398092508316
+                    }
+                  ],
+                  "selected_weight_sum": 0.45122118294239044,
+                  "top1_top2_margin": 0.05309305340051651
+                },
+                {
+                  "position": 25,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.12179290503263474
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.09725924581289291
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.09423467516899109
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.09347380697727203
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.07409657537937164
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.07257192581892014
+                    }
+                  ],
+                  "selected_weight_sum": 0.5534291341900826,
+                  "top1_top2_margin": 0.02453365921974182
+                }
+              ]
+            },
+            {
+              "id": "en_architecture",
+              "label": "English architecture",
+              "tokens": 18,
+              "load": [
+                3,
+                0,
+                5,
+                0,
+                7,
+                2,
+                0,
+                4,
+                1,
+                0,
+                0,
+                0,
+                1,
+                1,
+                2,
+                0,
+                1,
+                8,
+                1,
+                2,
+                0,
+                1,
+                1,
+                1,
+                0,
+                12,
+                0,
+                0,
+                0,
+                1,
+                10,
+                3,
+                2,
+                0,
+                0,
+                0,
+                3,
+                3,
+                3,
+                3,
+                0,
+                1,
+                1,
+                2,
+                1,
+                1,
+                1,
+                1,
+                1,
+                1,
+                1,
+                0,
+                2,
+                2,
+                0,
+                1,
+                2,
+                7,
+                1,
+                1,
+                0,
+                0,
+                0,
+                0
+              ],
+              "metrics": {
+                "routes": 108,
+                "used_experts": 42,
+                "zero_experts": 22,
+                "mean": 1.6875,
+                "std": 2.429154872894287,
+                "cv": 1.4394991397857666,
+                "gini": 0.6342592835426331,
+                "entropy_nats": 3.3749022483825684,
+                "effective_experts": 29.221427199847042,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 3.6999969482421875,
+                "max": 12.0
+              },
+              "top_experts": [
+                {
+                  "expert": 25,
+                  "routes": 12
+                },
+                {
+                  "expert": 30,
+                  "routes": 10
+                },
+                {
+                  "expert": 17,
+                  "routes": 8
+                },
+                {
+                  "expert": 4,
+                  "routes": 7
+                },
+                {
+                  "expert": 57,
+                  "routes": 7
+                },
+                {
+                  "expert": 2,
+                  "routes": 5
+                },
+                {
+                  "expert": 7,
+                  "routes": 4
+                },
+                {
+                  "expert": 0,
+                  "routes": 3
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.5437576770782471
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.02887026220560074
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.022947292774915695
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.020739872008562088
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.019217059016227722
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.01740080676972866
+                    }
+                  ],
+                  "selected_weight_sum": 0.652932969853282,
+                  "top1_top2_margin": 0.5148874148726463
+                },
+                {
+                  "position": 1,
+                  "token_id": 99297,
+                  "token": "Explain",
+                  "decoded_piece": "Explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 19,
+                      "weight": 0.10629317164421082
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.09721477329730988
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.09492044895887375
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.05780201405286789
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04771720618009567
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.041242267936468124
+                    }
+                  ],
+                  "selected_weight_sum": 0.4451898820698261,
+                  "top1_top2_margin": 0.00907839834690094
+                },
+                {
+                  "position": 2,
+                  "token_id": 946,
+                  "token": "Ġhow",
+                  "decoded_piece": " how",
+                  "experts_by_weight": [
+                    {
+                      "expert": 19,
+                      "weight": 0.12368354946374893
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.09157591313123703
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.05463002249598503
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.050672709941864014
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.047128066420555115
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03596038371324539
+                    }
+                  ],
+                  "selected_weight_sum": 0.4036506451666355,
+                  "top1_top2_margin": 0.0321076363325119
+                },
+                {
+                  "position": 3,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 52,
+                      "weight": 0.0887380987405777
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.08550800383090973
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.07985977083444595
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.06503360718488693
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.04625506326556206
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.0421152301132679
+                    }
+                  ],
+                  "selected_weight_sum": 0.40750977396965027,
+                  "top1_top2_margin": 0.0032300949096679688
+                },
+                {
+                  "position": 4,
+                  "token_id": 29640,
+                  "token": "Ġcompressed",
+                  "decoded_piece": " compressed",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.1338948905467987
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.08762071281671524
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.0809745118021965
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.0679486021399498
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.06139545142650604
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.05613873153924942
+                    }
+                  ],
+                  "selected_weight_sum": 0.4879729002714157,
+                  "top1_top2_margin": 0.046274177730083466
+                },
+                {
+                  "position": 5,
+                  "token_id": 2131,
+                  "token": "Ġkey",
+                  "decoded_piece": " key",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.12676671147346497
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.11693397909402847
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.10827948153018951
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.07470021396875381
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.05182885378599167
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.04812060296535492
+                    }
+                  ],
+                  "selected_weight_sum": 0.5266298428177834,
+                  "top1_top2_margin": 0.009832732379436493
+                },
+                {
+                  "position": 6,
+                  "token_id": 12,
+                  "token": "-",
+                  "decoded_piece": "-",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.06554465740919113
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.05510614812374115
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.04951103776693344
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.04858021065592766
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.0470929853618145
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04168501868844032
+                    }
+                  ],
+                  "selected_weight_sum": 0.3075200580060482,
+                  "top1_top2_margin": 0.010438509285449982
+                },
+                {
+                  "position": 7,
+                  "token_id": 2834,
+                  "token": "value",
+                  "decoded_piece": "value",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.1504407376050949
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.10922424495220184
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.08334412425756454
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.06348013132810593
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.05469481274485588
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.05172499269247055
+                    }
+                  ],
+                  "selected_weight_sum": 0.5129090435802937,
+                  "top1_top2_margin": 0.041216492652893066
+                },
+                {
+                  "position": 8,
+                  "token_id": 28937,
+                  "token": "Ġlatent",
+                  "decoded_piece": " latent",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.15120594203472137
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.09751661121845245
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.07147576659917831
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.05824965238571167
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.05139882490038872
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.04974324256181717
+                    }
+                  ],
+                  "selected_weight_sum": 0.4795900397002697,
+                  "top1_top2_margin": 0.05368933081626892
+                },
+                {
+                  "position": 9,
+                  "token_id": 4183,
+                  "token": "Ġchanges",
+                  "decoded_piece": " changes",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.11196812242269516
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.09889062494039536
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.06677774339914322
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.044559650123119354
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.04172351956367493
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.037076983600854874
+                    }
+                  ],
+                  "selected_weight_sum": 0.4009966440498829,
+                  "top1_top2_margin": 0.013077497482299805
+                },
+                {
+                  "position": 10,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.1058885008096695
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.07895072549581528
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.06406665593385696
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.05847375839948654
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.054556623101234436
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.034156013280153275
+                    }
+                  ],
+                  "selected_weight_sum": 0.396092277020216,
+                  "top1_top2_margin": 0.026937775313854218
+                },
+                {
+                  "position": 11,
+                  "token_id": 4887,
+                  "token": "Ġmemory",
+                  "decoded_piece": " memory",
+                  "experts_by_weight": [
+                    {
+                      "expert": 30,
+                      "weight": 0.09737883508205414
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.08197834342718124
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.05927024781703949
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.04981620982289314
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.04698258265852928
+                    },
+                    {
+                      "expert": 38,
+                      "weight": 0.04634204879403114
+                    }
+                  ],
+                  "selected_weight_sum": 0.38176826760172844,
+                  "top1_top2_margin": 0.015400491654872894
+                },
+                {
+                  "position": 12,
+                  "token_id": 2295,
+                  "token": "Ġcost",
+                  "decoded_piece": " cost",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.15634430944919586
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.12142481654882431
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.11154589802026749
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.09600093215703964
+                    },
+                    {
+                      "expert": 53,
+                      "weight": 0.028693636879324913
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.027443602681159973
+                    }
+                  ],
+                  "selected_weight_sum": 0.5414531957358122,
+                  "top1_top2_margin": 0.03491949290037155
+                },
+                {
+                  "position": 13,
+                  "token_id": 280,
+                  "token": "Ġof",
+                  "decoded_piece": " of",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.10794661939144135
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.0671105906367302
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.05974861606955528
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.05525089055299759
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.04587714374065399
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.03875768184661865
+                    }
+                  ],
+                  "selected_weight_sum": 0.37469154223799706,
+                  "top1_top2_margin": 0.04083602875471115
+                },
+                {
+                  "position": 14,
+                  "token_id": 77939,
+                  "token": "Ġautore",
+                  "decoded_piece": " autore",
+                  "experts_by_weight": [
+                    {
+                      "expert": 57,
+                      "weight": 0.06619898229837418
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.05084220692515373
+                    },
+                    {
+                      "expert": 14,
+                      "weight": 0.03945222496986389
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.038507360965013504
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.03479800373315811
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.03332655876874924
+                    }
+                  ],
+                  "selected_weight_sum": 0.26312533766031265,
+                  "top1_top2_margin": 0.015356775373220444
+                },
+                {
+                  "position": 15,
+                  "token_id": 79184,
+                  "token": "gressive",
+                  "decoded_piece": "gressive",
+                  "experts_by_weight": [
+                    {
+                      "expert": 17,
+                      "weight": 0.21340444684028625
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.14111490547657013
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.07872821390628815
+                    },
+                    {
+                      "expert": 30,
+                      "weight": 0.04866563901305199
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.04710451513528824
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.02478797361254692
+                    }
+                  ],
+                  "selected_weight_sum": 0.5538056939840317,
+                  "top1_top2_margin": 0.07228954136371613
+                },
+                {
+                  "position": 16,
+                  "token_id": 39671,
+                  "token": "Ġdecoding",
+                  "decoded_piece": " decoding",
+                  "experts_by_weight": [
+                    {
+                      "expert": 30,
+                      "weight": 0.19381695985794067
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.16847439110279083
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.10773279517889023
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.06173955276608467
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.038706641644239426
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.027917681261897087
+                    }
+                  ],
+                  "selected_weight_sum": 0.5983880218118429,
+                  "top1_top2_margin": 0.02534256875514984
+                },
+                {
+                  "position": 17,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.12360352277755737
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.1027035191655159
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.09440722316503525
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.09196682274341583
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.08447214215993881
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.08360416442155838
+                    }
+                  ],
+                  "selected_weight_sum": 0.5807573944330215,
+                  "top1_top2_margin": 0.020900003612041473
+                }
+              ]
+            },
+            {
+              "id": "code",
+              "label": "Python code",
+              "tokens": 19,
+              "load": [
+                0,
+                0,
+                3,
+                1,
+                12,
+                3,
+                0,
+                5,
+                4,
+                5,
+                0,
+                1,
+                2,
+                1,
+                0,
+                4,
+                2,
+                2,
+                1,
+                4,
+                0,
+                1,
+                1,
+                2,
+                1,
+                5,
+                1,
+                3,
+                3,
+                4,
+                0,
+                2,
+                2,
+                0,
+                1,
+                1,
+                5,
+                2,
+                0,
+                0,
+                4,
+                1,
+                3,
+                1,
+                4,
+                0,
+                1,
+                1,
+                1,
+                2,
+                2,
+                0,
+                2,
+                0,
+                0,
+                0,
+                3,
+                1,
+                2,
+                0,
+                1,
+                0,
+                0,
+                1
+              ],
+              "metrics": {
+                "routes": 114,
+                "used_experts": 46,
+                "zero_experts": 18,
+                "mean": 1.78125,
+                "std": 2.0037167072296143,
+                "cv": 1.1248935461044312,
+                "gini": 0.5419408082962036,
+                "entropy_nats": 3.5931899547576904,
+                "effective_experts": 36.349845541114355,
+                "min": 0.0,
+                "p50": 1.0,
+                "p90": 4.0,
+                "max": 12.0
+              },
+              "top_experts": [
+                {
+                  "expert": 4,
+                  "routes": 12
+                },
+                {
+                  "expert": 7,
+                  "routes": 5
+                },
+                {
+                  "expert": 9,
+                  "routes": 5
+                },
+                {
+                  "expert": 25,
+                  "routes": 5
+                },
+                {
+                  "expert": 36,
+                  "routes": 5
+                },
+                {
+                  "expert": 8,
+                  "routes": 4
+                },
+                {
+                  "expert": 15,
+                  "routes": 4
+                },
+                {
+                  "expert": 19,
+                  "routes": 4
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.5437576770782471
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.02887026220560074
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.022947292774915695
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.020739872008562088
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.019217059016227722
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.01740080676972866
+                    }
+                  ],
+                  "selected_weight_sum": 0.652932969853282,
+                  "top1_top2_margin": 0.5148874148726463
+                },
+                {
+                  "position": 1,
+                  "token_id": 9081,
+                  "token": "Write",
+                  "decoded_piece": "Write",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.13753922283649445
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.09548301994800568
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.061647929251194
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.055846862494945526
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05239253491163254
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.047255437821149826
+                    }
+                  ],
+                  "selected_weight_sum": 0.450165007263422,
+                  "top1_top2_margin": 0.04205620288848877
+                },
+                {
+                  "position": 2,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 8,
+                      "weight": 0.09976524114608765
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.09795945137739182
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.08393261581659317
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.07459735125303268
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.05948468670248985
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.03450009599328041
+                    }
+                  ],
+                  "selected_weight_sum": 0.4502394422888756,
+                  "top1_top2_margin": 0.0018057897686958313
+                },
+                {
+                  "position": 3,
+                  "token_id": 12974,
+                  "token": "ĠPython",
+                  "decoded_piece": " Python",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.1781829595565796
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.11575023084878922
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.10451281070709229
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.07266294956207275
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.051248300820589066
+                    },
+                    {
+                      "expert": 48,
+                      "weight": 0.043809834867715836
+                    }
+                  ],
+                  "selected_weight_sum": 0.5661670863628387,
+                  "top1_top2_margin": 0.062432728707790375
+                },
+                {
+                  "position": 4,
+                  "token_id": 1157,
+                  "token": "Ġfunction",
+                  "decoded_piece": " function",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.19366487860679626
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.11906357854604721
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.03140224888920784
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.03139836713671684
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.031194070354104042
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.02878504991531372
+                    }
+                  ],
+                  "selected_weight_sum": 0.4355081934481859,
+                  "top1_top2_margin": 0.07460130006074905
+                },
+                {
+                  "position": 5,
+                  "token_id": 344,
+                  "token": "Ġthat",
+                  "decoded_piece": " that",
+                  "experts_by_weight": [
+                    {
+                      "expert": 40,
+                      "weight": 0.09220249950885773
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.06857212632894516
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.06615664809942245
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.04282122477889061
+                    },
+                    {
+                      "expert": 35,
+                      "weight": 0.04161207005381584
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.031904879957437515
+                    }
+                  ],
+                  "selected_weight_sum": 0.3432694487273693,
+                  "top1_top2_margin": 0.023630373179912567
+                },
+                {
+                  "position": 6,
+                  "token_id": 7578,
+                  "token": "Ġreturns",
+                  "decoded_piece": " returns",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.15439683198928833
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.11837179958820343
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.09164318442344666
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03915099427103996
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.03787941113114357
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.03748919814825058
+                    }
+                  ],
+                  "selected_weight_sum": 0.47893141955137253,
+                  "top1_top2_margin": 0.0360250324010849
+                },
+                {
+                  "position": 7,
+                  "token_id": 254,
+                  "token": "Ġthe",
+                  "decoded_piece": " the",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.1356981247663498
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.09688761830329895
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.05860389024019241
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.04973827674984932
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.04758499935269356
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.035689134150743484
+                    }
+                  ],
+                  "selected_weight_sum": 0.4242020435631275,
+                  "top1_top2_margin": 0.03881050646305084
+                },
+                {
+                  "position": 8,
+                  "token_id": 1022,
+                  "token": "Ġfirst",
+                  "decoded_piece": " first",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.18592320382595062
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.17726802825927734
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.06434556841850281
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.058037169277668
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.04549534246325493
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.03907840698957443
+                    }
+                  ],
+                  "selected_weight_sum": 0.5701477192342281,
+                  "top1_top2_margin": 0.008655175566673279
+                },
+                {
+                  "position": 9,
+                  "token_id": 9842,
+                  "token": "Ġrepeated",
+                  "decoded_piece": " repeated",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.15881554782390594
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.1423594057559967
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.13588689267635345
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.08650468289852142
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03664877638220787
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.0329442135989666
+                    }
+                  ],
+                  "selected_weight_sum": 0.593159519135952,
+                  "top1_top2_margin": 0.01645614206790924
+                },
+                {
+                  "position": 10,
+                  "token_id": 3699,
+                  "token": "Ġelement",
+                  "decoded_piece": " element",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.15322136878967285
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.09143341332674026
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.08360946178436279
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.05222221836447716
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04394563287496567
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.04124316945672035
+                    }
+                  ],
+                  "selected_weight_sum": 0.4656752645969391,
+                  "top1_top2_margin": 0.06178795546293259
+                },
+                {
+                  "position": 11,
+                  "token_id": 279,
+                  "token": "Ġin",
+                  "decoded_piece": " in",
+                  "experts_by_weight": [
+                    {
+                      "expert": 49,
+                      "weight": 0.16705435514450073
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.06526098400354385
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.0634346604347229
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03620058670639992
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.035932786762714386
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.029904931783676147
+                    }
+                  ],
+                  "selected_weight_sum": 0.39778830483555794,
+                  "top1_top2_margin": 0.10179337114095688
+                },
+                {
+                  "position": 12,
+                  "token_id": 245,
+                  "token": "Ġa",
+                  "decoded_piece": " a",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.14008726179599762
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.12481290847063065
+                    },
+                    {
+                      "expert": 8,
+                      "weight": 0.07218452543020248
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.05240301042795181
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.040102530270814896
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.03472679853439331
+                    }
+                  ],
+                  "selected_weight_sum": 0.46431703492999077,
+                  "top1_top2_margin": 0.015274353325366974
+                },
+                {
+                  "position": 13,
+                  "token_id": 1525,
+                  "token": "Ġlist",
+                  "decoded_piece": " list",
+                  "experts_by_weight": [
+                    {
+                      "expert": 4,
+                      "weight": 0.10503388941287994
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.09194021672010422
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.07014487683773041
+                    },
+                    {
+                      "expert": 3,
+                      "weight": 0.048768382519483566
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.04372379556298256
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.03325546905398369
+                    }
+                  ],
+                  "selected_weight_sum": 0.3928666301071644,
+                  "top1_top2_margin": 0.013093672692775726
+                },
+                {
+                  "position": 14,
+                  "token_id": 285,
+                  "token": "Ġand",
+                  "decoded_piece": " and",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.11566011607646942
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.08769606053829193
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.06660442799329758
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.05762474611401558
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.04991111159324646
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.03884124383330345
+                    }
+                  ],
+                  "selected_weight_sum": 0.4163377061486244,
+                  "top1_top2_margin": 0.02796405553817749
+                },
+                {
+                  "position": 15,
+                  "token_id": 6716,
+                  "token": "Ġexplain",
+                  "decoded_piece": " explain",
+                  "experts_by_weight": [
+                    {
+                      "expert": 7,
+                      "weight": 0.13019512593746185
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.07663741707801819
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.05870531126856804
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.05767242982983589
+                    },
+                    {
+                      "expert": 9,
+                      "weight": 0.054065629839897156
+                    },
+                    {
+                      "expert": 2,
+                      "weight": 0.046069856733083725
+                    }
+                  ],
+                  "selected_weight_sum": 0.42334577068686485,
+                  "top1_top2_margin": 0.053557708859443665
+                },
+                {
+                  "position": 16,
+                  "token_id": 895,
+                  "token": "Ġits",
+                  "decoded_piece": " its",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.1395944356918335
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.10110253840684891
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.09792105853557587
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.09229704737663269
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.03483238443732262
+                    },
+                    {
+                      "expert": 24,
+                      "weight": 0.020935704931616783
+                    }
+                  ],
+                  "selected_weight_sum": 0.48668316937983036,
+                  "top1_top2_margin": 0.03849189728498459
+                },
+                {
+                  "position": 17,
+                  "token_id": 13913,
+                  "token": "Ġcomplexity",
+                  "decoded_piece": " complexity",
+                  "experts_by_weight": [
+                    {
+                      "expert": 2,
+                      "weight": 0.1412113904953003
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.1274697482585907
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.08632361143827438
+                    },
+                    {
+                      "expert": 29,
+                      "weight": 0.043605998158454895
+                    },
+                    {
+                      "expert": 60,
+                      "weight": 0.03586355224251747
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.03131873905658722
+                    }
+                  ],
+                  "selected_weight_sum": 0.46579303964972496,
+                  "top1_top2_margin": 0.013741642236709595
+                },
+                {
+                  "position": 18,
+                  "token_id": 13,
+                  "token": ".",
+                  "decoded_piece": ".",
+                  "experts_by_weight": [
+                    {
+                      "expert": 50,
+                      "weight": 0.13818378746509552
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.12461185455322266
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.11088532954454422
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.10322917252779007
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.08050823956727982
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.059470295906066895
+                    }
+                  ],
+                  "selected_weight_sum": 0.6168886795639992,
+                  "top1_top2_margin": 0.013571932911872864
+                }
+              ]
+            },
+            {
+              "id": "math",
+              "label": "数学推理",
+              "tokens": 27,
+              "load": [
+                6,
+                2,
+                0,
+                0,
+                5,
+                1,
+                6,
+                2,
+                0,
+                0,
+                0,
+                2,
+                2,
+                2,
+                0,
+                1,
+                4,
+                3,
+                1,
+                2,
+                4,
+                3,
+                2,
+                1,
+                0,
+                10,
+                4,
+                2,
+                6,
+                0,
+                0,
+                3,
+                2,
+                1,
+                4,
+                0,
+                6,
+                8,
+                0,
+                3,
+                4,
+                5,
+                3,
+                3,
+                1,
+                3,
+                4,
+                2,
+                0,
+                2,
+                3,
+                2,
+                2,
+                0,
+                1,
+                5,
+                6,
+                1,
+                4,
+                7,
+                0,
+                0,
+                0,
+                6
+              ],
+              "metrics": {
+                "routes": 162,
+                "used_experts": 48,
+                "zero_experts": 16,
+                "mean": 2.53125,
+                "std": 2.3160500526428223,
+                "cv": 0.9149827361106873,
+                "gini": 0.4951774775981903,
+                "entropy_nats": 3.697890281677246,
+                "effective_experts": 40.362061891369216,
+                "min": 0.0,
+                "p50": 2.0,
+                "p90": 6.0,
+                "max": 10.0
+              },
+              "top_experts": [
+                {
+                  "expert": 25,
+                  "routes": 10
+                },
+                {
+                  "expert": 37,
+                  "routes": 8
+                },
+                {
+                  "expert": 59,
+                  "routes": 7
+                },
+                {
+                  "expert": 0,
+                  "routes": 6
+                },
+                {
+                  "expert": 6,
+                  "routes": 6
+                },
+                {
+                  "expert": 28,
+                  "routes": 6
+                },
+                {
+                  "expert": 36,
+                  "routes": 6
+                },
+                {
+                  "expert": 56,
+                  "routes": 6
+                }
+              ],
+              "token_routes": [
+                {
+                  "position": 0,
+                  "token_id": 100000,
+                  "token": "<｜begin▁of▁sentence｜>",
+                  "decoded_piece": "<｜begin▁of▁sentence｜>",
+                  "experts_by_weight": [
+                    {
+                      "expert": 22,
+                      "weight": 0.5437576770782471
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.02887026220560074
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.022947292774915695
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.020739872008562088
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.019217059016227722
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.01740080676972866
+                    }
+                  ],
+                  "selected_weight_sum": 0.652932969853282,
+                  "top1_top2_margin": 0.5148874148726463
+                },
+                {
+                  "position": 1,
+                  "token_id": 8246,
+                  "token": "èĭ¥",
+                  "decoded_piece": "若",
+                  "experts_by_weight": [
+                    {
+                      "expert": 46,
+                      "weight": 0.14609992504119873
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.1228821724653244
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.10514180362224579
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.07545209676027298
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.054526668041944504
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.04116235300898552
+                    }
+                  ],
+                  "selected_weight_sum": 0.5452650189399719,
+                  "top1_top2_margin": 0.02321775257587433
+                },
+                {
+                  "position": 2,
+                  "token_id": 2206,
+                  "token": "æŃ£",
+                  "decoded_piece": "正",
+                  "experts_by_weight": [
+                    {
+                      "expert": 26,
+                      "weight": 0.0876239538192749
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.07643693685531616
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.057786520570516586
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.040895964950323105
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.040521040558815
+                    },
+                    {
+                      "expert": 13,
+                      "weight": 0.03683890402317047
+                    }
+                  ],
+                  "selected_weight_sum": 0.34010332077741623,
+                  "top1_top2_margin": 0.01118701696395874
+                },
+                {
+                  "position": 3,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.12693962454795837
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.11401762068271637
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.05557835474610329
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.04544328525662422
+                    },
+                    {
+                      "expert": 57,
+                      "weight": 0.03626488149166107
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.0351380929350853
+                    }
+                  ],
+                  "selected_weight_sum": 0.4133818596601486,
+                  "top1_top2_margin": 0.012922003865242004
+                },
+                {
+                  "position": 4,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 41,
+                      "weight": 0.15854693949222565
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.0794890820980072
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.06637027114629745
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.03351316973567009
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.03202974796295166
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.02984650433063507
+                    }
+                  ],
+                  "selected_weight_sum": 0.3997957147657871,
+                  "top1_top2_margin": 0.07905785739421844
+                },
+                {
+                  "position": 5,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 39,
+                      "weight": 0.048063986003398895
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.03813319653272629
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.03565631061792374
+                    },
+                    {
+                      "expert": 23,
+                      "weight": 0.033526234328746796
+                    },
+                    {
+                      "expert": 22,
+                      "weight": 0.02928059734404087
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.02924760803580284
+                    }
+                  ],
+                  "selected_weight_sum": 0.21390793286263943,
+                  "top1_top2_margin": 0.009930789470672607
+                },
+                {
+                  "position": 6,
+                  "token_id": 13211,
+                  "token": "满足",
+                  "decoded_piece": "满足",
+                  "experts_by_weight": [
+                    {
+                      "expert": 58,
+                      "weight": 0.12237146496772766
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.07029765099287033
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.059514883905649185
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.05232641473412514
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.05189927667379379
+                    },
+                    {
+                      "expert": 5,
+                      "weight": 0.05173082277178764
+                    }
+                  ],
+                  "selected_weight_sum": 0.40814051404595375,
+                  "top1_top2_margin": 0.05207381397485733
+                },
+                {
+                  "position": 7,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 41,
+                      "weight": 0.14802685379981995
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.11230576783418655
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.045813966542482376
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.04318483546376228
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.037408940494060516
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.03519348427653313
+                    }
+                  ],
+                  "selected_weight_sum": 0.4219338484108448,
+                  "top1_top2_margin": 0.03572108596563339
+                },
+                {
+                  "position": 8,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 56,
+                      "weight": 0.15301837027072906
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.10797952860593796
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.10473429411649704
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.06219639629125595
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.03330612555146217
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.029627667739987373
+                    }
+                  ],
+                  "selected_weight_sum": 0.49086238257586956,
+                  "top1_top2_margin": 0.04503884166479111
+                },
+                {
+                  "position": 9,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 34,
+                      "weight": 0.14497965574264526
+                    },
+                    {
+                      "expert": 55,
+                      "weight": 0.14415185153484344
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.03581972047686577
+                    },
+                    {
+                      "expert": 45,
+                      "weight": 0.032462701201438904
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.03084731660783291
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.02867933362722397
+                    }
+                  ],
+                  "selected_weight_sum": 0.41694057919085026,
+                  "top1_top2_margin": 0.0008278042078018188
+                },
+                {
+                  "position": 10,
+                  "token_id": 12985,
+                  "token": "å®ĥçļĦ",
+                  "decoded_piece": "它的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.25599992275238037
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.10082335025072098
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.06489169597625732
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.04302974417805672
+                    },
+                    {
+                      "expert": 15,
+                      "weight": 0.030839329585433006
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.02814270369708538
+                    }
+                  ],
+                  "selected_weight_sum": 0.5237267464399338,
+                  "top1_top2_margin": 0.1551765725016594
+                },
+                {
+                  "position": 11,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 25,
+                      "weight": 0.09931953251361847
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.05962561443448067
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.05795827880501747
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.04644583910703659
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.03724290803074837
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03425101935863495
+                    }
+                  ],
+                  "selected_weight_sum": 0.3348431922495365,
+                  "top1_top2_margin": 0.0396939180791378
+                },
+                {
+                  "position": 12,
+                  "token_id": 2266,
+                  "token": "æķ°",
+                  "decoded_piece": "数",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.15665771067142487
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.11005081236362457
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.09605225920677185
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.06298534572124481
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.0517939031124115
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.04924814775586128
+                    }
+                  ],
+                  "selected_weight_sum": 0.5267881788313389,
+                  "top1_top2_margin": 0.04660689830780029
+                },
+                {
+                  "position": 13,
+                  "token_id": 37360,
+                  "token": "çŃīäºİ",
+                  "decoded_piece": "等于",
+                  "experts_by_weight": [
+                    {
+                      "expert": 55,
+                      "weight": 0.15661580860614777
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.09555711597204208
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.0472816526889801
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.0455225333571434
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.03498144820332527
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.02906597964465618
+                    }
+                  ],
+                  "selected_weight_sum": 0.4090245384722948,
+                  "top1_top2_margin": 0.06105869263410568
+                },
+                {
+                  "position": 14,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 56,
+                      "weight": 0.1745140701532364
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.1652524471282959
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.16063380241394043
+                    },
+                    {
+                      "expert": 16,
+                      "weight": 0.09852830320596695
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.07098746299743652
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.05819505825638771
+                    }
+                  ],
+                  "selected_weight_sum": 0.7281111441552639,
+                  "top1_top2_margin": 0.00926162302494049
+                },
+                {
+                  "position": 15,
+                  "token_id": 18,
+                  "token": "3",
+                  "decoded_piece": "3",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.10042109340429306
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.08639753609895706
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.05279490351676941
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.040343768894672394
+                    },
+                    {
+                      "expert": 42,
+                      "weight": 0.0377325676381588
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.0330699160695076
+                    }
+                  ],
+                  "selected_weight_sum": 0.3507597856223583,
+                  "top1_top2_margin": 0.014023557305335999
+                },
+                {
+                  "position": 16,
+                  "token_id": 19304,
+                  "token": "ï¼Į",
+                  "decoded_piece": "，",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.10927784442901611
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.10593557357788086
+                    },
+                    {
+                      "expert": 18,
+                      "weight": 0.08180209249258041
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.07350264489650726
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.06115064397454262
+                    },
+                    {
+                      "expert": 54,
+                      "weight": 0.05411747470498085
+                    }
+                  ],
+                  "selected_weight_sum": 0.4857862740755081,
+                  "top1_top2_margin": 0.003342270851135254
+                },
+                {
+                  "position": 17,
+                  "token_id": 2989,
+                  "token": "æ±Ĥ",
+                  "decoded_piece": "求",
+                  "experts_by_weight": [
+                    {
+                      "expert": 0,
+                      "weight": 0.0868334174156189
+                    },
+                    {
+                      "expert": 7,
+                      "weight": 0.07825968414545059
+                    },
+                    {
+                      "expert": 19,
+                      "weight": 0.04057038575410843
+                    },
+                    {
+                      "expert": 44,
+                      "weight": 0.037275321781635284
+                    },
+                    {
+                      "expert": 52,
+                      "weight": 0.03410649672150612
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.032320231199264526
+                    }
+                  ],
+                  "selected_weight_sum": 0.30936553701758385,
+                  "top1_top2_margin": 0.008573733270168304
+                },
+                {
+                  "position": 18,
+                  "token_id": 1376,
+                  "token": "Ġx",
+                  "decoded_piece": " x",
+                  "experts_by_weight": [
+                    {
+                      "expert": 41,
+                      "weight": 0.1231520026922226
+                    },
+                    {
+                      "expert": 6,
+                      "weight": 0.06557122617959976
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.0597735270857811
+                    },
+                    {
+                      "expert": 4,
+                      "weight": 0.05677611008286476
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.03178378567099571
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.031433671712875366
+                    }
+                  ],
+                  "selected_weight_sum": 0.3684903234243393,
+                  "top1_top2_margin": 0.05758077651262283
+                },
+                {
+                  "position": 19,
+                  "token_id": 207,
+                  "token": "Ġ",
+                  "decoded_piece": " ",
+                  "experts_by_weight": [
+                    {
+                      "expert": 56,
+                      "weight": 0.09447808563709259
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.07532291114330292
+                    },
+                    {
+                      "expert": 46,
+                      "weight": 0.06776554137468338
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.05218034237623215
+                    },
+                    {
+                      "expert": 39,
+                      "weight": 0.03654729574918747
+                    },
+                    {
+                      "expert": 33,
+                      "weight": 0.032638877630233765
+                    }
+                  ],
+                  "selected_weight_sum": 0.35893305391073227,
+                  "top1_top2_margin": 0.019155174493789673
+                },
+                {
+                  "position": 20,
+                  "token_id": 337,
+                  "token": "çļĦ",
+                  "decoded_piece": "的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 36,
+                      "weight": 0.1181933805346489
+                    },
+                    {
+                      "expert": 32,
+                      "weight": 0.04208538308739662
+                    },
+                    {
+                      "expert": 21,
+                      "weight": 0.0391385555267334
+                    },
+                    {
+                      "expert": 40,
+                      "weight": 0.03826816380023956
+                    },
+                    {
+                      "expert": 43,
+                      "weight": 0.036307066679000854
+                    },
+                    {
+                      "expert": 1,
+                      "weight": 0.03165046498179436
+                    }
+                  ],
+                  "selected_weight_sum": 0.3056430146098137,
+                  "top1_top2_margin": 0.07610799744725227
+                },
+                {
+                  "position": 21,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 63,
+                      "weight": 0.09191884845495224
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.0589422807097435
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.043951794505119324
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.04311515390872955
+                    },
+                    {
+                      "expert": 31,
+                      "weight": 0.04050914943218231
+                    },
+                    {
+                      "expert": 51,
+                      "weight": 0.032655391842126846
+                    }
+                  ],
+                  "selected_weight_sum": 0.3110926188528538,
+                  "top1_top2_margin": 0.03297656774520874
+                },
+                {
+                  "position": 22,
+                  "token_id": 14855,
+                  "token": "åĬłä¸Ĭ",
+                  "decoded_piece": "加上",
+                  "experts_by_weight": [
+                    {
+                      "expert": 55,
+                      "weight": 0.14588066935539246
+                    },
+                    {
+                      "expert": 34,
+                      "weight": 0.14225462079048157
+                    },
+                    {
+                      "expert": 58,
+                      "weight": 0.034157682210206985
+                    },
+                    {
+                      "expert": 49,
+                      "weight": 0.029166897758841515
+                    },
+                    {
+                      "expert": 0,
+                      "weight": 0.028123507276177406
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.025089992210268974
+                    }
+                  ],
+                  "selected_weight_sum": 0.4046733696013689,
+                  "top1_top2_margin": 0.0036260485649108887
+                },
+                {
+                  "position": 23,
+                  "token_id": 8270,
+                  "token": "åĢĴ",
+                  "decoded_piece": "倒",
+                  "experts_by_weight": [
+                    {
+                      "expert": 20,
+                      "weight": 0.057385675609111786
+                    },
+                    {
+                      "expert": 26,
+                      "weight": 0.054804299026727676
+                    },
+                    {
+                      "expert": 59,
+                      "weight": 0.054245784878730774
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.05022210627794266
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.039826568216085434
+                    },
+                    {
+                      "expert": 41,
+                      "weight": 0.03494826331734657
+                    }
+                  ],
+                  "selected_weight_sum": 0.2914326973259449,
+                  "top1_top2_margin": 0.0025813765823841095
+                },
+                {
+                  "position": 24,
+                  "token_id": 51971,
+                  "token": "æķ°çļĦ",
+                  "decoded_piece": "数的",
+                  "experts_by_weight": [
+                    {
+                      "expert": 28,
+                      "weight": 0.0728067085146904
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.07093488425016403
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.0637853667140007
+                    },
+                    {
+                      "expert": 17,
+                      "weight": 0.05612759292125702
+                    },
+                    {
+                      "expert": 36,
+                      "weight": 0.047605425119400024
+                    },
+                    {
+                      "expert": 20,
+                      "weight": 0.0461735725402832
+                    }
+                  ],
+                  "selected_weight_sum": 0.3574335500597954,
+                  "top1_top2_margin": 0.0018718242645263672
+                },
+                {
+                  "position": 25,
+                  "token_id": 18718,
+                  "token": "å¹³æĸ¹",
+                  "decoded_piece": "平方",
+                  "experts_by_weight": [
+                    {
+                      "expert": 59,
+                      "weight": 0.13647541403770447
+                    },
+                    {
+                      "expert": 63,
+                      "weight": 0.10184752941131592
+                    },
+                    {
+                      "expert": 25,
+                      "weight": 0.059981655329465866
+                    },
+                    {
+                      "expert": 28,
+                      "weight": 0.0428638830780983
+                    },
+                    {
+                      "expert": 27,
+                      "weight": 0.0374276340007782
+                    },
+                    {
+                      "expert": 47,
+                      "weight": 0.031549129635095596
+                    }
+                  ],
+                  "selected_weight_sum": 0.41014524549245834,
+                  "top1_top2_margin": 0.03462788462638855
+                },
+                {
+                  "position": 26,
+                  "token_id": 398,
+                  "token": "ãĢĤ",
+                  "decoded_piece": "。",
+                  "experts_by_weight": [
+                    {
+                      "expert": 16,
+                      "weight": 0.11203551292419434
+                    },
+                    {
+                      "expert": 11,
+                      "weight": 0.1093420758843422
+                    },
+                    {
+                      "expert": 12,
+                      "weight": 0.08985795080661774
+                    },
+                    {
+                      "expert": 37,
+                      "weight": 0.08588015288114548
+                    },
+                    {
+                      "expert": 56,
+                      "weight": 0.07133956253528595
+                    },
+                    {
+                      "expert": 50,
+                      "weight": 0.06675778329372406
+                    }
+                  ],
+                  "selected_weight_sum": 0.5352130383253098,
+                  "top1_top2_margin": 0.0026934370398521423
+                }
+              ]
+            }
+          ],
+          "prompt_pair_jaccard": [
+            {
+              "left": "zh_explanation",
+              "right": "en_architecture",
+              "used_expert_jaccard": 0.6545454545454545,
+              "shared_experts": 36,
+              "union_experts": 55
+            },
+            {
+              "left": "zh_explanation",
+              "right": "code",
+              "used_expert_jaccard": 0.7272727272727273,
+              "shared_experts": 40,
+              "union_experts": 55
+            },
+            {
+              "left": "zh_explanation",
+              "right": "math",
+              "used_expert_jaccard": 0.7321428571428571,
+              "shared_experts": 41,
+              "union_experts": 56
+            },
+            {
+              "left": "en_architecture",
+              "right": "code",
+              "used_expert_jaccard": 0.6,
+              "shared_experts": 33,
+              "union_experts": 55
+            },
+            {
+              "left": "en_architecture",
+              "right": "math",
+              "used_expert_jaccard": 0.6071428571428571,
+              "shared_experts": 34,
+              "union_experts": 56
+            },
+            {
+              "left": "code",
+              "right": "math",
+              "used_expert_jaccard": 0.6491228070175439,
+              "shared_experts": 37,
+              "union_experts": 57
+            }
+          ]
+        }
+      }
+    ],
+    "final_hidden": {
+      "valid_tokens": 90,
+      "hidden_rms": {
+        "mean": 3.3090813159942627,
+        "p10": 0.2696530222892761,
+        "p50": 0.31665489077568054,
+        "p90": 0.3847275674343109
+      },
+      "l2_norm": {
+        "mean": 149.75193786621094,
+        "p10": 12.203103065490723,
+        "p50": 14.330163955688477,
+        "p90": 17.410781860351562
+      },
+      "finite": true,
+      "sha256_fp32": "1ffd41521b5a32af38695b767a5ecc556ddeae1e1efe5339743e5c8c7644030e"
+    }
+  },
+  "cache_accounting": {
+    "dtype": "BF16",
+    "latent_elements_per_token_layer": 576,
+    "latent_bytes_per_token_layer": 1152,
+    "hf_eager_elements_per_token_layer": 5120,
+    "hf_eager_bytes_per_token_layer": 10240,
+    "eager_over_latent_ratio": 8.88888888888889,
+    "latent_reduction_vs_eager": 0.8875,
+    "components": {
+      "latent_content": 512,
+      "rope_key": 64,
+      "expanded_key": 3072,
+      "expanded_value": 2048
+    }
+  }
+}
diff --git a/src/pages/deepseek/index.astro b/src/pages/deepseek/index.astro
index 0b8b795..574ff8c 100644
--- a/src/pages/deepseek/index.astro
+++ b/src/pages/deepseek/index.astro
@@ -2,6 +2,7 @@
 import BaseLayout from "@/layouts/BaseLayout.astro";
 import DeepSeekLineage from "@/components/DeepSeekLineage.astro";
 import DeepSeekLab from "@/components/DeepSeekLab.astro";
+import DeepSeekArtifactLab from "@/components/DeepSeekArtifactLab.astro";
 import { deepseekBranches, deepseekLedgers, deepseekPaperChain, deepseekWaves } from "@/data/deepseek";
 
 const toc = [
@@ -27,21 +28,22 @@ const toc = [
   ["19", "v4-state", "V4：异构状态与稳定性"],
   ["20", "k3", "与 K3 的继承边界"],
   ["21", "lab", "四联交互实验"],
-  ["22", "branches", "别漏掉旁支"],
-  ["23", "audit", "事实、推导与教学模型"],
+  ["22", "artifact", "真实权重执行"],
+  ["23", "branches", "别漏掉旁支"],
+  ["24", "audit", "事实、推导与教学模型"],
   ["↳", "papers", "六十节点阅读链"],
 ];
 ---
 
 <BaseLayout
-  title="DeepSeek 技术谱系深读：从 Dense、MoE、MLA 到 R1 与 V4"
-  description="用二十四张问题账、十次技术转向、四个交互实验和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
+  title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
+  description="用二十四张问题账、十次技术转向、八个交互实验、真实 V2-Lite 权重 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
   section="deepseek"
 >
   <header class="page-hero deepseek-hero">
     <div class="page-hero-inner">
       <div>
-        <p class="eyebrow"><span>SPOTLIGHT / DEEPSEEK · ROUND 02</span> ALGORITHM × SYSTEM × EVIDENCE</p>
+        <p class="eyebrow"><span>SPOTLIGHT / DEEPSEEK · ROUND 03</span> ALGORITHM × SYSTEM × REAL WEIGHTS</p>
         <h1>不要背模型名<br />要看懂每次为什么转向</h1>
         <p class="lead">
           这不是七篇报告的摘要，而是一套可追问、可计算、可反驳的技术谱系：
@@ -53,9 +55,9 @@ const toc = [
         <div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
         <div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
         <div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
-        <div><dt>LABS</dt><dd>4 个可操作实验</dd></div>
+        <div><dt>LABS</dt><dd>8 个可操作实验</dd></div>
         <div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
-        <div><dt>STATUS</dt><dd>重点专题 · 二轮深读</dd></div>
+        <div><dt>STATUS</dt><dd>三轮 · 真实权重执行</dd></div>
       </dl>
     </div>
   </header>
@@ -762,8 +764,25 @@ const toc = [
         <DeepSeekLab />
       </section>
 
+      <section class="article-section" id="artifact">
+        <p class="eyebrow"><span>22</span> OFFICIAL WEIGHTS / EXECUTED</p>
+        <h2>从“MLA 与 MoE 的概念”再往前一步：让官方 V2-Lite 权重真的跑起来</h2>
+        <p class="lede">
+          前面的四联实验负责建立公式与角色合同；下面的四联工件实验固定官方 revision、tokenizer、
+          模型代码和 checkpoint 第一分片，在 RTX 5090 上连续执行 layer 0–6。它把真实观测、shape 推导、
+          实现差距和未覆盖范围放在同一张证据图里。
+        </p>
+        <div class="artifact-callout">
+          <article><span>X / FORWARD</span><b>7 / 27 layers</b><p>1 个 dense 层 + 6 个 MoE 层；layer 7 因跨分片停止。</p></article>
+          <article><span>X / ROUTES</span><b>3,240</b><p>90 个有效 token × 6 层 × top-6 routed experts。</p></article>
+          <article><span>X + D / CACHE</span><b>576 ↔ 5,120</b><p>latent 合同与 HF eager 实际展开元素，两张账同时保留。</p></article>
+          <article><span>X / RERUN</span><b>31 / 31 exact</b><p>hidden hashes、MLA shapes、loads 与全部 token routes。</p></article>
+        </div>
+        <DeepSeekArtifactLab />
+      </section>
+
       <section class="article-section" id="branches">
-        <p class="eyebrow"><span>22</span> THE MAIN LINE IS NOT THE WHOLE TREE</p>
+        <p class="eyebrow"><span>23</span> THE MAIN LINE IS NOT THE WHOLE TREE</p>
         <h2>如果只读 V2 → V3 → R1 → V4，会漏掉五条反过来影响主线的旁支</h2>
         <div class="branch-grid">
           {deepseekBranches.map(([name, line, text, url]) => (
@@ -783,7 +802,7 @@ const toc = [
       </section>
 
       <section class="article-section" id="audit">
-        <p class="eyebrow"><span>23</span> EVIDENCE AUDIT</p>
+        <p class="eyebrow"><span>24</span> EVIDENCE AUDIT</p>
         <h2>同一张页面里有三种知识，它们的语气必须不同</h2>
         <div class="audit-grid">
           <article class="reported">
@@ -837,6 +856,43 @@ const toc = [
         radial-gradient(circle at 65% 38%, rgba(84, 124, 116, 0.13), transparent 24%);
     }
 
+    .artifact-callout {
+      display: grid;
+      grid-template-columns: repeat(4, minmax(0, 1fr));
+      max-width: 980px;
+      margin: 30px 0;
+      border-top: 1px solid var(--line);
+      border-left: 1px solid var(--line);
+    }
+
+    .artifact-callout article {
+      min-height: 155px;
+      padding: 20px;
+      border-right: 1px solid var(--line);
+      border-bottom: 1px solid var(--line);
+      background: var(--paper-raised);
+    }
+
+    .artifact-callout span {
+      display: block;
+      color: var(--copper);
+      font: 700 0.58rem/1 var(--mono);
+      letter-spacing: 0.08em;
+    }
+
+    .artifact-callout b {
+      display: block;
+      margin-top: 18px;
+      font-size: 1.14rem;
+    }
+
+    .artifact-callout p {
+      margin: 11px 0 0;
+      color: var(--muted);
+      font-size: 0.68rem;
+      line-height: 1.55;
+    }
+
     .moe-compare,
     .four-layer {
       display: grid;
@@ -1674,6 +1730,7 @@ const toc = [
       .followup-grid,
       .state-machines,
       .v4-contract,
+      .artifact-callout,
       .branch-grid,
       .audit-grid,
       .paper-chain.expanded {
diff --git a/src/pages/index.astro b/src/pages/index.astro
index d782178..c93c809 100644
--- a/src/pages/index.astro
+++ b/src/pages/index.astro
@@ -145,17 +145,17 @@ const paths = [
       </a>
       <a class="release-card deepseek-release" href="/deepseek/">
         <div>
-          <p class="eyebrow"><span>NEW / DEEPSEEK ROUND 02</span> CAPACITY · STATE · SYSTEM · REASONING</p>
+          <p class="eyebrow"><span>NEW / DEEPSEEK ROUND 03</span> LINEAGE · REAL WEIGHTS · ROUTES · CACHE</p>
           <h2>从 Dense 到百万上下文：每次创新都在偿还上一代最贵的一张账</h2>
           <p>
-            用二十四张问题账和十次技术转向，从 DeepSeek LLM、MoE、V2 的 MLA 权重吸收，
-            走到 V3 的 FP8 / DualPipe / MTP、R1 与 DAPO / Dr.GRPO 反查、V3.2 Agent 环境和 V4 异构长状态。
+            用二十四张问题账和十次技术转向走完 Dense→V4，再固定官方 V2-Lite 权重执行 7/27 层：
+            逐 token 检查 3,240 次专家选择，并把 latent 状态与 HF eager cache 的实现差距摆在同一张账上。
           </p>
         </div>
         <dl>
           <div><dt>LINEAGE</dt><dd>1991 → 2026 · 10 次转向</dd></div>
           <div><dt>NODES</dt><dd>60 个一手 / 官方节点</dd></div>
-          <div><dt>LAB</dt><dd>MoE · MLA · V3 协同 · RL 偏差</dd></div>
+          <div><dt>LAB</dt><dd>4 公式实验 · 4 真实工件实验</dd></div>
         </dl>
         <span class="release-arrow" aria-hidden="true">进入 DeepSeek 完整技术谱系 →</span>
       </a>
diff --git a/src/pages/progress/index.astro b/src/pages/progress/index.astro
index 566b8af..c7f1eea 100644
--- a/src/pages/progress/index.astro
+++ b/src/pages/progress/index.astro
@@ -15,7 +15,7 @@ const workstreams = [
   { label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
   { label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
   { label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
-  { label: "DeepSeek 专题", value: 83, next: "加入真实专家负载、MLA kernel、RL 训练 traces 与独立复现" },
+  { label: "DeepSeek 专题", value: 87, next: "真实 latent-cache kernel、更大样本负载、FP8/pipeline 与 R1-like RL 复现" },
   { label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
   { label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
   { label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
@@ -50,7 +50,7 @@ const workstreams = [
         <div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
         <div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
         <div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
-        <div><dt>UPDATED</dt><dd>2026-07-29 12:40 CST</dd></div>
+        <div><dt>UPDATED</dt><dd>2026-07-29 14:15 CST</dd></div>
         <div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
       </dl>
     </div>
@@ -97,12 +97,12 @@ const workstreams = [
       <article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
       <article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
       <article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
-      <article><span>✓</span><h3>七十一个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，以及语言模型前史、Transformer、表示深度、DeepSeek、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
+      <article><span>✓</span><h3>七十五个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验与四联真实权重实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
       <article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
       <article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
       <article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
       <article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
-      <article><span>✓</span><h3>DeepSeek 技术谱系二轮深读</h3><p>二十四张问题账、十次技术转向、60 个一手/官方节点，以及稀疏容量—MLA 缓存—V3 协同—RL 偏差四联实验。</p></article>
+      <article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上，新增 V2-Lite 7/27 层连续 forward、3,240 次真实专家选择、MLA/eager cache 实现账与 31/31 exact 复跑四联实验。</p></article>
       <article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
       <article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
       <article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
@@ -134,7 +134,7 @@ const workstreams = [
     <div class="queue-table">
       <div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
       <div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
-      <div><span>P0</span><strong>DeepSeek 三轮</strong><p>真实 expert load / MLA kernel → FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
+      <div><span>P0</span><strong>DeepSeek 三轮</strong><p>真实 latent-cache kernel / 更大负载样本 → FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
       <div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
       <div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
       <div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>