feat: trace DeepSeek Chat completion depth

2026-07-30 00:24:27 +08:00
parent fb44d15bb8
commit 8bb488f275
23 changed files with 1507052 additions and 33 deletions
@@ -586,3 +586,89 @@ See `research/DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_PROTOCOL.md` and
 `research/DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_AUDIT.md` for the preregistered
 contract, output-divergence table, offload device map, truncation boundary,
 rerun coverage, execution fixes, primary sources, and forbidden conclusions.
+
+## Completion-aware 512-token evaluation
+
+The completion run uses the same 16 sources and eight-condition batches but
+reruns all 128 cells with one uniform 512-token budget. The successful formal
+placement leaves embeddings and layers 0–23 on CUDA and offloads layers 24–26,
+final norm, and LM head. `expandable_segments:True` is part of the recorded
+runtime contract.
+
+```bash
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+PYTHONPATH=/path/to/transformers-4.41.2-deps \
+python -B experiments/deepseek/v2_lite_chat_special_token_behavior_probe.py \
+  --artifact-dir /path/to/deepseek-v2-lite-chat \
+  --reference-routing-json \
+    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
+  --human-eval /path/to/HumanEval.jsonl.gz \
+  --gsm8k /path/to/gsm8k/test.jsonl \
+  --tnews /path/to/tnews/test.json \
+  --tnews-archive /path/to/tnews_public.zip \
+  --wikitext /path/to/wikitext-validation.parquet \
+  --output src/data/deepseek-v2-lite-chat-completion-512.json \
+  --per-domain 4 \
+  --max-new-tokens 512 \
+  --gpu-memory 28GiB \
+  --cpu-memory 80GiB
+```
+
+The evaluator separates stopping, semantic terminal state, evaluator coverage,
+and correctness. Each HumanEval candidate runs in a fresh pinned, networkless,
+read-only Docker container with no host mounts:
+
+```bash
+python -B experiments/deepseek/v2_lite_chat_completion_evaluator.py \
+  --behavior-json \
+    src/data/deepseek-v2-lite-chat-completion-512.json \
+  --baseline-json src/data/deepseek-v2-lite-chat-behavior.json \
+  --human-eval /path/to/HumanEval.jsonl.gz \
+  --gsm8k /path/to/gsm8k/test.jsonl \
+  --sandbox-image \
+    python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
+  --output \
+    src/data/deepseek-v2-lite-chat-completion-512-eval.json
+```
+
+The formal run reaches natural EOS in 121/128 cells, up from 31/128 at the
+128-token budget. Strict-complete GSM8K numeric exact is 23/32; HumanEval
+official-test pass is 24/32. These are four source tasks per domain, not
+benchmark estimates. A fresh-process one-source-per-domain rerun reproduces
+all 32 complete generated token sequences.
+
+## Full 27-layer Chat hidden-state and router trace
+
+`v2_lite_chat_full_depth_trace.py` performs a prompt-only `use_cache=False`
+forward on the same checkpoint and inputs. It captures hashes and statistics
+for embedding, every decoder-layer output, final norm, and every MoE gate.
+Boundary-crossing tokenizer tokens are excluded from the exact target-content
+scope.
+
+```bash
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+PYTHONPATH=/path/to/transformers-4.41.2-deps \
+python -B experiments/deepseek/v2_lite_chat_full_depth_trace.py \
+  --artifact-dir /path/to/deepseek-v2-lite-chat \
+  --reference-routing-json \
+    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
+  --human-eval /path/to/HumanEval.jsonl.gz \
+  --gsm8k /path/to/gsm8k/test.jsonl \
+  --tnews /path/to/tnews/test.json \
+  --tnews-archive /path/to/tnews_public.zip \
+  --wikitext /path/to/wikitext-validation.parquet \
+  --output src/data/deepseek-v2-lite-chat-full-depth.json \
+  --per-domain 4 \
+  --gpu-memory 28GiB \
+  --cpu-memory 80GiB
+```
+
+The formal trace covers 1,537 exact interior content tokens per condition,
+29 hidden stages, 26 gates, and 1,918,176 top-6 route decisions. A fresh
+four-source rerun reproduces 1,856 hidden tensor hashes, 1,664 ordered route
+hashes, 1,664 route-weight hashes, and every derived comparison.
+
+See `research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_PROTOCOL.md` and
+`research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md` for the OOM
+amendment, task evaluators, completion table, depth curves, reproduction
+audit, artifact hashes, and claim boundaries.