feat: add DeepSeek chat-template routing probe
This commit is contained in:
@@ -153,3 +153,55 @@ python -B experiments/deepseek/compare_routing_length_control.py \
|
||||
See `research/DEEPSEEK_ROUTING_LENGTH_CONTROL_AUDIT.md` for the sampling bias
|
||||
audit, paired CV/JSD deltas, total-variation accounting, and interpretation
|
||||
boundaries.
|
||||
|
||||
## Official chat-template sensitivity
|
||||
|
||||
`v2_lite_routing_template_probe.py` renders three variants of one fixed
|
||||
23-content-token prefix:
|
||||
|
||||
```text
|
||||
raw BOS + content
|
||||
user BOS + "User: " + content + "\n\n"
|
||||
generation user prefix + "Assistant:"
|
||||
```
|
||||
|
||||
The latter two use the pinned official `chat_template` through
|
||||
`apply_chat_template`. All three variants of one source prompt execute in the
|
||||
same padded batch. Statistics are split between:
|
||||
|
||||
- the full operational input, including wrapper tokens;
|
||||
- the exact intersection of `(relative character span, token ID)` inside the
|
||||
source content across all three variants.
|
||||
|
||||
The `user → generation` comparison is a causal negative control: the appended
|
||||
suffix must not change routes on their shared prefix.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||||
python -B experiments/deepseek/v2_lite_routing_template_probe.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-template.json \
|
||||
--per-domain 32 \
|
||||
--content-tokens 23 \
|
||||
--batch-prompts 8 \
|
||||
--layers 7 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T08:30:00+00:00
|
||||
```
|
||||
|
||||
The three variants add 382,032 real top-6 route selections. Across six MoE
|
||||
layers, all 21,852 `user → generation` shared-prefix token routes are
|
||||
ordered-top-6 exact. The committed run and independent rerun are byte-exact:
|
||||
|
||||
```text
|
||||
da1f10333b2fa269e64f9716a0ca6c1a656d23d3e70b8a25d6e59f8a5b3bc1b9
|
||||
```
|
||||
|
||||
See `research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md` for the aligned-content
|
||||
contract, paired intervals, per-token route stability, and claim boundaries.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user