feat: isolate DeepSeek role marker head token

This commit is contained in:
wuyang
2026-07-29 20:44:43 +08:00
parent 8ef2d33437
commit ea49ed0511
15 changed files with 3676450 additions and 33 deletions
+54
View File
@@ -371,3 +371,57 @@ See `research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md` for all paired
intervals, per-token route alignment, scope split, BF16 batch-shape audit,
primary literature, and the boundary between an input-ID intervention and a
chat-turn semantic claim.
## Role-marker-head single-token control
`v2_lite_routing_role_marker_head_control.py` keeps the official filler
history, EOS, colon, target span, generation prompt, mask, length, and 32-row
batch shape, while changing one ordinary token ID:
```text
system off/on × official / pre-target User→Assistant /
pre-target User→x / post-target Assistant→User
```
The pinned tokenizer maps `User`, `Assistant`, `:`, and `x` to IDs `5726`,
`77398`, `25`, and `87`. The pre-target controls identify the effect of the
first role-marker token only; the colon remains. The post-target replacement
is a causal suffix negative control. None of the three counterfactuals is a
valid official chat serialization.
```bash
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_role_marker_head_control.py \
--artifact-dir /path/to/deepseek-v2-lite \
--human-eval /path/to/HumanEval.jsonl.gz \
--gsm8k /path/to/gsm8k/test.jsonl \
--tnews /path/to/tnews/test.json \
--tnews-archive /path/to/tnews_public.zip \
--wikitext /path/to/wikitext-validation.parquet \
--output src/data/deepseek-v2-lite-routing-role-marker-head-control.json \
--per-domain 32 \
--content-tokens 23 \
--batch-prompts 4 \
--layers 7 \
--bootstrap 2000 \
--seed 20260729 \
--captured-at 2026-07-29T12:30:00+00:00
```
The eight cells add 2,044,224 real top-6 route selections. All 256
source×system groups preserve length and target position; every one of the
768 counterfactuals differs from its official sequence at exactly one ID. The
committed run and independent rerun are byte-exact:
```text
9dc0e37fbce6581269428dcfcb84c7a17b466c5239c8171e6741f66d5eeb8caf
```
For exact target content, direct replacement TV is about `.02–.03`, but the
system-edge contrast has mixed direction: `12↑12↓` for User→Assistant and
`13↑11↓` for User→x. The post-target control is exact for all 34,488 aligned
target-token ordered top-6 routes, with zero target TV, JSD, and ΔCV. See
`research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md` for all paired intervals,
depth maps, token-level alignment, cross-experiment BF16 batch-content audit,
primary sources, and the boundary against full role semantics or Chat-model
behavior.