feat: isolate DeepSeek role marker head token
This commit is contained in:
@@ -371,3 +371,57 @@ See `research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md` for all paired
|
||||
intervals, per-token route alignment, scope split, BF16 batch-shape audit,
|
||||
primary literature, and the boundary between an input-ID intervention and a
|
||||
chat-turn semantic claim.
|
||||
|
||||
## Role-marker-head single-token control
|
||||
|
||||
`v2_lite_routing_role_marker_head_control.py` keeps the official filler
|
||||
history, EOS, colon, target span, generation prompt, mask, length, and 32-row
|
||||
batch shape, while changing one ordinary token ID:
|
||||
|
||||
```text
|
||||
system off/on × official / pre-target User→Assistant /
|
||||
pre-target User→x / post-target Assistant→User
|
||||
```
|
||||
|
||||
The pinned tokenizer maps `User`, `Assistant`, `:`, and `x` to IDs `5726`,
|
||||
`77398`, `25`, and `87`. The pre-target controls identify the effect of the
|
||||
first role-marker token only; the colon remains. The post-target replacement
|
||||
is a causal suffix negative control. None of the three counterfactuals is a
|
||||
valid official chat serialization.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||||
python -B experiments/deepseek/v2_lite_routing_role_marker_head_control.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-role-marker-head-control.json \
|
||||
--per-domain 32 \
|
||||
--content-tokens 23 \
|
||||
--batch-prompts 4 \
|
||||
--layers 7 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T12:30:00+00:00
|
||||
```
|
||||
|
||||
The eight cells add 2,044,224 real top-6 route selections. All 256
|
||||
source×system groups preserve length and target position; every one of the
|
||||
768 counterfactuals differs from its official sequence at exactly one ID. The
|
||||
committed run and independent rerun are byte-exact:
|
||||
|
||||
```text
|
||||
9dc0e37fbce6581269428dcfcb84c7a17b466c5239c8171e6741f66d5eeb8caf
|
||||
```
|
||||
|
||||
For exact target content, direct replacement TV is about `.02–.03`, but the
|
||||
system-edge contrast has mixed direction: `12↑12↓` for User→Assistant and
|
||||
`13↑11↓` for User→x. The post-target control is exact for all 34,488 aligned
|
||||
target-token ordered top-6 routes, with zero target TV, JSD, and ΔCV. See
|
||||
`research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md` for all paired intervals,
|
||||
depth maps, token-level alignment, cross-experiment BF16 batch-content audit,
|
||||
primary sources, and the boundary against full role semantics or Chat-model
|
||||
behavior.
|
||||
|
||||
Reference in New Issue
Block a user