feat: isolate DeepSeek history boundary token

This commit is contained in:
wuyang
2026-07-29 20:00:37 +08:00
parent f5a69170c7
commit f4d4a05dfa
14 changed files with 3676654 additions and 33 deletions
+52
View File
@@ -319,3 +319,55 @@ in 24/24 layer×domain cells, with all 24 paired intervals below zero. See
`research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md` for the full
table, token-level route stability, BF16 batch-shape boundary, literature
context, and non-claims.
## History-boundary single-token control
`v2_lite_routing_history_boundary_token_control.py` keeps the repeated-token
history from the preceding probe and changes exactly one token ID at the
completed assistant boundary:
```text
system off/on × official EOS / x / period / newline
```
The official template places EOS between the filler assistant content and the
next `User:` marker. The three controls replace only that EOS ID after official
tokenization. They are explicit counterfactual token sequences, not valid
official chat serializations. All eight conditions preserve sequence length,
target position, role markers, attention mask, and within-run batch shape.
```bash
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_boundary_token_control.py \
--artifact-dir /path/to/deepseek-v2-lite \
--human-eval /path/to/HumanEval.jsonl.gz \
--gsm8k /path/to/gsm8k/test.jsonl \
--tnews /path/to/tnews/test.json \
--tnews-archive /path/to/tnews_public.zip \
--wikitext /path/to/wikitext-validation.parquet \
--output src/data/deepseek-v2-lite-routing-history-boundary-token-control.json \
--per-domain 32 \
--content-tokens 23 \
--batch-prompts 4 \
--layers 7 \
--bootstrap 2000 \
--seed 20260729 \
--captured-at 2026-07-29T11:12:00+00:00
```
The eight cells add 2,044,224 real top-6 route selections. All 256
source×system groups are equal-length and equal-position; all 768
counterfactual cells differ from their official sequence at exactly one input
ID. The committed run and independent rerun are byte-exact:
```text
9bb93834ffd8536aeebe4325e45d2179ba590554c6ff6b6fcceaba2f499b9c37
```
For exact target content under prompt-balanced aggregation, mean system-edge
TV is `.0374 / .0539 / .0553 / .0492` for EOS / x / period / newline. X and
period exceed EOS in 24/24 layer×domain cells, while newline does so in 23/24.
See `research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md` for all paired
intervals, per-token route alignment, scope split, BF16 batch-shape audit,
primary literature, and the boundary between an input-ID intervention and a
chat-turn semantic claim.