feat: isolate DeepSeek history boundary token
This commit is contained in:
@@ -319,3 +319,55 @@ in 24/24 layer×domain cells, with all 24 paired intervals below zero. See
|
||||
`research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md` for the full
|
||||
table, token-level route stability, BF16 batch-shape boundary, literature
|
||||
context, and non-claims.
|
||||
|
||||
## History-boundary single-token control
|
||||
|
||||
`v2_lite_routing_history_boundary_token_control.py` keeps the repeated-token
|
||||
history from the preceding probe and changes exactly one token ID at the
|
||||
completed assistant boundary:
|
||||
|
||||
```text
|
||||
system off/on × official EOS / x / period / newline
|
||||
```
|
||||
|
||||
The official template places EOS between the filler assistant content and the
|
||||
next `User:` marker. The three controls replace only that EOS ID after official
|
||||
tokenization. They are explicit counterfactual token sequences, not valid
|
||||
official chat serializations. All eight conditions preserve sequence length,
|
||||
target position, role markers, attention mask, and within-run batch shape.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||||
python -B experiments/deepseek/v2_lite_routing_history_boundary_token_control.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-history-boundary-token-control.json \
|
||||
--per-domain 32 \
|
||||
--content-tokens 23 \
|
||||
--batch-prompts 4 \
|
||||
--layers 7 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T11:12:00+00:00
|
||||
```
|
||||
|
||||
The eight cells add 2,044,224 real top-6 route selections. All 256
|
||||
source×system groups are equal-length and equal-position; all 768
|
||||
counterfactual cells differ from their official sequence at exactly one input
|
||||
ID. The committed run and independent rerun are byte-exact:
|
||||
|
||||
```text
|
||||
9bb93834ffd8536aeebe4325e45d2179ba590554c6ff6b6fcceaba2f499b9c37
|
||||
```
|
||||
|
||||
For exact target content under prompt-balanced aggregation, mean system-edge
|
||||
TV is `.0374 / .0539 / .0553 / .0492` for EOS / x / period / newline. X and
|
||||
period exceed EOS in 24/24 layer×domain cells, while newline does so in 23/24.
|
||||
See `research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md` for all paired
|
||||
intervals, per-token route alignment, scope split, BF16 batch-shape audit,
|
||||
primary literature, and the boundary between an input-ID intervention and a
|
||||
chat-turn semantic claim.
|
||||
|
||||
Reference in New Issue
Block a user