feat: add DeepSeek message-history factorial probe
This commit is contained in:
@@ -205,3 +205,62 @@ da1f10333b2fa269e64f9716a0ca6c1a656d23d3e70b8a25d6e59f8a5b3bc1b9
|
||||
|
||||
See `research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md` for the aligned-content
|
||||
contract, paired intervals, per-token route stability, and claim boundaries.
|
||||
|
||||
## System × one-shot message-history factorial
|
||||
|
||||
`v2_lite_routing_history_factorial_probe.py` reuses the exact 128-source,
|
||||
23-content-token cohort from the official-template probe and renders four
|
||||
official chat histories:
|
||||
|
||||
```text
|
||||
S0F0 target user
|
||||
S1F0 fixed system + target user
|
||||
S0F1 fixed demo user/assistant + target user
|
||||
S1F1 fixed system + fixed demo user/assistant + target user
|
||||
```
|
||||
|
||||
Every condition uses `add_generation_prompt=True`. The fixed system treatment
|
||||
adds 16 tokens per source in both `F0` and `F1`; the fixed one-shot treatment
|
||||
adds 17 tokens per source in both `S0` and `S1`. All four variants of one
|
||||
source execute in the same padded batch.
|
||||
|
||||
The output contains:
|
||||
|
||||
- full-input and exact target-content scopes;
|
||||
- token-weighted and prompt-balanced aggregation;
|
||||
- four paired factor edges;
|
||||
- system and one-shot main effects;
|
||||
- difference-in-differences interaction;
|
||||
- per-target-token ordered/set top-6 stability and Jaccard;
|
||||
- 2,000 source-prompt bootstrap replicates shared by all four cells.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||||
python -B experiments/deepseek/v2_lite_routing_history_factorial_probe.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-history-factorial.json \
|
||||
--per-domain 32 \
|
||||
--content-tokens 23 \
|
||||
--batch-prompts 8 \
|
||||
--layers 7 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T09:36:00+00:00
|
||||
```
|
||||
|
||||
The four cells add 865,440 real top-6 route selections. The committed run and
|
||||
independent rerun are byte-exact:
|
||||
|
||||
```text
|
||||
5765fbf85fa6fd948cca90f11e2237254fd58c32f373670f44530cff0a70c1fb
|
||||
```
|
||||
|
||||
See `research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md` for the factorial
|
||||
definition, all CV intervals, the 24/24 system-edge TV attenuation pattern,
|
||||
per-token stability, and the strict boundary between a message-history effect
|
||||
and a role-semantic claim.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user