llm-atlas/experiments/deepseek/README.md

# DeepSeek real-weight execution probes

The probes in this directory use official DeepSeek artifacts and keep their
scope deliberately narrower than a full-model benchmark.

## DeepSeek-V2-Lite truncated trace

`v2_lite_trace.py` executes layers 0–6 from the official BF16 checkpoint. Those
seven layers are fully contained in shard 1; layer 7 is split across shards 1
and 2 and is therefore outside the default evidence boundary.

Pinned model:

```text
deepseek-ai/DeepSeek-V2-Lite@604d5664dddd88a0433dbae533b7fe9472482de0
```

Required Python stack:

```text
torch==2.11.0+cu128
transformers==4.41.2
safetensors==0.8.0
```

The 2024 remote code does not import under Transformers 5.5 because
`is_torch_fx_available` was removed. The probe imports the official files as a
read-only local package; it does not patch the model source.

Download the metadata, tokenizer, remote code, index, and first shard with the
Hugging Face CLI, then run:

```bash
python experiments/deepseek/v2_lite_trace.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --output src/data/deepseek-v2-lite-trace.json
```

The result contains:

- real tokenizer pieces and model-derived hidden states;
- the actual `[B,T,576]` MLA compressed projection at each executed layer;
- the expanded key/value tensors stored by the Hugging Face eager cache;
- token-level top-6 routed expert IDs and weights for six MoE layers;
- per-layer and per-prompt expert-load summaries;
- explicit boundaries against global load, expert semantics, training traces,
  full-model generation, and production serving claims.

Run the probe twice and compare deterministic evidence while excluding timing:

```bash
python experiments/deepseek/compare_v2_lite_traces.py \
  --first /path/to/trace-1.json \
  --second /path/to/trace-2.json \
  --output src/data/deepseek-v2-lite-trace-repro.json
```

## Fixed public routing corpus

`v2_lite_routing_corpus.py` keeps the same official layer 0–6 execution boundary
but replaces the four authored prompts with 128 source-addressable public
prompts:

- 32 WikiText-2 raw validation passages;
- 32 CLUE TNEWS public-test sentences;
- 32 OpenAI HumanEval prompts, without solutions/tests or code execution;
- 32 OpenAI GSM8K test questions, without answers.

Selection is the ascending SHA-256 rank of a fixed salt, domain, and source ID.
Inputs are truncated to 96 DeepSeek tokens. The six MoE layers therefore produce
304,560 actual top-6 routed-expert selections over 8,460 valid tokens.

The output includes both token-weighted and prompt-balanced distributions. Its
95% intervals use 2,000 prompt-level bootstrap resamples within each domain,
rather than treating correlated tokens as independent observations.

```bash
PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_routing_corpus.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-corpus.json \
  --per-domain 32 \
  --max-tokens 96 \
  --batch-size 16 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T07:45:00+00:00
```

The committed independent rerun is byte-exact. Both JSON files have SHA-256:

```text
4678a1d15395de93ffba757598cc3642bf35e9e07f71d82ddd87c27fc38a09e4
```

See `research/DEEPSEEK_ROUTING_CORPUS_AUDIT.md` for corpus revisions and hashes,
metric definitions, interval semantics, results, and claim boundaries.

## Paired length-control cohort

The corpus runner can also select one fixed cohort by untruncated source length
and execute nested prefixes. The committed 16-token and 24-token traces use the
same 128 source prompts, all selected from records with at least 24 DeepSeek
tokens:

```bash
common_args=(
  --per-domain 32
  --batch-size 16
  --bootstrap 2000
  --sample-salt llm-atlas-deepseek-routing-length-control-v1
  --eligibility-min-tokens 24
)

python -B experiments/deepseek/v2_lite_routing_corpus.py \
  ...source arguments... \
  "${common_args[@]}" \
  --max-tokens 16 \
  --output src/data/deepseek-v2-lite-routing-matched16.json

python -B experiments/deepseek/v2_lite_routing_corpus.py \
  ...source arguments... \
  "${common_args[@]}" \
  --max-tokens 24 \
  --output src/data/deepseek-v2-lite-routing-matched24.json
```

The two real traces add 184,320 top-6 route selections. Their independent
reruns are byte-exact:

```text
matched-16  f8d437d5379ffb41ac8dca5a8e97c0f44ba10ce7b63f95d7be0b7c88ac0baebd
matched-24  bed54835ad243ca2ab46bf9574e53137e6c2c0e19267f581719b5f2f65546436
```

Use the paired comparison runner to resample identical prompt indices in the
short and long traces:

```bash
python -B experiments/deepseek/compare_routing_length_control.py \
  --short src/data/deepseek-v2-lite-routing-matched16.json \
  --long src/data/deepseek-v2-lite-routing-matched24.json \
  --output src/data/deepseek-v2-lite-routing-length-sensitivity.json \
  --bootstrap 2000 \
  --seed 20260729
```

See `research/DEEPSEEK_ROUTING_LENGTH_CONTROL_AUDIT.md` for the sampling bias
audit, paired CV/JSD deltas, total-variation accounting, and interpretation
boundaries.

## Official chat-template sensitivity

`v2_lite_routing_template_probe.py` renders three variants of one fixed
23-content-token prefix:

```text
raw         BOS + content
user        BOS + "User: " + content + "\n\n"
generation  user prefix + "Assistant:"
```

The latter two use the pinned official `chat_template` through
`apply_chat_template`. All three variants of one source prompt execute in the
same padded batch. Statistics are split between:

- the full operational input, including wrapper tokens;
- the exact intersection of `(relative character span, token ID)` inside the
  source content across all three variants.

The `user → generation` comparison is a causal negative control: the appended
suffix must not change routes on their shared prefix.

```bash
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_template_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-template.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 8 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T08:30:00+00:00
```

The three variants add 382,032 real top-6 route selections. Across six MoE
layers, all 21,852 `user → generation` shared-prefix token routes are
ordered-top-6 exact. The committed run and independent rerun are byte-exact:

```text
da1f10333b2fa269e64f9716a0ca6c1a656d23d3e70b8a25d6e59f8a5b3bc1b9
```

See `research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md` for the aligned-content
contract, paired intervals, per-token route stability, and claim boundaries.

## System × one-shot message-history factorial

`v2_lite_routing_history_factorial_probe.py` reuses the exact 128-source,
23-content-token cohort from the official-template probe and renders four
official chat histories:

```text
S0F0  target user
S1F0  fixed system + target user
S0F1  fixed demo user/assistant + target user
S1F1  fixed system + fixed demo user/assistant + target user
```

Every condition uses `add_generation_prompt=True`. The fixed system treatment
adds 16 tokens per source in both `F0` and `F1`; the fixed one-shot treatment
adds 17 tokens per source in both `S0` and `S1`. All four variants of one
source execute in the same padded batch.

The output contains:

- full-input and exact target-content scopes;
- token-weighted and prompt-balanced aggregation;
- four paired factor edges;
- system and one-shot main effects;
- difference-in-differences interaction;
- per-target-token ordered/set top-6 stability and Jaccard;
- 2,000 source-prompt bootstrap replicates shared by all four cells.

```bash
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_factorial_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-factorial.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 8 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T09:36:00+00:00
```

The four cells add 865,440 real top-6 route selections. The committed run and
independent rerun are byte-exact:

```text
5765fbf85fa6fd948cca90f11e2237254fd58c32f373670f44530cff0a70c1fb
```

See `research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md` for the factorial
definition, all CV intervals, the 24/24 system-edge TV attenuation pattern,
per-token stability, and the strict boundary between a message-history effect
and a role-semantic claim.

## Equal-length filler history control

`v2_lite_routing_history_distance_control.py` extends the preceding 2×2 probe
to six cells:

```text
system off/on × no history / repeated-token filler / fixed one-shot
```

The filler turn is:

```text
User: x x x x x x x x x
Assistant: x
```

Under the pinned official template, both filler and one-shot add exactly 17
tokens per source on both system levels. They therefore share the same
user/assistant roles, assistant EOS, target position, generation prompt, and
batch shape. The filler is deliberately called low-information rather than
semantics-free: repeated `x` tokens remain learned inputs.

```bash
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_distance_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-distance-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 5 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T10:17:00+00:00
```

The six cells add 1,376,496 real top-6 route selections. The committed run and
independent rerun are byte-exact:

```text
423a095d92738d4bc6c2ace2afa9b35627e61811458c7883dca7b44aee4e648e
```

For exact target content under prompt-balanced aggregation, mean system-edge
TV is `0.0738 → 0.0378 → 0.0188` for none → filler → demo. Both steps decrease
in 24/24 layer×domain cells, with all 24 paired intervals below zero. See
`research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md` for the full
table, token-level route stability, BF16 batch-shape boundary, literature
context, and non-claims.

## History-boundary single-token control

`v2_lite_routing_history_boundary_token_control.py` keeps the repeated-token
history from the preceding probe and changes exactly one token ID at the
completed assistant boundary:

```text
system off/on × official EOS / x / period / newline
```

The official template places EOS between the filler assistant content and the
next `User:` marker. The three controls replace only that EOS ID after official
tokenization. They are explicit counterfactual token sequences, not valid
official chat serializations. All eight conditions preserve sequence length,
target position, role markers, attention mask, and within-run batch shape.

```bash
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_boundary_token_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-boundary-token-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T11:12:00+00:00
```

The eight cells add 2,044,224 real top-6 route selections. All 256
source×system groups are equal-length and equal-position; all 768
counterfactual cells differ from their official sequence at exactly one input
ID. The committed run and independent rerun are byte-exact:

```text
9bb93834ffd8536aeebe4325e45d2179ba590554c6ff6b6fcceaba2f499b9c37
```

For exact target content under prompt-balanced aggregation, mean system-edge
TV is `.0374 / .0539 / .0553 / .0492` for EOS / x / period / newline. X and
period exceed EOS in 24/24 layer×domain cells, while newline does so in 23/24.
See `research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md` for all paired
intervals, per-token route alignment, scope split, BF16 batch-shape audit,
primary literature, and the boundary between an input-ID intervention and a
chat-turn semantic claim.

## Role-marker-head single-token control

`v2_lite_routing_role_marker_head_control.py` keeps the official filler
history, EOS, colon, target span, generation prompt, mask, length, and 32-row
batch shape, while changing one ordinary token ID:

```text
system off/on × official / pre-target User→Assistant /
                    pre-target User→x / post-target Assistant→User
```

The pinned tokenizer maps `User`, `Assistant`, `:`, and `x` to IDs `5726`,
`77398`, `25`, and `87`. The pre-target controls identify the effect of the
first role-marker token only; the colon remains. The post-target replacement
is a causal suffix negative control. None of the three counterfactuals is a
valid official chat serialization.

```bash
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_role_marker_head_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-role-marker-head-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T12:30:00+00:00
```

The eight cells add 2,044,224 real top-6 route selections. All 256
source×system groups preserve length and target position; every one of the
768 counterfactuals differs from its official sequence at exactly one ID. The
committed run and independent rerun are byte-exact:

```text
9dc0e37fbce6581269428dcfcb84c7a17b466c5239c8171e6741f66d5eeb8caf
```

For exact target content, direct replacement TV is about `.02–.03`, but the
system-edge contrast has mixed direction: `12↑12↓` for User→Assistant and
`13↑11↓` for User→x. The post-target control is exact for all 34,488 aligned
target-token ordered top-6 routes, with zero target TV, JSD, and ΔCV. See
`research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md` for all paired intervals,
depth maps, token-level alignment, cross-experiment BF16 batch-content audit,
primary sources, and the boundary against full role semantics or Chat-model
behavior.