llm-atlas/experiments/deepseek

DeepSeek real-weight execution probes

The probes in this directory use official DeepSeek artifacts and keep their scope deliberately narrower than a full-model benchmark.

DeepSeek-V2-Lite truncated trace

v2_lite_trace.py executes layers 0–6 from the official BF16 checkpoint. Those seven layers are fully contained in shard 1; layer 7 is split across shards 1 and 2 and is therefore outside the default evidence boundary.

Pinned model:

deepseek-ai/DeepSeek-V2-Lite@604d5664dddd88a0433dbae533b7fe9472482de0

Required Python stack:

torch==2.11.0+cu128
transformers==4.41.2
safetensors==0.8.0

The 2024 remote code does not import under Transformers 5.5 because is_torch_fx_available was removed. The probe imports the official files as a read-only local package; it does not patch the model source.

Download the metadata, tokenizer, remote code, index, and first shard with the Hugging Face CLI, then run:

python experiments/deepseek/v2_lite_trace.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --output src/data/deepseek-v2-lite-trace.json

The result contains:

• real tokenizer pieces and model-derived hidden states;
• the actual [B,T,576] MLA compressed projection at each executed layer;
• the expanded key/value tensors stored by the Hugging Face eager cache;
• token-level top-6 routed expert IDs and weights for six MoE layers;
• per-layer and per-prompt expert-load summaries;
• explicit boundaries against global load, expert semantics, training traces, full-model generation, and production serving claims.

Run the probe twice and compare deterministic evidence while excluding timing:

python experiments/deepseek/compare_v2_lite_traces.py \
  --first /path/to/trace-1.json \
  --second /path/to/trace-2.json \
  --output src/data/deepseek-v2-lite-trace-repro.json

Fixed public routing corpus

v2_lite_routing_corpus.py keeps the same official layer 0–6 execution boundary but replaces the four authored prompts with 128 source-addressable public prompts:

• 32 WikiText-2 raw validation passages;
• 32 CLUE TNEWS public-test sentences;
• 32 OpenAI HumanEval prompts, without solutions/tests or code execution;
• 32 OpenAI GSM8K test questions, without answers.

Selection is the ascending SHA-256 rank of a fixed salt, domain, and source ID. Inputs are truncated to 96 DeepSeek tokens. The six MoE layers therefore produce 304,560 actual top-6 routed-expert selections over 8,460 valid tokens.

The output includes both token-weighted and prompt-balanced distributions. Its 95% intervals use 2,000 prompt-level bootstrap resamples within each domain, rather than treating correlated tokens as independent observations.

PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_routing_corpus.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-corpus.json \
  --per-domain 32 \
  --max-tokens 96 \
  --batch-size 16 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T07:45:00+00:00

The committed independent rerun is byte-exact. Both JSON files have SHA-256:

4678a1d15395de93ffba757598cc3642bf35e9e07f71d82ddd87c27fc38a09e4

See research/DEEPSEEK_ROUTING_CORPUS_AUDIT.md for corpus revisions and hashes, metric definitions, interval semantics, results, and claim boundaries.

Paired length-control cohort

The corpus runner can also select one fixed cohort by untruncated source length and execute nested prefixes. The committed 16-token and 24-token traces use the same 128 source prompts, all selected from records with at least 24 DeepSeek tokens:

common_args=(
  --per-domain 32
  --batch-size 16
  --bootstrap 2000
  --sample-salt llm-atlas-deepseek-routing-length-control-v1
  --eligibility-min-tokens 24
)

python -B experiments/deepseek/v2_lite_routing_corpus.py \
  ...source arguments... \
  "${common_args[@]}" \
  --max-tokens 16 \
  --output src/data/deepseek-v2-lite-routing-matched16.json

python -B experiments/deepseek/v2_lite_routing_corpus.py \
  ...source arguments... \
  "${common_args[@]}" \
  --max-tokens 24 \
  --output src/data/deepseek-v2-lite-routing-matched24.json

The two real traces add 184,320 top-6 route selections. Their independent reruns are byte-exact:

matched-16  f8d437d5379ffb41ac8dca5a8e97c0f44ba10ce7b63f95d7be0b7c88ac0baebd
matched-24  bed54835ad243ca2ab46bf9574e53137e6c2c0e19267f581719b5f2f65546436

Use the paired comparison runner to resample identical prompt indices in the short and long traces:

python -B experiments/deepseek/compare_routing_length_control.py \
  --short src/data/deepseek-v2-lite-routing-matched16.json \
  --long src/data/deepseek-v2-lite-routing-matched24.json \
  --output src/data/deepseek-v2-lite-routing-length-sensitivity.json \
  --bootstrap 2000 \
  --seed 20260729

See research/DEEPSEEK_ROUTING_LENGTH_CONTROL_AUDIT.md for the sampling bias audit, paired CV/JSD deltas, total-variation accounting, and interpretation boundaries.

Official chat-template sensitivity

v2_lite_routing_template_probe.py renders three variants of one fixed 23-content-token prefix:

raw         BOS + content
user        BOS + "User: " + content + "\n\n"
generation  user prefix + "Assistant:"

The latter two use the pinned official chat_template through apply_chat_template. All three variants of one source prompt execute in the same padded batch. Statistics are split between:

• the full operational input, including wrapper tokens;
• the exact intersection of (relative character span, token ID) inside the source content across all three variants.

The user → generation comparison is a causal negative control: the appended suffix must not change routes on their shared prefix.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_template_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-template.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 8 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T08:30:00+00:00

The three variants add 382,032 real top-6 route selections. Across six MoE layers, all 21,852 user → generation shared-prefix token routes are ordered-top-6 exact. The committed run and independent rerun are byte-exact:

da1f10333b2fa269e64f9716a0ca6c1a656d23d3e70b8a25d6e59f8a5b3bc1b9

See research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md for the aligned-content contract, paired intervals, per-token route stability, and claim boundaries.

System × one-shot message-history factorial

v2_lite_routing_history_factorial_probe.py reuses the exact 128-source, 23-content-token cohort from the official-template probe and renders four official chat histories:

S0F0  target user
S1F0  fixed system + target user
S0F1  fixed demo user/assistant + target user
S1F1  fixed system + fixed demo user/assistant + target user

Every condition uses add_generation_prompt=True. The fixed system treatment adds 16 tokens per source in both F0 and F1; the fixed one-shot treatment adds 17 tokens per source in both S0 and S1. All four variants of one source execute in the same padded batch.

The output contains:

• full-input and exact target-content scopes;
• token-weighted and prompt-balanced aggregation;
• four paired factor edges;
• system and one-shot main effects;
• difference-in-differences interaction;
• per-target-token ordered/set top-6 stability and Jaccard;
• 2,000 source-prompt bootstrap replicates shared by all four cells.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_factorial_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-factorial.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 8 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T09:36:00+00:00

The four cells add 865,440 real top-6 route selections. The committed run and independent rerun are byte-exact:

5765fbf85fa6fd948cca90f11e2237254fd58c32f373670f44530cff0a70c1fb

See research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md for the factorial definition, all CV intervals, the 24/24 system-edge TV attenuation pattern, per-token stability, and the strict boundary between a message-history effect and a role-semantic claim.