wuyang/llm-atlas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fb44d15bb8e71ed170affa69752cd67557d5f9c6
llm-atlas/experiments/deepseek/README.md
T
wuyang 96443d4dfc feat: add DeepSeek Chat behavior evidence
2026-07-29 22:47:49 +08:00

22 KiB
Raw Blame History

DeepSeek real-weight execution probes

The probes in this directory use official DeepSeek artifacts and keep their scope deliberately narrower than a full-model benchmark.

DeepSeek-V2-Lite truncated trace

v2_lite_trace.py executes layers 0–6 from the official BF16 checkpoint. Those seven layers are fully contained in shard 1; layer 7 is split across shards 1 and 2 and is therefore outside the default evidence boundary.

Pinned model:

deepseek-ai/DeepSeek-V2-Lite@604d5664dddd88a0433dbae533b7fe9472482de0

Required Python stack:

torch==2.11.0+cu128
transformers==4.41.2
safetensors==0.8.0

The 2024 remote code does not import under Transformers 5.5 because is_torch_fx_available was removed. The probe imports the official files as a read-only local package; it does not patch the model source.

Download the metadata, tokenizer, remote code, index, and first shard with the Hugging Face CLI, then run:

python experiments/deepseek/v2_lite_trace.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --output src/data/deepseek-v2-lite-trace.json

The result contains:

  • real tokenizer pieces and model-derived hidden states;
  • the actual [B,T,576] MLA compressed projection at each executed layer;
  • the expanded key/value tensors stored by the Hugging Face eager cache;
  • token-level top-6 routed expert IDs and weights for six MoE layers;
  • per-layer and per-prompt expert-load summaries;
  • explicit boundaries against global load, expert semantics, training traces, full-model generation, and production serving claims.

Run the probe twice and compare deterministic evidence while excluding timing:

python experiments/deepseek/compare_v2_lite_traces.py \
  --first /path/to/trace-1.json \
  --second /path/to/trace-2.json \
  --output src/data/deepseek-v2-lite-trace-repro.json

Fixed public routing corpus

v2_lite_routing_corpus.py keeps the same official layer 0–6 execution boundary but replaces the four authored prompts with 128 source-addressable public prompts:

  • 32 WikiText-2 raw validation passages;
  • 32 CLUE TNEWS public-test sentences;
  • 32 OpenAI HumanEval prompts, without solutions/tests or code execution;
  • 32 OpenAI GSM8K test questions, without answers.

Selection is the ascending SHA-256 rank of a fixed salt, domain, and source ID. Inputs are truncated to 96 DeepSeek tokens. The six MoE layers therefore produce 304,560 actual top-6 routed-expert selections over 8,460 valid tokens.

The output includes both token-weighted and prompt-balanced distributions. Its 95% intervals use 2,000 prompt-level bootstrap resamples within each domain, rather than treating correlated tokens as independent observations.

PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_routing_corpus.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-corpus.json \
  --per-domain 32 \
  --max-tokens 96 \
  --batch-size 16 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T07:45:00+00:00

The committed independent rerun is byte-exact. Both JSON files have SHA-256:

4678a1d15395de93ffba757598cc3642bf35e9e07f71d82ddd87c27fc38a09e4

See research/DEEPSEEK_ROUTING_CORPUS_AUDIT.md for corpus revisions and hashes, metric definitions, interval semantics, results, and claim boundaries.

Paired length-control cohort

The corpus runner can also select one fixed cohort by untruncated source length and execute nested prefixes. The committed 16-token and 24-token traces use the same 128 source prompts, all selected from records with at least 24 DeepSeek tokens:

common_args=(
  --per-domain 32
  --batch-size 16
  --bootstrap 2000
  --sample-salt llm-atlas-deepseek-routing-length-control-v1
  --eligibility-min-tokens 24
)

python -B experiments/deepseek/v2_lite_routing_corpus.py \
  ...source arguments... \
  "${common_args[@]}" \
  --max-tokens 16 \
  --output src/data/deepseek-v2-lite-routing-matched16.json

python -B experiments/deepseek/v2_lite_routing_corpus.py \
  ...source arguments... \
  "${common_args[@]}" \
  --max-tokens 24 \
  --output src/data/deepseek-v2-lite-routing-matched24.json

The two real traces add 184,320 top-6 route selections. Their independent reruns are byte-exact:

matched-16  f8d437d5379ffb41ac8dca5a8e97c0f44ba10ce7b63f95d7be0b7c88ac0baebd
matched-24  bed54835ad243ca2ab46bf9574e53137e6c2c0e19267f581719b5f2f65546436

Use the paired comparison runner to resample identical prompt indices in the short and long traces:

python -B experiments/deepseek/compare_routing_length_control.py \
  --short src/data/deepseek-v2-lite-routing-matched16.json \
  --long src/data/deepseek-v2-lite-routing-matched24.json \
  --output src/data/deepseek-v2-lite-routing-length-sensitivity.json \
  --bootstrap 2000 \
  --seed 20260729

See research/DEEPSEEK_ROUTING_LENGTH_CONTROL_AUDIT.md for the sampling bias audit, paired CV/JSD deltas, total-variation accounting, and interpretation boundaries.

Official chat-template sensitivity

v2_lite_routing_template_probe.py renders three variants of one fixed 23-content-token prefix:

raw         BOS + content
user        BOS + "User: " + content + "\n\n"
generation  user prefix + "Assistant:"

The latter two use the pinned official chat_template through apply_chat_template. All three variants of one source prompt execute in the same padded batch. Statistics are split between:

  • the full operational input, including wrapper tokens;
  • the exact intersection of (relative character span, token ID) inside the source content across all three variants.

The user → generation comparison is a causal negative control: the appended suffix must not change routes on their shared prefix.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_template_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-template.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 8 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T08:30:00+00:00

The three variants add 382,032 real top-6 route selections. Across six MoE layers, all 21,852 user → generation shared-prefix token routes are ordered-top-6 exact. The committed run and independent rerun are byte-exact:

da1f10333b2fa269e64f9716a0ca6c1a656d23d3e70b8a25d6e59f8a5b3bc1b9

See research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md for the aligned-content contract, paired intervals, per-token route stability, and claim boundaries.

System × one-shot message-history factorial

v2_lite_routing_history_factorial_probe.py reuses the exact 128-source, 23-content-token cohort from the official-template probe and renders four official chat histories:

S0F0  target user
S1F0  fixed system + target user
S0F1  fixed demo user/assistant + target user
S1F1  fixed system + fixed demo user/assistant + target user

Every condition uses add_generation_prompt=True. The fixed system treatment adds 16 tokens per source in both F0 and F1; the fixed one-shot treatment adds 17 tokens per source in both S0 and S1. All four variants of one source execute in the same padded batch.

The output contains:

  • full-input and exact target-content scopes;
  • token-weighted and prompt-balanced aggregation;
  • four paired factor edges;
  • system and one-shot main effects;
  • difference-in-differences interaction;
  • per-target-token ordered/set top-6 stability and Jaccard;
  • 2,000 source-prompt bootstrap replicates shared by all four cells.
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_factorial_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-factorial.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 8 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T09:36:00+00:00

The four cells add 865,440 real top-6 route selections. The committed run and independent rerun are byte-exact:

5765fbf85fa6fd948cca90f11e2237254fd58c32f373670f44530cff0a70c1fb

See research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md for the factorial definition, all CV intervals, the 24/24 system-edge TV attenuation pattern, per-token stability, and the strict boundary between a message-history effect and a role-semantic claim.

Equal-length filler history control

v2_lite_routing_history_distance_control.py extends the preceding 2×2 probe to six cells:

system off/on × no history / repeated-token filler / fixed one-shot

The filler turn is:

User: x x x x x x x x x
Assistant: x

Under the pinned official template, both filler and one-shot add exactly 17 tokens per source on both system levels. They therefore share the same user/assistant roles, assistant EOS, target position, generation prompt, and batch shape. The filler is deliberately called low-information rather than semantics-free: repeated x tokens remain learned inputs.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_distance_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-distance-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 5 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T10:17:00+00:00

The six cells add 1,376,496 real top-6 route selections. The committed run and independent rerun are byte-exact:

423a095d92738d4bc6c2ace2afa9b35627e61811458c7883dca7b44aee4e648e

For exact target content under prompt-balanced aggregation, mean system-edge TV is 0.0738 → 0.0378 → 0.0188 for none → filler → demo. Both steps decrease in 24/24 layer×domain cells, with all 24 paired intervals below zero. See research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md for the full table, token-level route stability, BF16 batch-shape boundary, literature context, and non-claims.

History-boundary single-token control

v2_lite_routing_history_boundary_token_control.py keeps the repeated-token history from the preceding probe and changes exactly one token ID at the completed assistant boundary:

system off/on × official EOS / x / period / newline

The official template places EOS between the filler assistant content and the next User: marker. The three controls replace only that EOS ID after official tokenization. They are explicit counterfactual token sequences, not valid official chat serializations. All eight conditions preserve sequence length, target position, role markers, attention mask, and within-run batch shape.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_boundary_token_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-boundary-token-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T11:12:00+00:00

The eight cells add 2,044,224 real top-6 route selections. All 256 source×system groups are equal-length and equal-position; all 768 counterfactual cells differ from their official sequence at exactly one input ID. The committed run and independent rerun are byte-exact:

9bb93834ffd8536aeebe4325e45d2179ba590554c6ff6b6fcceaba2f499b9c37

For exact target content under prompt-balanced aggregation, mean system-edge TV is .0374 / .0539 / .0553 / .0492 for EOS / x / period / newline. X and period exceed EOS in 24/24 layer×domain cells, while newline does so in 23/24. See research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md for all paired intervals, per-token route alignment, scope split, BF16 batch-shape audit, primary literature, and the boundary between an input-ID intervention and a chat-turn semantic claim.

Role-marker-head single-token control

v2_lite_routing_role_marker_head_control.py keeps the official filler history, EOS, colon, target span, generation prompt, mask, length, and 32-row batch shape, while changing one ordinary token ID:

system off/on × official / pre-target User→Assistant /
                    pre-target User→x / post-target Assistant→User

The pinned tokenizer maps User, Assistant, :, and x to IDs 5726, 77398, 25, and 87. The pre-target controls identify the effect of the first role-marker token only; the colon remains. The post-target replacement is a causal suffix negative control. None of the three counterfactuals is a valid official chat serialization.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_role_marker_head_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-role-marker-head-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T12:30:00+00:00

The eight cells add 2,044,224 real top-6 route selections. All 256 source×system groups preserve length and target position; every one of the 768 counterfactuals differs from its official sequence at exactly one ID. The committed run and independent rerun are byte-exact:

9dc0e37fbce6581269428dcfcb84c7a17b466c5239c8171e6741f66d5eeb8caf

For exact target content, direct replacement TV is about .02–.03, but the system-edge contrast has mixed direction: 12↑12↓ for User→Assistant and 13↑11↓ for User→x. The post-target control is exact for all 34,488 aligned target-token ordered top-6 routes, with zero target TV, JSD, and ΔCV. See research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md for all paired intervals, depth maps, token-level alignment, cross-experiment BF16 batch-content audit, primary sources, and the boundary against full role semantics or Chat-model behavior.

Special-token family control

v2_lite_routing_special_token_family_control.py keeps the same repeated-token history and changes the completed assistant boundary to four single IDs:

system off/on × EOS / BOS / x / period

The pinned tokenizer has exactly two special-token IDs: BOS 100000 and EOS 100001; PAD aliases EOS. EOS/BOS therefore exhaust the special inventory, while x and period are only two selected ordinary controls. The 2-vs-2 family summary is descriptive for these four IDs and is not a population-level specialness claim.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_special_token_family_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T12:57:00+00:00

The eight cells add 2,044,224 real top-6 route selections. All 256 source×system groups preserve length and target position, and all 768 counterfactuals change exactly one ID. The committed run and independent rerun are byte-exact:

c372c1b03a8b15f615b54ded5d9257a8fc2cdb7728001735d3d4c1d8534af5bf

For exact target content under prompt-balanced aggregation, mean system-edge TV is .037518 / .048018 / .053975 / .055289 for EOS / BOS / x / period. BOS−EOS is positive in 22/24 layer×domain cells; x−EOS and period−EOS are positive in 24/24. The selected ordinary-control mean exceeds the complete special-inventory mean by .011864 in these four IDs only. See research/DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md for all paired intervals, direct edges, scope split, alignment, batch-content audit, literature context, and non-claims.

Full two-token role-marker factorial

v2_lite_routing_role_marker_block_factorial.py treats the pre-target two-token marker as two independent factors:

system off/on × head User/Assistant × delimiter colon/x

User:       [5726, 25]      Assistant:  [77398, 25]
User x      [5726, 87]      Assistant x [77398, 87]

The official assistant EOS and post-target generation suffix remain unchanged. The four blocks have zero, one, one, and two edited IDs relative to official User:; every edit is verified while length, target position, attention mask, and 32-row batch shape stay fixed.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_role_marker_block_factorial.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-role-marker-block-factorial.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T13:06:00+00:00

The eight cells add 2,044,224 real top-6 route selections. The 256 official, 512 one-ID, and 256 two-ID cells all pass their exact edit contracts. The committed run and independent rerun are byte-exact:

a703dddb6d04182b1a608c213bfd341cfc32e1801be42c417dca30a505087e82

For exact target content under prompt-balanced aggregation, the four mean system-edge TVs are .037304 / .037015 / .037239 / .036220. The head, delimiter, and head×delimiter effects are small and mixed across the 24 layer×domain cells. Direct head and delimiter edges remain nonzero; replacing colon with the single x control modestly reduces User-vs-Assistant direct TV in most cells, without a uniform system modulation. See research/DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md for the exact factor coding, intervals, direct dependencies, full-input split, route alignment, cross-batch audit, sources, and non-claims.

Full Chat-checkpoint generation behavior

v2_lite_chat_special_token_behavior_probe.py carries the special-token grid into the full official SFT Chat checkpoint:

deepseek-ai/DeepSeek-V2-Lite-Chat
@85864749cd611b4353ce1decdb286193298f64c7

system off/on × EOS / BOS / x / period

It reuses the pinned Base-routing source IDs and ranks but generates from the full source text. All eight variants of one source run in one left-padded batch. Decoding is deterministic greedy; BOS / x / period remain invalid-chat single-ID counterfactuals.

The official BF16 checkpoint contains 31,412,968,448 tensor bytes. Because the local RTX 5090 has less than the official 40GB single-GPU boundary, the formal run uses Accelerate device_map=auto: embeddings and layers 0–24 are placed on CUDA, while layers 25–26, final norm, and LM head are CPU-offloaded.

PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_chat_special_token_behavior_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite-chat \
  --reference-routing-json \
    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-chat-behavior.json \
  --per-domain 4 \
  --max-new-tokens 128 \
  --gpu-memory 29GiB \
  --cpu-memory 80GiB

The formal artifact contains 16 sources and 128 outputs. Only 31 outputs hit EOS; 97 reach the 128-token cap, so task-score fields are diagnostic and are not used for capability ranking. A 4-source / 32-cell long-sequence rerun is exact for prompt hashes, generated token IDs, decoded text, and EOS state.

formal  54496955d0dd20a3116e6b758a55f56e2444c43dd99bfef94a2cb7b140e6196e
repro   b74c31606c0fc70a8e52cc6c3d6135c4e916d698b4d5dae51863124d025b0c72

See research/DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_PROTOCOL.md and research/DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_AUDIT.md for the preregistered contract, output-divergence table, offload device map, truncation boundary, rerun coverage, execution fixes, primary sources, and forbidden conclusions.

Reference in New Issue View Git Blame Copy Permalink