wuyang/llm-atlas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
54c80fb115994b1f6997544855c4903db81c4c1d
llm-atlas/experiments/deepseek/README.md
T
wuyang f041c15e81 research: audit task bootstrap CRN experiment
2026-07-30 05:59:38 +08:00

38 KiB
Raw Blame History

DeepSeek real-weight execution probes

The probes in this directory use official DeepSeek artifacts and keep their scope deliberately narrower than a full-model benchmark.

DeepSeek-V2-Lite truncated trace

v2_lite_trace.py executes layers 0–6 from the official BF16 checkpoint. Those seven layers are fully contained in shard 1; layer 7 is split across shards 1 and 2 and is therefore outside the default evidence boundary.

Pinned model:

deepseek-ai/DeepSeek-V2-Lite@604d5664dddd88a0433dbae533b7fe9472482de0

Required Python stack:

torch==2.11.0+cu128
transformers==4.41.2
safetensors==0.8.0

The 2024 remote code does not import under Transformers 5.5 because is_torch_fx_available was removed. The probe imports the official files as a read-only local package; it does not patch the model source.

Download the metadata, tokenizer, remote code, index, and first shard with the Hugging Face CLI, then run:

python experiments/deepseek/v2_lite_trace.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --output src/data/deepseek-v2-lite-trace.json

The result contains:

  • real tokenizer pieces and model-derived hidden states;
  • the actual [B,T,576] MLA compressed projection at each executed layer;
  • the expanded key/value tensors stored by the Hugging Face eager cache;
  • token-level top-6 routed expert IDs and weights for six MoE layers;
  • per-layer and per-prompt expert-load summaries;
  • explicit boundaries against global load, expert semantics, training traces, full-model generation, and production serving claims.

Run the probe twice and compare deterministic evidence while excluding timing:

python experiments/deepseek/compare_v2_lite_traces.py \
  --first /path/to/trace-1.json \
  --second /path/to/trace-2.json \
  --output src/data/deepseek-v2-lite-trace-repro.json

Fixed public routing corpus

v2_lite_routing_corpus.py keeps the same official layer 0–6 execution boundary but replaces the four authored prompts with 128 source-addressable public prompts:

  • 32 WikiText-2 raw validation passages;
  • 32 CLUE TNEWS public-test sentences;
  • 32 OpenAI HumanEval prompts, without solutions/tests or code execution;
  • 32 OpenAI GSM8K test questions, without answers.

Selection is the ascending SHA-256 rank of a fixed salt, domain, and source ID. Inputs are truncated to 96 DeepSeek tokens. The six MoE layers therefore produce 304,560 actual top-6 routed-expert selections over 8,460 valid tokens.

The output includes both token-weighted and prompt-balanced distributions. Its 95% intervals use 2,000 prompt-level bootstrap resamples within each domain, rather than treating correlated tokens as independent observations.

PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_routing_corpus.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-corpus.json \
  --per-domain 32 \
  --max-tokens 96 \
  --batch-size 16 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T07:45:00+00:00

The committed independent rerun is byte-exact. Both JSON files have SHA-256:

4678a1d15395de93ffba757598cc3642bf35e9e07f71d82ddd87c27fc38a09e4

See research/DEEPSEEK_ROUTING_CORPUS_AUDIT.md for corpus revisions and hashes, metric definitions, interval semantics, results, and claim boundaries.

Paired length-control cohort

The corpus runner can also select one fixed cohort by untruncated source length and execute nested prefixes. The committed 16-token and 24-token traces use the same 128 source prompts, all selected from records with at least 24 DeepSeek tokens:

common_args=(
  --per-domain 32
  --batch-size 16
  --bootstrap 2000
  --sample-salt llm-atlas-deepseek-routing-length-control-v1
  --eligibility-min-tokens 24
)

python -B experiments/deepseek/v2_lite_routing_corpus.py \
  ...source arguments... \
  "${common_args[@]}" \
  --max-tokens 16 \
  --output src/data/deepseek-v2-lite-routing-matched16.json

python -B experiments/deepseek/v2_lite_routing_corpus.py \
  ...source arguments... \
  "${common_args[@]}" \
  --max-tokens 24 \
  --output src/data/deepseek-v2-lite-routing-matched24.json

The two real traces add 184,320 top-6 route selections. Their independent reruns are byte-exact:

matched-16  f8d437d5379ffb41ac8dca5a8e97c0f44ba10ce7b63f95d7be0b7c88ac0baebd
matched-24  bed54835ad243ca2ab46bf9574e53137e6c2c0e19267f581719b5f2f65546436

Use the paired comparison runner to resample identical prompt indices in the short and long traces:

python -B experiments/deepseek/compare_routing_length_control.py \
  --short src/data/deepseek-v2-lite-routing-matched16.json \
  --long src/data/deepseek-v2-lite-routing-matched24.json \
  --output src/data/deepseek-v2-lite-routing-length-sensitivity.json \
  --bootstrap 2000 \
  --seed 20260729

See research/DEEPSEEK_ROUTING_LENGTH_CONTROL_AUDIT.md for the sampling bias audit, paired CV/JSD deltas, total-variation accounting, and interpretation boundaries.

Official chat-template sensitivity

v2_lite_routing_template_probe.py renders three variants of one fixed 23-content-token prefix:

raw         BOS + content
user        BOS + "User: " + content + "\n\n"
generation  user prefix + "Assistant:"

The latter two use the pinned official chat_template through apply_chat_template. All three variants of one source prompt execute in the same padded batch. Statistics are split between:

  • the full operational input, including wrapper tokens;
  • the exact intersection of (relative character span, token ID) inside the source content across all three variants.

The user → generation comparison is a causal negative control: the appended suffix must not change routes on their shared prefix.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_template_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-template.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 8 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T08:30:00+00:00

The three variants add 382,032 real top-6 route selections. Across six MoE layers, all 21,852 user → generation shared-prefix token routes are ordered-top-6 exact. The committed run and independent rerun are byte-exact:

da1f10333b2fa269e64f9716a0ca6c1a656d23d3e70b8a25d6e59f8a5b3bc1b9

See research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md for the aligned-content contract, paired intervals, per-token route stability, and claim boundaries.

System × one-shot message-history factorial

v2_lite_routing_history_factorial_probe.py reuses the exact 128-source, 23-content-token cohort from the official-template probe and renders four official chat histories:

S0F0  target user
S1F0  fixed system + target user
S0F1  fixed demo user/assistant + target user
S1F1  fixed system + fixed demo user/assistant + target user

Every condition uses add_generation_prompt=True. The fixed system treatment adds 16 tokens per source in both F0 and F1; the fixed one-shot treatment adds 17 tokens per source in both S0 and S1. All four variants of one source execute in the same padded batch.

The output contains:

  • full-input and exact target-content scopes;
  • token-weighted and prompt-balanced aggregation;
  • four paired factor edges;
  • system and one-shot main effects;
  • difference-in-differences interaction;
  • per-target-token ordered/set top-6 stability and Jaccard;
  • 2,000 source-prompt bootstrap replicates shared by all four cells.
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_factorial_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-factorial.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 8 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T09:36:00+00:00

The four cells add 865,440 real top-6 route selections. The committed run and independent rerun are byte-exact:

5765fbf85fa6fd948cca90f11e2237254fd58c32f373670f44530cff0a70c1fb

See research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md for the factorial definition, all CV intervals, the 24/24 system-edge TV attenuation pattern, per-token stability, and the strict boundary between a message-history effect and a role-semantic claim.

Equal-length filler history control

v2_lite_routing_history_distance_control.py extends the preceding 2×2 probe to six cells:

system off/on × no history / repeated-token filler / fixed one-shot

The filler turn is:

User: x x x x x x x x x
Assistant: x

Under the pinned official template, both filler and one-shot add exactly 17 tokens per source on both system levels. They therefore share the same user/assistant roles, assistant EOS, target position, generation prompt, and batch shape. The filler is deliberately called low-information rather than semantics-free: repeated x tokens remain learned inputs.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_distance_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-distance-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 5 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T10:17:00+00:00

The six cells add 1,376,496 real top-6 route selections. The committed run and independent rerun are byte-exact:

423a095d92738d4bc6c2ace2afa9b35627e61811458c7883dca7b44aee4e648e

For exact target content under prompt-balanced aggregation, mean system-edge TV is 0.0738 → 0.0378 → 0.0188 for none → filler → demo. Both steps decrease in 24/24 layer×domain cells, with all 24 paired intervals below zero. See research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md for the full table, token-level route stability, BF16 batch-shape boundary, literature context, and non-claims.

History-boundary single-token control

v2_lite_routing_history_boundary_token_control.py keeps the repeated-token history from the preceding probe and changes exactly one token ID at the completed assistant boundary:

system off/on × official EOS / x / period / newline

The official template places EOS between the filler assistant content and the next User: marker. The three controls replace only that EOS ID after official tokenization. They are explicit counterfactual token sequences, not valid official chat serializations. All eight conditions preserve sequence length, target position, role markers, attention mask, and within-run batch shape.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_history_boundary_token_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-history-boundary-token-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T11:12:00+00:00

The eight cells add 2,044,224 real top-6 route selections. All 256 source×system groups are equal-length and equal-position; all 768 counterfactual cells differ from their official sequence at exactly one input ID. The committed run and independent rerun are byte-exact:

9bb93834ffd8536aeebe4325e45d2179ba590554c6ff6b6fcceaba2f499b9c37

For exact target content under prompt-balanced aggregation, mean system-edge TV is .0374 / .0539 / .0553 / .0492 for EOS / x / period / newline. X and period exceed EOS in 24/24 layer×domain cells, while newline does so in 23/24. See research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md for all paired intervals, per-token route alignment, scope split, BF16 batch-shape audit, primary literature, and the boundary between an input-ID intervention and a chat-turn semantic claim.

Role-marker-head single-token control

v2_lite_routing_role_marker_head_control.py keeps the official filler history, EOS, colon, target span, generation prompt, mask, length, and 32-row batch shape, while changing one ordinary token ID:

system off/on × official / pre-target User→Assistant /
                    pre-target User→x / post-target Assistant→User

The pinned tokenizer maps User, Assistant, :, and x to IDs 5726, 77398, 25, and 87. The pre-target controls identify the effect of the first role-marker token only; the colon remains. The post-target replacement is a causal suffix negative control. None of the three counterfactuals is a valid official chat serialization.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_role_marker_head_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-role-marker-head-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T12:30:00+00:00

The eight cells add 2,044,224 real top-6 route selections. All 256 source×system groups preserve length and target position; every one of the 768 counterfactuals differs from its official sequence at exactly one ID. The committed run and independent rerun are byte-exact:

9dc0e37fbce6581269428dcfcb84c7a17b466c5239c8171e6741f66d5eeb8caf

For exact target content, direct replacement TV is about .02–.03, but the system-edge contrast has mixed direction: 12↑12↓ for User→Assistant and 13↑11↓ for User→x. The post-target control is exact for all 34,488 aligned target-token ordered top-6 routes, with zero target TV, JSD, and ΔCV. See research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md for all paired intervals, depth maps, token-level alignment, cross-experiment BF16 batch-content audit, primary sources, and the boundary against full role semantics or Chat-model behavior.

Special-token family control

v2_lite_routing_special_token_family_control.py keeps the same repeated-token history and changes the completed assistant boundary to four single IDs:

system off/on × EOS / BOS / x / period

The pinned tokenizer has exactly two special-token IDs: BOS 100000 and EOS 100001; PAD aliases EOS. EOS/BOS therefore exhaust the special inventory, while x and period are only two selected ordinary controls. The 2-vs-2 family summary is descriptive for these four IDs and is not a population-level specialness claim.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_special_token_family_control.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T12:57:00+00:00

The eight cells add 2,044,224 real top-6 route selections. All 256 source×system groups preserve length and target position, and all 768 counterfactuals change exactly one ID. The committed run and independent rerun are byte-exact:

c372c1b03a8b15f615b54ded5d9257a8fc2cdb7728001735d3d4c1d8534af5bf

For exact target content under prompt-balanced aggregation, mean system-edge TV is .037518 / .048018 / .053975 / .055289 for EOS / BOS / x / period. BOS−EOS is positive in 22/24 layer×domain cells; x−EOS and period−EOS are positive in 24/24. The selected ordinary-control mean exceeds the complete special-inventory mean by .011864 in these four IDs only. See research/DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md for all paired intervals, direct edges, scope split, alignment, batch-content audit, literature context, and non-claims.

Full two-token role-marker factorial

v2_lite_routing_role_marker_block_factorial.py treats the pre-target two-token marker as two independent factors:

system off/on × head User/Assistant × delimiter colon/x

User:       [5726, 25]      Assistant:  [77398, 25]
User x      [5726, 87]      Assistant x [77398, 87]

The official assistant EOS and post-target generation suffix remain unchanged. The four blocks have zero, one, one, and two edited IDs relative to official User:; every edit is verified while length, target position, attention mask, and 32-row batch shape stay fixed.

PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_role_marker_block_factorial.py \
  --artifact-dir /path/to/deepseek-v2-lite \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-routing-role-marker-block-factorial.json \
  --per-domain 32 \
  --content-tokens 23 \
  --batch-prompts 4 \
  --layers 7 \
  --bootstrap 2000 \
  --seed 20260729 \
  --captured-at 2026-07-29T13:06:00+00:00

The eight cells add 2,044,224 real top-6 route selections. The 256 official, 512 one-ID, and 256 two-ID cells all pass their exact edit contracts. The committed run and independent rerun are byte-exact:

a703dddb6d04182b1a608c213bfd341cfc32e1801be42c417dca30a505087e82

For exact target content under prompt-balanced aggregation, the four mean system-edge TVs are .037304 / .037015 / .037239 / .036220. The head, delimiter, and head×delimiter effects are small and mixed across the 24 layer×domain cells. Direct head and delimiter edges remain nonzero; replacing colon with the single x control modestly reduces User-vs-Assistant direct TV in most cells, without a uniform system modulation. See research/DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md for the exact factor coding, intervals, direct dependencies, full-input split, route alignment, cross-batch audit, sources, and non-claims.

Full Chat-checkpoint generation behavior

v2_lite_chat_special_token_behavior_probe.py carries the special-token grid into the full official SFT Chat checkpoint:

deepseek-ai/DeepSeek-V2-Lite-Chat
@85864749cd611b4353ce1decdb286193298f64c7

system off/on × EOS / BOS / x / period

It reuses the pinned Base-routing source IDs and ranks but generates from the full source text. All eight variants of one source run in one left-padded batch. Decoding is deterministic greedy; BOS / x / period remain invalid-chat single-ID counterfactuals.

The official BF16 checkpoint contains 31,412,968,448 tensor bytes. Because the local RTX 5090 has less than the official 40GB single-GPU boundary, the formal run uses Accelerate device_map=auto: embeddings and layers 0–24 are placed on CUDA, while layers 25–26, final norm, and LM head are CPU-offloaded.

PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_chat_special_token_behavior_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite-chat \
  --reference-routing-json \
    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-chat-behavior.json \
  --per-domain 4 \
  --max-new-tokens 128 \
  --gpu-memory 29GiB \
  --cpu-memory 80GiB

The formal artifact contains 16 sources and 128 outputs. Only 31 outputs hit EOS; 97 reach the 128-token cap, so task-score fields are diagnostic and are not used for capability ranking. A 4-source / 32-cell long-sequence rerun is exact for prompt hashes, generated token IDs, decoded text, and EOS state.

formal  54496955d0dd20a3116e6b758a55f56e2444c43dd99bfef94a2cb7b140e6196e
repro   b74c31606c0fc70a8e52cc6c3d6135c4e916d698b4d5dae51863124d025b0c72

See research/DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_PROTOCOL.md and research/DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_AUDIT.md for the preregistered contract, output-divergence table, offload device map, truncation boundary, rerun coverage, execution fixes, primary sources, and forbidden conclusions.

Completion-aware 512-token evaluation

The completion run uses the same 16 sources and eight-condition batches but reruns all 128 cells with one uniform 512-token budget. The successful formal placement leaves embeddings and layers 0–23 on CUDA and offloads layers 24–26, final norm, and LM head. expandable_segments:True is part of the recorded runtime contract.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_chat_special_token_behavior_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite-chat \
  --reference-routing-json \
    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-chat-completion-512.json \
  --per-domain 4 \
  --max-new-tokens 512 \
  --gpu-memory 28GiB \
  --cpu-memory 80GiB

The evaluator separates stopping, semantic terminal state, evaluator coverage, and correctness. Each HumanEval candidate runs in a fresh pinned, networkless, read-only Docker container with no host mounts:

python -B experiments/deepseek/v2_lite_chat_completion_evaluator.py \
  --behavior-json \
    src/data/deepseek-v2-lite-chat-completion-512.json \
  --baseline-json src/data/deepseek-v2-lite-chat-behavior.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --sandbox-image \
    python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
  --output \
    src/data/deepseek-v2-lite-chat-completion-512-eval.json

The formal run reaches natural EOS in 121/128 cells, up from 31/128 at the 128-token budget. Strict-complete GSM8K numeric exact is 23/32; HumanEval official-test pass is 24/32. These are four source tasks per domain, not benchmark estimates. A fresh-process one-source-per-domain rerun reproduces all 32 complete generated token sequences.

Full 27-layer Chat hidden-state and router trace

v2_lite_chat_full_depth_trace.py performs a prompt-only use_cache=False forward on the same checkpoint and inputs. It captures hashes and statistics for embedding, every decoder-layer output, final norm, and every MoE gate. Boundary-crossing tokenizer tokens are excluded from the exact target-content scope.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_chat_full_depth_trace.py \
  --artifact-dir /path/to/deepseek-v2-lite-chat \
  --reference-routing-json \
    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --output src/data/deepseek-v2-lite-chat-full-depth.json \
  --per-domain 4 \
  --gpu-memory 28GiB \
  --cpu-memory 80GiB

The formal trace covers 1,537 exact interior content tokens per condition, 29 hidden stages, 26 gates, and 1,918,176 top-6 route decisions. A fresh four-source rerun reproduces 1,856 hidden tensor hashes, 1,664 ordered route hashes, 1,664 route-weight hashes, and every derived comparison.

See research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_PROTOCOL.md and research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md for the OOM amendment, task evaluators, completion table, depth curves, reproduction audit, artifact hashes, and claim boundaries.

Preregistered multi-seed Chat sampling

v2_lite_chat_sampling_probe.py keeps the same four first-ranked sources, eight prompt conditions, 512-token cap, BF16 checkpoint, and offload contract. It enables the pinned official sampling values and explicitly disables top-k:

do_sample=true · temperature=.3 · top_p=.95 · top_k=0
4 sources × 8 SHA-256-derived seeds × 8 conditions = 256 outputs

The eight rows of one source remain in a fixed batch. They are batch-seed aligned, not common-random-number pairs: Transformers samples the whole batch with one torch.multinomial call.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_chat_sampling_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite-chat \
  --reference-routing-json \
    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --greedy-json src/data/deepseek-v2-lite-chat-completion-512.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --base-seeds \
    19683830 1560062173 3978401375 1280933274 \
    1467459869 1297359489 2722953988 3330978061 \
  --max-new-tokens 512 \
  --gpu-memory 28GiB \
  --cpu-memory 80GiB \
  --output src/data/deepseek-v2-lite-chat-sampling.json

The independent evaluator preserves stopping, semantic terminal state, coverage, and correctness as separate ledgers. Exact duplicate code candidates reuse a cache keyed by candidate, task tests, and sandbox-harness hashes:

python -B experiments/deepseek/v2_lite_chat_sampling_evaluator.py \
  --sampling-json src/data/deepseek-v2-lite-chat-sampling.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --sandbox-image \
    python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
  --output src/data/deepseek-v2-lite-chat-sampling-eval.json

A fresh Python process reruns R0/R1. The reproduction comparer checks run seeds, prompt hashes, complete token IDs, text, stopping state, and CPU/CUDA RNG pre-state hashes:

python -B experiments/deepseek/v2_lite_chat_sampling_reproduction.py \
  --formal-json src/data/deepseek-v2-lite-chat-sampling.json \
  --rerun-json \
    src/data/deepseek-v2-lite-chat-sampling-repro-r0r1.json \
  --output \
    src/data/deepseek-v2-lite-chat-sampling-reproduction.json

Formal results are 251/256 natural EOS and 242/256 globally unique complete token trajectories. Strict GSM8K exact is 62/64; HumanEval official-test pass is 63/64. These repeat one task per domain and are not benchmark estimates. The R0/R1 fresh-process rerun reproduces all eight preregistered fields in 64/64 cells.

formal  46c7edfce1409e798d6b0f06e905dd3a9d4ab6911acd8e42e2ec9d06d65345af
eval    078f486e021ffd0e4af8ad942d001a9f2429933b2fb0fb3523b7d1da5b24c4d8
rerun   72d050eace531bb38fb2592e39dc7f1b9c9003306c92c568dcd4acaaa691f6a0
compare 4d59a775943459604ea2d9976bb3be01ff7b3b08d71eb0ca22c376122ef82d15

See research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md and research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md for seed derivation, smoke gates, source-condition diversity tables, edge-set comparisons, task failure cases, exact reproduction scope, primary sources, and non-claims.

Source-blocked cross-source Chat sampling

v2_lite_chat_cross_source_sampling_probe.py reuses the audited sampling runtime but changes the coverage contract before any outputs are inspected:

16 preregistered sources
× 4 SHA-256-derived seeds
× 4 conditions (system off/on × EOS/period)
= 256 outputs

The four first-ranked sources in each of WikiText-2, TNEWS, HumanEval, and GSM8K come from the frozen routing-corpus selection contract. Source is the primary coverage unit; seed is a within-source repeat. The period condition is a directed Round-06 follow-up, not a blind independent confirmation.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B \
  experiments/deepseek/v2_lite_chat_cross_source_sampling_probe.py \
  --artifact-dir /path/to/deepseek-v2-lite-chat \
  --reference-routing-json \
    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --greedy-json src/data/deepseek-v2-lite-chat-completion-512.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --base-seeds 2101325316 2511573438 1677220094 2412346607 \
  --max-new-tokens 512 \
  --gpu-memory 28GiB \
  --cpu-memory 80GiB \
  --output \
    src/data/deepseek-v2-lite-chat-cross-source-sampling.json

The independent evaluator uses the same pinned, networkless, read-only HumanEval sandbox and avoids pooling final-answer frequencies across different GSM8K tasks:

python -B \
  experiments/deepseek/v2_lite_chat_cross_source_sampling_evaluator.py \
  --sampling-json \
    src/data/deepseek-v2-lite-chat-cross-source-sampling.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --sandbox-image \
    python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
  --output \
    src/data/deepseek-v2-lite-chat-cross-source-sampling-eval.json

The source-blocked analysis computes each contrast within source, then reports the four source directions in each domain. It deliberately emits no p-values or population confidence intervals:

python -B \
  experiments/deepseek/v2_lite_chat_cross_source_sampling_analysis.py \
  --sampling-json \
    src/data/deepseek-v2-lite-chat-cross-source-sampling.json \
  --evaluation-json \
    src/data/deepseek-v2-lite-chat-cross-source-sampling-eval.json \
  --reproduction-json \
    src/data/deepseek-v2-lite-chat-cross-source-sampling-reproduction.json \
  --output \
    src/data/deepseek-v2-lite-chat-cross-source-sampling-analysis.json

A fresh Python process reruns R0 for all 16 sources. The existing reproduction comparer verifies the eight preregistered fields in all 64 source-condition cells.

Formal results are 250/256 natural EOS and 247/256 unique complete token trajectories. Strict task totals are Math 47/64 and Code 52/64, but the per-task totals range from 8/16 to 16/16 in both domains. Period shortens mean generation length for 11/16 sources. English is the important counterexample: the first source is −121 tokens while the other three are +44.5, +41, and +1.375, so the domain mean is negative even though three of four sources are positive.

formal   f013132485f27adce008f03f781bed9982efc0d7939f13faede01c9f6f3d7f7c
eval     e88b274599fc5951561f9e5e7438fb4d6d25f341ae3bc4a2c4121877a0db8975
rerun    143dc9d0f7c914db4781e36b1401cdc9fbc2971a0dca71188bab0f8cedb002a6
compare  ec4a47894953f5d73bb62211b588632ef7032da0e915218992b7fc3ef9c2a556
analysis d0dece388998fee419d34ff33f140695a9fedef6e79799cdf42eb283b047bc84
compact  d0ab65646c6119bdeafeb451103dc6afebff3624a1e05f13a45a52ad965be1af

See research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md and research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md for the preregistered source frame, task matrices, source-direction contrasts, failure identities, hash chain, exact replay scope, and non-claims.

Selected-task bootstrap with explicit common random numbers

v2_lite_chat_task_bootstrap_crn_probe.py expands the task frame to all 32 frozen HumanEval and 32 frozen GSM8K sources. The primary analysis runs one preregistered tape T0 on every task. Four tasks per domain additionally run T1-T3 as a crossed tape-sensitivity diagnostic:

64 tasks × T0 × 4 conditions                         = 256 outputs
8 diagnostic tasks × 3 additional tapes × 4 conditions = 96 outputs
formal total                                            = 352 outputs

Unlike the preceding batch-seed-aligned studies, Round 08 constructs a true step-index common-random-number tape. The same SHA-256-derived u_t is mapped through each condition's official .3/.95 temperature/top-p distribution using a float32 token-ID-ordered inverse CDF. It does not call transformers.generate or torch.multinomial.

The frozen manifest and all 256 Chat prompt hashes must be built before any model outputs:

node scripts/build-deepseek-chat-task-bootstrap-manifest.mjs \
  --reference \
    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --output \
    research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json

PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B scripts/freeze-deepseek-chat-task-bootstrap-prompts.py \
  --artifact-dir /path/to/deepseek-v2-lite-chat \
  --manifest \
    research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
  --reference-routing-json \
    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet

The formal runner writes one hash-checked journal per source and supports --resume:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
PYTHONPATH=/path/to/transformers-4.41.2-deps:experiments/deepseek \
python -B \
  experiments/deepseek/v2_lite_chat_task_bootstrap_crn_probe.py \
  --execution-mode formal \
  --artifact-dir /path/to/deepseek-v2-lite-chat \
  --manifest \
    research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
  --reference-routing-json \
    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --tnews /path/to/tnews/test.json \
  --tnews-archive /path/to/tnews_public.zip \
  --wikitext /path/to/wikitext-validation.parquet \
  --journal-dir /path/to/formal-journal \
  --resume \
  --output \
    src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json

Evaluation keeps T0 primary rows separate from T1-T3 diagnostics:

PYTHONPATH=experiments/deepseek \
python -B \
  experiments/deepseek/v2_lite_chat_task_bootstrap_crn_evaluator.py \
  --sampling-json \
    src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
  --human-eval /path/to/HumanEval.jsonl.gz \
  --gsm8k /path/to/gsm8k/test.jsonl \
  --sandbox-image \
    python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
  --output \
    src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json

The analysis resamples the 32 selected tasks 10,000 times within each domain while preserving condition pairing. These are selected-task resampling bands under T0, not benchmark-population or generation-seed confidence intervals:

python -B \
  experiments/deepseek/v2_lite_chat_task_bootstrap_crn_analysis.py \
  --sampling-json \
    src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
  --evaluation-json \
    src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json \
  --reproduction-json \
    src/data/deepseek-v2-lite-chat-task-bootstrap-crn-reproduction.json \
  --output \
    src/data/deepseek-v2-lite-chat-task-bootstrap-crn-analysis.json

The formal grid contains 343/352 natural EOS outputs and 320 unique complete trajectories. T0 correctness totals are Code 59/128 and Math 71/128 across 32 tasks × four prompt conditions; they are not standard benchmark scores. All correctness contrast bands cross zero. The clearest length interaction is system-on minus system-off under the period boundary: Code is -130.875 tokens with band [-178.688, -83.218], while Math is +25.125 with band [7.874, 44.531]. The preregistered 64-cell cross-process replay is exact in all twelve audited fields.

formal   ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809
eval     82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab
replay   6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508
compare  63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5
analysis 9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae
compact  082e3c56373e5f91d51d88b1430d14ad3b68b0f200e80901171c2fb62a1e57bf

Run the complete offline hash and invariant audit with:

node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs

See research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md and research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_CRN_AUDIT.md for the pre-output corrections, explicit sampler, task bootstrap, transition tables, multi-tape matrices, exact replay, reported gold-loading process deviation, and claim boundaries.

Reference in New Issue View Git Blame Copy Permalink