wuyang
208 lines
7.0 KiB
Markdown
208 lines
7.0 KiB
Markdown
# DeepSeek real-weight execution probes
|
||
|
||
The probes in this directory use official DeepSeek artifacts and keep their
|
||
scope deliberately narrower than a full-model benchmark.
|
||
|
||
## DeepSeek-V2-Lite truncated trace
|
||
|
||
`v2_lite_trace.py` executes layers 0–6 from the official BF16 checkpoint. Those
|
||
seven layers are fully contained in shard 1; layer 7 is split across shards 1
|
||
and 2 and is therefore outside the default evidence boundary.
|
||
|
||
Pinned model:
|
||
|
||
```text
|
||
deepseek-ai/DeepSeek-V2-Lite@604d5664dddd88a0433dbae533b7fe9472482de0
|
||
```
|
||
|
||
Required Python stack:
|
||
|
||
```text
|
||
torch==2.11.0+cu128
|
||
transformers==4.41.2
|
||
safetensors==0.8.0
|
||
```
|
||
|
||
The 2024 remote code does not import under Transformers 5.5 because
|
||
`is_torch_fx_available` was removed. The probe imports the official files as a
|
||
read-only local package; it does not patch the model source.
|
||
|
||
Download the metadata, tokenizer, remote code, index, and first shard with the
|
||
Hugging Face CLI, then run:
|
||
|
||
```bash
|
||
python experiments/deepseek/v2_lite_trace.py \
|
||
--artifact-dir /path/to/deepseek-v2-lite \
|
||
--output src/data/deepseek-v2-lite-trace.json
|
||
```
|
||
|
||
The result contains:
|
||
|
||
- real tokenizer pieces and model-derived hidden states;
|
||
- the actual `[B,T,576]` MLA compressed projection at each executed layer;
|
||
- the expanded key/value tensors stored by the Hugging Face eager cache;
|
||
- token-level top-6 routed expert IDs and weights for six MoE layers;
|
||
- per-layer and per-prompt expert-load summaries;
|
||
- explicit boundaries against global load, expert semantics, training traces,
|
||
full-model generation, and production serving claims.
|
||
|
||
Run the probe twice and compare deterministic evidence while excluding timing:
|
||
|
||
```bash
|
||
python experiments/deepseek/compare_v2_lite_traces.py \
|
||
--first /path/to/trace-1.json \
|
||
--second /path/to/trace-2.json \
|
||
--output src/data/deepseek-v2-lite-trace-repro.json
|
||
```
|
||
|
||
## Fixed public routing corpus
|
||
|
||
`v2_lite_routing_corpus.py` keeps the same official layer 0–6 execution boundary
|
||
but replaces the four authored prompts with 128 source-addressable public
|
||
prompts:
|
||
|
||
- 32 WikiText-2 raw validation passages;
|
||
- 32 CLUE TNEWS public-test sentences;
|
||
- 32 OpenAI HumanEval prompts, without solutions/tests or code execution;
|
||
- 32 OpenAI GSM8K test questions, without answers.
|
||
|
||
Selection is the ascending SHA-256 rank of a fixed salt, domain, and source ID.
|
||
Inputs are truncated to 96 DeepSeek tokens. The six MoE layers therefore produce
|
||
304,560 actual top-6 routed-expert selections over 8,460 valid tokens.
|
||
|
||
The output includes both token-weighted and prompt-balanced distributions. Its
|
||
95% intervals use 2,000 prompt-level bootstrap resamples within each domain,
|
||
rather than treating correlated tokens as independent observations.
|
||
|
||
```bash
|
||
PYTHONPATH=/path/to/transformers-4.41.2-deps \
|
||
python -B experiments/deepseek/v2_lite_routing_corpus.py \
|
||
--artifact-dir /path/to/deepseek-v2-lite \
|
||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||
--tnews /path/to/tnews/test.json \
|
||
--tnews-archive /path/to/tnews_public.zip \
|
||
--wikitext /path/to/wikitext-validation.parquet \
|
||
--output src/data/deepseek-v2-lite-routing-corpus.json \
|
||
--per-domain 32 \
|
||
--max-tokens 96 \
|
||
--batch-size 16 \
|
||
--bootstrap 2000 \
|
||
--seed 20260729 \
|
||
--captured-at 2026-07-29T07:45:00+00:00
|
||
```
|
||
|
||
The committed independent rerun is byte-exact. Both JSON files have SHA-256:
|
||
|
||
```text
|
||
4678a1d15395de93ffba757598cc3642bf35e9e07f71d82ddd87c27fc38a09e4
|
||
```
|
||
|
||
See `research/DEEPSEEK_ROUTING_CORPUS_AUDIT.md` for corpus revisions and hashes,
|
||
metric definitions, interval semantics, results, and claim boundaries.
|
||
|
||
## Paired length-control cohort
|
||
|
||
The corpus runner can also select one fixed cohort by untruncated source length
|
||
and execute nested prefixes. The committed 16-token and 24-token traces use the
|
||
same 128 source prompts, all selected from records with at least 24 DeepSeek
|
||
tokens:
|
||
|
||
```bash
|
||
common_args=(
|
||
--per-domain 32
|
||
--batch-size 16
|
||
--bootstrap 2000
|
||
--sample-salt llm-atlas-deepseek-routing-length-control-v1
|
||
--eligibility-min-tokens 24
|
||
)
|
||
|
||
python -B experiments/deepseek/v2_lite_routing_corpus.py \
|
||
...source arguments... \
|
||
"${common_args[@]}" \
|
||
--max-tokens 16 \
|
||
--output src/data/deepseek-v2-lite-routing-matched16.json
|
||
|
||
python -B experiments/deepseek/v2_lite_routing_corpus.py \
|
||
...source arguments... \
|
||
"${common_args[@]}" \
|
||
--max-tokens 24 \
|
||
--output src/data/deepseek-v2-lite-routing-matched24.json
|
||
```
|
||
|
||
The two real traces add 184,320 top-6 route selections. Their independent
|
||
reruns are byte-exact:
|
||
|
||
```text
|
||
matched-16 f8d437d5379ffb41ac8dca5a8e97c0f44ba10ce7b63f95d7be0b7c88ac0baebd
|
||
matched-24 bed54835ad243ca2ab46bf9574e53137e6c2c0e19267f581719b5f2f65546436
|
||
```
|
||
|
||
Use the paired comparison runner to resample identical prompt indices in the
|
||
short and long traces:
|
||
|
||
```bash
|
||
python -B experiments/deepseek/compare_routing_length_control.py \
|
||
--short src/data/deepseek-v2-lite-routing-matched16.json \
|
||
--long src/data/deepseek-v2-lite-routing-matched24.json \
|
||
--output src/data/deepseek-v2-lite-routing-length-sensitivity.json \
|
||
--bootstrap 2000 \
|
||
--seed 20260729
|
||
```
|
||
|
||
See `research/DEEPSEEK_ROUTING_LENGTH_CONTROL_AUDIT.md` for the sampling bias
|
||
audit, paired CV/JSD deltas, total-variation accounting, and interpretation
|
||
boundaries.
|
||
|
||
## Official chat-template sensitivity
|
||
|
||
`v2_lite_routing_template_probe.py` renders three variants of one fixed
|
||
23-content-token prefix:
|
||
|
||
```text
|
||
raw BOS + content
|
||
user BOS + "User: " + content + "\n\n"
|
||
generation user prefix + "Assistant:"
|
||
```
|
||
|
||
The latter two use the pinned official `chat_template` through
|
||
`apply_chat_template`. All three variants of one source prompt execute in the
|
||
same padded batch. Statistics are split between:
|
||
|
||
- the full operational input, including wrapper tokens;
|
||
- the exact intersection of `(relative character span, token ID)` inside the
|
||
source content across all three variants.
|
||
|
||
The `user → generation` comparison is a causal negative control: the appended
|
||
suffix must not change routes on their shared prefix.
|
||
|
||
```bash
|
||
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||
python -B experiments/deepseek/v2_lite_routing_template_probe.py \
|
||
--artifact-dir /path/to/deepseek-v2-lite \
|
||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||
--tnews /path/to/tnews/test.json \
|
||
--tnews-archive /path/to/tnews_public.zip \
|
||
--wikitext /path/to/wikitext-validation.parquet \
|
||
--output src/data/deepseek-v2-lite-routing-template.json \
|
||
--per-domain 32 \
|
||
--content-tokens 23 \
|
||
--batch-prompts 8 \
|
||
--layers 7 \
|
||
--bootstrap 2000 \
|
||
--seed 20260729 \
|
||
--captured-at 2026-07-29T08:30:00+00:00
|
||
```
|
||
|
||
The three variants add 382,032 real top-6 route selections. Across six MoE
|
||
layers, all 21,852 `user → generation` shared-prefix token routes are
|
||
ordered-top-6 exact. The committed run and independent rerun are byte-exact:
|
||
|
||
```text
|
||
da1f10333b2fa269e64f9716a0ca6c1a656d23d3e70b8a25d6e59f8a5b3bc1b9
|
||
```
|
||
|
||
See `research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md` for the aligned-content
|
||
contract, paired intervals, per-token route stability, and claim boundaries.
|