feat: map DeepSeek Chat sampling robustness
This commit is contained in:
@@ -672,3 +672,85 @@ See `research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md` for the OOM
|
||||
amendment, task evaluators, completion table, depth curves, reproduction
|
||||
audit, artifact hashes, and claim boundaries.
|
||||
|
||||
## Preregistered multi-seed Chat sampling
|
||||
|
||||
`v2_lite_chat_sampling_probe.py` keeps the same four first-ranked sources,
|
||||
eight prompt conditions, 512-token cap, BF16 checkpoint, and offload contract.
|
||||
It enables the pinned official sampling values and explicitly disables top-k:
|
||||
|
||||
```text
|
||||
do_sample=true · temperature=.3 · top_p=.95 · top_k=0
|
||||
4 sources × 8 SHA-256-derived seeds × 8 conditions = 256 outputs
|
||||
```
|
||||
|
||||
The eight rows of one source remain in a fixed batch. They are batch-seed
|
||||
aligned, not common-random-number pairs: Transformers samples the whole batch
|
||||
with one `torch.multinomial` call.
|
||||
|
||||
```bash
|
||||
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps \
|
||||
python -B experiments/deepseek/v2_lite_chat_sampling_probe.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite-chat \
|
||||
--reference-routing-json \
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--greedy-json src/data/deepseek-v2-lite-chat-completion-512.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--base-seeds \
|
||||
19683830 1560062173 3978401375 1280933274 \
|
||||
1467459869 1297359489 2722953988 3330978061 \
|
||||
--max-new-tokens 512 \
|
||||
--gpu-memory 28GiB \
|
||||
--cpu-memory 80GiB \
|
||||
--output src/data/deepseek-v2-lite-chat-sampling.json
|
||||
```
|
||||
|
||||
The independent evaluator preserves stopping, semantic terminal state,
|
||||
coverage, and correctness as separate ledgers. Exact duplicate code candidates
|
||||
reuse a cache keyed by candidate, task tests, and sandbox-harness hashes:
|
||||
|
||||
```bash
|
||||
python -B experiments/deepseek/v2_lite_chat_sampling_evaluator.py \
|
||||
--sampling-json src/data/deepseek-v2-lite-chat-sampling.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--sandbox-image \
|
||||
python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
|
||||
--output src/data/deepseek-v2-lite-chat-sampling-eval.json
|
||||
```
|
||||
|
||||
A fresh Python process reruns R0/R1. The reproduction comparer checks run
|
||||
seeds, prompt hashes, complete token IDs, text, stopping state, and CPU/CUDA
|
||||
RNG pre-state hashes:
|
||||
|
||||
```bash
|
||||
python -B experiments/deepseek/v2_lite_chat_sampling_reproduction.py \
|
||||
--formal-json src/data/deepseek-v2-lite-chat-sampling.json \
|
||||
--rerun-json \
|
||||
src/data/deepseek-v2-lite-chat-sampling-repro-r0r1.json \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-sampling-reproduction.json
|
||||
```
|
||||
|
||||
Formal results are 251/256 natural EOS and 242/256 globally unique complete
|
||||
token trajectories. Strict GSM8K exact is 62/64; HumanEval official-test pass
|
||||
is 63/64. These repeat one task per domain and are not benchmark estimates.
|
||||
The R0/R1 fresh-process rerun reproduces all eight preregistered fields in
|
||||
64/64 cells.
|
||||
|
||||
```text
|
||||
formal 46c7edfce1409e798d6b0f06e905dd3a9d4ab6911acd8e42e2ec9d06d65345af
|
||||
eval 078f486e021ffd0e4af8ad942d001a9f2429933b2fb0fb3523b7d1da5b24c4d8
|
||||
rerun 72d050eace531bb38fb2592e39dc7f1b9c9003306c92c568dcd4acaaa691f6a0
|
||||
compare 4d59a775943459604ea2d9976bb3be01ff7b3b08d71eb0ca22c376122ef82d15
|
||||
```
|
||||
|
||||
See `research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md` for seed derivation,
|
||||
smoke gates, source-condition diversity tables, edge-set comparisons, task
|
||||
failure cases, exact reproduction scope, primary sources, and non-claims.
|
||||
|
||||
Reference in New Issue
Block a user