feat: audit DeepSeek Chat across sources

This commit is contained in:
wuyang
2026-07-30 03:56:55 +08:00
parent 29ceae4e1b
commit 9211333234
24 changed files with 123017 additions and 42 deletions
+99
View File
@@ -754,3 +754,102 @@ See `research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md` and
`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md` for seed derivation,
smoke gates, source-condition diversity tables, edge-set comparisons, task
failure cases, exact reproduction scope, primary sources, and non-claims.
## Source-blocked cross-source Chat sampling
`v2_lite_chat_cross_source_sampling_probe.py` reuses the audited sampling
runtime but changes the coverage contract before any outputs are inspected:
```text
16 preregistered sources
× 4 SHA-256-derived seeds
× 4 conditions (system off/on × EOS/period)
= 256 outputs
```
The four first-ranked sources in each of WikiText-2, TNEWS, HumanEval, and
GSM8K come from the frozen routing-corpus selection contract. Source is the
primary coverage unit; seed is a within-source repeat. The period condition is
a directed Round-06 follow-up, not a blind independent confirmation.
```bash
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B \
experiments/deepseek/v2_lite_chat_cross_source_sampling_probe.py \
--artifact-dir /path/to/deepseek-v2-lite-chat \
--reference-routing-json \
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
--greedy-json src/data/deepseek-v2-lite-chat-completion-512.json \
--human-eval /path/to/HumanEval.jsonl.gz \
--gsm8k /path/to/gsm8k/test.jsonl \
--tnews /path/to/tnews/test.json \
--tnews-archive /path/to/tnews_public.zip \
--wikitext /path/to/wikitext-validation.parquet \
--base-seeds 2101325316 2511573438 1677220094 2412346607 \
--max-new-tokens 512 \
--gpu-memory 28GiB \
--cpu-memory 80GiB \
--output \
src/data/deepseek-v2-lite-chat-cross-source-sampling.json
```
The independent evaluator uses the same pinned, networkless, read-only
HumanEval sandbox and avoids pooling final-answer frequencies across different
GSM8K tasks:
```bash
python -B \
experiments/deepseek/v2_lite_chat_cross_source_sampling_evaluator.py \
--sampling-json \
src/data/deepseek-v2-lite-chat-cross-source-sampling.json \
--human-eval /path/to/HumanEval.jsonl.gz \
--gsm8k /path/to/gsm8k/test.jsonl \
--sandbox-image \
python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
--output \
src/data/deepseek-v2-lite-chat-cross-source-sampling-eval.json
```
The source-blocked analysis computes each contrast within source, then reports
the four source directions in each domain. It deliberately emits no p-values
or population confidence intervals:
```bash
python -B \
experiments/deepseek/v2_lite_chat_cross_source_sampling_analysis.py \
--sampling-json \
src/data/deepseek-v2-lite-chat-cross-source-sampling.json \
--evaluation-json \
src/data/deepseek-v2-lite-chat-cross-source-sampling-eval.json \
--reproduction-json \
src/data/deepseek-v2-lite-chat-cross-source-sampling-reproduction.json \
--output \
src/data/deepseek-v2-lite-chat-cross-source-sampling-analysis.json
```
A fresh Python process reruns R0 for all 16 sources. The existing reproduction
comparer verifies the eight preregistered fields in all 64
source-condition cells.
Formal results are 250/256 natural EOS and 247/256 unique complete token
trajectories. Strict task totals are Math 47/64 and Code 52/64, but the
per-task totals range from 8/16 to 16/16 in both domains. Period shortens mean
generation length for 11/16 sources. English is the important counterexample:
the first source is −121 tokens while the other three are +44.5, +41, and
+1.375, so the domain mean is negative even though three of four sources are
positive.
```text
formal f013132485f27adce008f03f781bed9982efc0d7939f13faede01c9f6f3d7f7c
eval e88b274599fc5951561f9e5e7438fb4d6d25f341ae3bc4a2c4121877a0db8975
rerun 143dc9d0f7c914db4781e36b1401cdc9fbc2971a0dca71188bab0f8cedb002a6
compare ec4a47894953f5d73bb62211b588632ef7032da0e915218992b7fc3ef9c2a556
analysis d0dece388998fee419d34ff33f140695a9fedef6e79799cdf42eb283b047bc84
compact d0ab65646c6119bdeafeb451103dc6afebff3624a1e05f13a45a52ad965be1af
```
See `research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md` and
`research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md` for the
preregistered source frame, task matrices, source-direction contrasts,
failure identities, hash chain, exact replay scope, and non-claims.