feat: audit DeepSeek Chat across sources
This commit is contained in:
@@ -754,3 +754,102 @@ See `research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md` for seed derivation,
|
||||
smoke gates, source-condition diversity tables, edge-set comparisons, task
|
||||
failure cases, exact reproduction scope, primary sources, and non-claims.
|
||||
|
||||
## Source-blocked cross-source Chat sampling
|
||||
|
||||
`v2_lite_chat_cross_source_sampling_probe.py` reuses the audited sampling
|
||||
runtime but changes the coverage contract before any outputs are inspected:
|
||||
|
||||
```text
|
||||
16 preregistered sources
|
||||
× 4 SHA-256-derived seeds
|
||||
× 4 conditions (system off/on × EOS/period)
|
||||
= 256 outputs
|
||||
```
|
||||
|
||||
The four first-ranked sources in each of WikiText-2, TNEWS, HumanEval, and
|
||||
GSM8K come from the frozen routing-corpus selection contract. Source is the
|
||||
primary coverage unit; seed is a within-source repeat. The period condition is
|
||||
a directed Round-06 follow-up, not a blind independent confirmation.
|
||||
|
||||
```bash
|
||||
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps \
|
||||
python -B \
|
||||
experiments/deepseek/v2_lite_chat_cross_source_sampling_probe.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite-chat \
|
||||
--reference-routing-json \
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--greedy-json src/data/deepseek-v2-lite-chat-completion-512.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--base-seeds 2101325316 2511573438 1677220094 2412346607 \
|
||||
--max-new-tokens 512 \
|
||||
--gpu-memory 28GiB \
|
||||
--cpu-memory 80GiB \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-cross-source-sampling.json
|
||||
```
|
||||
|
||||
The independent evaluator uses the same pinned, networkless, read-only
|
||||
HumanEval sandbox and avoids pooling final-answer frequencies across different
|
||||
GSM8K tasks:
|
||||
|
||||
```bash
|
||||
python -B \
|
||||
experiments/deepseek/v2_lite_chat_cross_source_sampling_evaluator.py \
|
||||
--sampling-json \
|
||||
src/data/deepseek-v2-lite-chat-cross-source-sampling.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--sandbox-image \
|
||||
python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-cross-source-sampling-eval.json
|
||||
```
|
||||
|
||||
The source-blocked analysis computes each contrast within source, then reports
|
||||
the four source directions in each domain. It deliberately emits no p-values
|
||||
or population confidence intervals:
|
||||
|
||||
```bash
|
||||
python -B \
|
||||
experiments/deepseek/v2_lite_chat_cross_source_sampling_analysis.py \
|
||||
--sampling-json \
|
||||
src/data/deepseek-v2-lite-chat-cross-source-sampling.json \
|
||||
--evaluation-json \
|
||||
src/data/deepseek-v2-lite-chat-cross-source-sampling-eval.json \
|
||||
--reproduction-json \
|
||||
src/data/deepseek-v2-lite-chat-cross-source-sampling-reproduction.json \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-cross-source-sampling-analysis.json
|
||||
```
|
||||
|
||||
A fresh Python process reruns R0 for all 16 sources. The existing reproduction
|
||||
comparer verifies the eight preregistered fields in all 64
|
||||
source-condition cells.
|
||||
|
||||
Formal results are 250/256 natural EOS and 247/256 unique complete token
|
||||
trajectories. Strict task totals are Math 47/64 and Code 52/64, but the
|
||||
per-task totals range from 8/16 to 16/16 in both domains. Period shortens mean
|
||||
generation length for 11/16 sources. English is the important counterexample:
|
||||
the first source is −121 tokens while the other three are +44.5, +41, and
|
||||
+1.375, so the domain mean is negative even though three of four sources are
|
||||
positive.
|
||||
|
||||
```text
|
||||
formal f013132485f27adce008f03f781bed9982efc0d7939f13faede01c9f6f3d7f7c
|
||||
eval e88b274599fc5951561f9e5e7438fb4d6d25f341ae3bc4a2c4121877a0db8975
|
||||
rerun 143dc9d0f7c914db4781e36b1401cdc9fbc2971a0dca71188bab0f8cedb002a6
|
||||
compare ec4a47894953f5d73bb62211b588632ef7032da0e915218992b7fc3ef9c2a556
|
||||
analysis d0dece388998fee419d34ff33f140695a9fedef6e79799cdf42eb283b047bc84
|
||||
compact d0ab65646c6119bdeafeb451103dc6afebff3624a1e05f13a45a52ad965be1af
|
||||
```
|
||||
|
||||
See `research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md` for the
|
||||
preregistered source frame, task matrices, source-direction contrasts,
|
||||
failure identities, hash chain, exact replay scope, and non-claims.
|
||||
|
||||
Reference in New Issue
Block a user