research: audit task bootstrap CRN experiment

2026-07-30 05:59:38 +08:00
parent 65a91ae178
commit f041c15e81
11 changed files with 162499 additions and 0 deletions
@@ -853,3 +853,135 @@ See `research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md` and
 `research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md` for the
 preregistered source frame, task matrices, source-direction contrasts,
 failure identities, hash chain, exact replay scope, and non-claims.
+
+## Selected-task bootstrap with explicit common random numbers
+
+`v2_lite_chat_task_bootstrap_crn_probe.py` expands the task frame to all 32
+frozen HumanEval and 32 frozen GSM8K sources.  The primary analysis runs one
+preregistered tape T0 on every task.  Four tasks per domain additionally run
+T1-T3 as a crossed tape-sensitivity diagnostic:
+
+```text
+64 tasks × T0 × 4 conditions                         = 256 outputs
+8 diagnostic tasks × 3 additional tapes × 4 conditions = 96 outputs
+formal total                                            = 352 outputs
+```
+
+Unlike the preceding batch-seed-aligned studies, Round 08 constructs a true
+step-index common-random-number tape.  The same SHA-256-derived `u_t` is
+mapped through each condition's official `.3/.95` temperature/top-p
+distribution using a float32 token-ID-ordered inverse CDF.  It does not call
+`transformers.generate` or `torch.multinomial`.
+
+The frozen manifest and all 256 Chat prompt hashes must be built before any
+model outputs:
+
+```bash
+node scripts/build-deepseek-chat-task-bootstrap-manifest.mjs \
+  --reference \
+    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
+  --output \
+    research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json
+
+PYTHONPATH=/path/to/transformers-4.41.2-deps \
+python -B scripts/freeze-deepseek-chat-task-bootstrap-prompts.py \
+  --artifact-dir /path/to/deepseek-v2-lite-chat \
+  --manifest \
+    research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
+  --reference-routing-json \
+    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
+  --human-eval /path/to/HumanEval.jsonl.gz \
+  --gsm8k /path/to/gsm8k/test.jsonl \
+  --tnews /path/to/tnews/test.json \
+  --tnews-archive /path/to/tnews_public.zip \
+  --wikitext /path/to/wikitext-validation.parquet
+```
+
+The formal runner writes one hash-checked journal per source and supports
+`--resume`:
+
+```bash
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+PYTHONPATH=/path/to/transformers-4.41.2-deps:experiments/deepseek \
+python -B \
+  experiments/deepseek/v2_lite_chat_task_bootstrap_crn_probe.py \
+  --execution-mode formal \
+  --artifact-dir /path/to/deepseek-v2-lite-chat \
+  --manifest \
+    research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
+  --reference-routing-json \
+    src/data/deepseek-v2-lite-routing-special-token-family-control.json \
+  --human-eval /path/to/HumanEval.jsonl.gz \
+  --gsm8k /path/to/gsm8k/test.jsonl \
+  --tnews /path/to/tnews/test.json \
+  --tnews-archive /path/to/tnews_public.zip \
+  --wikitext /path/to/wikitext-validation.parquet \
+  --journal-dir /path/to/formal-journal \
+  --resume \
+  --output \
+    src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json
+```
+
+Evaluation keeps T0 primary rows separate from T1-T3 diagnostics:
+
+```bash
+PYTHONPATH=experiments/deepseek \
+python -B \
+  experiments/deepseek/v2_lite_chat_task_bootstrap_crn_evaluator.py \
+  --sampling-json \
+    src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
+  --human-eval /path/to/HumanEval.jsonl.gz \
+  --gsm8k /path/to/gsm8k/test.jsonl \
+  --sandbox-image \
+    python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
+  --output \
+    src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json
+```
+
+The analysis resamples the 32 selected tasks 10,000 times within each domain
+while preserving condition pairing.  These are selected-task resampling
+bands under T0, not benchmark-population or generation-seed confidence
+intervals:
+
+```bash
+python -B \
+  experiments/deepseek/v2_lite_chat_task_bootstrap_crn_analysis.py \
+  --sampling-json \
+    src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
+  --evaluation-json \
+    src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json \
+  --reproduction-json \
+    src/data/deepseek-v2-lite-chat-task-bootstrap-crn-reproduction.json \
+  --output \
+    src/data/deepseek-v2-lite-chat-task-bootstrap-crn-analysis.json
+```
+
+The formal grid contains 343/352 natural EOS outputs and 320 unique complete
+trajectories.  T0 correctness totals are Code 59/128 and Math 71/128 across
+32 tasks × four prompt conditions; they are not standard benchmark scores.
+All correctness contrast bands cross zero.  The clearest length interaction
+is system-on minus system-off under the period boundary: Code is -130.875
+tokens with band [-178.688, -83.218], while Math is +25.125 with band
+[7.874, 44.531].  The preregistered 64-cell cross-process replay is exact in
+all twelve audited fields.
+
+```text
+formal   ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809
+eval     82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab
+replay   6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508
+compare  63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5
+analysis 9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae
+compact  082e3c56373e5f91d51d88b1430d14ad3b68b0f200e80901171c2fb62a1e57bf
+```
+
+Run the complete offline hash and invariant audit with:
+
+```bash
+node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs
+```
+
+See `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md` and
+`research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_CRN_AUDIT.md` for the
+pre-output corrections, explicit sampler, task bootstrap, transition tables,
+multi-tape matrices, exact replay, reported gold-loading process deviation,
+and claim boundaries.