wuyang
59 lines
2.1 KiB
Markdown
59 lines
2.1 KiB
Markdown
# Attention Residuals train-time forward intervention
|
|
|
|
This directory implements preregistered protocol
|
|
`llm-atlas-k3-attnres-forward-training-v1`:
|
|
|
|
- `research/K3_ATTNRES_FORWARD_TRAINING_SCOPING.md`
|
|
- `research/K3_ATTNRES_FORWARD_TRAINING_PROTOCOL.md`
|
|
- `research/K3_ATTNRES_FORWARD_TRAINING_GROK_REVIEW.md`
|
|
|
|
It is a depth-32 reduced Block AttnRes architecture ablation. It is not a
|
|
Kimi-K3 checkpoint forward pass and does not claim to recover unpublished
|
|
Figure 5 telemetry.
|
|
|
|
## Frozen environment
|
|
|
|
```text
|
|
Python /home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python
|
|
PyTorch 2.11.0+cu128
|
|
GPU NVIDIA GeForce RTX 5090
|
|
CUBLAS_WORKSPACE_CONFIG=:4096:8
|
|
maximum concurrency 2
|
|
```
|
|
|
|
## Pre-result gates
|
|
|
|
The checked-in gate artifacts must pass before formal output:
|
|
|
|
```bash
|
|
CUBLAS_WORKSPACE_CONFIG=:4096:8 \
|
|
/home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python \
|
|
experiments/k3/attnres_forward/verify.py step-zero \
|
|
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
|
--parent-manifest experiments/k3/attnres_gradient/manifest.json \
|
|
--study-manifest experiments/k3/attnres_forward/manifest.json \
|
|
--output experiments/k3/attnres_forward/results/gates/step-zero.json
|
|
```
|
|
|
|
`learned_reference` is smoke-only. Its 20-step result is compared with a
|
|
fresh parent Round 05 smoke using `verify.py smoke-compare`.
|
|
|
|
## Formal matrix
|
|
|
|
```bash
|
|
/home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python \
|
|
experiments/k3/attnres_forward/run_matrix.py \
|
|
--python /home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python \
|
|
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
|
--parent-manifest experiments/k3/attnres_gradient/manifest.json \
|
|
--study-manifest experiments/k3/attnres_forward/manifest.json \
|
|
--output-dir experiments/k3/attnres_forward/results/raw \
|
|
--phase all \
|
|
--concurrency 2
|
|
```
|
|
|
|
This runs 12 formal cells and one full replay. The analyzer reads all cells,
|
|
the frozen historical paired references, and generates the only authoritative
|
|
status, interaction map, and website compact artifact.
|
|
|