wuyang
58 lines
1.8 KiB
Markdown
58 lines
1.8 KiB
Markdown
# Attention Residuals activation-gradient/depth study
|
|
|
|
This directory implements preregistered protocol
|
|
`llm-atlas-k3-attnres-gradient-scale-v1`:
|
|
|
|
- `research/K3_ATTNRES_GRADIENT_DEFINITION_AUDIT.md`
|
|
- `research/K3_ATTNRES_GRADIENT_SCALE_PROTOCOL.md`
|
|
|
|
It is an independent reduced mechanism experiment. It is not a Kimi K3
|
|
checkpoint forward pass and does not claim to recover the paper's unpublished
|
|
Figure 5 telemetry definition.
|
|
|
|
## Frozen environment
|
|
|
|
```text
|
|
Python /home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python
|
|
PyTorch 2.11.0+cu128
|
|
GPU NVIDIA GeForce RTX 5090
|
|
CUBLAS_WORKSPACE_CONFIG=:4096:8
|
|
```
|
|
|
|
## Build the manifest
|
|
|
|
```bash
|
|
python experiments/k3/attnres_gradient/build_dataset.py \
|
|
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
|
--manifest experiments/k3/attnres_gradient/manifest.json
|
|
```
|
|
|
|
## Run a smoke cell
|
|
|
|
```bash
|
|
CUBLAS_WORKSPACE_CONFIG=:4096:8 \
|
|
python experiments/k3/attnres_gradient/train.py \
|
|
--run-kind smoke \
|
|
--architecture block \
|
|
--depth 32 \
|
|
--seed 2026073001 \
|
|
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
|
--manifest experiments/k3/attnres_gradient/manifest.json \
|
|
--output /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1/smoke-a/depth-32-block.json
|
|
```
|
|
|
|
Smoke is fixed to 20 steps. Formal and replay runs are fixed to 8,000 steps;
|
|
the runner rejects alternative budgets. The same command uses
|
|
`--run-kind formal` or `--run-kind replay` and omits an explicit `--steps`.
|
|
|
|
Formal output keys use:
|
|
|
|
```text
|
|
formal/depth-{16|32}-{baseline|block}-seed-{seed}.json
|
|
replay/depth-32-block-seed-2026073001.json
|
|
```
|
|
|
|
Raw parquet/binary files and working runs remain in the local cache. The
|
|
manifest, runner, complete result JSON, compact website payload, reproduction
|
|
hashes, protocol, and audit enter the public repository.
|