feat: factor DeepSeek boundary and role blocks
This commit is contained in:
wuyang
@@ -425,3 +425,109 @@ target-token ordered top-6 routes, with zero target TV, JSD, and ΔCV. See
|
||||
depth maps, token-level alignment, cross-experiment BF16 batch-content audit,
|
||||
primary sources, and the boundary against full role semantics or Chat-model
|
||||
behavior.
|
||||
|
||||
## Special-token family control
|
||||
|
||||
`v2_lite_routing_special_token_family_control.py` keeps the same repeated-token
|
||||
history and changes the completed assistant boundary to four single IDs:
|
||||
|
||||
```text
|
||||
system off/on × EOS / BOS / x / period
|
||||
```
|
||||
|
||||
The pinned tokenizer has exactly two special-token IDs: BOS `100000` and EOS
|
||||
`100001`; PAD aliases EOS. EOS/BOS therefore exhaust the special inventory,
|
||||
while `x` and period are only two selected ordinary controls. The 2-vs-2
|
||||
family summary is descriptive for these four IDs and is not a
|
||||
population-level specialness claim.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||||
python -B experiments/deepseek/v2_lite_routing_special_token_family_control.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--per-domain 32 \
|
||||
--content-tokens 23 \
|
||||
--batch-prompts 4 \
|
||||
--layers 7 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T12:57:00+00:00
|
||||
```
|
||||
|
||||
The eight cells add 2,044,224 real top-6 route selections. All 256
|
||||
source×system groups preserve length and target position, and all 768
|
||||
counterfactuals change exactly one ID. The committed run and independent
|
||||
rerun are byte-exact:
|
||||
|
||||
```text
|
||||
c372c1b03a8b15f615b54ded5d9257a8fc2cdb7728001735d3d4c1d8534af5bf
|
||||
```
|
||||
|
||||
For exact target content under prompt-balanced aggregation, mean system-edge
|
||||
TV is `.037518 / .048018 / .053975 / .055289` for EOS / BOS / x / period.
|
||||
BOS−EOS is positive in 22/24 layer×domain cells; x−EOS and period−EOS are
|
||||
positive in 24/24. The selected ordinary-control mean exceeds the complete
|
||||
special-inventory mean by `.011864` in these four IDs only. See
|
||||
`research/DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md` for all paired
|
||||
intervals, direct edges, scope split, alignment, batch-content audit,
|
||||
literature context, and non-claims.
|
||||
|
||||
## Full two-token role-marker factorial
|
||||
|
||||
`v2_lite_routing_role_marker_block_factorial.py` treats the pre-target
|
||||
two-token marker as two independent factors:
|
||||
|
||||
```text
|
||||
system off/on × head User/Assistant × delimiter colon/x
|
||||
|
||||
User: [5726, 25] Assistant: [77398, 25]
|
||||
User x [5726, 87] Assistant x [77398, 87]
|
||||
```
|
||||
|
||||
The official assistant EOS and post-target generation suffix remain unchanged.
|
||||
The four blocks have zero, one, one, and two edited IDs relative to official
|
||||
`User:`; every edit is verified while length, target position, attention mask,
|
||||
and 32-row batch shape stay fixed.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||||
python -B experiments/deepseek/v2_lite_routing_role_marker_block_factorial.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-role-marker-block-factorial.json \
|
||||
--per-domain 32 \
|
||||
--content-tokens 23 \
|
||||
--batch-prompts 4 \
|
||||
--layers 7 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T13:06:00+00:00
|
||||
```
|
||||
|
||||
The eight cells add 2,044,224 real top-6 route selections. The 256 official,
|
||||
512 one-ID, and 256 two-ID cells all pass their exact edit contracts. The
|
||||
committed run and independent rerun are byte-exact:
|
||||
|
||||
```text
|
||||
a703dddb6d04182b1a608c213bfd341cfc32e1801be42c417dca30a505087e82
|
||||
```
|
||||
|
||||
For exact target content under prompt-balanced aggregation, the four mean
|
||||
system-edge TVs are `.037304 / .037015 / .037239 / .036220`. The head,
|
||||
delimiter, and head×delimiter effects are small and mixed across the 24
|
||||
layer×domain cells. Direct head and delimiter edges remain nonzero; replacing
|
||||
colon with the single `x` control modestly reduces User-vs-Assistant direct TV
|
||||
in most cells, without a uniform system modulation. See
|
||||
`research/DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md` for the exact factor
|
||||
coding, intervals, direct dependencies, full-input split, route alignment,
|
||||
cross-batch audit, sources, and non-claims.
|
||||
|
||||
@@ -737,6 +737,9 @@ def finalize_result(path: Path) -> dict[str, Any]:
|
||||
|
||||
def main() -> None:
|
||||
install_control_contract()
|
||||
if {"-h", "--help"} & set(sys.argv[1:]):
|
||||
base.main()
|
||||
return
|
||||
output = output_path_from_argv()
|
||||
with open(os.devnull, "w", encoding="utf-8") as sink:
|
||||
with contextlib.redirect_stdout(sink):
|
||||
|
||||
@@ -0,0 +1,894 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run a 2 x 2 x 2 target role-marker-block control on DeepSeek-V2-Lite.
|
||||
|
||||
The official target role block is two ordinary token IDs:
|
||||
|
||||
User : -> [5726, 25]
|
||||
|
||||
This experiment crosses system off/on with a complete two-token block:
|
||||
|
||||
role head: User / Assistant
|
||||
delimiter: colon / x
|
||||
|
||||
The four blocks are therefore ``User:``, ``Assistant:``, ``User x``, and
|
||||
``Assistant x``. The official EOS, repeated-token history, target content,
|
||||
generation-prompt ``Assistant:``, sequence length, target position, attention
|
||||
mask, and 32-row padded batch shape are fixed. One cell is an official chat
|
||||
serialization; the other three are explicit pre-target token-ID
|
||||
counterfactuals.
|
||||
|
||||
The design identifies role-head, delimiter, and head-by-delimiter routing
|
||||
effects inside this fixed base-checkpoint forward contract. It does not
|
||||
identify complete role semantics, Chat/SFT behavior, answer quality, or a
|
||||
population-level punctuation effect from one chosen delimiter control.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
import v2_lite_routing_role_marker_head_control as role_prior
|
||||
|
||||
|
||||
boundary_prior = role_prior.prior
|
||||
base = role_prior.base
|
||||
|
||||
SYSTEM_MESSAGE = role_prior.SYSTEM_MESSAGE
|
||||
FILLER_USER = role_prior.FILLER_USER
|
||||
FILLER_ASSISTANT = role_prior.FILLER_ASSISTANT
|
||||
|
||||
BLOCK_LEVELS = (
|
||||
"user_colon",
|
||||
"assistant_colon",
|
||||
"user_x",
|
||||
"assistant_x",
|
||||
)
|
||||
REFERENCE_LEVEL = "user_colon"
|
||||
LEVEL_FACTORS = {
|
||||
"user_colon": {"head": "user", "delimiter": "colon"},
|
||||
"assistant_colon": {"head": "assistant", "delimiter": "colon"},
|
||||
"user_x": {"head": "user", "delimiter": "x"},
|
||||
"assistant_x": {"head": "assistant", "delimiter": "x"},
|
||||
}
|
||||
CONDITIONS = tuple(
|
||||
f"s{system}_{level}"
|
||||
for level in BLOCK_LEVELS
|
||||
for system in (0, 1)
|
||||
)
|
||||
FACTORS = {
|
||||
condition: {
|
||||
"system": int(condition[1]),
|
||||
"history": "filler",
|
||||
"assistant_boundary": "official_eos",
|
||||
"role_block": condition.split("_", 1)[1],
|
||||
**LEVEL_FACTORS[condition.split("_", 1)[1]],
|
||||
}
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
SYSTEM_CELLS = {
|
||||
level: (f"s0_{level}", f"s1_{level}")
|
||||
for level in BLOCK_LEVELS
|
||||
}
|
||||
SYSTEM_EDGE_CONTRASTS = {
|
||||
f"{level}_minus_user_colon": (REFERENCE_LEVEL, level)
|
||||
for level in BLOCK_LEVELS
|
||||
if level != REFERENCE_LEVEL
|
||||
}
|
||||
COMPARISONS = tuple(
|
||||
[
|
||||
(f"system_{level}", f"s0_{level}", f"s1_{level}")
|
||||
for level in BLOCK_LEVELS
|
||||
]
|
||||
+ [
|
||||
(
|
||||
f"{level}_at_s{system}",
|
||||
f"s{system}_{REFERENCE_LEVEL}",
|
||||
f"s{system}_{level}",
|
||||
)
|
||||
for level in BLOCK_LEVELS
|
||||
if level != REFERENCE_LEVEL
|
||||
for system in (0, 1)
|
||||
]
|
||||
+ [
|
||||
(
|
||||
f"head_at_x_s{system}",
|
||||
f"s{system}_user_x",
|
||||
f"s{system}_assistant_x",
|
||||
)
|
||||
for system in (0, 1)
|
||||
]
|
||||
+ [
|
||||
(
|
||||
f"delimiter_at_assistant_s{system}",
|
||||
f"s{system}_assistant_colon",
|
||||
f"s{system}_assistant_x",
|
||||
)
|
||||
for system in (0, 1)
|
||||
]
|
||||
)
|
||||
ALIGNMENT_COMPARISONS = COMPARISONS
|
||||
|
||||
RENDER_AUDIT: list[dict[str, Any]] = []
|
||||
ROLE_TOKEN_IDS: dict[str, int] = {}
|
||||
ORIGINAL_BOUNDARY_DOMAIN = boundary_prior.boundary_control_domain
|
||||
|
||||
|
||||
def condition_messages(
|
||||
content: str,
|
||||
condition: str,
|
||||
) -> list[dict[str, str]]:
|
||||
factors = FACTORS[condition]
|
||||
messages: list[dict[str, str]] = []
|
||||
if factors["system"]:
|
||||
messages.append({"role": "system", "content": SYSTEM_MESSAGE})
|
||||
messages.extend(
|
||||
[
|
||||
{"role": "user", "content": FILLER_USER},
|
||||
{"role": "assistant", "content": FILLER_ASSISTANT},
|
||||
{"role": "user", "content": content},
|
||||
]
|
||||
)
|
||||
return messages
|
||||
|
||||
|
||||
def role_token_ids(tokenizer: Any) -> dict[str, int]:
|
||||
ids = role_prior.role_token_ids(tokenizer)
|
||||
if ids != {
|
||||
"user": 5726,
|
||||
"assistant": 77398,
|
||||
"x": 87,
|
||||
"colon": 25,
|
||||
"eos": 100001,
|
||||
}:
|
||||
raise RuntimeError(f"pinned role token contract changed: {ids}")
|
||||
return ids
|
||||
|
||||
|
||||
def render_role_block_variant(
|
||||
tokenizer: Any,
|
||||
content: str,
|
||||
condition: str,
|
||||
) -> dict[str, Any]:
|
||||
messages = condition_messages(content, condition)
|
||||
rendered = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=False,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
official_ids = list(
|
||||
tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
)
|
||||
token_ids, offsets = base.tokenize_with_offsets(
|
||||
tokenizer,
|
||||
rendered,
|
||||
add_special_tokens=False,
|
||||
)
|
||||
if token_ids != official_ids:
|
||||
raise RuntimeError(
|
||||
f"{condition} rendered IDs differ from apply_chat_template"
|
||||
)
|
||||
|
||||
ids = role_token_ids(tokenizer)
|
||||
ROLE_TOKEN_IDS.update(ids)
|
||||
content_start = rendered.rfind(content)
|
||||
if content_start < 0:
|
||||
raise RuntimeError(f"{condition} target content is absent")
|
||||
content_end = content_start + len(content)
|
||||
positions, records, crossing = base.content_positions(
|
||||
token_ids,
|
||||
offsets,
|
||||
content_start,
|
||||
content_end,
|
||||
)
|
||||
if not positions:
|
||||
raise RuntimeError(f"{condition} has no target-content tokens")
|
||||
target_first = min(positions)
|
||||
target_last = max(positions)
|
||||
|
||||
target_user_candidates = [
|
||||
index
|
||||
for index, token_id in enumerate(official_ids)
|
||||
if token_id == ids["user"] and index < target_first
|
||||
]
|
||||
if len(target_user_candidates) < 2:
|
||||
raise RuntimeError(
|
||||
f"{condition} cannot locate filler and target User heads"
|
||||
)
|
||||
target_head_position = max(target_user_candidates)
|
||||
target_delimiter_position = target_head_position + 1
|
||||
if official_ids[target_delimiter_position] != ids["colon"]:
|
||||
raise RuntimeError("target User head is not followed by colon")
|
||||
|
||||
suffix_assistant_candidates = [
|
||||
index
|
||||
for index, token_id in enumerate(official_ids)
|
||||
if token_id == ids["assistant"] and index > target_last
|
||||
]
|
||||
if len(suffix_assistant_candidates) != 1:
|
||||
raise RuntimeError(
|
||||
f"{condition} expected one suffix Assistant head; "
|
||||
f"got {suffix_assistant_candidates}"
|
||||
)
|
||||
suffix_head_position = suffix_assistant_candidates[0]
|
||||
if official_ids[suffix_head_position + 1] != ids["colon"]:
|
||||
raise RuntimeError("suffix Assistant head is not followed by colon")
|
||||
if ids["eos"] not in official_ids[:target_head_position]:
|
||||
raise RuntimeError("official history EOS is absent before target role")
|
||||
|
||||
level = FACTORS[condition]["role_block"]
|
||||
desired_head = ids[LEVEL_FACTORS[level]["head"]]
|
||||
desired_delimiter = ids[LEVEL_FACTORS[level]["delimiter"]]
|
||||
token_ids = list(token_ids)
|
||||
token_ids[target_head_position] = desired_head
|
||||
token_ids[target_delimiter_position] = desired_delimiter
|
||||
changed_positions = [
|
||||
index
|
||||
for index, (left, right) in enumerate(
|
||||
zip(official_ids, token_ids, strict=True)
|
||||
)
|
||||
if left != right
|
||||
]
|
||||
expected_differences = (
|
||||
int(LEVEL_FACTORS[level]["head"] != "user")
|
||||
+ int(LEVEL_FACTORS[level]["delimiter"] != "colon")
|
||||
)
|
||||
if len(changed_positions) != expected_differences:
|
||||
raise RuntimeError(
|
||||
f"{condition} changed {changed_positions}; "
|
||||
f"expected {expected_differences} positions"
|
||||
)
|
||||
if any(position >= target_first for position in changed_positions):
|
||||
raise RuntimeError("role-block edit is not strictly pre-target")
|
||||
|
||||
decoded = tokenizer.decode(
|
||||
token_ids,
|
||||
skip_special_tokens=False,
|
||||
clean_up_tokenization_spaces=False,
|
||||
)
|
||||
RENDER_AUDIT.append(
|
||||
{
|
||||
"content_sha256": base.text_sha256(content),
|
||||
"condition": condition,
|
||||
"role_block": level,
|
||||
"head": LEVEL_FACTORS[level]["head"],
|
||||
"delimiter": LEVEL_FACTORS[level]["delimiter"],
|
||||
"target_head_position": target_head_position,
|
||||
"target_delimiter_position": target_delimiter_position,
|
||||
"suffix_head_position": suffix_head_position,
|
||||
"changed_positions": changed_positions,
|
||||
"changed_token_ids_vs_official": len(changed_positions),
|
||||
"official_eos_token_id": ids["eos"],
|
||||
"input_tokens": len(token_ids),
|
||||
"target_first_position": target_first,
|
||||
"target_last_position": target_last,
|
||||
"counterfactual_decoded_sha256": base.text_sha256(decoded),
|
||||
}
|
||||
)
|
||||
return {
|
||||
"condition": condition,
|
||||
"messages_sha256": base.canonical_hash(messages),
|
||||
"rendered_sha256": base.text_sha256(rendered),
|
||||
"token_ids": token_ids,
|
||||
"tokens": len(token_ids),
|
||||
"token_ids_sha256": base.canonical_hash(token_ids),
|
||||
"content_positions": positions,
|
||||
"content_records": records,
|
||||
"content_tokens": len(positions),
|
||||
"wrapper_tokens": len(token_ids) - len(positions),
|
||||
"boundary_crossing_tokens": crossing,
|
||||
}
|
||||
|
||||
|
||||
def factorial_effects(
|
||||
values: dict[str, np.ndarray],
|
||||
) -> dict[str, np.ndarray]:
|
||||
return {
|
||||
"head_main": 0.5 * (
|
||||
values["assistant_colon"]
|
||||
+ values["assistant_x"]
|
||||
- values["user_colon"]
|
||||
- values["user_x"]
|
||||
),
|
||||
"delimiter_main": 0.5 * (
|
||||
values["user_x"]
|
||||
+ values["assistant_x"]
|
||||
- values["user_colon"]
|
||||
- values["assistant_colon"]
|
||||
),
|
||||
"head_by_delimiter": (
|
||||
values["assistant_x"]
|
||||
- values["user_x"]
|
||||
- values["assistant_colon"]
|
||||
+ values["user_colon"]
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def role_block_domain(
|
||||
loads: dict[str, np.ndarray],
|
||||
mode: str,
|
||||
replicates: int,
|
||||
seed: int,
|
||||
scope: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Add paired role-head, delimiter, and interaction statistics."""
|
||||
result = ORIGINAL_BOUNDARY_DOMAIN(
|
||||
loads,
|
||||
mode,
|
||||
replicates,
|
||||
seed,
|
||||
scope,
|
||||
)
|
||||
shapes = {value.shape for value in loads.values()}
|
||||
if len(shapes) != 1:
|
||||
raise ValueError(f"role-block shape mismatch: {sorted(shapes)}")
|
||||
rows = next(iter(loads.values())).shape[0]
|
||||
rng = np.random.default_rng(base.scoped_seed(seed, scope))
|
||||
sampled = rng.integers(
|
||||
0,
|
||||
rows,
|
||||
size=(replicates, rows),
|
||||
endpoint=False,
|
||||
)
|
||||
point = {
|
||||
condition: base.distribution(value, mode)
|
||||
for condition, value in loads.items()
|
||||
}
|
||||
boot = {
|
||||
condition: base.bootstrap_distributions(value, mode, sampled)
|
||||
for condition, value in loads.items()
|
||||
}
|
||||
|
||||
point_system_vectors = {
|
||||
level: point[after] - point[before]
|
||||
for level, (before, after) in SYSTEM_CELLS.items()
|
||||
}
|
||||
boot_system_vectors = {
|
||||
level: boot[after] - boot[before]
|
||||
for level, (before, after) in SYSTEM_CELLS.items()
|
||||
}
|
||||
point_vector_effects = factorial_effects(point_system_vectors)
|
||||
boot_vector_effects = factorial_effects(boot_system_vectors)
|
||||
distribution_effects = {}
|
||||
for name in point_vector_effects:
|
||||
point_magnitude = 0.5 * np.abs(
|
||||
point_vector_effects[name]
|
||||
).sum()
|
||||
boot_magnitude = 0.5 * np.abs(
|
||||
boot_vector_effects[name]
|
||||
).sum(axis=1)
|
||||
distribution_effects[name] = {
|
||||
"half_l1_magnitude": {
|
||||
"point": float(point_magnitude),
|
||||
"ci95": base.interval(boot_magnitude),
|
||||
},
|
||||
"signed_expert_share_effect": (
|
||||
point_vector_effects[name].tolist()
|
||||
),
|
||||
"signed_expert_share_effect_ci95": (
|
||||
base.interval(boot_vector_effects[name])
|
||||
),
|
||||
}
|
||||
|
||||
point_tv: dict[str, float] = {}
|
||||
boot_tv: dict[str, np.ndarray] = {}
|
||||
point_jsd: dict[str, float] = {}
|
||||
boot_jsd: dict[str, np.ndarray] = {}
|
||||
for level, (before, after) in SYSTEM_CELLS.items():
|
||||
point_tv[level] = float(
|
||||
0.5 * np.abs(point[after] - point[before]).sum()
|
||||
)
|
||||
boot_tv[level] = 0.5 * np.abs(
|
||||
boot[after] - boot[before]
|
||||
).sum(axis=1)
|
||||
point_jsd[level] = float(
|
||||
base.js_divergence(point[before], point[after])[0]
|
||||
)
|
||||
boot_jsd[level] = base.js_divergence(
|
||||
boot[before],
|
||||
boot[after],
|
||||
)
|
||||
|
||||
def scalar_factorial(
|
||||
point_values: dict[str, float],
|
||||
boot_values: dict[str, np.ndarray],
|
||||
unit: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
point_arrays = {
|
||||
key: np.asarray([value])
|
||||
for key, value in point_values.items()
|
||||
}
|
||||
point_effects = factorial_effects(point_arrays)
|
||||
boot_effects = factorial_effects(boot_values)
|
||||
payload = {
|
||||
name: {
|
||||
"point": float(point_effects[name][0]),
|
||||
"ci95": base.interval(boot_effects[name]),
|
||||
}
|
||||
for name in point_effects
|
||||
}
|
||||
if unit is not None:
|
||||
for value in payload.values():
|
||||
value["unit"] = unit
|
||||
return payload
|
||||
|
||||
metric_effects: dict[str, Any] = {}
|
||||
point_metrics = {
|
||||
condition: base.metric_vector(value)
|
||||
for condition, value in point.items()
|
||||
}
|
||||
boot_metrics = {
|
||||
condition: base.metric_vector(value)
|
||||
for condition, value in boot.items()
|
||||
}
|
||||
for metric in point_metrics[f"s0_{REFERENCE_LEVEL}"]:
|
||||
point_edges = {
|
||||
level: (
|
||||
point_metrics[after][metric]
|
||||
- point_metrics[before][metric]
|
||||
)
|
||||
for level, (before, after) in SYSTEM_CELLS.items()
|
||||
}
|
||||
boot_edges = {
|
||||
level: (
|
||||
boot_metrics[after][metric]
|
||||
- boot_metrics[before][metric]
|
||||
)
|
||||
for level, (before, after) in SYSTEM_CELLS.items()
|
||||
}
|
||||
point_effects = factorial_effects(point_edges)
|
||||
boot_effects = factorial_effects(boot_edges)
|
||||
metric_effects[metric] = {
|
||||
name: {
|
||||
"point": float(point_effects[name][0]),
|
||||
"ci95": base.interval(boot_effects[name]),
|
||||
}
|
||||
for name in point_effects
|
||||
}
|
||||
|
||||
def direct_distance(
|
||||
before_level: str,
|
||||
after_level: str,
|
||||
system: int,
|
||||
) -> dict[str, Any]:
|
||||
before = f"s{system}_{before_level}"
|
||||
after = f"s{system}_{after_level}"
|
||||
point_delta = point[after] - point[before]
|
||||
tv_boot = 0.5 * np.abs(
|
||||
boot[after] - boot[before]
|
||||
).sum(axis=1)
|
||||
jsd_boot = base.js_divergence(boot[before], boot[after])
|
||||
return {
|
||||
"total_variation": {
|
||||
"point": float(0.5 * np.abs(point_delta).sum()),
|
||||
"ci95": base.interval(tv_boot),
|
||||
"_bootstrap": tv_boot,
|
||||
},
|
||||
"js_divergence": {
|
||||
"point": float(
|
||||
base.js_divergence(point[before], point[after])[0]
|
||||
),
|
||||
"ci95": base.interval(jsd_boot),
|
||||
"unit": "nats",
|
||||
"_bootstrap": jsd_boot,
|
||||
},
|
||||
}
|
||||
|
||||
direct_specs = {
|
||||
"head_at_colon": ("user_colon", "assistant_colon"),
|
||||
"head_at_x": ("user_x", "assistant_x"),
|
||||
"delimiter_at_user": ("user_colon", "user_x"),
|
||||
"delimiter_at_assistant": (
|
||||
"assistant_colon",
|
||||
"assistant_x",
|
||||
),
|
||||
}
|
||||
direct_raw = {
|
||||
name: {
|
||||
system: direct_distance(
|
||||
before_level,
|
||||
after_level,
|
||||
system,
|
||||
)
|
||||
for system in (0, 1)
|
||||
}
|
||||
for name, (before_level, after_level) in direct_specs.items()
|
||||
}
|
||||
direct: dict[str, Any] = {}
|
||||
for name in direct_specs:
|
||||
cells = direct_raw[name]
|
||||
direct[name] = {}
|
||||
for system in (0, 1):
|
||||
direct[name][f"at_s{system}"] = {
|
||||
metric: {
|
||||
key: value
|
||||
for key, value in payload.items()
|
||||
if key != "_bootstrap"
|
||||
}
|
||||
for metric, payload in cells[system].items()
|
||||
}
|
||||
direct[name]["s1_minus_s0"] = {}
|
||||
for metric in ("total_variation", "js_divergence"):
|
||||
before_payload = cells[0][metric]
|
||||
after_payload = cells[1][metric]
|
||||
direct[name]["s1_minus_s0"][
|
||||
f"{metric}_delta"
|
||||
] = {
|
||||
"point": (
|
||||
after_payload["point"] - before_payload["point"]
|
||||
),
|
||||
"ci95": base.interval(
|
||||
after_payload["_bootstrap"]
|
||||
- before_payload["_bootstrap"]
|
||||
),
|
||||
**(
|
||||
{"unit": "nats"}
|
||||
if metric == "js_divergence"
|
||||
else {}
|
||||
),
|
||||
}
|
||||
|
||||
dependency_specs = {
|
||||
"delimiter_dependence_of_head_direct": (
|
||||
"head_at_colon",
|
||||
"head_at_x",
|
||||
),
|
||||
"head_dependence_of_delimiter_direct": (
|
||||
"delimiter_at_user",
|
||||
"delimiter_at_assistant",
|
||||
),
|
||||
}
|
||||
dependencies: dict[str, Any] = {}
|
||||
for name, (before_edge, after_edge) in dependency_specs.items():
|
||||
dependencies[name] = {}
|
||||
system_payloads: dict[int, dict[str, Any]] = {}
|
||||
for system in (0, 1):
|
||||
system_payloads[system] = {}
|
||||
for metric in ("total_variation", "js_divergence"):
|
||||
before_payload = direct_raw[before_edge][system][metric]
|
||||
after_payload = direct_raw[after_edge][system][metric]
|
||||
payload = {
|
||||
"point": (
|
||||
after_payload["point"] - before_payload["point"]
|
||||
),
|
||||
"ci95": base.interval(
|
||||
after_payload["_bootstrap"]
|
||||
- before_payload["_bootstrap"]
|
||||
),
|
||||
"_bootstrap": (
|
||||
after_payload["_bootstrap"]
|
||||
- before_payload["_bootstrap"]
|
||||
),
|
||||
}
|
||||
if metric == "js_divergence":
|
||||
payload["unit"] = "nats"
|
||||
system_payloads[system][metric] = payload
|
||||
dependencies[name].setdefault(
|
||||
f"at_s{system}",
|
||||
{},
|
||||
)[f"{metric}_delta"] = {
|
||||
key: value
|
||||
for key, value in payload.items()
|
||||
if key != "_bootstrap"
|
||||
}
|
||||
dependencies[name]["s1_minus_s0"] = {}
|
||||
for metric in ("total_variation", "js_divergence"):
|
||||
before_payload = system_payloads[0][metric]
|
||||
after_payload = system_payloads[1][metric]
|
||||
dependencies[name]["s1_minus_s0"][
|
||||
f"{metric}_difference_in_differences"
|
||||
] = {
|
||||
"point": (
|
||||
after_payload["point"] - before_payload["point"]
|
||||
),
|
||||
"ci95": base.interval(
|
||||
after_payload["_bootstrap"]
|
||||
- before_payload["_bootstrap"]
|
||||
),
|
||||
**(
|
||||
{"unit": "nats"}
|
||||
if metric == "js_divergence"
|
||||
else {}
|
||||
),
|
||||
}
|
||||
|
||||
result["role_block_factorial"] = {
|
||||
"factor_coding": {
|
||||
"head_main": (
|
||||
"0.5 * [(Assistant:-User:) + "
|
||||
"(Assistant x-User x)]"
|
||||
),
|
||||
"delimiter_main": (
|
||||
"0.5 * [(User x-User:) + "
|
||||
"(Assistant x-Assistant:)]"
|
||||
),
|
||||
"head_by_delimiter": (
|
||||
"(Assistant x-User x) - (Assistant:-User:)"
|
||||
),
|
||||
},
|
||||
"system_edge_distance_effects": {
|
||||
"total_variation": scalar_factorial(point_tv, boot_tv),
|
||||
"js_divergence": scalar_factorial(
|
||||
point_jsd,
|
||||
boot_jsd,
|
||||
unit="nats",
|
||||
),
|
||||
},
|
||||
"metric_system_edge_effects": metric_effects,
|
||||
"distribution_system_edge_effects": distribution_effects,
|
||||
"direct_factor_edges": direct,
|
||||
"direct_effect_dependencies": dependencies,
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def layer_statistics(
|
||||
prompt_rows: list[dict[str, Any]],
|
||||
replicates: int,
|
||||
seed: int,
|
||||
layer_index: int,
|
||||
) -> dict[str, Any]:
|
||||
scopes = boundary_prior.layer_statistics(
|
||||
prompt_rows,
|
||||
replicates,
|
||||
seed,
|
||||
layer_index,
|
||||
)
|
||||
for scope in scopes.values():
|
||||
for mode in scope["modes"].values():
|
||||
mode["role_block_control"] = mode.pop("boundary_control")
|
||||
return scopes
|
||||
|
||||
|
||||
def install_control_contract() -> None:
|
||||
boundary_prior.BOUNDARY_LEVELS = BLOCK_LEVELS
|
||||
boundary_prior.REFERENCE_LEVEL = REFERENCE_LEVEL
|
||||
boundary_prior.CONDITIONS = CONDITIONS
|
||||
boundary_prior.FACTORS = FACTORS
|
||||
boundary_prior.SYSTEM_CELLS = SYSTEM_CELLS
|
||||
boundary_prior.SYSTEM_EDGE_CONTRASTS = SYSTEM_EDGE_CONTRASTS
|
||||
boundary_prior.COMPARISONS = COMPARISONS
|
||||
boundary_prior.ALIGNMENT_COMPARISONS = ALIGNMENT_COMPARISONS
|
||||
boundary_prior.boundary_control_domain = role_block_domain
|
||||
|
||||
base.CONDITIONS = CONDITIONS
|
||||
base.FACTORS = FACTORS
|
||||
base.COMPARISONS = COMPARISONS
|
||||
base.ALIGNMENT_COMPARISONS = ALIGNMENT_COMPARISONS
|
||||
base.condition_messages = condition_messages
|
||||
base.render_variant = render_role_block_variant
|
||||
base.layer_statistics = layer_statistics
|
||||
|
||||
|
||||
def output_path_from_argv() -> Path:
|
||||
try:
|
||||
return Path(sys.argv[sys.argv.index("--output") + 1])
|
||||
except (ValueError, IndexError) as error:
|
||||
raise ValueError("--output is required") from error
|
||||
|
||||
|
||||
def render_contract_summary() -> dict[str, Any]:
|
||||
if not RENDER_AUDIT:
|
||||
raise RuntimeError("render audit is empty")
|
||||
by_content: dict[str, list[dict[str, Any]]] = {}
|
||||
for row in RENDER_AUDIT:
|
||||
by_content.setdefault(row["content_sha256"], []).append(row)
|
||||
if len(by_content) != 128 and "--per-domain" not in sys.argv:
|
||||
raise RuntimeError(
|
||||
f"expected 128 selected contents; observed {len(by_content)}"
|
||||
)
|
||||
|
||||
equal_lengths = 0
|
||||
equal_target_positions = 0
|
||||
exact_by_level = {
|
||||
level: 0
|
||||
for level in BLOCK_LEVELS
|
||||
}
|
||||
for rows in by_content.values():
|
||||
for system in (0, 1):
|
||||
cells = [
|
||||
row
|
||||
for row in rows
|
||||
if FACTORS[row["condition"]]["system"] == system
|
||||
]
|
||||
if len(cells) != len(BLOCK_LEVELS):
|
||||
raise RuntimeError(
|
||||
"render audit lacks one or more role-block cells"
|
||||
)
|
||||
if len({row["input_tokens"] for row in cells}) == 1:
|
||||
equal_lengths += 1
|
||||
target_spans = {
|
||||
(
|
||||
row["target_first_position"],
|
||||
row["target_last_position"],
|
||||
)
|
||||
for row in cells
|
||||
}
|
||||
if len(target_spans) == 1:
|
||||
equal_target_positions += 1
|
||||
for row in cells:
|
||||
expected = (
|
||||
int(row["head"] != "user")
|
||||
+ int(row["delimiter"] != "colon")
|
||||
)
|
||||
exact_by_level[row["role_block"]] += int(
|
||||
row["changed_token_ids_vs_official"] == expected
|
||||
)
|
||||
|
||||
groups = 2 * len(by_content)
|
||||
return {
|
||||
"selected_contents": len(by_content),
|
||||
"system_groups": groups,
|
||||
"equal_input_length_groups": equal_lengths,
|
||||
"equal_target_position_groups": equal_target_positions,
|
||||
"exact_edit_contract_cells_by_level": exact_by_level,
|
||||
"official_zero_id_cells_exact": exact_by_level["user_colon"],
|
||||
"one_id_counterfactual_cells_exact": (
|
||||
exact_by_level["assistant_colon"]
|
||||
+ exact_by_level["user_x"]
|
||||
),
|
||||
"two_id_counterfactual_cells_exact": (
|
||||
exact_by_level["assistant_x"]
|
||||
),
|
||||
"all_group_lengths_equal": equal_lengths == groups,
|
||||
"all_target_positions_equal": equal_target_positions == groups,
|
||||
"all_edit_contracts_exact": all(
|
||||
value == groups
|
||||
for value in exact_by_level.values()
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def finalize_result(path: Path) -> dict[str, Any]:
|
||||
result = json.loads(path.read_text(encoding="utf-8"))
|
||||
result["schema_version"] = 4
|
||||
result["evidence_identity"] = (
|
||||
"X / official BF16 weights and pinned tokenizer; target two-ID "
|
||||
"role head by delimiter factorial counterfactuals"
|
||||
)
|
||||
boundary = result["boundary"]
|
||||
boundary.pop("factorial_claim", None)
|
||||
boundary.update(
|
||||
{
|
||||
"role_marker_block_factorial": True,
|
||||
"official_serialization_by_role_block": {
|
||||
"user_colon": True,
|
||||
"assistant_colon": False,
|
||||
"user_x": False,
|
||||
"assistant_x": False,
|
||||
},
|
||||
"official_assistant_eos_held_fixed": True,
|
||||
"generation_prompt_held_fixed": True,
|
||||
"target_position_held_fixed": True,
|
||||
"complete_role_semantics_identified": False,
|
||||
"task_performance": False,
|
||||
"causal_boundary": (
|
||||
"Within this fixed base-checkpoint batch, edits intervene on "
|
||||
"the two target role-block input positions before target "
|
||||
"content. They identify head, one delimiter control, and "
|
||||
"their interaction, not complete role semantics, Chat/SFT "
|
||||
"behavior, or answer quality."
|
||||
),
|
||||
}
|
||||
)
|
||||
|
||||
old_contract = result.pop("message_history_contract")
|
||||
result["role_marker_block_contract"] = {
|
||||
"chat_template_revision": base.MODEL_REVISION,
|
||||
"chat_template": old_contract["chat_template"],
|
||||
"chat_template_sha256": old_contract["chat_template_sha256"],
|
||||
"official_sequence": (
|
||||
"Assistant: {filler} + eos_token + User: {target} + "
|
||||
"generation-prompt Assistant:"
|
||||
),
|
||||
"system_message": SYSTEM_MESSAGE,
|
||||
"system_message_sha256": base.text_sha256(SYSTEM_MESSAGE),
|
||||
"filler_user": FILLER_USER,
|
||||
"filler_user_sha256": base.text_sha256(FILLER_USER),
|
||||
"filler_assistant": FILLER_ASSISTANT,
|
||||
"filler_assistant_sha256": base.text_sha256(FILLER_ASSISTANT),
|
||||
"block_levels": list(BLOCK_LEVELS),
|
||||
"level_factors": LEVEL_FACTORS,
|
||||
"role_token_ids": ROLE_TOKEN_IDS,
|
||||
"official_target_block_ids": [
|
||||
ROLE_TOKEN_IDS["user"],
|
||||
ROLE_TOKEN_IDS["colon"],
|
||||
],
|
||||
"conditions": FACTORS,
|
||||
"comparisons": [
|
||||
{"name": name, "before": before, "after": after}
|
||||
for name, before, after in COMPARISONS
|
||||
],
|
||||
"render_validation": render_contract_summary(),
|
||||
"target_role": "user",
|
||||
"add_generation_prompt": True,
|
||||
"scope_split": {
|
||||
"full_input": (
|
||||
"all official or counterfactually edited BOS, system/history, "
|
||||
"target, newline, and generation-prompt token IDs"
|
||||
),
|
||||
"target_content": (
|
||||
"exact intersection of (relative character span, token ID) "
|
||||
"inside target user content across all eight conditions"
|
||||
),
|
||||
},
|
||||
}
|
||||
result["inference_contract"]["batch_grouping"] = (
|
||||
"all eight system x head x delimiter variants of one source prompt "
|
||||
"execute in the same 32-row right-padded batch"
|
||||
)
|
||||
statistical = result["statistical_contract"]
|
||||
statistical["paired_indices"] = (
|
||||
"one sampled source-prompt index matrix is reused across all eight "
|
||||
"cells for every role-block main effect and interaction within each "
|
||||
"domain/layer/scope/mode"
|
||||
)
|
||||
statistical["role_block_factor_coding"] = {
|
||||
"head_main": (
|
||||
"average Assistant-minus-User edge across colon and x"
|
||||
),
|
||||
"delimiter_main": (
|
||||
"average x-minus-colon edge across User and Assistant"
|
||||
),
|
||||
"head_by_delimiter": (
|
||||
"(Assistant x-User x) - (Assistant:-User:)"
|
||||
),
|
||||
}
|
||||
|
||||
path.write_text(
|
||||
json.dumps(result, indent=2, ensure_ascii=False) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
install_control_contract()
|
||||
if {"-h", "--help"} & set(sys.argv[1:]):
|
||||
base.main()
|
||||
return
|
||||
output = output_path_from_argv()
|
||||
with open(os.devnull, "w", encoding="utf-8") as sink:
|
||||
with contextlib.redirect_stdout(sink):
|
||||
base.main()
|
||||
result = finalize_result(output)
|
||||
payload = output.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(output),
|
||||
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"bytes": len(payload),
|
||||
"source_prompts": result["inference_contract"][
|
||||
"total_source_prompts"
|
||||
],
|
||||
"prompt_variants": result["inference_contract"][
|
||||
"total_prompt_variants"
|
||||
],
|
||||
"total_routes": result["inference_contract"][
|
||||
"total_routes_all_conditions_all_moe_layers"
|
||||
],
|
||||
"render_validation": result[
|
||||
"role_marker_block_contract"
|
||||
]["render_validation"],
|
||||
},
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,375 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run a pinned special-token-family boundary control on DeepSeek-V2-Lite.
|
||||
|
||||
The official tokenizer at revision 604d5664 has exactly two special token IDs:
|
||||
|
||||
bos: 100000, <｜begin▁of▁sentence｜>
|
||||
eos: 100001, <｜end▁of▁sentence｜> (also used as pad_token)
|
||||
|
||||
This experiment keeps the audited repeated-token history and changes exactly
|
||||
one completed-assistant boundary ID:
|
||||
|
||||
system off/on x official EOS / counterfactual BOS / x / period
|
||||
|
||||
EOS and BOS exhaust the tokenizer's special-token inventory. X and period are
|
||||
ordinary one-token controls. The three counterfactuals are not valid official
|
||||
chat serializations. This design can identify differences among these four
|
||||
pinned IDs inside one fixed BF16 batch contract; it cannot establish a general
|
||||
"specialness" property from two special and two selected ordinary tokens.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
import v2_lite_routing_history_boundary_token_control as prior
|
||||
|
||||
|
||||
base = prior.base
|
||||
|
||||
BOUNDARY_LEVELS = ("eos", "bos", "x", "period")
|
||||
REFERENCE_LEVEL = "eos"
|
||||
BOUNDARY_TEXT = {
|
||||
"x": "x",
|
||||
"period": ".",
|
||||
}
|
||||
CONDITIONS = tuple(
|
||||
f"s{system}_{boundary}"
|
||||
for boundary in BOUNDARY_LEVELS
|
||||
for system in (0, 1)
|
||||
)
|
||||
FACTORS = {
|
||||
condition: {
|
||||
"system": int(condition[1]),
|
||||
"history": "filler",
|
||||
"boundary": condition.split("_", 1)[1],
|
||||
"token_class": (
|
||||
"special"
|
||||
if condition.split("_", 1)[1] in {"eos", "bos"}
|
||||
else "ordinary_control"
|
||||
),
|
||||
}
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
SYSTEM_CELLS = {
|
||||
boundary: (f"s0_{boundary}", f"s1_{boundary}")
|
||||
for boundary in BOUNDARY_LEVELS
|
||||
}
|
||||
SYSTEM_EDGE_CONTRASTS = {
|
||||
f"{boundary}_minus_eos": ("eos", boundary)
|
||||
for boundary in BOUNDARY_LEVELS
|
||||
if boundary != "eos"
|
||||
}
|
||||
COMPARISONS = tuple(
|
||||
[
|
||||
(
|
||||
f"system_{boundary}",
|
||||
f"s0_{boundary}",
|
||||
f"s1_{boundary}",
|
||||
)
|
||||
for boundary in BOUNDARY_LEVELS
|
||||
]
|
||||
+ [
|
||||
(
|
||||
f"{boundary}_at_s{system}",
|
||||
f"s{system}_eos",
|
||||
f"s{system}_{boundary}",
|
||||
)
|
||||
for boundary in BOUNDARY_LEVELS
|
||||
if boundary != "eos"
|
||||
for system in (0, 1)
|
||||
]
|
||||
)
|
||||
ALIGNMENT_COMPARISONS = COMPARISONS
|
||||
ORIGINAL_BOUNDARY_DOMAIN = prior.boundary_control_domain
|
||||
|
||||
|
||||
def special_family_token_ids(tokenizer: Any) -> dict[str, int]:
|
||||
"""Resolve and validate the complete pinned special-token inventory."""
|
||||
if tokenizer.bos_token_id is None or tokenizer.eos_token_id is None:
|
||||
raise RuntimeError("tokenizer must expose both BOS and EOS")
|
||||
special_ids = [int(value) for value in tokenizer.all_special_ids]
|
||||
expected_special = {
|
||||
int(tokenizer.bos_token_id),
|
||||
int(tokenizer.eos_token_id),
|
||||
}
|
||||
if set(special_ids) != expected_special or len(special_ids) != 2:
|
||||
raise RuntimeError(
|
||||
"pinned special-token inventory changed: "
|
||||
f"observed={special_ids} expected={sorted(expected_special)}"
|
||||
)
|
||||
if tokenizer.pad_token_id != tokenizer.eos_token_id:
|
||||
raise RuntimeError(
|
||||
"pinned tokenizer no longer aliases pad_token_id to eos_token_id"
|
||||
)
|
||||
|
||||
ids = {
|
||||
"eos": int(tokenizer.eos_token_id),
|
||||
"bos": int(tokenizer.bos_token_id),
|
||||
}
|
||||
for name, text in BOUNDARY_TEXT.items():
|
||||
encoded = list(
|
||||
tokenizer(text, add_special_tokens=False).input_ids
|
||||
)
|
||||
if len(encoded) != 1:
|
||||
raise RuntimeError(
|
||||
f"{name} boundary control is not one token: {encoded}"
|
||||
)
|
||||
if encoded[0] in expected_special:
|
||||
raise RuntimeError(
|
||||
f"{name} boundary control unexpectedly uses a special token"
|
||||
)
|
||||
ids[name] = int(encoded[0])
|
||||
if len(set(ids.values())) != len(ids):
|
||||
raise RuntimeError(f"boundary token IDs are not distinct: {ids}")
|
||||
return ids
|
||||
|
||||
|
||||
def special_family_domain(
|
||||
loads: dict[str, np.ndarray],
|
||||
mode: str,
|
||||
replicates: int,
|
||||
seed: int,
|
||||
scope: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Add a paired descriptive 2-special versus 2-ordinary summary."""
|
||||
result = ORIGINAL_BOUNDARY_DOMAIN(
|
||||
loads,
|
||||
mode,
|
||||
replicates,
|
||||
seed,
|
||||
scope,
|
||||
)
|
||||
shapes = {value.shape for value in loads.values()}
|
||||
if len(shapes) != 1:
|
||||
raise ValueError(
|
||||
f"special-family shape mismatch: {sorted(shapes)}"
|
||||
)
|
||||
rows = next(iter(loads.values())).shape[0]
|
||||
rng = np.random.default_rng(base.scoped_seed(seed, scope))
|
||||
sampled = rng.integers(
|
||||
0,
|
||||
rows,
|
||||
size=(replicates, rows),
|
||||
endpoint=False,
|
||||
)
|
||||
point = {
|
||||
condition: base.distribution(value, mode)
|
||||
for condition, value in loads.items()
|
||||
}
|
||||
boot = {
|
||||
condition: base.bootstrap_distributions(value, mode, sampled)
|
||||
for condition, value in loads.items()
|
||||
}
|
||||
|
||||
point_tv: dict[str, float] = {}
|
||||
boot_tv: dict[str, np.ndarray] = {}
|
||||
point_jsd: dict[str, float] = {}
|
||||
boot_jsd: dict[str, np.ndarray] = {}
|
||||
for level, (before, after) in SYSTEM_CELLS.items():
|
||||
point_tv[level] = float(
|
||||
0.5 * np.abs(point[after] - point[before]).sum()
|
||||
)
|
||||
boot_tv[level] = 0.5 * np.abs(
|
||||
boot[after] - boot[before]
|
||||
).sum(axis=1)
|
||||
point_jsd[level] = float(
|
||||
base.js_divergence(point[before], point[after])[0]
|
||||
)
|
||||
boot_jsd[level] = base.js_divergence(
|
||||
boot[before],
|
||||
boot[after],
|
||||
)
|
||||
|
||||
def paired_family(
|
||||
point_metric: dict[str, float],
|
||||
boot_metric: dict[str, np.ndarray],
|
||||
unit: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
special_point = 0.5 * (
|
||||
point_metric["eos"] + point_metric["bos"]
|
||||
)
|
||||
ordinary_point = 0.5 * (
|
||||
point_metric["x"] + point_metric["period"]
|
||||
)
|
||||
special_boot = 0.5 * (
|
||||
boot_metric["eos"] + boot_metric["bos"]
|
||||
)
|
||||
ordinary_boot = 0.5 * (
|
||||
boot_metric["x"] + boot_metric["period"]
|
||||
)
|
||||
payload = {
|
||||
"special_mean": {
|
||||
"point": special_point,
|
||||
"ci95": base.interval(special_boot),
|
||||
},
|
||||
"ordinary_control_mean": {
|
||||
"point": ordinary_point,
|
||||
"ci95": base.interval(ordinary_boot),
|
||||
},
|
||||
"ordinary_minus_special": {
|
||||
"point": ordinary_point - special_point,
|
||||
"ci95": base.interval(ordinary_boot - special_boot),
|
||||
},
|
||||
}
|
||||
if unit is not None:
|
||||
for value in payload.values():
|
||||
value["unit"] = unit
|
||||
return payload
|
||||
|
||||
result["descriptive_family_summary"] = {
|
||||
"definition": (
|
||||
"mean(EOS,BOS) versus mean(x,period) inside the same shared "
|
||||
"source bootstrap; descriptive for these four IDs only"
|
||||
),
|
||||
"total_variation": paired_family(point_tv, boot_tv),
|
||||
"js_divergence": paired_family(
|
||||
point_jsd,
|
||||
boot_jsd,
|
||||
unit="nats",
|
||||
),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def install_control_contract() -> None:
|
||||
"""Install four pinned boundary IDs into the audited eight-cell runner."""
|
||||
prior.BOUNDARY_LEVELS = BOUNDARY_LEVELS
|
||||
prior.REFERENCE_LEVEL = REFERENCE_LEVEL
|
||||
prior.BOUNDARY_TEXT = BOUNDARY_TEXT
|
||||
prior.CONDITIONS = CONDITIONS
|
||||
prior.FACTORS = FACTORS
|
||||
prior.SYSTEM_CELLS = SYSTEM_CELLS
|
||||
prior.SYSTEM_EDGE_CONTRASTS = SYSTEM_EDGE_CONTRASTS
|
||||
prior.COMPARISONS = COMPARISONS
|
||||
prior.ALIGNMENT_COMPARISONS = ALIGNMENT_COMPARISONS
|
||||
prior.RENDER_AUDIT.clear()
|
||||
prior.BOUNDARY_TOKEN_IDS.clear()
|
||||
prior.boundary_token_ids = special_family_token_ids
|
||||
prior.boundary_control_domain = special_family_domain
|
||||
prior.install_control_contract()
|
||||
|
||||
|
||||
def output_path_from_argv() -> Path:
|
||||
try:
|
||||
return Path(sys.argv[sys.argv.index("--output") + 1])
|
||||
except (ValueError, IndexError) as error:
|
||||
raise ValueError("--output is required") from error
|
||||
|
||||
|
||||
def finalize_result(path: Path) -> dict[str, Any]:
|
||||
result = prior.finalize_result(path)
|
||||
result["schema_version"] = 4
|
||||
result["evidence_identity"] = (
|
||||
"X / official BF16 weights and pinned tokenizer; official EOS plus "
|
||||
"BOS and two ordinary single-ID boundary counterfactuals"
|
||||
)
|
||||
boundary = result["boundary"]
|
||||
boundary.update(
|
||||
{
|
||||
"special_token_family_control": True,
|
||||
"complete_pinned_special_inventory": True,
|
||||
"specialness_generalized": False,
|
||||
"causal_boundary": (
|
||||
"All replacement contrasts causally intervene on exactly one "
|
||||
"prior boundary input ID inside this run. BOS versus EOS "
|
||||
"exhausts the pinned tokenizer's two special IDs, while x "
|
||||
"and period are selected ordinary controls; four IDs do not "
|
||||
"identify a universal special-token category effect."
|
||||
),
|
||||
}
|
||||
)
|
||||
boundary["official_serialization_by_boundary"] = {
|
||||
"eos": True,
|
||||
"bos": False,
|
||||
"x": False,
|
||||
"period": False,
|
||||
}
|
||||
|
||||
old = result.pop("history_boundary_token_contract")
|
||||
result["special_token_family_contract"] = {
|
||||
**old,
|
||||
"boundary_levels": list(BOUNDARY_LEVELS),
|
||||
"boundary_token_ids": dict(prior.BOUNDARY_TOKEN_IDS),
|
||||
"boundary_text_controls": BOUNDARY_TEXT,
|
||||
"conditions": FACTORS,
|
||||
"tokenizer_special_inventory": {
|
||||
"all_special_tokens": [
|
||||
"<｜begin▁of▁sentence｜>",
|
||||
"<｜end▁of▁sentence｜>",
|
||||
],
|
||||
"all_special_ids": [100000, 100001],
|
||||
"bos_token_id": 100000,
|
||||
"eos_token_id": 100001,
|
||||
"pad_token_id": 100001,
|
||||
"pad_aliases_eos": True,
|
||||
"inventory_size": 2,
|
||||
},
|
||||
"class_comparison_boundary": (
|
||||
"EOS/BOS are the complete special inventory, but x/period are "
|
||||
"only two chosen ordinary controls; report individual-ID "
|
||||
"contrasts and a descriptive 2-vs-2 family summary, not a "
|
||||
"population-level specialness claim"
|
||||
),
|
||||
}
|
||||
result["inference_contract"]["batch_grouping"] = (
|
||||
"all eight EOS/BOS/x/period variants of one source prompt execute "
|
||||
"in the same 32-row right-padded batch"
|
||||
)
|
||||
|
||||
path.write_text(
|
||||
json.dumps(result, indent=2, ensure_ascii=False) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
install_control_contract()
|
||||
if {"-h", "--help"} & set(sys.argv[1:]):
|
||||
base.main()
|
||||
return
|
||||
output = output_path_from_argv()
|
||||
with open(os.devnull, "w", encoding="utf-8") as sink:
|
||||
with contextlib.redirect_stdout(sink):
|
||||
base.main()
|
||||
result = finalize_result(output)
|
||||
payload = output.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(output),
|
||||
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"bytes": len(payload),
|
||||
"source_prompts": result["inference_contract"][
|
||||
"total_source_prompts"
|
||||
],
|
||||
"prompt_variants": result["inference_contract"][
|
||||
"total_prompt_variants"
|
||||
],
|
||||
"total_routes": result["inference_contract"][
|
||||
"total_routes_all_conditions_all_moe_layers"
|
||||
],
|
||||
"token_inventory": result[
|
||||
"special_token_family_contract"
|
||||
]["tokenizer_special_inventory"],
|
||||
"render_validation": result[
|
||||
"special_token_family_contract"
|
||||
]["render_validation"],
|
||||
},
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user