Add model iteration skill

skills/model-iteration/README.md

@@ -0,0 +1,66 @@
+# Skill: model-iteration
+
+本目录是小爱中控模型迭代 skill，用于评测、错误分析、数据增强、SFT 训练和 CML workflow 追踪。
+
+## 目录结构
+
+```
+model-iteration/
+├── config.yaml                 # 评估阈值 + CML 工作流配置
+├── knowledge/                  # 分流知识库
+│   ├── navigation_routing_rules.md   # 地图导航 Agent/ComplexTask 分流
+│   ├── travel_routing_rules.md       # 旅游 Agent/ComplexTask 分流
+│   └── multi_command_rules.md        # 多指令 vs ComplexTask 判定
+├── scripts/
+│   ├── prepare_and_train_sft.py      # 数据组装 + 本地训练
+│   ├── submit_sft_via_cml.sh         # CML 云端训练提交（R29 起统一用此）
+│   └── sft_train_job.yaml.tpl        # CML 训练 job YAML 模板
+├── SKILL.md                    # 完整迭代协议（触发规则、Step 0-6、数据增强/清洗规范）
+└── README.md                   # 本文件
+```
+
+## 快速上手
+
+### 1. 评测当前模型
+
+```bash
+cml workflow run \
+  --workflow_id f-20260408161444-wu3pz --version v28 \
+  --global_inputs runDic=<N> \
+  --global_inputs model_path_new=<新模型路径> \
+  --global_inputs model_path_old=<基线模型路径>
+```
+
+### 2. SFT 训练（CML 云端）
+
+```bash
+AUTORESEARCH_ROOT=/mnt/wangsenhao/autoresearch-zk \
+  ./scripts/submit_sft_via_cml.sh <RUNDIC> [PREV_RUNDIC]
+```
+
+训练完成后自动起评测 workflow。
+
+### 3. SFT 训练（本地）
+
+```bash
+AUTORESEARCH_ROOT=/mnt/wangsenhao/autoresearch-zk \
+  python scripts/prepare_and_train_sft.py prepare --output_dir ./sft_data
+
+AUTORESEARCH_ROOT=/mnt/wangsenhao/autoresearch-zk \
+  python scripts/prepare_and_train_sft.py train \
+  --data_dir ./sft_data \
+  --model_path /mnt/wangsenhao/verl_zk/Qwen3-4B-Instruct-2507 \
+  --model_type qwen3 \
+  --train_output ./sft_output \
+  --epochs 3 --lr 1e-5
+```
+
+## 关键规则
+
+- **达标标准**：需求集合 ≥ 95%，大盘车载降幅 ≤ 0.3%，specific test 降幅 ≤ 1%
+- **数据清洗**：case 驱动、禁止批量、必须备份、只改 output 不动 instruction
+- **训练环境**：FSDP2 + torch 2.6 需要 accelerate==1.7.0，不要改训练代码去适配
+
+## 完整协议
+
+详见 [SKILL.md](SKILL.md)

skills/model-iteration/config.yaml

@@ -0,0 +1,13 @@
+# 小爱中控 autoresearch 迭代框架配置
+
+# 评估阈值
+thresholds:
+  overall_max_drop: 0.003
+  specific_max_drop: 0.01
+  requirement_pass_rate: 0.95
+
+# CML 评测工作流配置
+cml_eval:
+  workflow_id: f-20260408161444-wu3pz
+  version: v28
+

skills/model-iteration/knowledge/multi_command_rules.md

@@ -0,0 +1,21 @@
+# 多指令 vs ComplexTask 分流规则
+
+## 核心判定
+
+| 场景 | 输出 | 示例 |
+|------|------|------|
+| 多个独立命令拼在一句话里 | 多个 `Agent(...)` | "导航到大姚县打开微信播放音乐" → 3个Agent |
+| 单个任务涉及多步骤/多目的地 | `ComplexTask(tag="...")` | "先去加油站再去机场" → 1个ComplexTask |
+
+## 判定要点
+
+- **多指令（Multi-command）≠ ComplexTask**：用户在一句话中说了多个独立的命令（导航+播放+打电话+控制），每个命令应该独立解析为各自的 Agent，不是一个 ComplexTask
+- **ComplexTask 是单任务多步**：只有当一个任务本身需要多步规划时才是 ComplexTask（如多目的地导航、带筛选条件的旅游规划）
+- **"然后"不一定是 ComplexTask 信号**：如果"然后"连接的是不同领域的独立命令（导航+控制+播放），那是多指令；只有"然后"连接同一任务的多个步骤（先去A再去B）才是 ComplexTask
+
+## 识别方法
+
+- output 中有多个 `Agent(...)` 调用 → 多指令，保持不变
+- output 中只有一个调用但 query 含多目的地 → 可能应该是 ComplexTask
+- query 中"然后/再/接着"连接不同 tag 的任务 → 多指令
+- query 中"然后/再/接着"连接同 tag 的多个地点 → ComplexTask

skills/model-iteration/knowledge/navigation_routing_rules.md

@@ -0,0 +1,21 @@
+# 地图导航 Agent / ComplexTask 分流规则
+
+## 核心判定逻辑
+
+| 场景 | 输出 | 示例 |
+|------|------|------|
+| 单POI + 任意数量形容词 | `Agent(tag="地图导航")` | 导航去最近的加油站、去附近最便宜的停车场 |
+| 多POI（多个目的地） | `ComplexTask(tag="地图导航")` | 先去加油站再去机场接人 |
+| 单POI + 一句话描述当前状态 | `ComplexTask(tag="地图导航")` | 加完油再去机场（"加完油"描述当前状态） |
+
+## 判定要点
+
+- **形容词不影响分流**：不管加多少形容词修饰（最近的、便宜的、大的），只要是单POI就是 Agent
+- **多目的地 = Complex**：只要 query 中出现多个地点/动作序列，就是 ComplexTask
+- **状态描述 = Complex**：单POI 但前面带了一句描述当前状态的话（如"加完油"、"吃完饭"），说明用户在做多步规划，属于 ComplexTask
+
+## 应用场景
+
+- 数据增强时：按此规则标注 ground_truth
+- Badcase 分析时：以此为标准区分"分流错误"和"语义错误"
+- Reward 函数：分流违反此规则的应被惩罚

skills/model-iteration/knowledge/travel_routing_rules.md

@@ -0,0 +1,26 @@
+# 旅游 Agent / ComplexTask 分流规则
+
+## 核心判定逻辑
+
+| 场景 | 输出 | 示例 |
+|------|------|------|
+| 规划路线，无筛选条件 | `Agent(tag="旅游出行")` | 规划一个从北京到上海的路线 |
+| 规划路线，有筛选条件 | `ComplexTask(tag="旅游出行")` | 规划一个从北京到上海的路线，人少风景好、三天两夜、3000预算 |
+
+## 判定要点
+
+- **无条件纯路线规划 = Agent**：只要求从A到B的路线，没有额外约束
+- **带筛选/约束条件 = Complex**：路线规划附带了时间、预算、偏好、天数等任何筛选条件，说明需要多维度规划，属于 ComplexTask
+
+## 筛选条件举例
+
+- 时间约束：三天两夜、五一假期、周末
+- 预算约束：3000预算、经济型
+- 偏好约束：人少、风景好、适合亲子、有美食
+- 交通约束：自驾、高铁优先
+
+## 应用场景
+
+- 数据增强时：按此规则标注 ground_truth
+- Badcase 分析时：以此为标准区分"分流错误"和"语义错误"
+- Reward 函数：分流违反此规则的应被惩罚

skills/model-iteration/scripts/prepare_and_train_sft.py

@@ -0,0 +1,317 @@
+#!/usr/bin/env python3
+"""
+从 ai-planning 组装 SFT 训练/验证集，并启动 zk_trainer SFT 训练。
+
+用法:
+  # 仅组装数据
+  AUTORESEARCH_ROOT=/mnt/wangsenhao/autoresearch-zk \
+    python prepare_and_train_sft.py prepare --output_dir /mnt/wangsenhao/autoresearch-zk/sft_data
+
+  # 启动训练（需先 prepare）
+  AUTORESEARCH_ROOT=/mnt/wangsenhao/autoresearch-zk \
+  python prepare_and_train_sft.py train \
+    --data_dir /mnt/wangsenhao/autoresearch-zk/sft_data \
+    --model_path /mnt/wangsenhao/verl_zk/Qwen3-4B-Instruct-2507 \
+    --model_type qwen3 \
+    --train_output /mnt/wangsenhao/autoresearch-zk/sft_output
+"""
+
+import argparse
+import json
+import logging
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
+log = logging.getLogger(__name__)
+
+
+def _path_from_env(name: str, fallback: Path) -> Path:
+    return Path(os.environ.get(name, str(fallback))).expanduser().resolve()
+
+
+AUTORESEARCH_ROOT = _path_from_env("AUTORESEARCH_ROOT", Path("/mnt/wangsenhao/autoresearch-zk"))
+AI_PLANNING = _path_from_env("AI_PLANNING_DIR", AUTORESEARCH_ROOT / "ai-planning")
+ZK_TRAINER_DIR = _path_from_env("ZK_TRAINER_DIR", AUTORESEARCH_ROOT / "zk_trainer")
+TRAIN_SET_DIR = AI_PLANNING / "data" / "train_set"
+DATA_TRAIN_DIR = AI_PLANNING / "data_train"
+GENERATE_TRAIN_SCRIPT = DATA_TRAIN_DIR / "generate_train.py"
+
+
+# ──────────────────────────────────────────────
+# 1. 数据组装
+# ──────────────────────────────────────────────
+
+def regen_jsonl_from_csv():
+    """调用 data_train/generate_train.py 从 CSV 重新生成 all_train.jsonl。"""
+    if not GENERATE_TRAIN_SCRIPT.exists():
+        log.error(f"generate_train.py 不存在: {GENERATE_TRAIN_SCRIPT}")
+        return False
+    log.info("从 CSV 重新生成 JSONL ...")
+    result = subprocess.run(
+        [sys.executable, str(GENERATE_TRAIN_SCRIPT)],
+        cwd=str(DATA_TRAIN_DIR),
+    )
+    if result.returncode != 0:
+        log.error("generate_train.py 执行失败")
+        return False
+    log.info("CSV → JSONL 完成")
+    return True
+
+
+def scan_train_set():
+    """扫描 train_set/ 下所有子目录，返回 (train_files, valid_files)。"""
+    train_files, valid_files = [], []
+    if not TRAIN_SET_DIR.exists():
+        log.error(f"train_set 目录不存在: {TRAIN_SET_DIR}")
+        return train_files, valid_files
+
+    for subdir in sorted(TRAIN_SET_DIR.iterdir()):
+        if not subdir.is_dir() or subdir.name.startswith("."):
+            continue
+        for f in sorted(subdir.iterdir()):
+            if not f.is_file() or f.suffix != ".jsonl":
+                continue
+            if "_valid" in f.name:
+                valid_files.append(f)
+            else:
+                train_files.append(f)
+    return train_files, valid_files
+
+
+def count_lines(path):
+    with open(path, "r", encoding="utf-8") as f:
+        return sum(1 for _ in f)
+
+
+def prepare_data(output_dir: str, regen: bool = False, dataset_name: str = "zk_sft_new_structure"):
+    """
+    组装训练集和验证集到 output_dir，生成 zk_trainer 所需的目录结构:
+      output_dir/
+        train/<dataset_name>/merged_train.jsonl
+        validation/<dataset_name>/part-0.jsonl
+    """
+    output = Path(output_dir).resolve()
+    train_dir = output / "train" / dataset_name
+    valid_dir = output / "validation" / dataset_name
+
+    if regen:
+        regen_jsonl_from_csv()
+
+    train_files, valid_files = scan_train_set()
+    if not train_files:
+        log.error("未找到训练文件")
+        return None
+
+    log.info(f"训练文件 ({len(train_files)}):")
+    for f in train_files:
+        log.info(f"  {f.relative_to(AI_PLANNING)}  ({count_lines(f)} 条)")
+    log.info(f"验证文件 ({len(valid_files)}):")
+    for f in valid_files:
+        log.info(f"  {f.relative_to(AI_PLANNING)}  ({count_lines(f)} 条)")
+
+    # 清理旧数据
+    for d in [train_dir, valid_dir]:
+        if d.exists():
+            shutil.rmtree(d)
+        d.mkdir(parents=True)
+
+    # 合并训练文件
+    merged_path = train_dir / "merged_train.jsonl"
+    total_train = 0
+    with open(merged_path, "w", encoding="utf-8") as out:
+        for fp in train_files:
+            with open(fp, "r", encoding="utf-8") as inp:
+                for line in inp:
+                    out.write(line)
+                    total_train += 1
+    log.info(f"合并训练集: {total_train} 条 → {merged_path}")
+
+    # 合并验证文件
+    merged_valid_path = valid_dir / "part-0.jsonl"
+    total_valid = 0
+    with open(merged_valid_path, "w", encoding="utf-8") as out:
+        for fp in valid_files:
+            with open(fp, "r", encoding="utf-8") as inp:
+                for line in inp:
+                    out.write(line)
+                    total_valid += 1
+    log.info(f"验证集: {total_valid} 条 → {merged_valid_path}")
+
+    log.info("=" * 50)
+    log.info(f"数据目录: {output}")
+    log.info(f"  train:      {total_train} 条")
+    log.info(f"  validation: {total_valid} 条")
+
+    return str(output)
+
+
+# ──────────────────────────────────────────────
+# 2. SFT 训练
+# ──────────────────────────────────────────────
+
+def run_sft_training(data_dir: str, model_path: str, model_type: str, train_output: str,
+                     epochs: int = 3, lr: float = 1e-5, max_seq_length: int = 1024,
+                     per_device_batch: int = 1, grad_accum: int = 1):
+    """基于 zk_trainer 的 accelerate launch 方式启动 SFT 训练。"""
+    zk_trainer_dir = str(ZK_TRAINER_DIR)
+    if not os.path.isdir(zk_trainer_dir):
+        log.error(f"zk_trainer 目录不存在: {zk_trainer_dir}")
+        return None
+
+    data_dir = str(Path(data_dir).resolve())
+    train_output = str(Path(train_output).resolve())
+    os.makedirs(train_output, exist_ok=True)
+
+    run_name = f"{model_type}-zk"
+
+    env = os.environ.copy()
+    env["PYTHONPATH"] = f"{zk_trainer_dir}:{env.get('PYTHONPATH', '')}"
+    env["WANDB_DISABLED"] = "true"
+    env["WANDB_MODE"] = "offline"
+    env.setdefault("VOLUME_PREFIX", "/mnt/xiaoai-zk-model-train-tj5")
+
+    # Step 1: 用 get_accelerate_config.py 动态生成 accelerate config
+    num_gpu_result = subprocess.run(
+        ["nvidia-smi", "--query-gpu=gpu_name", "--format=csv,noheader"],
+        capture_output=True, text=True
+    )
+    num_processes = len(num_gpu_result.stdout.strip().split("\n")) if num_gpu_result.returncode == 0 else 1
+    log.info(f"检测到 {num_processes} 张 GPU")
+
+    acc_config_dir = zk_trainer_dir
+    gen_config_cmd = [
+        sys.executable, f"{zk_trainer_dir}/llm/train/get_accelerate_config.py",
+        "--dt_framework=fsdp",
+        "--fsdp_sharding_strategy=1",
+        "--machine_rank=0",
+        "--num_machines=1",
+        f"--num_processes={num_processes}",
+        f"--output_dir={acc_config_dir}",
+        "--main_process_ip=127.0.0.1",
+        "--main_process_port=56390",
+        "--mixed_precision=bf16",
+        f"--model_type={model_type}",
+        "--ds_stage=3",
+        "--fsdp_version=2",
+        "--fsdp_wrap_cls=",
+    ]
+    log.info("生成 accelerate config ...")
+    result = subprocess.run(gen_config_cmd, env=env, cwd=zk_trainer_dir)
+    if result.returncode != 0:
+        log.error("生成 accelerate config 失败")
+        return None
+
+    acc_config = os.path.join(acc_config_dir, "accelerate_config_0.yaml")
+    if not os.path.exists(acc_config):
+        log.error(f"accelerate config 未生成: {acc_config}")
+        return None
+    log.info(f"accelerate config: {acc_config}")
+
+    # Step 2: 启动训练
+    train_cmd = [
+        "accelerate", "launch",
+        "--main_process_port", "12345",
+        "--config_file", acc_config,
+        f"{zk_trainer_dir}/llm/train/trainer_general.py",
+        "--model_name_or_path", model_path,
+        "--model_type", model_type,
+        "--use_auto_class", "true",
+        "--output_dir", train_output,
+        "--dataset_type", "zk_sft",
+        "--train_data_dir", os.path.join(data_dir, "train"),
+        "--valid_data_dir", os.path.join(data_dir, "validation"),
+        "--do_eval",
+        "--fp16_full_eval",
+        "--per_device_eval_batch_size", "1",
+        "--include_for_metrics", "inputs",
+        "--batch_eval_metrics",
+        "--concat_samples", "False",
+        "--mix_at_eval", "False",
+        "--num_train_epochs", str(epochs),
+        "--learning_rate", str(lr),
+        "--gradient_accumulation_steps", str(grad_accum),
+        "--ignore_data_skip", "False",
+        "--resume_train_if_ckpt_exists", "false",
+        "--save_strategy", "no",
+        "--logging_steps", "5",
+        "--save_total_limit", "0",
+        "--max_seq_length", str(max_seq_length),
+        "--adam_beta1", "0.9",
+        "--adam_beta2", "0.95",
+        "--adam_epsilon", "1e-9",
+        "--warmup_ratio", "0.1",
+        "--lr_scheduler_type", "cosine",
+        "--use_shuffle", "true",
+        "--batch_eval_metrics",
+        "--run_name", run_name,
+        "--report_to", "none",
+        "--bf16",
+    ]
+
+    log.info("=" * 50)
+    log.info("启动 SFT 训练")
+    log.info(f"  模型:   {model_path}")
+    log.info(f"  数据:   {data_dir}")
+    log.info(f"  输出:   {train_output}")
+    log.info(f"  GPU:    {num_processes}")
+    log.info(f"  epochs: {epochs}, lr: {lr}, max_seq: {max_seq_length}")
+
+    result = subprocess.run(train_cmd, env=env, cwd=zk_trainer_dir)
+
+    if result.returncode != 0:
+        log.error(f"训练失败，返回码: {result.returncode}")
+        return None
+
+    log.info(f"训练完成，输出: {train_output}")
+    return train_output
+
+
+# ──────────────────────────────────────────────
+# CLI
+# ──────────────────────────────────────────────
+
+def main():
+    parser = argparse.ArgumentParser(description="ai-planning SFT 数据组装 & 训练")
+    subparsers = parser.add_subparsers(dest="command")
+
+    # prepare
+    p_prepare = subparsers.add_parser("prepare", help="仅组装数据")
+    p_prepare.add_argument("--output_dir", required=True, help="数据输出目录")
+    p_prepare.add_argument("--regen_from_csv", action="store_true", help="从 CSV 重新生成 JSONL")
+    p_prepare.add_argument("--dataset_name", default="zk_sft_new_structure")
+
+    # train
+    p_train = subparsers.add_parser("train", help="启动训练（需先 prepare）")
+    p_train.add_argument("--data_dir", required=True, help="已组装的数据目录（prepare 的 output_dir）")
+    p_train.add_argument("--model_path", required=True, help="基模路径")
+    p_train.add_argument("--model_type", default="qwen3", help="模型类型")
+    p_train.add_argument("--train_output", required=True, help="训练输出目录")
+    p_train.add_argument("--epochs", type=int, default=3)
+    p_train.add_argument("--lr", type=float, default=1e-5)
+    p_train.add_argument("--max_seq_length", type=int, default=1024)
+
+    args = parser.parse_args()
+
+    if args.command == "prepare":
+        prepare_data(args.output_dir, regen=args.regen_from_csv, dataset_name=args.dataset_name)
+
+    elif args.command == "train":
+        run_sft_training(
+            data_dir=args.data_dir,
+            model_path=args.model_path,
+            model_type=args.model_type,
+            train_output=args.train_output,
+            epochs=args.epochs,
+            lr=args.lr,
+            max_seq_length=args.max_seq_length,
+        )
+    else:
+        parser.print_help()
+
+
+if __name__ == "__main__":
+    main()

skills/model-iteration/scripts/sft_train_job.yaml.tpl

@@ -0,0 +1,117 @@
+jobName: "sft-train-r{RUNDIC}"
+description: "AutoResearch R{RUNDIC} SFT training (40k+ samples, qwen3-4B base, 3 epochs FSDP2)"
+accessType: PUBLIC
+
+imageConfig:
+  imageUrl: micr.cloud.mioffice.cn/vllm-image/ai-arch-llm-prod:vllm-v0.12.0-f098b188
+  imageCommand: |-
+    set -ex
+    cd {AUTORESEARCH_ROOT}
+
+    # 0a. 验环境 + 装缺的包（vllm 镜像 py3.12 + torch 已含）
+    python3 --version
+    python3 -c "import torch; print(f'torch={torch.__version__}')"
+    pip install -q --no-deps "accelerate==1.7.0" 2>&1 | tail -3
+    pip install -q --ignore-installed blinker 2>&1 | tail -2
+    pip install -q peft 2>&1 | tail -3
+    pip install -q wandb bitsandbytes 2>&1 | tail -3
+    pip install -q luigi mlflow scikit-learn openpyxl pyyaml sentencepiece tiktoken protobuf pynvml datasets 2>&1 | tail -3
+    pip install -q "transformers>=4.45" 2>&1 | tail -3
+    python3 -c 'import torch, accelerate, transformers, peft; print(torch.__version__, accelerate.__version__, transformers.__version__, peft.__version__)'
+
+    # 0c. 把 HF cache 重定向到容器本地大盘（默认在 ~/.cache 可能是 juicefs，mmap 易 SIGBUS）
+    export HF_HOME=/tmp/hf_cache
+    export HF_DATASETS_CACHE=/tmp/hf_cache/datasets
+    export TRANSFORMERS_CACHE=/tmp/hf_cache/transformers
+    mkdir -p /tmp/hf_cache/datasets /tmp/hf_cache/transformers
+    df -h /tmp /dev/shm 2>/dev/null || true
+
+    # 强制 datasets 加载进内存而不是 arrow mmap（避免 /dev/shm 溢出 SIGBUS）
+    export HF_DATASETS_IN_MEMORY_MAX_SIZE=20000000000
+    export HF_DATASETS_NUM_PROC=1
+    export TOKENIZERS_PARALLELISM=false
+    export OMP_NUM_THREADS=1
+    export MKL_NUM_THREADS=1
+    # 关闭 NCCL 用 shm 通信（改 socket 通道）
+    export NCCL_SHM_DISABLE=1
+    export NCCL_P2P_DISABLE=0
+    export NCCL_DEBUG=WARN
+
+    # 0b. 把上一轮产出搬走（cml job 内幂等，本地搬过就跳过）
+    if [ -d sft_output ] && [ ! -d sft_output_r{PREV_RUNDIC} ]; then
+      mv sft_output sft_output_r{PREV_RUNDIC}
+    fi
+    rm -rf sft_output
+
+    # 1. 组装数据
+    python3 {AUTORESEARCH_ROOT}/prepare_and_train_sft.py prepare \
+      --output_dir {AUTORESEARCH_ROOT}/sft_data
+
+    # 2. 启动训练（绝对路径，避免相对路径在容器内找不到 zk_trainer）
+    python3 {AUTORESEARCH_ROOT}/prepare_and_train_sft.py train \
+      --data_dir {AUTORESEARCH_ROOT}/sft_data \
+      --model_path /mnt/wangsenhao/verl_zk/Qwen3-4B-Instruct-2507 \
+      --model_type qwen3 \
+      --train_output {AUTORESEARCH_ROOT}/sft_output \
+      --epochs 3 --lr 1e-5
+
+    # 3. 训练成功标记（被 watcher 检测）
+    [ -f {AUTORESEARCH_ROOT}/sft_output/config.json ] && \
+      touch {AUTORESEARCH_ROOT}/sft_output/_SUCCESS
+
+# 挂载 wangsenhao + xiaoai-zk-model-train-tj5 + verl_zk 所在卷
+juiceFsMountConfigs:
+  - volume: wangsenhao
+    juiceFsCluster: tj5-common
+    subPath: /
+    mountPath: /mnt/wangsenhao
+    readOnly: false
+  - volume: xiaoai-zk-model-train-tj5
+    juiceFsCluster: tj5-common
+    subPath: /
+    mountPath: /mnt/xiaoai-zk-model-train-tj5
+    readOnly: false
+
+envConfigs:
+  - key: PYTHONPATH
+    value: {AUTORESEARCH_ROOT}/zk_trainer
+  - key: WANDB_DISABLED
+    value: "true"
+  - key: WANDB_MODE
+    value: offline
+  - key: VOLUME_PREFIX
+    value: /mnt/xiaoai-zk-model-train-tj5
+  - key: HF_HOME
+    value: /mnt/wangsenhao/.hf_cache
+
+# h20-96g 8 卡 FSDP2
+queueId: "6052"
+priority: 5
+preemptible: false
+framework: pytorch
+resourceConfigs:
+  - nodeRole: worker
+    nodeNumber: 1
+    perNodeResourceSpec:
+      resourcePriority: GUARANTEED
+      resourceName: cloudml.ng2h20-8-8.20-199
+      resourceNumber: 8
+
+# 故障自动重试（节点级失败）
+retryConfig:
+  enableRetry: true
+  maxRetryTimes: 2
+  policySets:
+    - NodeFailure
+
+# 失败/完成飞书告警
+alertConfig:
+  enableAlert: true
+  alertItems:
+    - alertConditions:
+        - FAILED
+        - SUCCEED
+      alertLevel: P2
+      alertReceivers:
+        persons:
+          - wangsenhao

skills/model-iteration/scripts/submit_sft_via_cml.sh

@@ -0,0 +1,104 @@
+#!/bin/bash
+# 用 cml custom_train submit 提交 SFT 训练，训练完成后自动起 cml workflow run 评测
+# 用法: ./submit_sft_via_cml.sh <RUNDIC>
+# 例:   ./submit_sft_via_cml.sh 17756
+
+set -euo pipefail
+
+RUNDIC=${1:?usage: $0 <RUNDIC>}
+PREV_RUNDIC=${2:-$((RUNDIC-1))}
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT=${AUTORESEARCH_ROOT:-/mnt/wangsenhao/autoresearch-zk}
+MODEL_OLD=${MODEL_OLD:-/mnt/wangsenhao/verl_zk/qwen4b_cispo_wokl_add_bvt_2/global_step_5/actor/huggingface}
+TPL=${SFT_TRAIN_JOB_TEMPLATE:-$SCRIPT_DIR/sft_train_job.yaml.tpl}
+YAML=/tmp/sft_train_job_r${RUNDIC}.yaml
+
+if [ -f ~/.cloudml-cli/.profile ]; then
+  source ~/.cloudml-cli/.profile
+else
+  echo "[cml-sft] ❌ 未找到 ~/.cloudml-cli/.profile，请先安装并初始化 cml" >&2
+  exit 1
+fi
+
+# 1. 渲染 yaml 模板
+sed \
+  -e "s|{RUNDIC}|${RUNDIC}|g" \
+  -e "s|{PREV_RUNDIC}|${PREV_RUNDIC}|g" \
+  -e "s|{AUTORESEARCH_ROOT}|${ROOT}|g" \
+  "$TPL" > "$YAML"
+echo "[cml-sft] yaml: $YAML"
+
+# 2. 提交训练任务
+SUBMIT_OUT=$(cml custom_train submit --filename "$YAML" 2>&1)
+echo "$SUBMIT_OUT"
+JOB_ID=$(echo "$SUBMIT_OUT" | grep -oE 't-[0-9]+-[a-z0-9]+' | head -1)
+if [ -z "$JOB_ID" ]; then
+  echo "[cml-sft] ❌ 提交失败" >&2
+  exit 1
+fi
+echo "[cml-sft] ✅ JobID: $JOB_ID"
+
+# 3. 后台 watcher：等训练成功 → 自动起评测
+WATCHER_LOG=/tmp/r${RUNDIC}_logs/cml_watcher.log
+mkdir -p "$(dirname "$WATCHER_LOG")"
+
+nohup bash -c '
+JOB_ID='"$JOB_ID"'
+RUNDIC='"$RUNDIC"'
+LOG='"$WATCHER_LOG"'
+ROOT='"$ROOT"'
+MODEL_NEW="$ROOT/sft_output"
+MODEL_OLD='"$MODEL_OLD"'
+EVAL_WORKFLOW_ID=f-20260408161444-wu3pz
+EVAL_VERSION=v28
+
+source ~/.cloudml-cli/.profile
+
+echo "[$(date)] 等待 cml job $JOB_ID 完成..." >> "$LOG"
+while true; do
+  STATE=$(cml custom_train describe "$JOB_ID" 2>/dev/null | grep -o "\"state\": \"[a-z]*\"" | head -1 | sed "s/.*: \"//;s/\"//")
+  case "$STATE" in
+    succeed)
+      echo "[$(date)] 训练成功" >> "$LOG"
+      break ;;
+    failed|killed)
+      echo "[$(date)] ❌ 训练 $STATE" >> "$LOG"
+      exit 1 ;;
+    *)
+      sleep 60 ;;
+  esac
+done
+
+# 验证产出
+[ ! -f "$MODEL_NEW/_SUCCESS" ] && [ ! -f "$MODEL_NEW/config.json" ] && {
+  echo "[$(date)] ❌ 产出缺失" >> "$LOG"; exit 1
+}
+
+# 起评测
+echo "[$(date)] 启动 CML 评测 runDic=$RUNDIC" >> "$LOG"
+cml workflow run \
+  --workflow_id $EVAL_WORKFLOW_ID --version $EVAL_VERSION \
+  --global_inputs runDic=$RUNDIC \
+  --global_inputs model_path_new=$MODEL_NEW \
+  --global_inputs model_path_old=$MODEL_OLD >> "$LOG" 2>&1
+echo "[$(date)] cml workflow run 提交完成" >> "$LOG"
+' > /tmp/r${RUNDIC}_logs/watcher_runner.log 2>&1 &
+
+WATCHER_PID=$!
+echo "[cml-sft] watcher PID: $WATCHER_PID（log: $WATCHER_LOG）"
+
+# 4. 立即输出可查看命令
+cat <<EOF
+
+[cml-sft] 任务提交完成，关键命令：
+  查看任务状态:  cml custom_train describe $JOB_ID
+  查看实时日志:  cml custom_train logs $JOB_ID --follow
+  停止任务:      cml custom_train kill $JOB_ID
+  watcher log:   tail -f $WATCHER_LOG
+
+训练完成后，evaluation workflow 会自动启动；评测产物在
+  /mnt/xiaoai-zk-model-train-tj5/workflow5/workflow${RUNDIC}/
+
+JobID: $JOB_ID
+RUNDIC: $RUNDIC
+EOF