Enhance model training lite workflow
This commit is contained in:
@@ -11,6 +11,7 @@ skills/model-iteration/scripts/submit_sft.sh
|
||||
skills/model-iteration/scripts/submit_cml_eval.sh
|
||||
skills/model-iteration/scripts/sft_train_job.yaml.tpl
|
||||
skills/model-iteration/assets/config.yaml
|
||||
skills/model-training-lite/scripts/summarize_eval_result.py
|
||||
```
|
||||
|
||||
## 默认基模
|
||||
@@ -28,6 +29,14 @@ git@git.n.xiaomi.com:ai-service/ai-planning.git
|
||||
branch: autoresearch-v1
|
||||
```
|
||||
|
||||
如果用户只提供数据分支、不提供 commit,训练前必须在远端工作区记录:
|
||||
|
||||
```bash
|
||||
git -C ai-planning rev-parse HEAD
|
||||
```
|
||||
|
||||
正式报告使用该 commit 作为本次训练的数据版本。
|
||||
|
||||
训练脚本会扫描:
|
||||
|
||||
```text
|
||||
@@ -49,6 +58,22 @@ ai-planning/data/train_set/*/*.jsonl
|
||||
<workspace>/sft_output/_SUCCESS
|
||||
```
|
||||
|
||||
## 默认 Jupyter 配置
|
||||
|
||||
本地默认配置文件:
|
||||
|
||||
```text
|
||||
skills/model-training-lite/.local/jupyter_defaults.json
|
||||
```
|
||||
|
||||
`.local/` 不提交。配置优先级:
|
||||
|
||||
```text
|
||||
脚本输入 > 环境变量 > .local/jupyter_defaults.json
|
||||
```
|
||||
|
||||
脚本输出只显示认证是否已配置,不回显密码、token 或 cookie。
|
||||
|
||||
## CML 环境
|
||||
|
||||
CloudML CLI 通常位于:
|
||||
@@ -71,6 +96,37 @@ cml config show
|
||||
cml custom_train describe <JOB_ID>
|
||||
```
|
||||
|
||||
## 默认评测
|
||||
|
||||
训练后评测复用:
|
||||
|
||||
```text
|
||||
skills/model-iteration/scripts/submit_cml_eval.sh
|
||||
skills/model-iteration/assets/config.yaml
|
||||
```
|
||||
|
||||
默认 workflow 配置从 `config.yaml` 读取:
|
||||
|
||||
```text
|
||||
cml_eval.workflow_id
|
||||
cml_eval.version
|
||||
```
|
||||
|
||||
评测产物目录:
|
||||
|
||||
```text
|
||||
/mnt/xiaoai-zk-model-train-tj5/workflow5/workflow<EVAL_RUNDIC>/metric_diff/
|
||||
```
|
||||
|
||||
轻量摘要脚本:
|
||||
|
||||
```bash
|
||||
python scripts/summarize_eval_result.py \
|
||||
--workflow-root /mnt/xiaoai-zk-model-train-tj5/workflow5 \
|
||||
--run-dic "$EVAL_RUNDIC" \
|
||||
--output results/eval_summary_${EVAL_RUNDIC}.md
|
||||
```
|
||||
|
||||
## 常见坑
|
||||
|
||||
- Jupyter POST 请求缺少 `_xsrf`:从 cookie 取 `_xsrf`,请求头带 `X-XSRFToken`。
|
||||
@@ -78,3 +134,5 @@ cml custom_train describe <JOB_ID>
|
||||
- `which cml` 为空:先 `source ~/.cloudml-cli/.profile`。
|
||||
- `resolve_run_ids.sh` 没有 `RUNDIC`:只使用 `SFT_RUNDIC` / `EVAL_RUNDIC`。
|
||||
- git clone 看似卡住:先检查目标目录是否已经完整、是否存在残留进程,不要重复提交训练。
|
||||
- 只给 branch 不给 commit:可训练,但必须回填实际 commit,否则结果不可复现。
|
||||
- 训练和评测不要放在同一个长 bash 里串到底;SFT `_SUCCESS` 落盘后再单独提交评测。
|
||||
|
||||
Reference in New Issue
Block a user