Improve online mining record conversion
This commit is contained in:
@@ -16,23 +16,48 @@
|
||||
| `scripts/elk_profile_index.py` | 采样索引 schema 和关键字段,帮助 Agent 判断字段可用性。 |
|
||||
| `scripts/elk_search_cases.py` | 按 query/domain/prompt/model output/candidate domain 等条件搜索候选 case。 |
|
||||
| `scripts/elk_fetch_by_request_ids.py` | 批量按 request id 拉取 main/pre_processing 原始摘要。 |
|
||||
| `scripts/elk_join_request_logs.py` | 把 `arch-flat-nlp-log-f-*` 和 `pre-processing-info*` 按 request id 合并。 |
|
||||
| `scripts/elk_join_request_logs.py` | 把 `arch-flat-nlp-log-f-*` 和 `pre-processing*` 按 request id 合并。 |
|
||||
| `scripts/build_dataset_draft.py` | 把 review 后的线上 case 转成 product-data dataset draft text。 |
|
||||
| `scripts/build_online_records.py` | 把 rid/候选 case 直接转成 canonical records,并可导出 records.csv、training.jsonl、eval_planning.csv。 |
|
||||
|
||||
## 数据源
|
||||
|
||||
- `main`:`arch-flat-nlp-log-f-*`
|
||||
- `pre_processing`:`pre-processing-info*`
|
||||
- `pre_processing`:`pre-processing*`
|
||||
|
||||
字段说明见 `knowledge/数据源字段说明.md`。
|
||||
|
||||
## 后处理
|
||||
|
||||
后处理复用 `skills/product-data/scripts/`:
|
||||
优先使用 `scripts/build_online_records.py` 一步完成线上候选到标准数据:
|
||||
|
||||
```json
|
||||
{
|
||||
"request_ids": ["<rid>"],
|
||||
"date": "YYYYMMDD",
|
||||
"dataset_label": "线上挖掘样本",
|
||||
"target": "Agent(tag=\"xxx\")",
|
||||
"complex": false,
|
||||
"output_dir": "output",
|
||||
"export_records": true,
|
||||
"export_training": false,
|
||||
"export_eval": false
|
||||
}
|
||||
```
|
||||
|
||||
它会补齐:
|
||||
|
||||
- `request_id`:主表 `request_id` / 前处理表 `requestId`
|
||||
- `timestamp`:优先主表 `timestamp`
|
||||
- `query`:优先前处理 query,缺失时用主表 query
|
||||
- `prev_session`:主表同 `session_id` 且早于当前 timestamp 的最多 10 轮
|
||||
- `target`:优先用户输入,否则用前处理 `code/planningOriginalResult`,再用主表仲裁信息兜底
|
||||
- `complex`:优先用户输入,默认 false
|
||||
|
||||
如果需要拆开调试,后处理仍可复用 `skills/product-data/scripts/`:
|
||||
|
||||
1. `normalize_dataset_draft.py`
|
||||
2. `validate_dataset_records.py`
|
||||
3. `export_dataset_records.py`
|
||||
4. 可选 `export_training_jsonl.py`
|
||||
5. 可选 `export_planning_eval_csv.py`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user