Unify skills under repository root
This commit is contained in:
@@ -63,5 +63,4 @@ include = ["src*", "backend*"]
|
||||
[tool.setuptools.package-data]
|
||||
src = [
|
||||
"reference_data/*.json",
|
||||
"skills/bundled/*/SKILL.md",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
# Local override only — committed profiles.json holds shared team creds
|
||||
.env
|
||||
.env.*
|
||||
|
||||
# Ad-hoc probes / local experiments
|
||||
_probe_*.py
|
||||
scratch_*.py
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.egg-info/
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# IDE / OS
|
||||
.idea/
|
||||
.vscode/
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
@@ -0,0 +1,140 @@
|
||||
# ELK 索引目录(schema reference)
|
||||
|
||||
> 每张索引一节:**集群 / 账号 / 关键字段 / JSON 字符串字段的内部结构**。纯 schema 参考。
|
||||
>
|
||||
> 业务侧"为什么这么查 / 怎么排查问题"在 [`business/`](business/) 目录:
|
||||
>
|
||||
> - [`business/keyword-free.md`](business/keyword-free.md) — 免唤醒判决(拒识 / kwfree 两张表的视角差异)
|
||||
> - [`business/intent-arbitrator.md`](business/intent-arbitrator.md) — 中控仲裁
|
||||
> - [`business/micar-onetrack.md`](business/micar-onetrack.md) — 小米汽车 OneTrack 端侧埋点(端到端可用性 / 全离线日志 / 离线NLP结果)
|
||||
|
||||
## 集群 & 账号
|
||||
|
||||
⚠️ **不止一个 ES 集群**。每个 profile 自带 `host` 字段指向所在集群。账号密码(团队共用)已 commit 在仓库的 `profiles.json` 里,clone 即可用。
|
||||
|
||||
| Profile Key | ES 集群 | 账号 | 对应 Kibana 前端 |
|
||||
| ----------- | ------ | --- | ---------------- |
|
||||
| `default` | `akaiservice.api.es.srv:80` | `ai_service_kibana` | `aiservice.ak.kibana.cloud.mioffice.cn` / `akelk.pt.ai.srv` |
|
||||
| `reject` | `akaiservice.api.es.srv:80` | `ai_service_kibana` | `akaiservice.kibana.pt.xiaomi.com` |
|
||||
| `micar` | `c3log.api.es.srv:80` | `xiaoai_micar_kibana` | `c3log.kibana.pt.xiaomi.com` |
|
||||
|
||||
`default` / `reject` 共用一个集群和账号;`micar` 是**完全独立的 c3log 集群**,跨集群账号互相不通(403)。
|
||||
|
||||
## 索引清单
|
||||
|
||||
| preset | 索引 pattern | profile | request id 字段 | 时间字段 | 内置过滤 | 主要业务 |
|
||||
| ------ | ----------- | ------- | -------------- | ------- | ------- | -------- |
|
||||
| `main` | `arch-flat-nlp-log-f-*` | default | `request_id` | `timestamp` | — | 主 NLP 日志 / 中控仲裁 |
|
||||
| `reject` | `aiservice_duplex_rejection_lcs_log*` | reject | `requestId` | `time` | — | 后置拒识 |
|
||||
| `kwfree` | `nlp_post_processing_lcs-*` | default | `requestId` | `timestamp` | `moduleName=keyword-free-log` | 后处理免唤醒 |
|
||||
| `micar` | `onetrack_xiaoai_micar*` | micar | `request_id` | `timestamp` | — | 小米汽车 OneTrack 端侧埋点 |
|
||||
|
||||
⚠️ **字段命名各表不统一**:snake (`request_id`) vs camel (`requestId`);时间字段 `timestamp` vs `time`。Kibana 显示的字段名不一定等于 ES 字段名。
|
||||
|
||||
---
|
||||
|
||||
## `arch-flat-nlp-log-f-*`(preset=`main`)
|
||||
|
||||
主 NLP 服务端日志,每次请求一条。
|
||||
|
||||
**顶层字段**(节选):
|
||||
`request_id`, `query`, `query_origin`, `domain`, `func`, `intention`, `code`, `latency`,
|
||||
`user_agent`, `session_id`, `app_id`, `app_name`, `device_id`, `mask_did`,
|
||||
`arbitrator_dialog_status`, `offline_arbitrate_domains`,
|
||||
`is_llm`, `is_llm_classify_enable`, `is_llm_classify_success`,
|
||||
`large_model_info`, `large_model_traceid`,
|
||||
`llm_intent`, `llm_intent_model_name`, `llm_intent_model_version`, `llm_content`, `llm_knowledge`, `llm_system_prompt`, `llm_history_size`, `llm_reject_intent`, `llm_access_control`, `llm_description`,
|
||||
`is_continuous_dialog`, `exit_continuous_dialog`, `is_multi_turn`, `is_multi_rewrite`, `multi_rewrite_method`,
|
||||
`reject_type`, `rejection_hint`, `rejection_info`, `is_shumei_reject`, `is_filtered`, `is_user_cancelled`,
|
||||
`requestInfo`, `responseInfo`, `text`, `display_text`, `to_read`, `to_speak`, `tts_speaker`, `tts_vendor`,
|
||||
`car_type`, `car_config`, `car_category`, `vehicle_driving_status`, `vehicle_wakeup_zone`, `is_real_vehicle`, `is_bench_vehicle`, `is_internal`
|
||||
|
||||
**关键嵌套对象**:
|
||||
|
||||
- `intention` (dict) — 意图判决总集,含:
|
||||
- `intention.intent_arbitrator_info` ⭐ — **中控仲裁对象**(顶层同名字段通常 null,**真东西在这里**),子结构详见 [`business/intent-arbitrator.md`](business/intent-arbitrator.md)
|
||||
- `intention.score` / `intention.func` / `intention.domain` / `intention.query`
|
||||
- `intention.dialog_status`, `intention.rewrite_infos`, `intention.domain_judge`, `intention.provider_domains`
|
||||
|
||||
---
|
||||
|
||||
## `aiservice_duplex_rejection_lcs_log*`(preset=`reject`)
|
||||
|
||||
后置拒识模块日志。**同一 requestId 通常多条(多轮)**,按 `time` 倒序取最新。
|
||||
|
||||
**顶层字段**:
|
||||
`requestId`, `time`, `query`, `appId`, `deviceId`, `sessionId`, `env`,
|
||||
`isDuplex`, `isExit`, `isWakeup`, `costInfo`, `exitInfo`, `instructionPriority`,
|
||||
`rejectHint`, `rejectInfo`, `rejectType`, `requestInfo`, `strategy_info`
|
||||
|
||||
**JSON 字符串字段**(脚本自动深度解析):
|
||||
- `rejectInfo` — 完整拒识判决,关键路径:
|
||||
- `rejectInfo.reject_reason` / `rejectInfo.rejection_info` / `rejectInfo.debug_info`
|
||||
- `rejectInfo.strategy_info.reject.<策略名>` — 各子策略
|
||||
- 常用子策略 `KEY_WORD_FREE_RESTRIC`(免唤醒):`process` / `skipped_reason` / `keywordFreeType` / `isCodeMatch` / `getProcessedQuery` / `Function List` —— 业务深入见 [`business/keyword-free.md`](business/keyword-free.md)
|
||||
|
||||
---
|
||||
|
||||
## `nlp_post_processing_lcs-*`(preset=`kwfree`)
|
||||
|
||||
后处理通用日志表。`kwfree` preset 内置 `moduleName=keyword-free-log` 过滤,锁定到免唤醒模块那条。
|
||||
|
||||
**顶层字段**:
|
||||
`requestId`, `timestamp`, `moduleName`, `appId`, `env`, `machine`, `maskUid`, `message`
|
||||
|
||||
**JSON 字符串字段**(脚本自动深度解析):
|
||||
|
||||
- `message` — 免唤醒模块本次处理的入参出参,关键路径:
|
||||
- `message.process` / `message.skipReason`
|
||||
- `message.route.flow` / `route.enableFullMigration` / `route.clawEnabled`
|
||||
- `message.request.query` / `keywordFreeType` / `zone` / `appId` / `maskUid` / `maskDeviceId`
|
||||
- `message.fallback.oldRejectType` / `fallback.oldDomain`
|
||||
- 业务深入见 [`business/keyword-free.md`](business/keyword-free.md)
|
||||
|
||||
---
|
||||
|
||||
## `onetrack_xiaoai_micar*`(preset=`micar`)
|
||||
|
||||
小米汽车小爱端侧 OneTrack 埋点。**c3log 独立集群**。按日 rolling,单日 ~300GB——**强烈建议 `--date YYYYMMDD`**。
|
||||
|
||||
⚠️ **同一 request_id 多条记录**,按 `tip` 区分业务点位。**不同 tip 的负载 schema 不统一**。
|
||||
|
||||
**顶层公共字段**:
|
||||
`request_id`, `tip`, `tip_id`, `tip_name`, `tip_module_name`, `tip_page_id`, `tip_pos`,
|
||||
`event_name` (`execute` / `state` / `start`),
|
||||
`device_id`, `vid`, `instance_id`, `app_id`, `app_package_name`, `pkg`,
|
||||
`path_id`, `clientTime`, `serverTime`, `timestamp`,
|
||||
`network`, `region`, `mfrs`, `model`, `car_type`, `car_config`,
|
||||
`voice_position` (`driver` / `passenger`), `wakeup_origin`,
|
||||
`srv_env`, `app_ver`, `micar_ver`, `os_ver`,
|
||||
`is_internal`, `is_bench_vehicle`, `sender`, `distinct_id`, `plugin_id`
|
||||
|
||||
**主要 tip 与负载位置**:
|
||||
|
||||
| tip | tip_name | 负载在 |
|
||||
| --- | -------- | ------ |
|
||||
| `1024.1.1.1.23177` | 小爱端到端可用性 | 顶层 `key_value`(性能时序) |
|
||||
| `1024.1.2.1.23176` | 小爱全离线日志 | 顶层 `request_infos[]` / `response_infos[]` / `other_infos`(**不在 `key_value`**) |
|
||||
| `1024.1.2.1.34496` | 离线NLP结果 | `key_value.instructions` |
|
||||
| `1024.4.1.1.25631` | ASR识别状态 | `key_value` 为 null,看 `event_name` / `other_infos` |
|
||||
| `1024.1.1.1.33610` | 第三方接口调用 | — |
|
||||
| `1024.1.3.1.23520` / `23521` | 小爱指令开始 / 结束 | — |
|
||||
|
||||
**JSON 字符串字段**(脚本自动深度解析):
|
||||
|
||||
- `key_value`(大多数 tip)
|
||||
- `request_infos[]` / `response_infos[]` / `other_infos[]`(23176)
|
||||
|
||||
各 tip 字段语义、模型相关字段(`nlp_debug_info` / `Arbitrate` / `nlp_model_version` 等)业务深入见 [`business/micar-onetrack.md`](business/micar-onetrack.md)。
|
||||
|
||||
---
|
||||
|
||||
## 新增索引流程
|
||||
|
||||
1. 从 Kibana URL 拿:Kibana index UUID、KQL 字段、filter 里的 `match_phrase` 条件
|
||||
2. 必要时跑临时 `_probe_*.py`(命名已 gitignore):用 `match_phrase` 条件 + `*` 索引跨库搜,从命中文档的 `_index` 反推真实索引名
|
||||
3. 确认 **request id 字段名**(`request_id` / `requestId`)和**时间字段**(`timestamp` / `time`)
|
||||
4. 如 `AuthorizationException(403)`,换 profile
|
||||
5. 字段为 JSON 字符串时(如 `rejectInfo` / `message` / `key_value`)脚本已自动深度解析,把业务路径记到本文档对应索引一节
|
||||
6. 在 `elk_query.py` 的 `PRESETS` 加一条,`extra_filters` 写清楚(如 `moduleName=...`)
|
||||
7. 如果是新业务主题,在 `business/` 下加一份业务深入文档
|
||||
@@ -0,0 +1,37 @@
|
||||
# elk-fetch
|
||||
|
||||
按 request id 查 ELK / Elasticsearch 日志的小工具 + Claude Code Skill。
|
||||
|
||||
**核心能力**
|
||||
- 绕开 Kibana CAS 登录,直接走 ES HTTP API
|
||||
- 多账号 profile(不同业务用不同 ES 账号,支持多 ES 集群)
|
||||
- 多业务 preset(业务语义 → 索引/字段/过滤条件,一键套用)
|
||||
- 深度 JSON 字段自动解析(如 `rejectInfo` / `message` / `key_value` / `response_infos` 里的嵌套 JSON)
|
||||
- 按点号路径(`a.b.0.c`)提取嵌套字段 / 仅输出指定字段
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
git clone git@git.n.xiaomi.com:zhongsiyao/elk-fetch.git
|
||||
cd elk-fetch
|
||||
|
||||
pip install "elasticsearch<8" urllib3
|
||||
|
||||
# 默认查主 NLP 日志
|
||||
python elk_query.py <your_request_id>
|
||||
```
|
||||
|
||||
账号密码(团队共用)已经在仓库里的 `profiles.json` 中,clone 完即可用。想用自己的账号则 `ELK_FETCH_PROFILES=/path/to/profiles.json` 指向别处。
|
||||
|
||||
## 完整用法
|
||||
|
||||
- **怎么选 preset / Kibana URL 转参数 / 命令模板** → [`SKILL.md`](SKILL.md)
|
||||
- **每张表的字段 schema / JSON 字符串内部结构 / 新增索引流程** → [`INDEX_CATALOG.md`](INDEX_CATALOG.md)
|
||||
- **业务深入**(每个主题一份排查 playbook)→ [`business/`](business/):
|
||||
- [`business/keyword-free.md`](business/keyword-free.md) — 免唤醒判决(拒识 / kwfree 两张表三个视角)
|
||||
- [`business/intent-arbitrator.md`](business/intent-arbitrator.md) — 中控仲裁
|
||||
- [`business/micar-onetrack.md`](business/micar-onetrack.md) — 小米汽车 OneTrack 端侧埋点
|
||||
|
||||
## 作为 Claude Code Skill 使用
|
||||
|
||||
把本仓库 clone / 软链到 `~/.claude/skills/elk-fetch/`,Claude Code 会根据 `SKILL.md` 的触发条件(给出 Kibana 链接、提到 requestId / 某类业务日志等)自动调用。
|
||||
@@ -0,0 +1,200 @@
|
||||
---
|
||||
name: elk-fetch
|
||||
description: 从小米内网 ELK 按 request id 拉取日志。支持多账号 profile、多 ES 集群、多业务 preset(主 NLP / 拒识表 / 免唤醒后处理日志 / 小米汽车 OneTrack 端侧埋点)、深度 JSON 字段自动解析。绕开 Kibana CAS 登录,直接走 ES HTTP API。触发条件:用户给出 Kibana 链接、提到 requestId / request_id 查日志、想看某次请求的 NLP 日志 / 中控仲裁日志(intent_arbitrator_info / llm_agent_info / hit_rules / score_domains)/ 拒识 rejectInfo / KEY_WORD_FREE_RESTRIC / 免唤醒判决(keyword-free-log)/ 小爱端到端可用性 / 小爱全离线日志 / OneTrack 埋点 / micar tip。
|
||||
when_to_use: 用户给出 Kibana 链接、request_id/requestId,或要求查询 NLP 日志、中控仲裁日志、拒识日志、免唤醒日志、小米汽车 OneTrack 埋点日志时使用。
|
||||
aliases: elk, elk-query, log-fetch, request-log
|
||||
allowed_tools: python_exec, python_package, read_file
|
||||
---
|
||||
|
||||
# ELK Fetch Skill
|
||||
|
||||
按 request id 查小米内网 ELK。**多个 ES 集群**:`default`/`reject` 走 `akaiservice.api.es.srv:80`,`micar` 走独立的 `c3log.api.es.srv:80`。每个 profile 自带 `host`,脚本会自动选集群。
|
||||
|
||||
账号密码(团队共用)已 commit 进仓库的 `profiles.json`,无需配置。需要换账号时用 `ELK_FETCH_PROFILES=/abs/path/to/profiles.json` 指向自己的文件。
|
||||
|
||||
## 在本数据 Agent 中使用
|
||||
|
||||
本 skill 已安装在项目目录 `skills/elk-fetch/`。在本项目中不要用 `bash` 执行 `python elk_query.py`,也不要用 `pip` 直接安装依赖。
|
||||
|
||||
实际查询必须使用 `python_exec`:
|
||||
|
||||
```json
|
||||
{
|
||||
"script_path": "skills/elk-fetch/elk_query.py",
|
||||
"args": ["<request_id>", "--preset", "main"],
|
||||
"timeout_seconds": 90,
|
||||
"max_output_chars": 20000
|
||||
}
|
||||
```
|
||||
|
||||
如果 `python_exec` 返回缺少 `elasticsearch` 或 `urllib3`,先使用 `python_package` 安装:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "install",
|
||||
"packages": ["elasticsearch<8", "urllib3"],
|
||||
"timeout_seconds": 120
|
||||
}
|
||||
```
|
||||
|
||||
下面所有 `python elk_query.py ...` 命令模板只表示参数选择;实际执行时都要转换为 `python_exec.script_path = "skills/elk-fetch/elk_query.py"` 和对应的 `args`。
|
||||
|
||||
## 关键文件
|
||||
|
||||
| 文件 | 用途 |
|
||||
|------|------|
|
||||
| `elk_query.py` | CLI 入口 |
|
||||
| `INDEX_CATALOG.md` | **每张表的字段 schema reference**(顶层字段 / JSON 字符串字段内部结构) |
|
||||
| `business/keyword-free.md` | 免唤醒判决 — 拒识 vs kwfree 两张表三个视角的区分 + 排查 playbook |
|
||||
| `business/intent-arbitrator.md` | 中控仲裁 — `intention.intent_arbitrator_info` 子字段语义 |
|
||||
| `business/micar-onetrack.md` | 小米汽车 OneTrack — 23177 端到端可用性 / 23176 全离线日志 / 34496 离线NLP结果 |
|
||||
|
||||
## Preset 速查
|
||||
|
||||
| preset | profile | 集群 | 索引 pattern | 字段名 | 时间字段 | 内置过滤 |
|
||||
|--------|---------|------|-------------|--------|---------|---------|
|
||||
| `main` (默认) | default | akaiservice | `arch-flat-nlp-log-f-*` | `request_id` | `timestamp` | 无 |
|
||||
| `reject` | reject | akaiservice | `aiservice_duplex_rejection_lcs_log*` | **`requestId`** | **`time`** | 无 |
|
||||
| `kwfree` | default | akaiservice | `nlp_post_processing_lcs-*` | `requestId` | `timestamp` | `moduleName = keyword-free-log` |
|
||||
| `micar` | micar | **c3log** | `onetrack_xiaoai_micar*` | `request_id` | `timestamp` | 无(一个 rid 多条 tip) |
|
||||
|
||||
## 选 preset 的决策表
|
||||
|
||||
| 用户想看 | preset | 额外参数 | 业务深入 |
|
||||
|---------|--------|---------|---------|
|
||||
| query / domain / func / 设备信息 | `main` | — | — |
|
||||
| **中控仲裁日志**(agent 路由 / score_domains / llm_agent_info / hit_rules) | `main` | `--json-path intention.intent_arbitrator_info` | [`business/intent-arbitrator.md`](business/intent-arbitrator.md) |
|
||||
| **拒识表整体**(rejectInfo 全部 / reject_reason / 各子策略) | `reject` | 不加 `--json-path` | [`business/keyword-free.md`](business/keyword-free.md) |
|
||||
| **拒识里的免唤醒**(后置拒识调到免唤醒那次的出入参) | `reject` | `--json-path rejectInfo.strategy_info.reject.KEY_WORD_FREE_RESTRIC` | [`business/keyword-free.md`](business/keyword-free.md) |
|
||||
| **后处理的免唤醒**(免唤醒模块被后处理调用时自己记的日志) | `kwfree` | — | [`business/keyword-free.md`](business/keyword-free.md) |
|
||||
| **小米汽车端侧埋点**(端到端可用性 / 全离线日志 / ASR 状态等 OneTrack tip) | `micar` | **`--date YYYYMMDD` 强烈建议** | [`business/micar-onetrack.md`](business/micar-onetrack.md) |
|
||||
|
||||
## 从 Kibana 链接提取参数
|
||||
|
||||
Kibana URL 形如:
|
||||
|
||||
```text
|
||||
...discover#/?...
|
||||
filters:(...match_phrase:(moduleName:keyword-free-log))
|
||||
index:d7ff4d65-... ← Kibana index pattern UUID(非 ES 索引名)
|
||||
query:(language:kuery,query:'requestId : 67f41dff...')
|
||||
time:(from:now-1d,to:now)
|
||||
```
|
||||
|
||||
反推方法:
|
||||
|
||||
- `requestId : xxx` / `request_id : xxx` → request id
|
||||
- `filters` 里的 `match_phrase:(moduleName:keyword-free-log)` → **`kwfree` preset**
|
||||
- 主机 `aiservice.ak.kibana.cloud.mioffice.cn` → `default` profile;主机 `akaiservice.kibana.pt.xiaomi.com` → `reject` profile;主机 `c3log.kibana.pt.xiaomi.com` → **`micar` profile(独立集群)**;主机 `akelk.pt.ai.srv` → `default` profile + `main` preset(中控仲裁的 Kibana 前端,底层就是主表)
|
||||
- index 名为 `onetrack_xiaoai_micar` 直接走 `micar` preset;filter 里有 `tip` / `tip_name` 也走 `micar`
|
||||
- `time:(from:now-1d,...)` → 默认近 48 小时,`from:'2026-04-22...'` 这种具体日期 → 传 `--date YYYYMMDD`
|
||||
|
||||
Kibana index UUID 不能直接反查 ES 索引名;如遇未知 UUID 跑一次 preset 试,打不中再 probe。
|
||||
|
||||
## 命令模板
|
||||
|
||||
> 路径形如 `python "C:\Users\Sivan\.claude\skills\elk-fetch\elk_query.py" ...`,下方为简写。每个场景的字段语义和排查思路见 `business/` 对应文档。
|
||||
|
||||
### 主 NLP 日志(默认)
|
||||
|
||||
```bash
|
||||
python elk_query.py <request_id>
|
||||
```
|
||||
|
||||
### 中控仲裁日志(→ [`business/intent-arbitrator.md`](business/intent-arbitrator.md))
|
||||
|
||||
```bash
|
||||
# 整个仲裁对象
|
||||
python elk_query.py <rid> --json-path intention.intent_arbitrator_info
|
||||
|
||||
# 命中的 agent + 召回规则 + 仲裁等级
|
||||
python elk_query.py <rid> \
|
||||
--fields request_id,query,domain,intention.intent_arbitrator_info.llm_agent_info,intention.intent_arbitrator_info.hit_rules,intention.intent_arbitrator_info.arbitrator_level
|
||||
|
||||
# 各候选 domain 的仲裁打分
|
||||
python elk_query.py <rid> --json-path intention.intent_arbitrator_info.score_domains
|
||||
```
|
||||
|
||||
⚠️ 真东西在 `intention.intent_arbitrator_info`,**顶层同名字段通常 null**。
|
||||
|
||||
### 拒识表 / `KEY_WORD_FREE_RESTRIC`(→ [`business/keyword-free.md`](business/keyword-free.md))
|
||||
|
||||
```bash
|
||||
# 整个拒识 rejectInfo
|
||||
python elk_query.py <rid> --preset reject
|
||||
|
||||
# 拒识里的免唤醒子字段
|
||||
python elk_query.py <rid> --preset reject \
|
||||
--json-path rejectInfo.strategy_info.reject.KEY_WORD_FREE_RESTRIC
|
||||
```
|
||||
|
||||
同一 requestId 在此索引常有多条(多轮),按 `time` 倒序返回。
|
||||
|
||||
### 后处理免唤醒(→ [`business/keyword-free.md`](business/keyword-free.md))
|
||||
|
||||
```bash
|
||||
# 整个 message
|
||||
python elk_query.py <rid> --preset kwfree --json-path message
|
||||
|
||||
# 是否实际处理 + 跳过原因
|
||||
python elk_query.py <rid> --preset kwfree \
|
||||
--fields requestId,message.process,message.skipReason,message.request.keywordFreeType
|
||||
```
|
||||
|
||||
### 小米汽车 OneTrack(→ [`business/micar-onetrack.md`](business/micar-onetrack.md))
|
||||
|
||||
⚠️ 必须传 `--date YYYYMMDD`(按日 rolling,单日 ~300GB)。
|
||||
|
||||
```bash
|
||||
# 这个 request 上报了哪些 tip
|
||||
python elk_query.py <rid> --preset micar --date 20260424 \
|
||||
--fields request_id,tip,tip_name,tip_module_name,event_name
|
||||
|
||||
# 端到端可用性(23177)的性能时序
|
||||
python elk_query.py <rid> --preset micar --date 20260424 \
|
||||
--fields request_id,tip,key_value.state_result_type,key_value.state_exec_result,key_value.wakeup_received_event,key_value.asr_final,key_value.nlp_finish_answer
|
||||
|
||||
# 全离线日志(23176)的模型 debug 输出
|
||||
python elk_query.py <rid> --preset micar --date 20260424 \
|
||||
--json-path response_infos.0.nlp_debug_info
|
||||
```
|
||||
|
||||
⚠️ 23176 schema 与其他 tip 不同:负载在 `request_infos[]` / `response_infos[]`,**不在 `key_value`**。`response_infos[0]` 含 `instructions` + `nlp_debug_info` 两块,研究模型行为时都要看,别只取 nlp_debug_info。
|
||||
|
||||
### 指定日期 / 自定义索引
|
||||
|
||||
```bash
|
||||
# 老请求超出 48h 窗口
|
||||
python elk_query.py <rid> --date 20260422
|
||||
|
||||
# probe 未知索引
|
||||
python elk_query.py <rid> \
|
||||
--index "some-other-*" --field requestId --time-field time --profile reject
|
||||
```
|
||||
|
||||
## 输出格式
|
||||
|
||||
始终单个 JSON 对象到 stdout:
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "...",
|
||||
"preset": "reject",
|
||||
"profile": "reject",
|
||||
"index": "aiservice_duplex_rejection_lcs_log*",
|
||||
"field": "requestId",
|
||||
"time_field": "time",
|
||||
"date": "past-48h",
|
||||
"total": 2,
|
||||
"hits": [ { "...": "_source(嵌套 JSON 字符串已自动解开)" } ]
|
||||
}
|
||||
```
|
||||
|
||||
出错时 JSON 里会有 `error` 键,退出码非零。
|
||||
|
||||
## 常见坑
|
||||
|
||||
- **请求 id 过期**:默认查近 48h,老请求必须 `--date YYYYMMDD`。
|
||||
- **字段名驼峰 vs 下划线**:`main` / `micar` 是 `request_id`,`reject` / `kwfree` 是 `requestId`;Kibana 界面显示的字段名不一定等于 ES 存储字段名。
|
||||
- **返回大量字段刷屏**:用 `--fields` 或 `--json-path` 精简。
|
||||
- **不要用 WebFetch 抓 Kibana URL**:会被 CAS 拦到登录页,永远走这个 skill。
|
||||
@@ -0,0 +1,60 @@
|
||||
# 中控仲裁日志
|
||||
|
||||
> "中控"= central intent arbitrator,决定一条 query 路由到哪个 domain/agent(`controlCopilot` / `iotCopilot` / `productAgent` / 闲聊 / ...)。
|
||||
|
||||
## 在哪查
|
||||
|
||||
主 NLP 表 `arch-flat-nlp-log-f-*` 里的 **`intention.intent_arbitrator_info`** 嵌套对象。`main` preset 直接能读,**不需要新 preset**。
|
||||
|
||||
⚠️ **坑**:顶层 `intent_arbitrator_info`(不带 `intention.`)通常是 `null`——同名但不同位置,别取错。
|
||||
|
||||
> Kibana URL 形如 `akelk.pt.ai.srv/app/discover#/?...index:b0bd62cb-...&query:(...request_id:...)`,这个 Kibana 前端虽然走 akelk 域名,底层 ES 就是 akaiservice 主表。
|
||||
|
||||
## 关键字段
|
||||
|
||||
路径前缀 `intention.intent_arbitrator_info`:
|
||||
|
||||
**仲裁结果**
|
||||
- `predict_result_domain` / `l2_domain` / `l2_func` / `before_final_rule_domain` — 各阶段判出的 domain
|
||||
- `single_result_domain` / `multi_result_domain` — 单轮 / 多轮结果
|
||||
- `dialog_status` — 对话状态(`FINISH` / ...)
|
||||
|
||||
**仲裁打分**
|
||||
- `score_domains` (dict) — 各候选 domain 的 `score` + `func`,仲裁打分原始数据
|
||||
- 例:`controlCopilot:1.0`, `soundboxControl:0.97`, `default:0.05`
|
||||
|
||||
**仲裁等级**
|
||||
- `arbitrator_level` — 仲裁等级(如 `Agent_LLM` 表示走 LLM agent 路径)
|
||||
- `is_llm_allowed` / `is_new_agent_logic`
|
||||
|
||||
**LLM agent 命中**
|
||||
- `llm_agent_info`:
|
||||
- `isEffective` — 是否生效
|
||||
- `agentType` — 类型(如 `车载控制`)
|
||||
- `agentName` — 名称(如 `controlCopilot|iotCopilot|productAgent`)
|
||||
- `agentSubType`
|
||||
- `hit_rules` — 命中的召回规则(如 `[agent_llm_recall_controlCopilot]`)
|
||||
|
||||
**LLM 仲裁子流程**
|
||||
- `llm_strategy_info`:`llm_domain` / `llm_intent` / `llm_strategy_stage` / `before_llm_*`
|
||||
|
||||
**confidence**
|
||||
- `intent_confidence_information`
|
||||
|
||||
## 命令模板
|
||||
|
||||
```bash
|
||||
# 整个仲裁对象
|
||||
python elk_query.py <rid> --json-path intention.intent_arbitrator_info
|
||||
|
||||
# 命中的 agent + 召回规则 + 仲裁等级(速览)
|
||||
python elk_query.py <rid> \
|
||||
--fields request_id,query,domain,intention.intent_arbitrator_info.llm_agent_info,intention.intent_arbitrator_info.hit_rules,intention.intent_arbitrator_info.arbitrator_level
|
||||
|
||||
# 各候选 domain 的仲裁打分
|
||||
python elk_query.py <rid> --json-path intention.intent_arbitrator_info.score_domains
|
||||
```
|
||||
|
||||
## 字段 schema
|
||||
|
||||
完整 schema 见 [`../INDEX_CATALOG.md`](../INDEX_CATALOG.md) 中 `arch-flat-nlp-log-f-*` 一节。
|
||||
@@ -0,0 +1,61 @@
|
||||
# 免唤醒判决日志
|
||||
|
||||
> "免唤醒"指车机上不需要每次说"小爱同学"的免唤醒指令模式(如开/关空调、调温度)。每次请求都会经过免唤醒判决——是否让这条 query 直接执行而不是走闲聊/拒识。
|
||||
|
||||
## 三个视角,两张表
|
||||
|
||||
| # | 业务视角 | 日志位置 | 命令 |
|
||||
| - | ------- | ------- | ---- |
|
||||
| 1 | **拒识模块视角**:每次请求拒识模块整体输出(含所有子策略,含调用免唤醒的结果) | `aiservice_duplex_rejection_lcs_log*` 整条 `rejectInfo` | `--preset reject` |
|
||||
| 2 | **拒识里的免唤醒子字段**:拒识调免唤醒那一次的入参出参 | 同表,`rejectInfo.strategy_info.reject.KEY_WORD_FREE_RESTRIC` | `--preset reject --json-path rejectInfo.strategy_info.reject.KEY_WORD_FREE_RESTRIC` |
|
||||
| 3 | **免唤醒模块视角**:免唤醒模块**被后处理调到时**自己写的日志(独立的另一张表) | `nlp_post_processing_lcs-*` 中 `moduleName=keyword-free-log` | `--preset kwfree` |
|
||||
|
||||
1 和 2 同表同条记录,2 只是取子字段;3 是完全独立的另一张表。
|
||||
|
||||
## 关键字段
|
||||
|
||||
### 视角 2:拒识里的免唤醒子字段
|
||||
|
||||
路径前缀 `rejectInfo.strategy_info.reject.KEY_WORD_FREE_RESTRIC`:
|
||||
- `process` — 是否走到免唤醒判决(true / false)
|
||||
- `skipped_reason` — 被跳过原因(如 `post_processing_takeover` 表示让后处理接管,进入视角 3)
|
||||
- `keywordFreeType`
|
||||
- `isCodeMatch`
|
||||
- `getProcessedQuery`
|
||||
- `Function List`
|
||||
|
||||
### 视角 3:后处理免唤醒模块自身
|
||||
|
||||
JSON 字段 `message` 顶层:
|
||||
- `process` — 是否实际走到处理
|
||||
- `skipReason` — 未处理原因(如 `rejectedByOtherStrategy` 已被其他策略先拒)
|
||||
- `route.flow` / `route.enableFullMigration` / `route.clawEnabled`
|
||||
- `request.query`
|
||||
- `request.keywordFreeType`(如 `INSTRUCTION_KEY_WORD_FREE`)
|
||||
- `request.zone`(如 `DRIVER`)
|
||||
- `request.appId` / `maskUid` / `maskDeviceId`
|
||||
- `fallback.oldRejectType` / `fallback.oldDomain`
|
||||
|
||||
## 排查 Playbook
|
||||
|
||||
**问题:免唤醒为什么没生效?**
|
||||
|
||||
1. 先查视角 2,看拒识有没有调到免唤醒:
|
||||
```bash
|
||||
python elk_query.py <rid> --preset reject \
|
||||
--json-path rejectInfo.strategy_info.reject.KEY_WORD_FREE_RESTRIC
|
||||
```
|
||||
- `process=true` → 走到免唤醒判决了,看 `keywordFreeType` 等子字段
|
||||
- `process=false` → 没走到,看 `skipped_reason`:
|
||||
- `post_processing_takeover` → 进视角 3 看后处理免唤醒
|
||||
- 其他原因 → 拒识自己挡了
|
||||
|
||||
2. 视角 3——后处理免唤醒模块自身:
|
||||
```bash
|
||||
python elk_query.py <rid> --preset kwfree --json-path message
|
||||
```
|
||||
- `process=false` 时看 `skipReason`(`rejectedByOtherStrategy` 表示已被先拒)
|
||||
|
||||
## 字段 schema
|
||||
|
||||
完整 schema 见 [`../INDEX_CATALOG.md`](../INDEX_CATALOG.md) 中 `aiservice_duplex_rejection_lcs_log*` 和 `nlp_post_processing_lcs-*` 两节。
|
||||
@@ -0,0 +1,127 @@
|
||||
# 小米汽车 OneTrack 端侧埋点
|
||||
|
||||
> 小米汽车小爱端侧上报的全量埋点(端到端可用性、ASR 状态、全离线日志、性能指标等)。与 NLP 服务端日志正交:从车端上报的客户端事件流。
|
||||
>
|
||||
> 索引 `onetrack_xiaoai_micar*`(**c3log 独立集群**),单日 ~300GB——**必须传 `--date YYYYMMDD`**。
|
||||
|
||||
## tip 体系
|
||||
|
||||
一个 request_id 对应多条记录,每条对应一个 `tip` 埋点点位。先 `--fields tip,tip_name` 看一眼有哪些 tip:
|
||||
|
||||
```bash
|
||||
python elk_query.py <rid> --preset micar --date 20260424 \
|
||||
--fields request_id,tip,tip_name,tip_module_name,event_name
|
||||
```
|
||||
|
||||
主要点位:
|
||||
|
||||
| tip | tip_name | tip_module_name | event_name | 含义 |
|
||||
| --- | -------- | --------------- | ---------- | ---- |
|
||||
| `1024.1.1.1.23177` | 小爱端到端可用性 | 性能类 | execute | 全链路时序埋点(wakeup → asr → nlp → exec) |
|
||||
| `1024.1.2.1.23176` | 小爱全离线日志 | 日志类 | execute | **离线模型相关日志** |
|
||||
| `1024.1.2.1.34496` | 离线NLP结果 | — | state | 离线 NLP 出参 |
|
||||
| `1024.4.1.1.25631` | ASR识别状态 | ASR识别 | state | ASR 中间状态 |
|
||||
| `1024.1.1.1.33610` | 第三方接口调用 | — | execute | — |
|
||||
| `1024.1.3.1.23520` / `23521` | 小爱指令开始 / 结束 | — | start / — | — |
|
||||
|
||||
⚠️ **不同 tip 的负载 schema 不统一**:
|
||||
- 23177 → 负载在顶层 `key_value`
|
||||
- 23176 → 负载在顶层 `request_infos[]` / `response_infos[]` / `other_infos`(**不在 `key_value`**)
|
||||
- 34496 → 负载在 `key_value.instructions`
|
||||
- 25631 → `key_value` 为 null,看 `event_name` / `other_infos`
|
||||
|
||||
---
|
||||
|
||||
## 23177 — 端到端可用性
|
||||
|
||||
负载在顶层 `key_value`(脚本自动解析)。常用字段:
|
||||
|
||||
**唤醒**:`wakeup_received_event` / `wakeup_ball_appear`
|
||||
|
||||
**ASR 时序**:`asr_first_partial` / `asr_first_text` / `asr_first_same_final` / `asr_final` / `asr_first_pack_sent`
|
||||
|
||||
**NLP 时序**:`nlp_start_answer` / `nlp_finish_answer`
|
||||
|
||||
**结果**:`state_result_type`(`success` / ...)/ `state_exec_result` / `state_cancel_msg`
|
||||
|
||||
**指令统计**:`state_exec_ins_total` / `state_exec_ins_success` / `state_exec_ins_failed` / `state_exec_ins_filtered`
|
||||
|
||||
**异常**:`state_nlp_unknown_instructions` (list) / `state_nlp_exceptions` / `state_vad_end_type`
|
||||
|
||||
**链路**:`state_duplex` / `state_request_id` / `state_asr_final_size`
|
||||
|
||||
```bash
|
||||
python elk_query.py <rid> --preset micar --date 20260424 \
|
||||
--fields request_id,tip,key_value.state_result_type,key_value.state_exec_result,key_value.wakeup_received_event,key_value.asr_final,key_value.nlp_finish_answer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 23176 — 全离线日志 ⭐ 离线模型核心
|
||||
|
||||
⚠️ schema 与其他 tip 不同:负载在顶层 `request_infos[]` / `response_infos[]` / `other_infos`(list,长度通常为 1,但每个 list 都有大量信息),**不在 `key_value`**。
|
||||
|
||||
### `request_infos[0]` — 送给离线引擎的入参
|
||||
|
||||
- 引擎/模型版本:`engine_id` / `engine_model`(如 `3.2.30-claw`)/ `offline_model_ver`(如 `2026012201`)/ `miai_ver` / `app_version`
|
||||
- 请求形态:`request_type` / `request_state_type` / `is_wakeup_req` / `is_llm` / `duplex` / `origin`
|
||||
- 链路环境:`srv_env` / `network` / `network_nlp` / `network_type` / `off_asr_avail`
|
||||
- 设备:`device_id` / `car_type` / `car_config` / `is_driving` / `zone_info` / `is_login` / `has_token` / `isInternal`
|
||||
- **`context_offline`** (list) — 离线引擎拿到的全部 context payload(含 `Nlp.OfflineSession` / `Map.MapState` / `UIController.InteractionInfoList` 等 namespace)。**研究模型行为时务必看这一段**
|
||||
- 其他:`ua` / `transaction_id` / `request_id` / `timeout_reason`
|
||||
|
||||
### `response_infos[0]` — 引擎返回的产物
|
||||
|
||||
⚠️ 不止 `nlp_debug_info`,三块都要看:
|
||||
|
||||
**`instructions[]`** — 引擎下发的全部指令序列。每条 `header.namespace.name` + `header.is_offline`(true/false 标识离/在线判决)+ `header.dialog_id` / `id` / `transaction_id` + `payload`。常见 namespace.name:
|
||||
- `System.Heartbeat` / `System.Abort`
|
||||
- `Offline.CloudStop`(`current_round_used_audio_duration` / `next_round_rid` / `stop_audio_duration`)
|
||||
- `SpeechRecognizer.RecognizeResult`(`is_final` / `results[].text` / `confidence` / `is_nlp_request` / `is_key_word_free_request`)
|
||||
- `Nlp.StartAnswer` / `Nlp.FinishAnswer` / `Nlp.IntentsWithRelation`(`payload.intent`)
|
||||
- `Template.Query`(`text`)
|
||||
- `Dialog.Reject`(`query` / `reject_type` 如 `NON_HUMAN`) / `Dialog.Finish`
|
||||
|
||||
**`nlp_debug_info`** ⭐ 离线 NLP 模型自身的调试输出(模型日志打点)。研究模型决策为什么这样做的核心:
|
||||
- **`Arbitrate`** — 仲裁器决策:`query` / **`logits`** (list[float], 各类目原始打分) / `label_id` / `category`(如 `Info`)/ `predict` / `label`(如 `baike#person`)/ `domains`(候选 domain)/ `time`
|
||||
- 各阶段耗时:`Rewrite.time` / `PreTrain.time`(`bertAvailable`)/ `EdgeIntentExecutor.time` / `GeneralParse.time` / `lookAndTalk.time`
|
||||
- 候选 domain:`domain_confidence` (list) / `domain_parsers` (list)
|
||||
- 各 domain parser 子模块(如 `music` / `phonecall` / `mapApp`):每个 parser 块下含 `time` / `score` / `parser_version` + 子模块耗时(如 `music.MusicKvParserCost` / `MusicJsgfParserCost` / `MusicIcsfModelParserCost` / `MusicGlobalCost`)
|
||||
- **`nlp_model_version`** (dict) — 各模型 checkpoint 版本(如 `arbitrator-l1: 20241205_offline` / `phonecall-icsf: 240205_offline` / `mapapp-icsf-single: 241224_offline`)
|
||||
- `nlp_edge_version` — 边端 NLP 引擎整体版本
|
||||
- `edge_track` — 上下文/能力 capability 追踪:`capabilitiesVersion` / `contexts`(用到的 context namespace list)/ `arbitrator_priority`
|
||||
- `llm_wait` — 是否等待 LLM
|
||||
|
||||
**`other_infos`** — 其他辅助信息(list)
|
||||
|
||||
### 命令模板
|
||||
|
||||
```bash
|
||||
# 整条全离线日志(含 request_infos[0]、response_infos[0]、other_infos)
|
||||
python elk_query.py <rid> --preset micar --date 20260424 \
|
||||
--fields request_id,tip,tip_name,request_infos,response_infos,other_infos
|
||||
|
||||
# 只看模型自身打的 debug 日志
|
||||
python elk_query.py <rid> --preset micar --date 20260424 \
|
||||
--json-path response_infos.0.nlp_debug_info
|
||||
|
||||
# 模型版本 + 仲裁结果速览
|
||||
python elk_query.py <rid> --preset micar --date 20260424 \
|
||||
--fields request_id,tip,response_infos.0.nlp_debug_info.Arbitrate,response_infos.0.nlp_debug_info.nlp_model_version,response_infos.0.nlp_debug_info.nlp_edge_version,request_infos.0.engine_model,request_infos.0.offline_model_ver
|
||||
|
||||
# 引擎全部指令(看 is_offline 区分离/在线)
|
||||
python elk_query.py <rid> --preset micar --date 20260424 \
|
||||
--json-path response_infos.0.instructions
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 34496 — 离线NLP结果
|
||||
|
||||
`key_value` 顶层就是 `{instructions: [...]}`,结构和 23176 的 `response_infos[0].instructions` 一致,但**只是 NLP 部分的指令子集**(不含 SpeechRecognizer / System / Offline 等)。没有 `nlp_debug_info`、没有 `request_infos`——想看模型决策细节去 23176。
|
||||
|
||||
---
|
||||
|
||||
## 字段 schema
|
||||
|
||||
完整 schema 见 [`../INDEX_CATALOG.md`](../INDEX_CATALOG.md) 中 `onetrack_xiaoai_micar*` 一节。
|
||||
@@ -0,0 +1,300 @@
|
||||
"""
|
||||
ELK / Elasticsearch 日志查询工具。
|
||||
|
||||
按 request id 查 ELK。支持:
|
||||
- 多账号 profile(不同业务用不同 ES 账号)
|
||||
- 多业务 preset(业务语义 → 索引/字段/过滤条件的映射)
|
||||
- 深度 JSON 字符串自动解析(rejectInfo / message 等场景)
|
||||
- 按点号路径提取嵌套字段(--json-path)
|
||||
|
||||
账号密码从同目录 profiles.json 加载(团队共用账号已 commit 进仓库;可用 ELK_FETCH_PROFILES 指向别处覆盖)。
|
||||
业务→索引→路径的详细说明见同目录 INDEX_CATALOG.md。
|
||||
|
||||
用法:
|
||||
python elk_query.py <request_id>
|
||||
[--preset main|reject|kwfree|micar]
|
||||
[--date YYYYMMDD]
|
||||
[--index PATTERN] [--field FIELD] [--time-field NAME] [--profile KEY]
|
||||
[--size N]
|
||||
[--fields f1,f2,...]
|
||||
[--json-path a.b.c] # 提取嵌套字段,自动解 JSON 字符串
|
||||
[--raw] # 关闭自动 JSON 字符串解析
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import warnings
|
||||
from copy import deepcopy
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import urllib3
|
||||
from elasticsearch import Elasticsearch
|
||||
from elasticsearch.exceptions import (
|
||||
ConnectionError as ESConnectionError,
|
||||
ElasticsearchWarning,
|
||||
NotFoundError,
|
||||
RequestError,
|
||||
AuthorizationException,
|
||||
AuthenticationException,
|
||||
)
|
||||
|
||||
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
|
||||
warnings.filterwarnings("ignore", category=ElasticsearchWarning)
|
||||
warnings.filterwarnings("ignore", category=DeprecationWarning)
|
||||
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(encoding="utf-8")
|
||||
|
||||
ES_PORT = int(os.getenv("ES_PORT", "80"))
|
||||
ES_HOST_FALLBACK = os.getenv("ES_HOST", "")
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
PROFILES_PATH = Path(os.getenv("ELK_FETCH_PROFILES", SCRIPT_DIR / "profiles.json"))
|
||||
|
||||
|
||||
def load_profiles() -> Dict[str, Dict[str, str]]:
|
||||
"""从 profiles.json 读取账号配置。缺文件 / 格式错误时给出可操作的错误信息。"""
|
||||
if not PROFILES_PATH.exists():
|
||||
raise FileNotFoundError(
|
||||
f"profiles.json not found at {PROFILES_PATH}. "
|
||||
f"Override path via ELK_FETCH_PROFILES=/path/to/profiles.json"
|
||||
)
|
||||
try:
|
||||
with PROFILES_PATH.open("r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"profiles.json is not valid JSON: {e}")
|
||||
if not isinstance(data, dict) or not data:
|
||||
raise ValueError("profiles.json must be a non-empty JSON object keyed by profile name")
|
||||
for key, prof in data.items():
|
||||
if not isinstance(prof, dict) or "user" not in prof or "password" not in prof:
|
||||
raise ValueError(f"profile '{key}' missing required fields 'user' / 'password'")
|
||||
return data
|
||||
|
||||
|
||||
# 业务→索引/字段/额外过滤 的映射。time_field 为毫秒时间戳字段名。
|
||||
# 每条 preset 对应一个明确的业务语义,详见 INDEX_CATALOG.md。
|
||||
PRESETS: Dict[str, Dict[str, Any]] = {
|
||||
"main": {
|
||||
"profile": "default",
|
||||
"index": "arch-flat-nlp-log-f-*",
|
||||
"field": "request_id",
|
||||
"time_field": "timestamp",
|
||||
"extra_filters": [],
|
||||
},
|
||||
"reject": {
|
||||
# 后置拒识模块的日志,内含调用免唤醒判决(KEY_WORD_FREE_RESTRIC)的结果
|
||||
"profile": "reject",
|
||||
"index": "aiservice_duplex_rejection_lcs_log*",
|
||||
"field": "requestId",
|
||||
"time_field": "time",
|
||||
"extra_filters": [],
|
||||
},
|
||||
"kwfree": {
|
||||
# 后处理阶段调用免唤醒模块自身的日志。message 是 JSON 字符串
|
||||
"profile": "default",
|
||||
"index": "nlp_post_processing_lcs-*",
|
||||
"field": "requestId",
|
||||
"time_field": "timestamp",
|
||||
"extra_filters": [{"match_phrase": {"moduleName": "keyword-free-log"}}],
|
||||
},
|
||||
"micar": {
|
||||
# 小米汽车小爱 OneTrack 埋点(端到端可用性 / 性能埋点等 tip)。
|
||||
# 索引按日 rolling 且非常大(~300GB/天),强烈建议传 --date 收敛日期。
|
||||
# key_value 是序列化 JSON 字符串,脚本会自动深度解析。
|
||||
"profile": "micar",
|
||||
"index": "onetrack_xiaoai_micar*",
|
||||
"field": "request_id",
|
||||
"time_field": "timestamp",
|
||||
"extra_filters": [],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def build_client(profiles: Dict[str, Dict[str, str]], profile_key: str) -> Elasticsearch:
|
||||
if profile_key not in profiles:
|
||||
available = ", ".join(sorted(profiles.keys())) or "(none)"
|
||||
raise KeyError(f"profile '{profile_key}' not in profiles.json (available: {available})")
|
||||
prof = profiles[profile_key]
|
||||
host = prof.get("host") or ES_HOST_FALLBACK
|
||||
if not host:
|
||||
raise ValueError(
|
||||
f"profile '{profile_key}' has no 'host' and ES_HOST env var is not set"
|
||||
)
|
||||
return Elasticsearch(
|
||||
hosts=[f"http://{host}:{ES_PORT}"],
|
||||
http_auth=(prof["user"], prof["password"]),
|
||||
timeout=60,
|
||||
max_retries=3,
|
||||
)
|
||||
|
||||
|
||||
def build_query(
|
||||
request_id: str,
|
||||
field: str,
|
||||
time_field: str,
|
||||
date_str: Optional[str],
|
||||
size: int,
|
||||
extra_filters: List[Dict[str, Any]],
|
||||
) -> Dict[str, Any]:
|
||||
if date_str:
|
||||
date_obj = datetime.strptime(date_str, "%Y%m%d")
|
||||
start_ms = int(date_obj.timestamp() * 1000)
|
||||
end_ms = int((date_obj + timedelta(days=1)).timestamp() * 1000) - 1
|
||||
else:
|
||||
end_ms = int(datetime.now().timestamp() * 1000)
|
||||
start_ms = int((datetime.now() - timedelta(hours=48)).timestamp() * 1000)
|
||||
|
||||
filters: List[Dict[str, Any]] = [
|
||||
{"wildcard": {field: {"value": f"{request_id}*"}}},
|
||||
]
|
||||
filters.extend(deepcopy(extra_filters))
|
||||
|
||||
return {
|
||||
"query": {
|
||||
"bool": {
|
||||
"must": [{"range": {time_field: {"gte": start_ms, "lte": end_ms}}}],
|
||||
"filter": filters,
|
||||
}
|
||||
},
|
||||
"size": size,
|
||||
"sort": [{time_field: {"order": "desc"}}],
|
||||
}
|
||||
|
||||
|
||||
def deep_parse(v: Any) -> Any:
|
||||
"""递归把看起来是 JSON 的字符串展开成 dict/list。"""
|
||||
if isinstance(v, str):
|
||||
s = v.strip()
|
||||
if (s.startswith("{") and s.endswith("}")) or (s.startswith("[") and s.endswith("]")):
|
||||
try:
|
||||
return deep_parse(json.loads(s))
|
||||
except Exception:
|
||||
return v
|
||||
return v
|
||||
if isinstance(v, dict):
|
||||
return {k: deep_parse(val) for k, val in v.items()}
|
||||
if isinstance(v, list):
|
||||
return [deep_parse(x) for x in v]
|
||||
return v
|
||||
|
||||
|
||||
def get_path(obj: Any, dotted: str) -> Any:
|
||||
cur = obj
|
||||
for part in dotted.split("."):
|
||||
if not part:
|
||||
continue
|
||||
if isinstance(cur, dict):
|
||||
cur = cur.get(part)
|
||||
elif isinstance(cur, list):
|
||||
try:
|
||||
cur = cur[int(part)]
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
else:
|
||||
return None
|
||||
return cur
|
||||
|
||||
|
||||
def project(
|
||||
source: Dict[str, Any],
|
||||
fields: Optional[List[str]],
|
||||
json_path: Optional[str],
|
||||
) -> Any:
|
||||
if json_path:
|
||||
return {json_path: get_path(source, json_path)}
|
||||
if fields:
|
||||
return {k: get_path(source, k) for k in fields}
|
||||
return source
|
||||
|
||||
|
||||
def run(args: argparse.Namespace, profiles: Dict[str, Dict[str, str]]) -> Dict[str, Any]:
|
||||
preset = PRESETS[args.preset]
|
||||
profile_key = args.profile or preset["profile"]
|
||||
index_pattern = args.index or preset["index"]
|
||||
field_name = args.field or preset["field"]
|
||||
time_field = args.time_field or preset["time_field"]
|
||||
extra_filters = preset.get("extra_filters", [])
|
||||
|
||||
body = build_query(
|
||||
args.request_id, field_name, time_field, args.date, args.size, extra_filters
|
||||
)
|
||||
try:
|
||||
client = build_client(profiles, profile_key)
|
||||
except (KeyError, ValueError) as e:
|
||||
return {"error": "ProfileError", "message": str(e)}
|
||||
|
||||
try:
|
||||
resp = client.search(index=index_pattern, body=body)
|
||||
except AuthenticationException as e:
|
||||
return {"error": "AuthenticationException", "profile": profile_key, "message": str(e)}
|
||||
except AuthorizationException as e:
|
||||
return {"error": "AuthorizationException", "profile": profile_key, "index": index_pattern, "message": str(e)}
|
||||
except NotFoundError as e:
|
||||
return {"error": "IndexNotFound", "index": index_pattern, "message": str(e)}
|
||||
except RequestError as e:
|
||||
return {"error": "QueryError", "message": str(e)}
|
||||
except ESConnectionError as e:
|
||||
return {"error": "ConnectionError", "message": str(e)}
|
||||
|
||||
hits_raw = resp.get("hits", {}).get("hits", [])
|
||||
fields = [f.strip() for f in args.fields.split(",")] if args.fields else None
|
||||
|
||||
records = []
|
||||
for h in hits_raw:
|
||||
src = h.get("_source", {})
|
||||
if not args.raw:
|
||||
src = deep_parse(src)
|
||||
records.append(project(src, fields, args.json_path))
|
||||
|
||||
return {
|
||||
"request_id": args.request_id,
|
||||
"preset": args.preset,
|
||||
"profile": profile_key,
|
||||
"index": index_pattern,
|
||||
"field": field_name,
|
||||
"time_field": time_field,
|
||||
"date": args.date or "past-48h",
|
||||
"total": len(records),
|
||||
"hits": records,
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
p = argparse.ArgumentParser(
|
||||
description="Query ELK by request id (multi-profile, deep-JSON aware)",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="业务→索引映射详见同目录 INDEX_CATALOG.md",
|
||||
)
|
||||
p.add_argument("request_id", help="目标 request id(支持前缀通配)")
|
||||
p.add_argument("--preset", choices=list(PRESETS.keys()), default="main",
|
||||
help="预置方案:" + " | ".join(PRESETS.keys()))
|
||||
p.add_argument("--profile", help="覆盖账号 profile(key 来自 profiles.json)")
|
||||
p.add_argument("--index", help="覆盖索引 pattern")
|
||||
p.add_argument("--field", help="覆盖 request id 字段名")
|
||||
p.add_argument("--time-field", help="覆盖时间字段名")
|
||||
p.add_argument("--date", help="指定日期 YYYYMMDD,不填默认近 48 小时")
|
||||
p.add_argument("--size", type=int, default=20, help="最多返回记录数(默认 20)")
|
||||
p.add_argument("--fields", help="只输出指定字段,逗号分隔,支持 a.b.c 嵌套")
|
||||
p.add_argument("--json-path", help="只提取某条嵌套路径的值")
|
||||
p.add_argument("--raw", action="store_true", help="关闭自动 JSON 字符串解析")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
try:
|
||||
profiles = load_profiles()
|
||||
except (FileNotFoundError, ValueError) as e:
|
||||
print(json.dumps({"error": "ConfigError", "message": str(e)}, indent=2, ensure_ascii=False))
|
||||
return 2
|
||||
result = run(args, profiles)
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
return 0 if "error" not in result else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"default": {
|
||||
"user": "ai_service_kibana",
|
||||
"password": "Ox7tLXSorOo5t5KG",
|
||||
"host": "akaiservice.api.es.srv"
|
||||
},
|
||||
"reject": {
|
||||
"user": "ai_service_kibana",
|
||||
"password": "Ox7tLXSorOo5t5KG",
|
||||
"host": "akaiservice.api.es.srv"
|
||||
},
|
||||
"micar": {
|
||||
"user": "xiaoai_micar_kibana",
|
||||
"password": "aF6pi9XfzXq2h543",
|
||||
"host": "c3log.api.es.srv"
|
||||
}
|
||||
}
|
||||
@@ -1,12 +1,12 @@
|
||||
"""Bundled skill definitions — prompt-type skills invocable via the Skill tool.
|
||||
"""Skill definitions — prompt-type skills invocable via the Skill tool.
|
||||
|
||||
Mirrors the npm ``src/skills/bundled/`` module.
|
||||
|
||||
Bundled skills differ from slash commands:
|
||||
Directory skills differ from slash commands:
|
||||
- They generate AI prompts sent to the model (prompt-type).
|
||||
- They carry ``when_to_use`` guidance for model auto-invocation.
|
||||
- They can restrict ``allowed_tools`` during execution.
|
||||
- They appear in system-reminder skill listings for model discovery.
|
||||
|
||||
Project-maintained skills live under the repository root ``skills/`` directory.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -196,7 +196,7 @@ def load_directory_skills(
|
||||
*,
|
||||
source: str = 'directory',
|
||||
) -> tuple[BundledSkill, ...]:
|
||||
skill_root = root or (Path(__file__).resolve().parent / 'skills' / 'bundled')
|
||||
skill_root = root or (Path(__file__).resolve().parent.parent / 'skills')
|
||||
if not skill_root.exists() or not skill_root.is_dir():
|
||||
return ()
|
||||
skills: list[BundledSkill] = []
|
||||
|
||||
Reference in New Issue
Block a user