diff --git a/docs/superpowers/specs/2026-07-09-n13-uia-selector-catalog-design.md b/docs/superpowers/specs/2026-07-09-n13-uia-selector-catalog-design.md new file mode 100644 index 0000000..cfb79d4 --- /dev/null +++ b/docs/superpowers/specs/2026-07-09-n13-uia-selector-catalog-design.md @@ -0,0 +1,597 @@ +# N13 UIA Selector Catalog Design + +日期:2026-07-09 + +## 1. 目标 + +N13 的目标是基于已 ACCEPT 的 N12R live UIA 证据,设计一套只读 selector catalog 和定位验证流程。 + +本阶段只做设计,不新增 helper op,不新增 MCP tool,不点击、不输入、不打开会话、不发送消息、不传输文件。 + +N13 完成后,下一步才可以写独立实现计划,考虑是否新增只读定位验证能力。 + +## 2. 输入证据 + +N12R 返回包已通过外网 validator: + +```text +runs/internal-sandbox-live-capture-reports/2026-07-09-retry-2/2026-07-08-internal-sandbox-a-validation.md +Decision: ACCEPT +PASS: 36 +WARN: 1 +FAIL: 0 +``` + +使用的 UIA dump: + +```text +runs/internal-sandbox-live-capture-intake/2026-07-09-retry-2/2026-07-08-internal-sandbox-a/uia/dump-uia-main.json +``` + +关键事实: + +```text +root control_type: Window +root automation_id: frmMain +root class_name: WindowsForms10.Window.8.app.0.d3a00f_r29_ad1 +framework_id: WinForm +UIA node count: 66 +max_depth: 7 +``` + +本设计只引用结构性字段,不依赖敏感可见文本。 + +## 3. 非目标 + +N13 不做以下事项: + +```text +自动登录 +搜索联系人 +打开会话 +点击控件 +输入文本 +发送消息 +发送文件 +接收文件 +读取或导出聊天内容 +注入进程 +hook +读取目标进程内存 +绕过终端安全/安防 +``` + +N13 也不把 `search_contacts`、`open_conversation`、`read_latest_messages` 作为当前实现目标。这些只能在 selector catalog 稳定、只读定位验证通过、并形成新的审批设计后再讨论。 + +## 4. 推荐方案 + +采用“selector catalog + 只读定位验证”方案。 + +核心思想: + +1. 用 N12R UIA dump 建立一份 selector catalog。 +2. 每个 selector 都只描述如何识别一个 UIA 节点。 +3. selector 的初始用途只允许 read-only locate。 +4. 后续如果实现 `locate_uia`,也只能返回匹配节点摘要,不执行任何 UI 动作。 +5. 任何真实动作类能力必须另写设计、计划和审批边界。 + +## 5. Selector 数据模型 + +建议 selector 条目使用以下字段: + +```json +{ + "id": "main_window", + "description": "主窗口", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "frmMain", + "control_type": "Window", + "framework_id": "WinForm" + }, + "preferred": { + "class_name_prefix": "WindowsForms10.Window.8.app.0", + "path_hint": "root" + }, + "fallback": [ + { + "control_type": "Window", + "class_name_prefix": "WindowsForms10.Window.8.app.0", + "child_automation_id_contains": ["panelLeft", "ucPluginTabControl1"] + } + ], + "stability_score": 95, + "evidence": { + "capture_id": "2026-07-08-internal-sandbox-a", + "uia_path": "root" + } +} +``` + +字段含义: + +- `id`:稳定、短小、英文蛇形命名。 +- `description`:中文说明。 +- `allowed_use`:N13 固定为 `read_only_locate`。 +- `required`:必须同时满足的字段。 +- `preferred`:加分项,用于减少误匹配。 +- `fallback`:主 selector 失效时的只读匹配策略。 +- `stability_score`:0 到 100,越高越稳定。 +- `evidence`:来源证据路径和 UIA 路径。 + +## 6. 初始 selector catalog + +### 6.1 主窗口:`main_window` + +```json +{ + "id": "main_window", + "description": "iSphere / IMPlatformClient 主窗口", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "frmMain", + "control_type": "Window", + "framework_id": "WinForm" + }, + "preferred": { + "class_name_prefix": "WindowsForms10.Window.8.app.0", + "path_hint": "root" + }, + "fallback": [ + { + "control_type": "Window", + "class_name_prefix": "WindowsForms10.Window.8.app.0", + "child_automation_id_contains": ["panelLeft", "ucPluginTabControl1"] + } + ], + "stability_score": 95, + "evidence": { + "uia_path": "root" + } +} +``` + +理由:`automation_id=frmMain` 出现在根节点;root 是 N12R 已确认主窗口,稳定性最高。 + +### 6.2 主 tab 容器:`main_plugin_tab` + +```json +{ + "id": "main_plugin_tab", + "description": "主功能 tab 容器", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "ucPluginTabControl1", + "control_type": "Pane", + "framework_id": "WinForm" + }, + "preferred": { + "path_hint": "root/3", + "ancestor_selector": "main_window" + }, + "fallback": [ + { + "ancestor_selector": "main_window", + "control_type": "Pane", + "child_automation_id_contains": ["FrmHomePage", "ucTabPage"] + } + ], + "stability_score": 90, + "evidence": { + "uia_path": "root/3" + } +} +``` + +理由:该节点直接挂在主窗口下,automation_id 明确,子结构包含首页和 tab 内容区。 + +### 6.3 首页窗口:`home_page_window` + +```json +{ + "id": "home_page_window", + "description": "首页内容窗口", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "FrmHomePage", + "control_type": "Window", + "framework_id": "WinForm" + }, + "preferred": { + "path_hint": "root/3/0/0", + "ancestor_selector": "main_plugin_tab" + }, + "fallback": [ + { + "ancestor_selector": "main_plugin_tab", + "control_type": "Window", + "class_name_prefix": "WindowsForms10.Window.8.app.0" + } + ], + "stability_score": 82, + "evidence": { + "uia_path": "root/3/0/0" + } +} +``` + +理由:该节点可用于确认当前主 UI 内容区存在,但不用于动作。 + +### 6.4 tab 内容窗口:`tab_page_content` + +```json +{ + "id": "tab_page_content", + "description": "tab 页内容窗口", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "FrmTabPageContent", + "control_type": "Window", + "framework_id": "WinForm" + }, + "preferred": { + "path_hint": "root/3/1/0", + "ancestor_selector": "main_plugin_tab" + }, + "fallback": [ + { + "ancestor_selector": "main_plugin_tab", + "control_type": "Window", + "child_control_type_contains": ["TitleBar"] + } + ], + "stability_score": 82, + "evidence": { + "uia_path": "root/3/1/0" + } +} +``` + +理由:可作为后续只读结构检查的内容区域锚点。 + +### 6.5 左侧面板:`left_panel` + +```json +{ + "id": "left_panel", + "description": "左侧主面板", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "panelLeft", + "control_type": "Pane", + "framework_id": "WinForm" + }, + "preferred": { + "path_hint": "root/4", + "ancestor_selector": "main_window" + }, + "fallback": [ + { + "ancestor_selector": "main_window", + "control_type": "Pane", + "child_automation_id_contains": ["ucPnlRoster1", "ucFuzzyQuery1"] + } + ], + "stability_score": 92, + "evidence": { + "uia_path": "root/4" + } +} +``` + +理由:该节点是联系人/搜索相关区域的稳定父节点。 + +### 6.6 组织/联系人容器:`roster_panel` + +```json +{ + "id": "roster_panel", + "description": "组织/联系人容器外层", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "ucPnlRoster1", + "control_type": "Pane", + "framework_id": "WinForm" + }, + "preferred": { + "path_hint": "root/4/0/0", + "ancestor_selector": "left_panel" + }, + "fallback": [ + { + "ancestor_selector": "left_panel", + "control_type": "Pane", + "child_automation_id_contains": ["ucRoster1"] + } + ], + "stability_score": 88, + "evidence": { + "uia_path": "root/4/0/0" + } +} +``` + +理由:该节点是 roster 结构外层,可以只读验证联系人区域是否存在。 + +### 6.7 组织/联系人列表:`roster_view` + +```json +{ + "id": "roster_view", + "description": "组织/联系人列表区域", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "ucRoster1", + "control_type": "Pane", + "framework_id": "WinForm" + }, + "preferred": { + "path_hint": "root/4/0/0/0", + "ancestor_selector": "roster_panel" + }, + "fallback": [ + { + "ancestor_selector": "roster_panel", + "control_type": "Pane", + "class_name_prefix": "WindowsForms10.Window.8.app.0" + } + ], + "stability_score": 86, + "evidence": { + "uia_path": "root/4/0/0/0" + } +} +``` + +理由:该节点本次 child_count 为 0,不能证明列表项结构;只能作为区域存在性 selector。 + +### 6.8 搜索组件外层:`fuzzy_query_panel` + +```json +{ + "id": "fuzzy_query_panel", + "description": "模糊搜索组件外层", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "ucFuzzyQuery1", + "control_type": "Pane", + "framework_id": "WinForm" + }, + "preferred": { + "path_hint": "root/4/1/0", + "ancestor_selector": "left_panel" + }, + "fallback": [ + { + "ancestor_selector": "left_panel", + "control_type": "Pane", + "child_automation_id_contains": ["FrmFuzzyQuery"] + } + ], + "stability_score": 90, + "evidence": { + "uia_path": "root/4/1/0" + } +} +``` + +理由:该组件是搜索输入框的稳定父级。 + +### 6.9 搜索窗口:`fuzzy_query_window` + +```json +{ + "id": "fuzzy_query_window", + "description": "模糊搜索内部窗口", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "FrmFuzzyQuery", + "control_type": "Pane", + "framework_id": "WinForm" + }, + "preferred": { + "path_hint": "root/4/1/0/0", + "ancestor_selector": "fuzzy_query_panel" + }, + "fallback": [ + { + "ancestor_selector": "fuzzy_query_panel", + "control_type": "Pane", + "child_automation_id_contains": ["skinAlphaTxt"] + } + ], + "stability_score": 88, + "evidence": { + "uia_path": "root/4/1/0/0" + } +} +``` + +理由:该节点包含搜索输入框,是后续只读定位验证中的关键父节点。 + +### 6.10 搜索输入框:`fuzzy_search_edit` + +```json +{ + "id": "fuzzy_search_edit", + "description": "模糊搜索输入框", + "allowed_use": "read_only_locate", + "required": { + "automation_id": "skinAlphaTxt", + "control_type": "Edit", + "framework_id": "WinForm" + }, + "preferred": { + "class_name": "WindowsForms10.EDIT.app.0.d3a00f_r29_ad1", + "path_hint": "root/4/1/0/0/2", + "ancestor_selector": "fuzzy_query_window" + }, + "fallback": [ + { + "ancestor_selector": "fuzzy_query_window", + "control_type": "Edit", + "class_name_prefix": "WindowsForms10.EDIT.app.0" + } + ], + "stability_score": 93, + "evidence": { + "uia_path": "root/4/1/0/0/2" + } +} +``` + +理由:该节点 automation_id、control_type 和 class_name 都明确。N13 只允许定位它,不允许输入搜索词。 + +## 7. 只读定位验证规则 + +后续如果实现定位验证,只能做以下事情: + +1. 接收 selector id。 +2. 读取当前窗口 UIA 树。 +3. 按 selector required 字段过滤节点。 +4. 用 preferred 字段排序或加分。 +5. 如果主 selector 无匹配,再尝试 fallback。 +6. 返回节点摘要:`selector_id`、`matched`、`match_count`、`path`、`control_type`、`automation_id`、`class_name`、`framework_id`、`is_enabled`、`is_offscreen`、`bounds`。 + +禁止返回完整 visible text。可见文本只能返回以下派生信息: + +```text +has_name: true/false +name_length: number +``` + +禁止执行: + +```text +InvokePattern +ValuePattern.SetValue +SelectionItemPattern.Select +ExpandCollapsePattern.Expand +点击 +键盘输入 +剪贴板操作 +窗口激活 +焦点切换 +``` + +## 8. 稳定性评分规则 + +建议评分: + +```text +95-100: root 或唯一主窗口级 anchor +90-94: automation_id 明确、control_type 明确、父子路径稳定 +80-89: automation_id 明确,但可能依赖 tab 或运行状态 +70-79: 缺 automation_id,但 class/control/path 组合可用 +0-69: 不进入首批 catalog +``` + +本次首批 selector 全部要求 automation_id 明确。 + +## 9. 错误处理设计 + +只读定位验证应返回结构化结果,不抛出模糊错误。 + +建议结果: + +```json +{ + "ok": true, + "selector_id": "fuzzy_search_edit", + "matched": true, + "match_count": 1, + "matches": [ + { + "path": "root/4/1/0/0/2", + "control_type": "Edit", + "automation_id": "skinAlphaTxt", + "class_name": "WindowsForms10.EDIT.app.0.d3a00f_r29_ad1", + "framework_id": "WinForm", + "has_name": true, + "name_length": 0 + } + ], + "warnings": [] +} +``` + +未匹配时: + +```json +{ + "ok": true, + "selector_id": "fuzzy_search_edit", + "matched": false, + "match_count": 0, + "matches": [], + "warnings": ["selector_not_found"] +} +``` + +selector 定义非法时: + +```json +{ + "ok": false, + "selector_id": "fuzzy_search_edit", + "error": { + "code": "INVALID_SELECTOR", + "message": "selector required fields are incomplete" + } +} +``` + +## 10. 测试策略 + +N13 设计阶段不写实现代码,但后续实现计划必须包含以下测试: + +1. 从 N12R dump fixture 读取 UIA 树。 +2. 验证 `main_window` 匹配唯一 root。 +3. 验证 `left_panel` 匹配 `root/4`。 +4. 验证 `fuzzy_search_edit` 匹配 `root/4/1/0/0/2`。 +5. 验证 selector 不返回完整 `name` 文本,只返回 `has_name` 和 `name_length`。 +6. 验证缺失 selector 返回 `matched=false`,不报崩溃。 +7. 验证动作模式字段或动作请求被拒绝。 +8. 验证 catalog 中不包含 `send`、`upload`、`download`、`conversation`、`login` 等动作词作为工具名。 + +## 11. 文件规划 + +如果后续进入实现计划,建议新增或修改: + +```text +internal/uiaselector/catalog.go +internal/uiaselector/catalog_test.go +internal/uiaselector/matcher.go +internal/uiaselector/matcher_test.go +internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json +scripts/verify-n13-uia-selectors.ps1 +``` + +暂不修改: + +```text +native/ISphereWinHelper/Program.cs +internal/tools/winhelper.go +cmd/isphere-mcp/main.go +``` + +除非新的实现计划明确批准新增只读定位工具,否则 Go MCP 仍只暴露现有四个只读工具。 + +## 12. N13 验收条件 + +N13 设计可接受条件: + +1. selector catalog 只依赖结构性 UIA 字段。 +2. 不依赖敏感可见文本。 +3. 首批 selector 都能追溯到 N12R ACCEPT dump 的路径。 +4. allowed_use 全部是 `read_only_locate`。 +5. 文档明确禁止点击、输入、打开会话、发送和文件操作。 +6. 后续实现计划必须 TDD,并先使用 fixture 测试 selector matcher。 + +## 13. 推荐下一步 + +下一步不是直接写代码,而是写一份 N13 implementation plan。 + +该计划应先实现纯离线 selector matcher,对 N12R UIA dump fixture 做只读匹配测试。 + +只有离线 selector matcher 通过后,才讨论是否新增 live `locate_uia` helper/MCP 只读工具。