docs: plan N13 UIA selector catalog implementation
This commit is contained in:
石头
@@ -0,0 +1,943 @@
|
||||
# N13 UIA Selector Catalog Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build an offline, read-only N13 UIA selector catalog and matcher that can validate selectors against the accepted N12R UIA dump without adding live helper or MCP actions.
|
||||
|
||||
**Architecture:** Add a focused Go package `internal/uiaselector` with selector data types, a static first-pass catalog, UIA dump loading, and pure in-memory matching. Tests use a redacted N12R fixture and assert that matching returns structural node summaries without visible text.
|
||||
|
||||
**Tech Stack:** Go 1.23.x, standard library `encoding/json`, existing repository test style, PowerShell verification wrapper.
|
||||
|
||||
---
|
||||
|
||||
## Scope and boundary
|
||||
|
||||
This plan implements only offline selector catalog and matching.
|
||||
|
||||
It does not modify these files:
|
||||
|
||||
```text
|
||||
native/ISphereWinHelper/Program.cs
|
||||
internal/tools/winhelper.go
|
||||
cmd/isphere-mcp/main.go
|
||||
```
|
||||
|
||||
It does not add MCP tools or helper ops. It does not click, type, open conversations, send messages, send files, receive files, or read chat content.
|
||||
|
||||
The accepted N12R evidence used by tests is:
|
||||
|
||||
```text
|
||||
runs/internal-sandbox-live-capture-intake/2026-07-09-retry-2/2026-07-08-internal-sandbox-a/uia/dump-uia-main-redacted.json
|
||||
```
|
||||
|
||||
## File structure
|
||||
|
||||
Create these files:
|
||||
|
||||
```text
|
||||
internal/uiaselector/catalog.go
|
||||
internal/uiaselector/catalog_test.go
|
||||
internal/uiaselector/dump.go
|
||||
internal/uiaselector/dump_test.go
|
||||
internal/uiaselector/matcher.go
|
||||
internal/uiaselector/matcher_test.go
|
||||
internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json
|
||||
scripts/verify-n13-uia-selectors.ps1
|
||||
```
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- `catalog.go`: selector data structures and initial N13 selector catalog.
|
||||
- `catalog_test.go`: catalog completeness and boundary tests.
|
||||
- `dump.go`: UIA dump response and node loading helpers.
|
||||
- `dump_test.go`: fixture parse tests.
|
||||
- `matcher.go`: pure in-memory selector matcher.
|
||||
- `matcher_test.go`: matching behavior and no-visible-text result tests.
|
||||
- `testdata/n12r-2026-07-09-uia-redacted.json`: redacted accepted N12R UIA fixture copied from the ignored run artifact.
|
||||
- `scripts/verify-n13-uia-selectors.ps1`: convenience verification script for Go-enabled environments.
|
||||
|
||||
## Task 1: Add N12R redacted UIA fixture
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json`
|
||||
|
||||
- [ ] **Step 1: Create package directory and copy the accepted redacted fixture**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
New-Item -ItemType Directory -Force -Path internal\uiaselector\testdata | Out-Null
|
||||
Copy-Item `
|
||||
-LiteralPath runs\internal-sandbox-live-capture-intake\2026-07-09-retry-2\2026-07-08-internal-sandbox-a\uia\dump-uia-main-redacted.json `
|
||||
-Destination internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json `
|
||||
-Force
|
||||
```
|
||||
|
||||
Expected: `internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json` exists.
|
||||
|
||||
- [ ] **Step 2: Verify fixture parses as JSON**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
Get-Content internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json -Encoding UTF8 | ConvertFrom-Json | Out-Null
|
||||
```
|
||||
|
||||
Expected: command exits 0.
|
||||
|
||||
- [ ] **Step 3: Commit fixture**
|
||||
|
||||
```powershell
|
||||
git add internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json
|
||||
git commit -m "test: add N13 redacted UIA fixture"
|
||||
```
|
||||
|
||||
## Task 2: Define selector catalog types and initial catalog
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/uiaselector/catalog_test.go`
|
||||
- Create: `internal/uiaselector/catalog.go`
|
||||
|
||||
- [ ] **Step 1: Write failing catalog tests**
|
||||
|
||||
Create `internal/uiaselector/catalog_test.go`:
|
||||
|
||||
```go
|
||||
package uiaselector
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDefaultCatalogContainsInitialN13Selectors(t *testing.T) {
|
||||
catalog := DefaultCatalog()
|
||||
wantIDs := []string{
|
||||
"main_window",
|
||||
"main_plugin_tab",
|
||||
"home_page_window",
|
||||
"tab_page_content",
|
||||
"left_panel",
|
||||
"roster_panel",
|
||||
"roster_view",
|
||||
"fuzzy_query_panel",
|
||||
"fuzzy_query_window",
|
||||
"fuzzy_search_edit",
|
||||
}
|
||||
|
||||
byID := map[string]Selector{}
|
||||
for _, selector := range catalog {
|
||||
byID[selector.ID] = selector
|
||||
}
|
||||
|
||||
for _, id := range wantIDs {
|
||||
selector, ok := byID[id]
|
||||
if !ok {
|
||||
t.Fatalf("DefaultCatalog missing selector %q", id)
|
||||
}
|
||||
if selector.AllowedUse != AllowedUseReadOnlyLocate {
|
||||
t.Fatalf("selector %q allowed_use = %q, want %q", id, selector.AllowedUse, AllowedUseReadOnlyLocate)
|
||||
}
|
||||
if selector.Required.AutomationID == "" {
|
||||
t.Fatalf("selector %q must require automation_id", id)
|
||||
}
|
||||
if selector.Required.ControlType == "" {
|
||||
t.Fatalf("selector %q must require control_type", id)
|
||||
}
|
||||
if selector.StabilityScore < 80 {
|
||||
t.Fatalf("selector %q stability_score = %d, want >= 80", id, selector.StabilityScore)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultCatalogDoesNotDescribeActionTools(t *testing.T) {
|
||||
for _, selector := range DefaultCatalog() {
|
||||
if selector.AllowedUse != AllowedUseReadOnlyLocate {
|
||||
t.Fatalf("selector %q allowed_use = %q, want read-only", selector.ID, selector.AllowedUse)
|
||||
}
|
||||
lower := strings.ToLower(selector.ID + " " + selector.Description)
|
||||
for _, forbidden := range []string{"send", "upload", "download", "receive", "conversation", "login", "file_transfer"} {
|
||||
if strings.Contains(lower, forbidden) {
|
||||
t.Fatalf("selector %q contains action word %q", selector.ID, forbidden)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run catalog tests and verify they fail**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/uiaselector -run TestDefaultCatalog -count=1
|
||||
```
|
||||
|
||||
Expected: FAIL because package `internal/uiaselector` or `DefaultCatalog` does not exist.
|
||||
|
||||
If this environment reports `go` not found, stop execution and record the environment gap. Continue only in a Go 1.23.x environment.
|
||||
|
||||
- [ ] **Step 3: Implement catalog types and selectors**
|
||||
|
||||
Create `internal/uiaselector/catalog.go`:
|
||||
|
||||
```go
|
||||
package uiaselector
|
||||
|
||||
const AllowedUseReadOnlyLocate = "read_only_locate"
|
||||
|
||||
type Selector struct {
|
||||
ID string `json:"id"`
|
||||
Description string `json:"description"`
|
||||
AllowedUse string `json:"allowed_use"`
|
||||
Required Criteria `json:"required"`
|
||||
Preferred PreferredCriteria `json:"preferred"`
|
||||
Fallback []FallbackCriteria `json:"fallback"`
|
||||
StabilityScore int `json:"stability_score"`
|
||||
Evidence Evidence `json:"evidence"`
|
||||
}
|
||||
|
||||
type Criteria struct {
|
||||
AutomationID string `json:"automation_id,omitempty"`
|
||||
ControlType string `json:"control_type,omitempty"`
|
||||
FrameworkID string `json:"framework_id,omitempty"`
|
||||
ClassName string `json:"class_name,omitempty"`
|
||||
ClassPrefix string `json:"class_name_prefix,omitempty"`
|
||||
AncestorID string `json:"ancestor_selector,omitempty"`
|
||||
}
|
||||
|
||||
type PreferredCriteria struct {
|
||||
ClassName string `json:"class_name,omitempty"`
|
||||
ClassPrefix string `json:"class_name_prefix,omitempty"`
|
||||
PathHint string `json:"path_hint,omitempty"`
|
||||
AncestorID string `json:"ancestor_selector,omitempty"`
|
||||
}
|
||||
|
||||
type FallbackCriteria struct {
|
||||
AncestorID string `json:"ancestor_selector,omitempty"`
|
||||
ControlType string `json:"control_type,omitempty"`
|
||||
ClassPrefix string `json:"class_name_prefix,omitempty"`
|
||||
ChildAutomationIDContains []string `json:"child_automation_id_contains,omitempty"`
|
||||
ChildControlTypeContains []string `json:"child_control_type_contains,omitempty"`
|
||||
}
|
||||
|
||||
type Evidence struct {
|
||||
CaptureID string `json:"capture_id"`
|
||||
UIAPath string `json:"uia_path"`
|
||||
}
|
||||
|
||||
func DefaultCatalog() []Selector {
|
||||
return []Selector{
|
||||
{
|
||||
ID: "main_window",
|
||||
Description: "iSphere / IMPlatformClient 主窗口",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "frmMain", ControlType: "Window", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{ClassPrefix: "WindowsForms10.Window.8.app.0", PathHint: "root"},
|
||||
Fallback: []FallbackCriteria{{ControlType: "Window", ClassPrefix: "WindowsForms10.Window.8.app.0", ChildAutomationIDContains: []string{"panelLeft", "ucPluginTabControl1"}}},
|
||||
StabilityScore: 95,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root"},
|
||||
},
|
||||
{
|
||||
ID: "main_plugin_tab",
|
||||
Description: "主功能 tab 容器",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "ucPluginTabControl1", ControlType: "Pane", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{PathHint: "root/3", AncestorID: "main_window"},
|
||||
Fallback: []FallbackCriteria{{AncestorID: "main_window", ControlType: "Pane", ChildAutomationIDContains: []string{"FrmHomePage", "ucTabPage"}}},
|
||||
StabilityScore: 90,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/3"},
|
||||
},
|
||||
{
|
||||
ID: "home_page_window",
|
||||
Description: "首页内容窗口",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "FrmHomePage", ControlType: "Window", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{PathHint: "root/3/0/0", AncestorID: "main_plugin_tab"},
|
||||
Fallback: []FallbackCriteria{{AncestorID: "main_plugin_tab", ControlType: "Window", ClassPrefix: "WindowsForms10.Window.8.app.0"}},
|
||||
StabilityScore: 82,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/3/0/0"},
|
||||
},
|
||||
{
|
||||
ID: "tab_page_content",
|
||||
Description: "tab 页内容窗口",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "FrmTabPageContent", ControlType: "Window", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{PathHint: "root/3/1/0", AncestorID: "main_plugin_tab"},
|
||||
Fallback: []FallbackCriteria{{AncestorID: "main_plugin_tab", ControlType: "Window", ChildControlTypeContains: []string{"TitleBar"}}},
|
||||
StabilityScore: 82,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/3/1/0"},
|
||||
},
|
||||
{
|
||||
ID: "left_panel",
|
||||
Description: "左侧主面板",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "panelLeft", ControlType: "Pane", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{PathHint: "root/4", AncestorID: "main_window"},
|
||||
Fallback: []FallbackCriteria{{AncestorID: "main_window", ControlType: "Pane", ChildAutomationIDContains: []string{"ucPnlRoster1", "ucFuzzyQuery1"}}},
|
||||
StabilityScore: 92,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4"},
|
||||
},
|
||||
{
|
||||
ID: "roster_panel",
|
||||
Description: "组织/联系人容器外层",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "ucPnlRoster1", ControlType: "Pane", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{PathHint: "root/4/0/0", AncestorID: "left_panel"},
|
||||
Fallback: []FallbackCriteria{{AncestorID: "left_panel", ControlType: "Pane", ChildAutomationIDContains: []string{"ucRoster1"}}},
|
||||
StabilityScore: 88,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/0/0"},
|
||||
},
|
||||
{
|
||||
ID: "roster_view",
|
||||
Description: "组织/联系人列表区域",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "ucRoster1", ControlType: "Pane", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{PathHint: "root/4/0/0/0", AncestorID: "roster_panel"},
|
||||
Fallback: []FallbackCriteria{{AncestorID: "roster_panel", ControlType: "Pane", ClassPrefix: "WindowsForms10.Window.8.app.0"}},
|
||||
StabilityScore: 86,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/0/0/0"},
|
||||
},
|
||||
{
|
||||
ID: "fuzzy_query_panel",
|
||||
Description: "模糊搜索组件外层",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "ucFuzzyQuery1", ControlType: "Pane", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{PathHint: "root/4/1/0", AncestorID: "left_panel"},
|
||||
Fallback: []FallbackCriteria{{AncestorID: "left_panel", ControlType: "Pane", ChildAutomationIDContains: []string{"FrmFuzzyQuery"}}},
|
||||
StabilityScore: 90,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/1/0"},
|
||||
},
|
||||
{
|
||||
ID: "fuzzy_query_window",
|
||||
Description: "模糊搜索内部窗口",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "FrmFuzzyQuery", ControlType: "Pane", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{PathHint: "root/4/1/0/0", AncestorID: "fuzzy_query_panel"},
|
||||
Fallback: []FallbackCriteria{{AncestorID: "fuzzy_query_panel", ControlType: "Pane", ChildAutomationIDContains: []string{"skinAlphaTxt"}}},
|
||||
StabilityScore: 88,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/1/0/0"},
|
||||
},
|
||||
{
|
||||
ID: "fuzzy_search_edit",
|
||||
Description: "模糊搜索输入框",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "skinAlphaTxt", ControlType: "Edit", FrameworkID: "WinForm"},
|
||||
Preferred: PreferredCriteria{ClassName: "WindowsForms10.EDIT.app.0.d3a00f_r29_ad1", PathHint: "root/4/1/0/0/2", AncestorID: "fuzzy_query_window"},
|
||||
Fallback: []FallbackCriteria{{AncestorID: "fuzzy_query_window", ControlType: "Edit", ClassPrefix: "WindowsForms10.EDIT.app.0"}},
|
||||
StabilityScore: 93,
|
||||
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/1/0/0/2"},
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run catalog tests and verify they pass**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/uiaselector -run TestDefaultCatalog -count=1
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit catalog**
|
||||
|
||||
```powershell
|
||||
git add internal\uiaselector\catalog.go internal\uiaselector\catalog_test.go
|
||||
git commit -m "feat: add N13 UIA selector catalog"
|
||||
```
|
||||
|
||||
## Task 3: Load redacted UIA dump fixture
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/uiaselector/dump_test.go`
|
||||
- Create: `internal/uiaselector/dump.go`
|
||||
|
||||
- [ ] **Step 1: Write failing dump loading tests**
|
||||
|
||||
Create `internal/uiaselector/dump_test.go`:
|
||||
|
||||
```go
|
||||
package uiaselector
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestLoadDumpFixtureParsesAcceptedN12RTree(t *testing.T) {
|
||||
root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
|
||||
if err != nil {
|
||||
t.Fatalf("LoadDumpFile: %v", err)
|
||||
}
|
||||
if root.AutomationID != "frmMain" {
|
||||
t.Fatalf("root automation_id = %q, want frmMain", root.AutomationID)
|
||||
}
|
||||
if root.ControlType != "Window" {
|
||||
t.Fatalf("root control_type = %q, want Window", root.ControlType)
|
||||
}
|
||||
if got := CountNodes(root); got != 66 {
|
||||
t.Fatalf("CountNodes = %d, want 66", got)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run dump tests and verify they fail**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/uiaselector -run TestLoadDumpFixture -count=1
|
||||
```
|
||||
|
||||
Expected: FAIL because `LoadDumpFile` and `CountNodes` do not exist.
|
||||
|
||||
- [ ] **Step 3: Implement dump loading**
|
||||
|
||||
Create `internal/uiaselector/dump.go`:
|
||||
|
||||
```go
|
||||
package uiaselector
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
)
|
||||
|
||||
type DumpResponse struct {
|
||||
OK bool `json:"ok"`
|
||||
Data *DumpData `json:"data"`
|
||||
Error any `json:"error"`
|
||||
}
|
||||
|
||||
type DumpData struct {
|
||||
Root *Node `json:"root"`
|
||||
}
|
||||
|
||||
type Node struct {
|
||||
Name string `json:"name"`
|
||||
ControlType string `json:"control_type"`
|
||||
AutomationID string `json:"automation_id"`
|
||||
ClassName string `json:"class_name"`
|
||||
FrameworkID string `json:"framework_id"`
|
||||
IsEnabled *bool `json:"is_enabled"`
|
||||
IsOffscreen *bool `json:"is_offscreen"`
|
||||
Bounds any `json:"bounds"`
|
||||
Children []Node `json:"children"`
|
||||
}
|
||||
|
||||
func LoadDumpFile(path string) (*Node, error) {
|
||||
content, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var response DumpResponse
|
||||
if err := json.Unmarshal(content, &response); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !response.OK {
|
||||
return nil, fmt.Errorf("dump response ok=false: %v", response.Error)
|
||||
}
|
||||
if response.Data == nil || response.Data.Root == nil {
|
||||
return nil, fmt.Errorf("dump response has no data.root")
|
||||
}
|
||||
return response.Data.Root, nil
|
||||
}
|
||||
|
||||
func CountNodes(root *Node) int {
|
||||
if root == nil {
|
||||
return 0
|
||||
}
|
||||
count := 1
|
||||
for i := range root.Children {
|
||||
count += CountNodes(&root.Children[i])
|
||||
}
|
||||
return count
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run dump tests and verify they pass**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/uiaselector -run TestLoadDumpFixture -count=1
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit dump loader**
|
||||
|
||||
```powershell
|
||||
git add internal\uiaselector\dump.go internal\uiaselector\dump_test.go
|
||||
git commit -m "feat: load N13 UIA dump fixture"
|
||||
```
|
||||
|
||||
## Task 4: Implement read-only selector matcher
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/uiaselector/matcher_test.go`
|
||||
- Create: `internal/uiaselector/matcher.go`
|
||||
|
||||
- [ ] **Step 1: Write failing matcher tests**
|
||||
|
||||
Create `internal/uiaselector/matcher_test.go`:
|
||||
|
||||
```go
|
||||
package uiaselector
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestMatchSelectorFindsKnownN12RNodes(t *testing.T) {
|
||||
root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
|
||||
if err != nil {
|
||||
t.Fatalf("LoadDumpFile: %v", err)
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
selectorID string
|
||||
path string
|
||||
control string
|
||||
autoID string
|
||||
}{
|
||||
{selectorID: "main_window", path: "root", control: "Window", autoID: "frmMain"},
|
||||
{selectorID: "left_panel", path: "root/4", control: "Pane", autoID: "panelLeft"},
|
||||
{selectorID: "fuzzy_search_edit", path: "root/4/1/0/0/2", control: "Edit", autoID: "skinAlphaTxt"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
result := MatchSelector(root, DefaultCatalog(), tt.selectorID)
|
||||
if !result.OK {
|
||||
t.Fatalf("%s OK=false: %#v", tt.selectorID, result.Error)
|
||||
}
|
||||
if !result.Matched {
|
||||
t.Fatalf("%s matched=false", tt.selectorID)
|
||||
}
|
||||
if result.MatchCount != 1 {
|
||||
t.Fatalf("%s match_count = %d, want 1", tt.selectorID, result.MatchCount)
|
||||
}
|
||||
match := result.Matches[0]
|
||||
if match.Path != tt.path || match.ControlType != tt.control || match.AutomationID != tt.autoID {
|
||||
t.Fatalf("%s match = %#v", tt.selectorID, match)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchSelectorDoesNotReturnVisibleText(t *testing.T) {
|
||||
root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
|
||||
if err != nil {
|
||||
t.Fatalf("LoadDumpFile: %v", err)
|
||||
}
|
||||
result := MatchSelector(root, DefaultCatalog(), "main_window")
|
||||
if !result.OK || !result.Matched || len(result.Matches) != 1 {
|
||||
t.Fatalf("unexpected result: %#v", result)
|
||||
}
|
||||
match := result.Matches[0]
|
||||
if match.Name != "" {
|
||||
t.Fatalf("match returned visible name text %q", match.Name)
|
||||
}
|
||||
if !match.HasName {
|
||||
t.Fatalf("main_window should report has_name=true")
|
||||
}
|
||||
if match.NameLength == 0 {
|
||||
t.Fatalf("main_window should report non-zero name_length")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchSelectorUnknownSelectorIsStructuredFailure(t *testing.T) {
|
||||
root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
|
||||
if err != nil {
|
||||
t.Fatalf("LoadDumpFile: %v", err)
|
||||
}
|
||||
result := MatchSelector(root, DefaultCatalog(), "missing_selector")
|
||||
if result.OK {
|
||||
t.Fatalf("OK=true for missing selector: %#v", result)
|
||||
}
|
||||
if result.Error.Code != "UNKNOWN_SELECTOR" {
|
||||
t.Fatalf("error code = %q, want UNKNOWN_SELECTOR", result.Error.Code)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run matcher tests and verify they fail**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/uiaselector -run TestMatchSelector -count=1
|
||||
```
|
||||
|
||||
Expected: FAIL because `MatchSelector` and result types do not exist.
|
||||
|
||||
- [ ] **Step 3: Implement matcher**
|
||||
|
||||
Create `internal/uiaselector/matcher.go`:
|
||||
|
||||
```go
|
||||
package uiaselector
|
||||
|
||||
import "strings"
|
||||
|
||||
type MatchResult struct {
|
||||
OK bool `json:"ok"`
|
||||
SelectorID string `json:"selector_id"`
|
||||
Matched bool `json:"matched"`
|
||||
MatchCount int `json:"match_count"`
|
||||
Matches []NodeMatch `json:"matches"`
|
||||
Warnings []string `json:"warnings"`
|
||||
Error MatchError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type MatchError struct {
|
||||
Code string `json:"code"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
type NodeMatch struct {
|
||||
Path string `json:"path"`
|
||||
ControlType string `json:"control_type"`
|
||||
AutomationID string `json:"automation_id"`
|
||||
ClassName string `json:"class_name"`
|
||||
FrameworkID string `json:"framework_id"`
|
||||
HasName bool `json:"has_name"`
|
||||
NameLength int `json:"name_length"`
|
||||
Name string `json:"-"`
|
||||
}
|
||||
|
||||
type nodeRef struct {
|
||||
path string
|
||||
node *Node
|
||||
}
|
||||
|
||||
func MatchSelector(root *Node, catalog []Selector, selectorID string) MatchResult {
|
||||
selector, ok := findSelector(catalog, selectorID)
|
||||
if !ok {
|
||||
return MatchResult{OK: false, SelectorID: selectorID, Error: MatchError{Code: "UNKNOWN_SELECTOR", Message: "selector is not in catalog"}}
|
||||
}
|
||||
if selector.AllowedUse != AllowedUseReadOnlyLocate {
|
||||
return MatchResult{OK: false, SelectorID: selectorID, Error: MatchError{Code: "INVALID_SELECTOR", Message: "selector is not read-only"}}
|
||||
}
|
||||
refs := flatten(root)
|
||||
matches := matchCriteria(refs, selector.Required)
|
||||
if len(matches) == 0 {
|
||||
for _, fallback := range selector.Fallback {
|
||||
matches = matchFallback(refs, fallback)
|
||||
if len(matches) > 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
result := MatchResult{OK: true, SelectorID: selectorID, Matched: len(matches) > 0, MatchCount: len(matches), Warnings: []string{}}
|
||||
for _, ref := range matches {
|
||||
result.Matches = append(result.Matches, summarizeNode(ref))
|
||||
}
|
||||
if !result.Matched {
|
||||
result.Warnings = append(result.Warnings, "selector_not_found")
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func findSelector(catalog []Selector, selectorID string) (Selector, bool) {
|
||||
for _, selector := range catalog {
|
||||
if selector.ID == selectorID {
|
||||
return selector, true
|
||||
}
|
||||
}
|
||||
return Selector{}, false
|
||||
}
|
||||
|
||||
func flatten(root *Node) []nodeRef {
|
||||
refs := []nodeRef{}
|
||||
var walk func(node *Node, path string)
|
||||
walk = func(node *Node, path string) {
|
||||
if node == nil {
|
||||
return
|
||||
}
|
||||
refs = append(refs, nodeRef{path: path, node: node})
|
||||
for i := range node.Children {
|
||||
walk(&node.Children[i], path+"/"+itoa(i))
|
||||
}
|
||||
}
|
||||
walk(root, "root")
|
||||
return refs
|
||||
}
|
||||
|
||||
func itoa(value int) string {
|
||||
if value == 0 {
|
||||
return "0"
|
||||
}
|
||||
digits := []byte{}
|
||||
for value > 0 {
|
||||
digits = append([]byte{byte('0' + value%10)}, digits...)
|
||||
value = value / 10
|
||||
}
|
||||
return string(digits)
|
||||
}
|
||||
|
||||
func matchCriteria(refs []nodeRef, criteria Criteria) []nodeRef {
|
||||
matches := []nodeRef{}
|
||||
for _, ref := range refs {
|
||||
if criteria.AutomationID != "" && ref.node.AutomationID != criteria.AutomationID {
|
||||
continue
|
||||
}
|
||||
if criteria.ControlType != "" && ref.node.ControlType != criteria.ControlType {
|
||||
continue
|
||||
}
|
||||
if criteria.FrameworkID != "" && ref.node.FrameworkID != criteria.FrameworkID {
|
||||
continue
|
||||
}
|
||||
if criteria.ClassName != "" && ref.node.ClassName != criteria.ClassName {
|
||||
continue
|
||||
}
|
||||
if criteria.ClassPrefix != "" && !strings.HasPrefix(ref.node.ClassName, criteria.ClassPrefix) {
|
||||
continue
|
||||
}
|
||||
matches = append(matches, ref)
|
||||
}
|
||||
return matches
|
||||
}
|
||||
|
||||
func matchFallback(refs []nodeRef, fallback FallbackCriteria) []nodeRef {
|
||||
matches := []nodeRef{}
|
||||
for _, ref := range refs {
|
||||
if fallback.ControlType != "" && ref.node.ControlType != fallback.ControlType {
|
||||
continue
|
||||
}
|
||||
if fallback.ClassPrefix != "" && !strings.HasPrefix(ref.node.ClassName, fallback.ClassPrefix) {
|
||||
continue
|
||||
}
|
||||
if !hasChildAutomationIDs(ref.node, fallback.ChildAutomationIDContains) {
|
||||
continue
|
||||
}
|
||||
if !hasChildControlTypes(ref.node, fallback.ChildControlTypeContains) {
|
||||
continue
|
||||
}
|
||||
matches = append(matches, ref)
|
||||
}
|
||||
return matches
|
||||
}
|
||||
|
||||
func hasChildAutomationIDs(node *Node, ids []string) bool {
|
||||
for _, id := range ids {
|
||||
found := false
|
||||
for i := range node.Children {
|
||||
if node.Children[i].AutomationID == id {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func hasChildControlTypes(node *Node, controlTypes []string) bool {
|
||||
for _, controlType := range controlTypes {
|
||||
found := false
|
||||
for i := range node.Children {
|
||||
if node.Children[i].ControlType == controlType {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func summarizeNode(ref nodeRef) NodeMatch {
|
||||
return NodeMatch{
|
||||
Path: ref.path,
|
||||
ControlType: ref.node.ControlType,
|
||||
AutomationID: ref.node.AutomationID,
|
||||
ClassName: ref.node.ClassName,
|
||||
FrameworkID: ref.node.FrameworkID,
|
||||
HasName: ref.node.Name != "",
|
||||
NameLength: len(ref.node.Name),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run matcher tests and verify they pass**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/uiaselector -run TestMatchSelector -count=1
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit matcher**
|
||||
|
||||
```powershell
|
||||
git add internal\uiaselector\matcher.go internal\uiaselector\matcher_test.go
|
||||
git commit -m "feat: match N13 UIA selectors offline"
|
||||
```
|
||||
|
||||
## Task 5: Add verification script
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/verify-n13-uia-selectors.ps1`
|
||||
|
||||
- [ ] **Step 1: Create verification script**
|
||||
|
||||
Create `scripts/verify-n13-uia-selectors.ps1`:
|
||||
|
||||
```powershell
|
||||
$ErrorActionPreference = "Stop"
|
||||
Set-StrictMode -Version Latest
|
||||
|
||||
$repo = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..")).Path
|
||||
Set-Location -LiteralPath $repo
|
||||
|
||||
if (-not (Get-Command go -ErrorAction SilentlyContinue)) {
|
||||
throw "go was not found. Run this verifier in a Go 1.23.x environment."
|
||||
}
|
||||
|
||||
Write-Host "## go test ./internal/uiaselector"
|
||||
go test ./internal/uiaselector -count=1
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "go test ./internal/uiaselector failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
|
||||
Write-Host "## boundary grep"
|
||||
$forbidden = @(
|
||||
"send_after_approval",
|
||||
"send_file_after_approval",
|
||||
"receive_file_after_approval",
|
||||
"open_conversation",
|
||||
"ValuePattern.SetValue",
|
||||
"InvokePattern"
|
||||
)
|
||||
$files = @(
|
||||
"internal\uiaselector\catalog.go",
|
||||
"internal\uiaselector\matcher.go"
|
||||
)
|
||||
foreach ($term in $forbidden) {
|
||||
$hit = Select-String -LiteralPath $files -Pattern $term -SimpleMatch -ErrorAction SilentlyContinue
|
||||
if ($hit) {
|
||||
throw "forbidden action term found in selector implementation: $term"
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "N13 UIA selector verification passed."
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run script and verify it passes**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n13-uia-selectors.ps1
|
||||
```
|
||||
|
||||
Expected in Go-enabled environment:
|
||||
|
||||
```text
|
||||
## go test ./internal/uiaselector
|
||||
ok isphere-ai-bridge/internal/uiaselector
|
||||
## boundary grep
|
||||
N13 UIA selector verification passed.
|
||||
```
|
||||
|
||||
If `go` is unavailable, expected failure is:
|
||||
|
||||
```text
|
||||
go was not found. Run this verifier in a Go 1.23.x environment.
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit verification script**
|
||||
|
||||
```powershell
|
||||
git add scripts\verify-n13-uia-selectors.ps1
|
||||
git commit -m "test: add N13 selector verifier"
|
||||
```
|
||||
|
||||
## Task 6: Final verification and checkpoint
|
||||
|
||||
**Files:**
|
||||
- No new files.
|
||||
- Verify: `internal/uiaselector/*`, `scripts/verify-n13-uia-selectors.ps1`
|
||||
|
||||
- [ ] **Step 1: Run package tests**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/uiaselector -count=1
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```text
|
||||
ok isphere-ai-bridge/internal/uiaselector
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run repository tests if Go is available**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Expected: PASS for all packages.
|
||||
|
||||
If the environment lacks Go, do not mark implementation complete. Move to a Go 1.23.x environment and rerun.
|
||||
|
||||
- [ ] **Step 3: Run N13 verifier**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n13-uia-selectors.ps1
|
||||
```
|
||||
|
||||
Expected: PASS in Go-enabled environment.
|
||||
|
||||
- [ ] **Step 4: Check no MCP/helper action surface changed**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
git diff --name-only origin/main..HEAD
|
||||
```
|
||||
|
||||
Expected changed implementation files are limited to:
|
||||
|
||||
```text
|
||||
internal/uiaselector/catalog.go
|
||||
internal/uiaselector/catalog_test.go
|
||||
internal/uiaselector/dump.go
|
||||
internal/uiaselector/dump_test.go
|
||||
internal/uiaselector/matcher.go
|
||||
internal/uiaselector/matcher_test.go
|
||||
internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json
|
||||
scripts/verify-n13-uia-selectors.ps1
|
||||
```
|
||||
|
||||
If `native/ISphereWinHelper/Program.cs`, `internal/tools/winhelper.go`, or `cmd/isphere-mcp/main.go` appears, stop and review because N13 is not supposed to add live tools.
|
||||
|
||||
- [ ] **Step 5: Commit final checkpoint if needed**
|
||||
|
||||
If any verification-only documentation changes were made, run:
|
||||
|
||||
```powershell
|
||||
git add <changed verification docs>
|
||||
git commit -m "docs: record N13 selector verification"
|
||||
```
|
||||
|
||||
If no files changed, do not create an empty commit.
|
||||
|
||||
## Self-review checklist
|
||||
|
||||
- Spec coverage: this plan covers selector catalog, fixture loading, read-only matcher, no visible text in result, and boundary verification.
|
||||
- Boundary: no helper op, no MCP tool, no click/input/send/file/conversation behavior.
|
||||
- TDD: every implementation task starts with failing tests before code.
|
||||
- Commit cadence: fixture, catalog, loader, matcher, verifier are separate commits.
|
||||
- Environment caveat: Go 1.23.x is required for implementation verification.
|
||||
Reference in New Issue
Block a user