944 lines
28 KiB
Markdown
944 lines
28 KiB
Markdown
# N13 UIA Selector Catalog Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Build an offline, read-only N13 UIA selector catalog and matcher that can validate selectors against the accepted N12R UIA dump without adding live helper or MCP actions.
|
|
|
|
**Architecture:** Add a focused Go package `internal/uiaselector` with selector data types, a static first-pass catalog, UIA dump loading, and pure in-memory matching. Tests use a redacted N12R fixture and assert that matching returns structural node summaries without visible text.
|
|
|
|
**Tech Stack:** Go 1.23.x, standard library `encoding/json`, existing repository test style, PowerShell verification wrapper.
|
|
|
|
---
|
|
|
|
## Scope and boundary
|
|
|
|
This plan implements only offline selector catalog and matching.
|
|
|
|
It does not modify these files:
|
|
|
|
```text
|
|
native/ISphereWinHelper/Program.cs
|
|
internal/tools/winhelper.go
|
|
cmd/isphere-mcp/main.go
|
|
```
|
|
|
|
It does not add MCP tools or helper ops. It does not click, type, open conversations, send messages, send files, receive files, or read chat content.
|
|
|
|
The accepted N12R evidence used by tests is:
|
|
|
|
```text
|
|
runs/internal-sandbox-live-capture-intake/2026-07-09-retry-2/2026-07-08-internal-sandbox-a/uia/dump-uia-main-redacted.json
|
|
```
|
|
|
|
## File structure
|
|
|
|
Create these files:
|
|
|
|
```text
|
|
internal/uiaselector/catalog.go
|
|
internal/uiaselector/catalog_test.go
|
|
internal/uiaselector/dump.go
|
|
internal/uiaselector/dump_test.go
|
|
internal/uiaselector/matcher.go
|
|
internal/uiaselector/matcher_test.go
|
|
internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json
|
|
scripts/verify-n13-uia-selectors.ps1
|
|
```
|
|
|
|
Responsibilities:
|
|
|
|
- `catalog.go`: selector data structures and initial N13 selector catalog.
|
|
- `catalog_test.go`: catalog completeness and boundary tests.
|
|
- `dump.go`: UIA dump response and node loading helpers.
|
|
- `dump_test.go`: fixture parse tests.
|
|
- `matcher.go`: pure in-memory selector matcher.
|
|
- `matcher_test.go`: matching behavior and no-visible-text result tests.
|
|
- `testdata/n12r-2026-07-09-uia-redacted.json`: redacted accepted N12R UIA fixture copied from the ignored run artifact.
|
|
- `scripts/verify-n13-uia-selectors.ps1`: convenience verification script for Go-enabled environments.
|
|
|
|
## Task 1: Add N12R redacted UIA fixture
|
|
|
|
**Files:**
|
|
- Create: `internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json`
|
|
|
|
- [ ] **Step 1: Create package directory and copy the accepted redacted fixture**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
New-Item -ItemType Directory -Force -Path internal\uiaselector\testdata | Out-Null
|
|
Copy-Item `
|
|
-LiteralPath runs\internal-sandbox-live-capture-intake\2026-07-09-retry-2\2026-07-08-internal-sandbox-a\uia\dump-uia-main-redacted.json `
|
|
-Destination internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json `
|
|
-Force
|
|
```
|
|
|
|
Expected: `internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json` exists.
|
|
|
|
- [ ] **Step 2: Verify fixture parses as JSON**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
Get-Content internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json -Encoding UTF8 | ConvertFrom-Json | Out-Null
|
|
```
|
|
|
|
Expected: command exits 0.
|
|
|
|
- [ ] **Step 3: Commit fixture**
|
|
|
|
```powershell
|
|
git add internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json
|
|
git commit -m "test: add N13 redacted UIA fixture"
|
|
```
|
|
|
|
## Task 2: Define selector catalog types and initial catalog
|
|
|
|
**Files:**
|
|
- Create: `internal/uiaselector/catalog_test.go`
|
|
- Create: `internal/uiaselector/catalog.go`
|
|
|
|
- [ ] **Step 1: Write failing catalog tests**
|
|
|
|
Create `internal/uiaselector/catalog_test.go`:
|
|
|
|
```go
|
|
package uiaselector
|
|
|
|
import (
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
func TestDefaultCatalogContainsInitialN13Selectors(t *testing.T) {
|
|
catalog := DefaultCatalog()
|
|
wantIDs := []string{
|
|
"main_window",
|
|
"main_plugin_tab",
|
|
"home_page_window",
|
|
"tab_page_content",
|
|
"left_panel",
|
|
"roster_panel",
|
|
"roster_view",
|
|
"fuzzy_query_panel",
|
|
"fuzzy_query_window",
|
|
"fuzzy_search_edit",
|
|
}
|
|
|
|
byID := map[string]Selector{}
|
|
for _, selector := range catalog {
|
|
byID[selector.ID] = selector
|
|
}
|
|
|
|
for _, id := range wantIDs {
|
|
selector, ok := byID[id]
|
|
if !ok {
|
|
t.Fatalf("DefaultCatalog missing selector %q", id)
|
|
}
|
|
if selector.AllowedUse != AllowedUseReadOnlyLocate {
|
|
t.Fatalf("selector %q allowed_use = %q, want %q", id, selector.AllowedUse, AllowedUseReadOnlyLocate)
|
|
}
|
|
if selector.Required.AutomationID == "" {
|
|
t.Fatalf("selector %q must require automation_id", id)
|
|
}
|
|
if selector.Required.ControlType == "" {
|
|
t.Fatalf("selector %q must require control_type", id)
|
|
}
|
|
if selector.StabilityScore < 80 {
|
|
t.Fatalf("selector %q stability_score = %d, want >= 80", id, selector.StabilityScore)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestDefaultCatalogDoesNotDescribeActionTools(t *testing.T) {
|
|
for _, selector := range DefaultCatalog() {
|
|
if selector.AllowedUse != AllowedUseReadOnlyLocate {
|
|
t.Fatalf("selector %q allowed_use = %q, want read-only", selector.ID, selector.AllowedUse)
|
|
}
|
|
lower := strings.ToLower(selector.ID + " " + selector.Description)
|
|
for _, forbidden := range []string{"send", "upload", "download", "receive", "conversation", "login", "file_transfer"} {
|
|
if strings.Contains(lower, forbidden) {
|
|
t.Fatalf("selector %q contains action word %q", selector.ID, forbidden)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Run catalog tests and verify they fail**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
go test ./internal/uiaselector -run TestDefaultCatalog -count=1
|
|
```
|
|
|
|
Expected: FAIL because package `internal/uiaselector` or `DefaultCatalog` does not exist.
|
|
|
|
If this environment reports `go` not found, stop execution and record the environment gap. Continue only in a Go 1.23.x environment.
|
|
|
|
- [ ] **Step 3: Implement catalog types and selectors**
|
|
|
|
Create `internal/uiaselector/catalog.go`:
|
|
|
|
```go
|
|
package uiaselector
|
|
|
|
const AllowedUseReadOnlyLocate = "read_only_locate"
|
|
|
|
type Selector struct {
|
|
ID string `json:"id"`
|
|
Description string `json:"description"`
|
|
AllowedUse string `json:"allowed_use"`
|
|
Required Criteria `json:"required"`
|
|
Preferred PreferredCriteria `json:"preferred"`
|
|
Fallback []FallbackCriteria `json:"fallback"`
|
|
StabilityScore int `json:"stability_score"`
|
|
Evidence Evidence `json:"evidence"`
|
|
}
|
|
|
|
type Criteria struct {
|
|
AutomationID string `json:"automation_id,omitempty"`
|
|
ControlType string `json:"control_type,omitempty"`
|
|
FrameworkID string `json:"framework_id,omitempty"`
|
|
ClassName string `json:"class_name,omitempty"`
|
|
ClassPrefix string `json:"class_name_prefix,omitempty"`
|
|
AncestorID string `json:"ancestor_selector,omitempty"`
|
|
}
|
|
|
|
type PreferredCriteria struct {
|
|
ClassName string `json:"class_name,omitempty"`
|
|
ClassPrefix string `json:"class_name_prefix,omitempty"`
|
|
PathHint string `json:"path_hint,omitempty"`
|
|
AncestorID string `json:"ancestor_selector,omitempty"`
|
|
}
|
|
|
|
type FallbackCriteria struct {
|
|
AncestorID string `json:"ancestor_selector,omitempty"`
|
|
ControlType string `json:"control_type,omitempty"`
|
|
ClassPrefix string `json:"class_name_prefix,omitempty"`
|
|
ChildAutomationIDContains []string `json:"child_automation_id_contains,omitempty"`
|
|
ChildControlTypeContains []string `json:"child_control_type_contains,omitempty"`
|
|
}
|
|
|
|
type Evidence struct {
|
|
CaptureID string `json:"capture_id"`
|
|
UIAPath string `json:"uia_path"`
|
|
}
|
|
|
|
func DefaultCatalog() []Selector {
|
|
return []Selector{
|
|
{
|
|
ID: "main_window",
|
|
Description: "iSphere / IMPlatformClient 主窗口",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "frmMain", ControlType: "Window", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{ClassPrefix: "WindowsForms10.Window.8.app.0", PathHint: "root"},
|
|
Fallback: []FallbackCriteria{{ControlType: "Window", ClassPrefix: "WindowsForms10.Window.8.app.0", ChildAutomationIDContains: []string{"panelLeft", "ucPluginTabControl1"}}},
|
|
StabilityScore: 95,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root"},
|
|
},
|
|
{
|
|
ID: "main_plugin_tab",
|
|
Description: "主功能 tab 容器",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "ucPluginTabControl1", ControlType: "Pane", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{PathHint: "root/3", AncestorID: "main_window"},
|
|
Fallback: []FallbackCriteria{{AncestorID: "main_window", ControlType: "Pane", ChildAutomationIDContains: []string{"FrmHomePage", "ucTabPage"}}},
|
|
StabilityScore: 90,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/3"},
|
|
},
|
|
{
|
|
ID: "home_page_window",
|
|
Description: "首页内容窗口",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "FrmHomePage", ControlType: "Window", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{PathHint: "root/3/0/0", AncestorID: "main_plugin_tab"},
|
|
Fallback: []FallbackCriteria{{AncestorID: "main_plugin_tab", ControlType: "Window", ClassPrefix: "WindowsForms10.Window.8.app.0"}},
|
|
StabilityScore: 82,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/3/0/0"},
|
|
},
|
|
{
|
|
ID: "tab_page_content",
|
|
Description: "tab 页内容窗口",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "FrmTabPageContent", ControlType: "Window", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{PathHint: "root/3/1/0", AncestorID: "main_plugin_tab"},
|
|
Fallback: []FallbackCriteria{{AncestorID: "main_plugin_tab", ControlType: "Window", ChildControlTypeContains: []string{"TitleBar"}}},
|
|
StabilityScore: 82,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/3/1/0"},
|
|
},
|
|
{
|
|
ID: "left_panel",
|
|
Description: "左侧主面板",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "panelLeft", ControlType: "Pane", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{PathHint: "root/4", AncestorID: "main_window"},
|
|
Fallback: []FallbackCriteria{{AncestorID: "main_window", ControlType: "Pane", ChildAutomationIDContains: []string{"ucPnlRoster1", "ucFuzzyQuery1"}}},
|
|
StabilityScore: 92,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4"},
|
|
},
|
|
{
|
|
ID: "roster_panel",
|
|
Description: "组织/联系人容器外层",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "ucPnlRoster1", ControlType: "Pane", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{PathHint: "root/4/0/0", AncestorID: "left_panel"},
|
|
Fallback: []FallbackCriteria{{AncestorID: "left_panel", ControlType: "Pane", ChildAutomationIDContains: []string{"ucRoster1"}}},
|
|
StabilityScore: 88,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/0/0"},
|
|
},
|
|
{
|
|
ID: "roster_view",
|
|
Description: "组织/联系人列表区域",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "ucRoster1", ControlType: "Pane", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{PathHint: "root/4/0/0/0", AncestorID: "roster_panel"},
|
|
Fallback: []FallbackCriteria{{AncestorID: "roster_panel", ControlType: "Pane", ClassPrefix: "WindowsForms10.Window.8.app.0"}},
|
|
StabilityScore: 86,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/0/0/0"},
|
|
},
|
|
{
|
|
ID: "fuzzy_query_panel",
|
|
Description: "模糊搜索组件外层",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "ucFuzzyQuery1", ControlType: "Pane", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{PathHint: "root/4/1/0", AncestorID: "left_panel"},
|
|
Fallback: []FallbackCriteria{{AncestorID: "left_panel", ControlType: "Pane", ChildAutomationIDContains: []string{"FrmFuzzyQuery"}}},
|
|
StabilityScore: 90,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/1/0"},
|
|
},
|
|
{
|
|
ID: "fuzzy_query_window",
|
|
Description: "模糊搜索内部窗口",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "FrmFuzzyQuery", ControlType: "Pane", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{PathHint: "root/4/1/0/0", AncestorID: "fuzzy_query_panel"},
|
|
Fallback: []FallbackCriteria{{AncestorID: "fuzzy_query_panel", ControlType: "Pane", ChildAutomationIDContains: []string{"skinAlphaTxt"}}},
|
|
StabilityScore: 88,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/1/0/0"},
|
|
},
|
|
{
|
|
ID: "fuzzy_search_edit",
|
|
Description: "模糊搜索输入框",
|
|
AllowedUse: AllowedUseReadOnlyLocate,
|
|
Required: Criteria{AutomationID: "skinAlphaTxt", ControlType: "Edit", FrameworkID: "WinForm"},
|
|
Preferred: PreferredCriteria{ClassName: "WindowsForms10.EDIT.app.0.d3a00f_r29_ad1", PathHint: "root/4/1/0/0/2", AncestorID: "fuzzy_query_window"},
|
|
Fallback: []FallbackCriteria{{AncestorID: "fuzzy_query_window", ControlType: "Edit", ClassPrefix: "WindowsForms10.EDIT.app.0"}},
|
|
StabilityScore: 93,
|
|
Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/1/0/0/2"},
|
|
},
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run catalog tests and verify they pass**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
go test ./internal/uiaselector -run TestDefaultCatalog -count=1
|
|
```
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 5: Commit catalog**
|
|
|
|
```powershell
|
|
git add internal\uiaselector\catalog.go internal\uiaselector\catalog_test.go
|
|
git commit -m "feat: add N13 UIA selector catalog"
|
|
```
|
|
|
|
## Task 3: Load redacted UIA dump fixture
|
|
|
|
**Files:**
|
|
- Create: `internal/uiaselector/dump_test.go`
|
|
- Create: `internal/uiaselector/dump.go`
|
|
|
|
- [ ] **Step 1: Write failing dump loading tests**
|
|
|
|
Create `internal/uiaselector/dump_test.go`:
|
|
|
|
```go
|
|
package uiaselector
|
|
|
|
import "testing"
|
|
|
|
func TestLoadDumpFixtureParsesAcceptedN12RTree(t *testing.T) {
|
|
root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
|
|
if err != nil {
|
|
t.Fatalf("LoadDumpFile: %v", err)
|
|
}
|
|
if root.AutomationID != "frmMain" {
|
|
t.Fatalf("root automation_id = %q, want frmMain", root.AutomationID)
|
|
}
|
|
if root.ControlType != "Window" {
|
|
t.Fatalf("root control_type = %q, want Window", root.ControlType)
|
|
}
|
|
if got := CountNodes(root); got != 66 {
|
|
t.Fatalf("CountNodes = %d, want 66", got)
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Run dump tests and verify they fail**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
go test ./internal/uiaselector -run TestLoadDumpFixture -count=1
|
|
```
|
|
|
|
Expected: FAIL because `LoadDumpFile` and `CountNodes` do not exist.
|
|
|
|
- [ ] **Step 3: Implement dump loading**
|
|
|
|
Create `internal/uiaselector/dump.go`:
|
|
|
|
```go
|
|
package uiaselector
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
"os"
|
|
)
|
|
|
|
type DumpResponse struct {
|
|
OK bool `json:"ok"`
|
|
Data *DumpData `json:"data"`
|
|
Error any `json:"error"`
|
|
}
|
|
|
|
type DumpData struct {
|
|
Root *Node `json:"root"`
|
|
}
|
|
|
|
type Node struct {
|
|
Name string `json:"name"`
|
|
ControlType string `json:"control_type"`
|
|
AutomationID string `json:"automation_id"`
|
|
ClassName string `json:"class_name"`
|
|
FrameworkID string `json:"framework_id"`
|
|
IsEnabled *bool `json:"is_enabled"`
|
|
IsOffscreen *bool `json:"is_offscreen"`
|
|
Bounds any `json:"bounds"`
|
|
Children []Node `json:"children"`
|
|
}
|
|
|
|
func LoadDumpFile(path string) (*Node, error) {
|
|
content, err := os.ReadFile(path)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
var response DumpResponse
|
|
if err := json.Unmarshal(content, &response); err != nil {
|
|
return nil, err
|
|
}
|
|
if !response.OK {
|
|
return nil, fmt.Errorf("dump response ok=false: %v", response.Error)
|
|
}
|
|
if response.Data == nil || response.Data.Root == nil {
|
|
return nil, fmt.Errorf("dump response has no data.root")
|
|
}
|
|
return response.Data.Root, nil
|
|
}
|
|
|
|
func CountNodes(root *Node) int {
|
|
if root == nil {
|
|
return 0
|
|
}
|
|
count := 1
|
|
for i := range root.Children {
|
|
count += CountNodes(&root.Children[i])
|
|
}
|
|
return count
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run dump tests and verify they pass**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
go test ./internal/uiaselector -run TestLoadDumpFixture -count=1
|
|
```
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 5: Commit dump loader**
|
|
|
|
```powershell
|
|
git add internal\uiaselector\dump.go internal\uiaselector\dump_test.go
|
|
git commit -m "feat: load N13 UIA dump fixture"
|
|
```
|
|
|
|
## Task 4: Implement read-only selector matcher
|
|
|
|
**Files:**
|
|
- Create: `internal/uiaselector/matcher_test.go`
|
|
- Create: `internal/uiaselector/matcher.go`
|
|
|
|
- [ ] **Step 1: Write failing matcher tests**
|
|
|
|
Create `internal/uiaselector/matcher_test.go`:
|
|
|
|
```go
|
|
package uiaselector
|
|
|
|
import "testing"
|
|
|
|
func TestMatchSelectorFindsKnownN12RNodes(t *testing.T) {
|
|
root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
|
|
if err != nil {
|
|
t.Fatalf("LoadDumpFile: %v", err)
|
|
}
|
|
|
|
tests := []struct {
|
|
selectorID string
|
|
path string
|
|
control string
|
|
autoID string
|
|
}{
|
|
{selectorID: "main_window", path: "root", control: "Window", autoID: "frmMain"},
|
|
{selectorID: "left_panel", path: "root/4", control: "Pane", autoID: "panelLeft"},
|
|
{selectorID: "fuzzy_search_edit", path: "root/4/1/0/0/2", control: "Edit", autoID: "skinAlphaTxt"},
|
|
}
|
|
|
|
for _, tt := range tests {
|
|
result := MatchSelector(root, DefaultCatalog(), tt.selectorID)
|
|
if !result.OK {
|
|
t.Fatalf("%s OK=false: %#v", tt.selectorID, result.Error)
|
|
}
|
|
if !result.Matched {
|
|
t.Fatalf("%s matched=false", tt.selectorID)
|
|
}
|
|
if result.MatchCount != 1 {
|
|
t.Fatalf("%s match_count = %d, want 1", tt.selectorID, result.MatchCount)
|
|
}
|
|
match := result.Matches[0]
|
|
if match.Path != tt.path || match.ControlType != tt.control || match.AutomationID != tt.autoID {
|
|
t.Fatalf("%s match = %#v", tt.selectorID, match)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestMatchSelectorDoesNotReturnVisibleText(t *testing.T) {
|
|
root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
|
|
if err != nil {
|
|
t.Fatalf("LoadDumpFile: %v", err)
|
|
}
|
|
result := MatchSelector(root, DefaultCatalog(), "main_window")
|
|
if !result.OK || !result.Matched || len(result.Matches) != 1 {
|
|
t.Fatalf("unexpected result: %#v", result)
|
|
}
|
|
match := result.Matches[0]
|
|
if match.Name != "" {
|
|
t.Fatalf("match returned visible name text %q", match.Name)
|
|
}
|
|
if !match.HasName {
|
|
t.Fatalf("main_window should report has_name=true")
|
|
}
|
|
if match.NameLength == 0 {
|
|
t.Fatalf("main_window should report non-zero name_length")
|
|
}
|
|
}
|
|
|
|
func TestMatchSelectorUnknownSelectorIsStructuredFailure(t *testing.T) {
|
|
root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
|
|
if err != nil {
|
|
t.Fatalf("LoadDumpFile: %v", err)
|
|
}
|
|
result := MatchSelector(root, DefaultCatalog(), "missing_selector")
|
|
if result.OK {
|
|
t.Fatalf("OK=true for missing selector: %#v", result)
|
|
}
|
|
if result.Error.Code != "UNKNOWN_SELECTOR" {
|
|
t.Fatalf("error code = %q, want UNKNOWN_SELECTOR", result.Error.Code)
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Run matcher tests and verify they fail**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
go test ./internal/uiaselector -run TestMatchSelector -count=1
|
|
```
|
|
|
|
Expected: FAIL because `MatchSelector` and result types do not exist.
|
|
|
|
- [ ] **Step 3: Implement matcher**
|
|
|
|
Create `internal/uiaselector/matcher.go`:
|
|
|
|
```go
|
|
package uiaselector
|
|
|
|
import "strings"
|
|
|
|
type MatchResult struct {
|
|
OK bool `json:"ok"`
|
|
SelectorID string `json:"selector_id"`
|
|
Matched bool `json:"matched"`
|
|
MatchCount int `json:"match_count"`
|
|
Matches []NodeMatch `json:"matches"`
|
|
Warnings []string `json:"warnings"`
|
|
Error MatchError `json:"error,omitempty"`
|
|
}
|
|
|
|
type MatchError struct {
|
|
Code string `json:"code"`
|
|
Message string `json:"message"`
|
|
}
|
|
|
|
type NodeMatch struct {
|
|
Path string `json:"path"`
|
|
ControlType string `json:"control_type"`
|
|
AutomationID string `json:"automation_id"`
|
|
ClassName string `json:"class_name"`
|
|
FrameworkID string `json:"framework_id"`
|
|
HasName bool `json:"has_name"`
|
|
NameLength int `json:"name_length"`
|
|
Name string `json:"-"`
|
|
}
|
|
|
|
type nodeRef struct {
|
|
path string
|
|
node *Node
|
|
}
|
|
|
|
func MatchSelector(root *Node, catalog []Selector, selectorID string) MatchResult {
|
|
selector, ok := findSelector(catalog, selectorID)
|
|
if !ok {
|
|
return MatchResult{OK: false, SelectorID: selectorID, Error: MatchError{Code: "UNKNOWN_SELECTOR", Message: "selector is not in catalog"}}
|
|
}
|
|
if selector.AllowedUse != AllowedUseReadOnlyLocate {
|
|
return MatchResult{OK: false, SelectorID: selectorID, Error: MatchError{Code: "INVALID_SELECTOR", Message: "selector is not read-only"}}
|
|
}
|
|
refs := flatten(root)
|
|
matches := matchCriteria(refs, selector.Required)
|
|
if len(matches) == 0 {
|
|
for _, fallback := range selector.Fallback {
|
|
matches = matchFallback(refs, fallback)
|
|
if len(matches) > 0 {
|
|
break
|
|
}
|
|
}
|
|
}
|
|
result := MatchResult{OK: true, SelectorID: selectorID, Matched: len(matches) > 0, MatchCount: len(matches), Warnings: []string{}}
|
|
for _, ref := range matches {
|
|
result.Matches = append(result.Matches, summarizeNode(ref))
|
|
}
|
|
if !result.Matched {
|
|
result.Warnings = append(result.Warnings, "selector_not_found")
|
|
}
|
|
return result
|
|
}
|
|
|
|
func findSelector(catalog []Selector, selectorID string) (Selector, bool) {
|
|
for _, selector := range catalog {
|
|
if selector.ID == selectorID {
|
|
return selector, true
|
|
}
|
|
}
|
|
return Selector{}, false
|
|
}
|
|
|
|
func flatten(root *Node) []nodeRef {
|
|
refs := []nodeRef{}
|
|
var walk func(node *Node, path string)
|
|
walk = func(node *Node, path string) {
|
|
if node == nil {
|
|
return
|
|
}
|
|
refs = append(refs, nodeRef{path: path, node: node})
|
|
for i := range node.Children {
|
|
walk(&node.Children[i], path+"/"+itoa(i))
|
|
}
|
|
}
|
|
walk(root, "root")
|
|
return refs
|
|
}
|
|
|
|
func itoa(value int) string {
|
|
if value == 0 {
|
|
return "0"
|
|
}
|
|
digits := []byte{}
|
|
for value > 0 {
|
|
digits = append([]byte{byte('0' + value%10)}, digits...)
|
|
value = value / 10
|
|
}
|
|
return string(digits)
|
|
}
|
|
|
|
func matchCriteria(refs []nodeRef, criteria Criteria) []nodeRef {
|
|
matches := []nodeRef{}
|
|
for _, ref := range refs {
|
|
if criteria.AutomationID != "" && ref.node.AutomationID != criteria.AutomationID {
|
|
continue
|
|
}
|
|
if criteria.ControlType != "" && ref.node.ControlType != criteria.ControlType {
|
|
continue
|
|
}
|
|
if criteria.FrameworkID != "" && ref.node.FrameworkID != criteria.FrameworkID {
|
|
continue
|
|
}
|
|
if criteria.ClassName != "" && ref.node.ClassName != criteria.ClassName {
|
|
continue
|
|
}
|
|
if criteria.ClassPrefix != "" && !strings.HasPrefix(ref.node.ClassName, criteria.ClassPrefix) {
|
|
continue
|
|
}
|
|
matches = append(matches, ref)
|
|
}
|
|
return matches
|
|
}
|
|
|
|
func matchFallback(refs []nodeRef, fallback FallbackCriteria) []nodeRef {
|
|
matches := []nodeRef{}
|
|
for _, ref := range refs {
|
|
if fallback.ControlType != "" && ref.node.ControlType != fallback.ControlType {
|
|
continue
|
|
}
|
|
if fallback.ClassPrefix != "" && !strings.HasPrefix(ref.node.ClassName, fallback.ClassPrefix) {
|
|
continue
|
|
}
|
|
if !hasChildAutomationIDs(ref.node, fallback.ChildAutomationIDContains) {
|
|
continue
|
|
}
|
|
if !hasChildControlTypes(ref.node, fallback.ChildControlTypeContains) {
|
|
continue
|
|
}
|
|
matches = append(matches, ref)
|
|
}
|
|
return matches
|
|
}
|
|
|
|
func hasChildAutomationIDs(node *Node, ids []string) bool {
|
|
for _, id := range ids {
|
|
found := false
|
|
for i := range node.Children {
|
|
if node.Children[i].AutomationID == id {
|
|
found = true
|
|
break
|
|
}
|
|
}
|
|
if !found {
|
|
return false
|
|
}
|
|
}
|
|
return true
|
|
}
|
|
|
|
func hasChildControlTypes(node *Node, controlTypes []string) bool {
|
|
for _, controlType := range controlTypes {
|
|
found := false
|
|
for i := range node.Children {
|
|
if node.Children[i].ControlType == controlType {
|
|
found = true
|
|
break
|
|
}
|
|
}
|
|
if !found {
|
|
return false
|
|
}
|
|
}
|
|
return true
|
|
}
|
|
|
|
func summarizeNode(ref nodeRef) NodeMatch {
|
|
return NodeMatch{
|
|
Path: ref.path,
|
|
ControlType: ref.node.ControlType,
|
|
AutomationID: ref.node.AutomationID,
|
|
ClassName: ref.node.ClassName,
|
|
FrameworkID: ref.node.FrameworkID,
|
|
HasName: ref.node.Name != "",
|
|
NameLength: len(ref.node.Name),
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run matcher tests and verify they pass**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
go test ./internal/uiaselector -run TestMatchSelector -count=1
|
|
```
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 5: Commit matcher**
|
|
|
|
```powershell
|
|
git add internal\uiaselector\matcher.go internal\uiaselector\matcher_test.go
|
|
git commit -m "feat: match N13 UIA selectors offline"
|
|
```
|
|
|
|
## Task 5: Add verification script
|
|
|
|
**Files:**
|
|
- Create: `scripts/verify-n13-uia-selectors.ps1`
|
|
|
|
- [ ] **Step 1: Create verification script**
|
|
|
|
Create `scripts/verify-n13-uia-selectors.ps1`:
|
|
|
|
```powershell
|
|
$ErrorActionPreference = "Stop"
|
|
Set-StrictMode -Version Latest
|
|
|
|
$repo = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..")).Path
|
|
Set-Location -LiteralPath $repo
|
|
|
|
if (-not (Get-Command go -ErrorAction SilentlyContinue)) {
|
|
throw "go was not found. Run this verifier in a Go 1.23.x environment."
|
|
}
|
|
|
|
Write-Host "## go test ./internal/uiaselector"
|
|
go test ./internal/uiaselector -count=1
|
|
if ($LASTEXITCODE -ne 0) {
|
|
throw "go test ./internal/uiaselector failed with exit code $LASTEXITCODE"
|
|
}
|
|
|
|
Write-Host "## boundary grep"
|
|
$forbidden = @(
|
|
"send_after_approval",
|
|
"send_file_after_approval",
|
|
"receive_file_after_approval",
|
|
"open_conversation",
|
|
"ValuePattern.SetValue",
|
|
"InvokePattern"
|
|
)
|
|
$files = @(
|
|
"internal\uiaselector\catalog.go",
|
|
"internal\uiaselector\matcher.go"
|
|
)
|
|
foreach ($term in $forbidden) {
|
|
$hit = Select-String -LiteralPath $files -Pattern $term -SimpleMatch -ErrorAction SilentlyContinue
|
|
if ($hit) {
|
|
throw "forbidden action term found in selector implementation: $term"
|
|
}
|
|
}
|
|
|
|
Write-Host "N13 UIA selector verification passed."
|
|
```
|
|
|
|
- [ ] **Step 2: Run script and verify it passes**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n13-uia-selectors.ps1
|
|
```
|
|
|
|
Expected in Go-enabled environment:
|
|
|
|
```text
|
|
## go test ./internal/uiaselector
|
|
ok isphere-ai-bridge/internal/uiaselector
|
|
## boundary grep
|
|
N13 UIA selector verification passed.
|
|
```
|
|
|
|
If `go` is unavailable, expected failure is:
|
|
|
|
```text
|
|
go was not found. Run this verifier in a Go 1.23.x environment.
|
|
```
|
|
|
|
- [ ] **Step 3: Commit verification script**
|
|
|
|
```powershell
|
|
git add scripts\verify-n13-uia-selectors.ps1
|
|
git commit -m "test: add N13 selector verifier"
|
|
```
|
|
|
|
## Task 6: Final verification and checkpoint
|
|
|
|
**Files:**
|
|
- No new files.
|
|
- Verify: `internal/uiaselector/*`, `scripts/verify-n13-uia-selectors.ps1`
|
|
|
|
- [ ] **Step 1: Run package tests**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
go test ./internal/uiaselector -count=1
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
ok isphere-ai-bridge/internal/uiaselector
|
|
```
|
|
|
|
- [ ] **Step 2: Run repository tests if Go is available**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
go test ./...
|
|
```
|
|
|
|
Expected: PASS for all packages.
|
|
|
|
If the environment lacks Go, do not mark implementation complete. Move to a Go 1.23.x environment and rerun.
|
|
|
|
- [ ] **Step 3: Run N13 verifier**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n13-uia-selectors.ps1
|
|
```
|
|
|
|
Expected: PASS in Go-enabled environment.
|
|
|
|
- [ ] **Step 4: Check no MCP/helper action surface changed**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
git diff --name-only origin/main..HEAD
|
|
```
|
|
|
|
Expected changed implementation files are limited to:
|
|
|
|
```text
|
|
internal/uiaselector/catalog.go
|
|
internal/uiaselector/catalog_test.go
|
|
internal/uiaselector/dump.go
|
|
internal/uiaselector/dump_test.go
|
|
internal/uiaselector/matcher.go
|
|
internal/uiaselector/matcher_test.go
|
|
internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json
|
|
scripts/verify-n13-uia-selectors.ps1
|
|
```
|
|
|
|
If `native/ISphereWinHelper/Program.cs`, `internal/tools/winhelper.go`, or `cmd/isphere-mcp/main.go` appears, stop and review because N13 is not supposed to add live tools.
|
|
|
|
- [ ] **Step 5: Commit final checkpoint if needed**
|
|
|
|
If any verification-only documentation changes were made, run:
|
|
|
|
```powershell
|
|
git add <changed verification docs>
|
|
git commit -m "docs: record N13 selector verification"
|
|
```
|
|
|
|
If no files changed, do not create an empty commit.
|
|
|
|
## Self-review checklist
|
|
|
|
- Spec coverage: this plan covers selector catalog, fixture loading, read-only matcher, no visible text in result, and boundary verification.
|
|
- Boundary: no helper op, no MCP tool, no click/input/send/file/conversation behavior.
|
|
- TDD: every implementation task starts with failing tests before code.
|
|
- Commit cadence: fixture, catalog, loader, matcher, verifier are separate commits.
|
|
- Environment caveat: Go 1.23.x is required for implementation verification.
|