Files

石头 5b5d33140b docs: plan N13 UIA selector catalog implementation

2026-07-09 14:57:51 +08:00

28 KiB

Raw Blame History

N13 UIA Selector Catalog Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build an offline, read-only N13 UIA selector catalog and matcher that can validate selectors against the accepted N12R UIA dump without adding live helper or MCP actions.

Architecture: Add a focused Go package internal/uiaselector with selector data types, a static first-pass catalog, UIA dump loading, and pure in-memory matching. Tests use a redacted N12R fixture and assert that matching returns structural node summaries without visible text.

Tech Stack: Go 1.23.x, standard library encoding/json, existing repository test style, PowerShell verification wrapper.

Scope and boundary

This plan implements only offline selector catalog and matching.

It does not modify these files:

native/ISphereWinHelper/Program.cs
internal/tools/winhelper.go
cmd/isphere-mcp/main.go

It does not add MCP tools or helper ops. It does not click, type, open conversations, send messages, send files, receive files, or read chat content.

The accepted N12R evidence used by tests is:

runs/internal-sandbox-live-capture-intake/2026-07-09-retry-2/2026-07-08-internal-sandbox-a/uia/dump-uia-main-redacted.json

File structure

Create these files:

internal/uiaselector/catalog.go
internal/uiaselector/catalog_test.go
internal/uiaselector/dump.go
internal/uiaselector/dump_test.go
internal/uiaselector/matcher.go
internal/uiaselector/matcher_test.go
internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json
scripts/verify-n13-uia-selectors.ps1

Responsibilities:

catalog.go: selector data structures and initial N13 selector catalog.
catalog_test.go: catalog completeness and boundary tests.
dump.go: UIA dump response and node loading helpers.
dump_test.go: fixture parse tests.
matcher.go: pure in-memory selector matcher.
matcher_test.go: matching behavior and no-visible-text result tests.
testdata/n12r-2026-07-09-uia-redacted.json: redacted accepted N12R UIA fixture copied from the ignored run artifact.
scripts/verify-n13-uia-selectors.ps1: convenience verification script for Go-enabled environments.

Task 1: Add N12R redacted UIA fixture

Files:

Create: internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json
Step 1: Create package directory and copy the accepted redacted fixture

Run:

New-Item -ItemType Directory -Force -Path internal\uiaselector\testdata | Out-Null
Copy-Item `
  -LiteralPath runs\internal-sandbox-live-capture-intake\2026-07-09-retry-2\2026-07-08-internal-sandbox-a\uia\dump-uia-main-redacted.json `
  -Destination internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json `
  -Force

Expected: internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json exists.

Step 2: Verify fixture parses as JSON

Run:

Get-Content internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json -Encoding UTF8 | ConvertFrom-Json | Out-Null

Expected: command exits 0.

Step 3: Commit fixture

git add internal\uiaselector\testdata\n12r-2026-07-09-uia-redacted.json
git commit -m "test: add N13 redacted UIA fixture"

Task 2: Define selector catalog types and initial catalog

Files:

Create: internal/uiaselector/catalog_test.go
Create: internal/uiaselector/catalog.go
Step 1: Write failing catalog tests

Create internal/uiaselector/catalog_test.go:

package uiaselector

import (
	"strings"
	"testing"
)

func TestDefaultCatalogContainsInitialN13Selectors(t *testing.T) {
	catalog := DefaultCatalog()
	wantIDs := []string{
		"main_window",
		"main_plugin_tab",
		"home_page_window",
		"tab_page_content",
		"left_panel",
		"roster_panel",
		"roster_view",
		"fuzzy_query_panel",
		"fuzzy_query_window",
		"fuzzy_search_edit",
	}

	byID := map[string]Selector{}
	for _, selector := range catalog {
		byID[selector.ID] = selector
	}

	for _, id := range wantIDs {
		selector, ok := byID[id]
		if !ok {
			t.Fatalf("DefaultCatalog missing selector %q", id)
		}
		if selector.AllowedUse != AllowedUseReadOnlyLocate {
			t.Fatalf("selector %q allowed_use = %q, want %q", id, selector.AllowedUse, AllowedUseReadOnlyLocate)
		}
		if selector.Required.AutomationID == "" {
			t.Fatalf("selector %q must require automation_id", id)
		}
		if selector.Required.ControlType == "" {
			t.Fatalf("selector %q must require control_type", id)
		}
		if selector.StabilityScore < 80 {
			t.Fatalf("selector %q stability_score = %d, want >= 80", id, selector.StabilityScore)
		}
	}
}

func TestDefaultCatalogDoesNotDescribeActionTools(t *testing.T) {
	for _, selector := range DefaultCatalog() {
		if selector.AllowedUse != AllowedUseReadOnlyLocate {
			t.Fatalf("selector %q allowed_use = %q, want read-only", selector.ID, selector.AllowedUse)
		}
		lower := strings.ToLower(selector.ID + " " + selector.Description)
		for _, forbidden := range []string{"send", "upload", "download", "receive", "conversation", "login", "file_transfer"} {
			if strings.Contains(lower, forbidden) {
				t.Fatalf("selector %q contains action word %q", selector.ID, forbidden)
			}
		}
	}
}

Step 2: Run catalog tests and verify they fail

Run:

go test ./internal/uiaselector -run TestDefaultCatalog -count=1

Expected: FAIL because package internal/uiaselector or DefaultCatalog does not exist.

If this environment reports go not found, stop execution and record the environment gap. Continue only in a Go 1.23.x environment.

Step 3: Implement catalog types and selectors

Create internal/uiaselector/catalog.go:

package uiaselector

const AllowedUseReadOnlyLocate = "read_only_locate"

type Selector struct {
	ID             string             `json:"id"`
	Description    string             `json:"description"`
	AllowedUse     string             `json:"allowed_use"`
	Required       Criteria           `json:"required"`
	Preferred      PreferredCriteria  `json:"preferred"`
	Fallback       []FallbackCriteria `json:"fallback"`
	StabilityScore int                `json:"stability_score"`
	Evidence       Evidence           `json:"evidence"`
}

type Criteria struct {
	AutomationID  string `json:"automation_id,omitempty"`
	ControlType   string `json:"control_type,omitempty"`
	FrameworkID   string `json:"framework_id,omitempty"`
	ClassName     string `json:"class_name,omitempty"`
	ClassPrefix   string `json:"class_name_prefix,omitempty"`
	AncestorID    string `json:"ancestor_selector,omitempty"`
}

type PreferredCriteria struct {
	ClassName    string `json:"class_name,omitempty"`
	ClassPrefix  string `json:"class_name_prefix,omitempty"`
	PathHint     string `json:"path_hint,omitempty"`
	AncestorID   string `json:"ancestor_selector,omitempty"`
}

type FallbackCriteria struct {
	AncestorID                  string   `json:"ancestor_selector,omitempty"`
	ControlType                 string   `json:"control_type,omitempty"`
	ClassPrefix                 string   `json:"class_name_prefix,omitempty"`
	ChildAutomationIDContains   []string `json:"child_automation_id_contains,omitempty"`
	ChildControlTypeContains    []string `json:"child_control_type_contains,omitempty"`
}

type Evidence struct {
	CaptureID string `json:"capture_id"`
	UIAPath   string `json:"uia_path"`
}

func DefaultCatalog() []Selector {
	return []Selector{
		{
			ID:          "main_window",
			Description: "iSphere / IMPlatformClient 主窗口",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "frmMain", ControlType: "Window", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{ClassPrefix: "WindowsForms10.Window.8.app.0", PathHint: "root"},
			Fallback: []FallbackCriteria{{ControlType: "Window", ClassPrefix: "WindowsForms10.Window.8.app.0", ChildAutomationIDContains: []string{"panelLeft", "ucPluginTabControl1"}}},
			StabilityScore: 95,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root"},
		},
		{
			ID:          "main_plugin_tab",
			Description: "主功能 tab 容器",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "ucPluginTabControl1", ControlType: "Pane", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{PathHint: "root/3", AncestorID: "main_window"},
			Fallback: []FallbackCriteria{{AncestorID: "main_window", ControlType: "Pane", ChildAutomationIDContains: []string{"FrmHomePage", "ucTabPage"}}},
			StabilityScore: 90,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/3"},
		},
		{
			ID:          "home_page_window",
			Description: "首页内容窗口",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "FrmHomePage", ControlType: "Window", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{PathHint: "root/3/0/0", AncestorID: "main_plugin_tab"},
			Fallback: []FallbackCriteria{{AncestorID: "main_plugin_tab", ControlType: "Window", ClassPrefix: "WindowsForms10.Window.8.app.0"}},
			StabilityScore: 82,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/3/0/0"},
		},
		{
			ID:          "tab_page_content",
			Description: "tab 页内容窗口",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "FrmTabPageContent", ControlType: "Window", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{PathHint: "root/3/1/0", AncestorID: "main_plugin_tab"},
			Fallback: []FallbackCriteria{{AncestorID: "main_plugin_tab", ControlType: "Window", ChildControlTypeContains: []string{"TitleBar"}}},
			StabilityScore: 82,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/3/1/0"},
		},
		{
			ID:          "left_panel",
			Description: "左侧主面板",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "panelLeft", ControlType: "Pane", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{PathHint: "root/4", AncestorID: "main_window"},
			Fallback: []FallbackCriteria{{AncestorID: "main_window", ControlType: "Pane", ChildAutomationIDContains: []string{"ucPnlRoster1", "ucFuzzyQuery1"}}},
			StabilityScore: 92,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4"},
		},
		{
			ID:          "roster_panel",
			Description: "组织/联系人容器外层",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "ucPnlRoster1", ControlType: "Pane", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{PathHint: "root/4/0/0", AncestorID: "left_panel"},
			Fallback: []FallbackCriteria{{AncestorID: "left_panel", ControlType: "Pane", ChildAutomationIDContains: []string{"ucRoster1"}}},
			StabilityScore: 88,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/0/0"},
		},
		{
			ID:          "roster_view",
			Description: "组织/联系人列表区域",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "ucRoster1", ControlType: "Pane", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{PathHint: "root/4/0/0/0", AncestorID: "roster_panel"},
			Fallback: []FallbackCriteria{{AncestorID: "roster_panel", ControlType: "Pane", ClassPrefix: "WindowsForms10.Window.8.app.0"}},
			StabilityScore: 86,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/0/0/0"},
		},
		{
			ID:          "fuzzy_query_panel",
			Description: "模糊搜索组件外层",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "ucFuzzyQuery1", ControlType: "Pane", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{PathHint: "root/4/1/0", AncestorID: "left_panel"},
			Fallback: []FallbackCriteria{{AncestorID: "left_panel", ControlType: "Pane", ChildAutomationIDContains: []string{"FrmFuzzyQuery"}}},
			StabilityScore: 90,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/1/0"},
		},
		{
			ID:          "fuzzy_query_window",
			Description: "模糊搜索内部窗口",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "FrmFuzzyQuery", ControlType: "Pane", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{PathHint: "root/4/1/0/0", AncestorID: "fuzzy_query_panel"},
			Fallback: []FallbackCriteria{{AncestorID: "fuzzy_query_panel", ControlType: "Pane", ChildAutomationIDContains: []string{"skinAlphaTxt"}}},
			StabilityScore: 88,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/1/0/0"},
		},
		{
			ID:          "fuzzy_search_edit",
			Description: "模糊搜索输入框",
			AllowedUse:  AllowedUseReadOnlyLocate,
			Required: Criteria{AutomationID: "skinAlphaTxt", ControlType: "Edit", FrameworkID: "WinForm"},
			Preferred: PreferredCriteria{ClassName: "WindowsForms10.EDIT.app.0.d3a00f_r29_ad1", PathHint: "root/4/1/0/0/2", AncestorID: "fuzzy_query_window"},
			Fallback: []FallbackCriteria{{AncestorID: "fuzzy_query_window", ControlType: "Edit", ClassPrefix: "WindowsForms10.EDIT.app.0"}},
			StabilityScore: 93,
			Evidence: Evidence{CaptureID: "2026-07-08-internal-sandbox-a", UIAPath: "root/4/1/0/0/2"},
		},
	}
}

Step 4: Run catalog tests and verify they pass

Run:

go test ./internal/uiaselector -run TestDefaultCatalog -count=1

Expected: PASS.

Step 5: Commit catalog

git add internal\uiaselector\catalog.go internal\uiaselector\catalog_test.go
git commit -m "feat: add N13 UIA selector catalog"

Task 3: Load redacted UIA dump fixture

Files:

Create: internal/uiaselector/dump_test.go
Create: internal/uiaselector/dump.go
Step 1: Write failing dump loading tests

Create internal/uiaselector/dump_test.go:

package uiaselector

import "testing"

func TestLoadDumpFixtureParsesAcceptedN12RTree(t *testing.T) {
	root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
	if err != nil {
		t.Fatalf("LoadDumpFile: %v", err)
	}
	if root.AutomationID != "frmMain" {
		t.Fatalf("root automation_id = %q, want frmMain", root.AutomationID)
	}
	if root.ControlType != "Window" {
		t.Fatalf("root control_type = %q, want Window", root.ControlType)
	}
	if got := CountNodes(root); got != 66 {
		t.Fatalf("CountNodes = %d, want 66", got)
	}
}

Step 2: Run dump tests and verify they fail

Run:

go test ./internal/uiaselector -run TestLoadDumpFixture -count=1

Expected: FAIL because LoadDumpFile and CountNodes do not exist.

Step 3: Implement dump loading

Create internal/uiaselector/dump.go:

package uiaselector

import (
	"encoding/json"
	"fmt"
	"os"
)

type DumpResponse struct {
	OK    bool      `json:"ok"`
	Data  *DumpData `json:"data"`
	Error any       `json:"error"`
}

type DumpData struct {
	Root *Node `json:"root"`
}

type Node struct {
	Name         string `json:"name"`
	ControlType  string `json:"control_type"`
	AutomationID string `json:"automation_id"`
	ClassName    string `json:"class_name"`
	FrameworkID  string `json:"framework_id"`
	IsEnabled    *bool  `json:"is_enabled"`
	IsOffscreen  *bool  `json:"is_offscreen"`
	Bounds       any    `json:"bounds"`
	Children     []Node `json:"children"`
}

func LoadDumpFile(path string) (*Node, error) {
	content, err := os.ReadFile(path)
	if err != nil {
		return nil, err
	}
	var response DumpResponse
	if err := json.Unmarshal(content, &response); err != nil {
		return nil, err
	}
	if !response.OK {
		return nil, fmt.Errorf("dump response ok=false: %v", response.Error)
	}
	if response.Data == nil || response.Data.Root == nil {
		return nil, fmt.Errorf("dump response has no data.root")
	}
	return response.Data.Root, nil
}

func CountNodes(root *Node) int {
	if root == nil {
		return 0
	}
	count := 1
	for i := range root.Children {
		count += CountNodes(&root.Children[i])
	}
	return count
}

Step 4: Run dump tests and verify they pass

Run:

go test ./internal/uiaselector -run TestLoadDumpFixture -count=1

Expected: PASS.

Step 5: Commit dump loader

git add internal\uiaselector\dump.go internal\uiaselector\dump_test.go
git commit -m "feat: load N13 UIA dump fixture"

Task 4: Implement read-only selector matcher

Files:

Create: internal/uiaselector/matcher_test.go
Create: internal/uiaselector/matcher.go
Step 1: Write failing matcher tests

Create internal/uiaselector/matcher_test.go:

package uiaselector

import "testing"

func TestMatchSelectorFindsKnownN12RNodes(t *testing.T) {
	root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
	if err != nil {
		t.Fatalf("LoadDumpFile: %v", err)
	}

	tests := []struct {
		selectorID string
		path       string
		control    string
		autoID     string
	}{
		{selectorID: "main_window", path: "root", control: "Window", autoID: "frmMain"},
		{selectorID: "left_panel", path: "root/4", control: "Pane", autoID: "panelLeft"},
		{selectorID: "fuzzy_search_edit", path: "root/4/1/0/0/2", control: "Edit", autoID: "skinAlphaTxt"},
	}

	for _, tt := range tests {
		result := MatchSelector(root, DefaultCatalog(), tt.selectorID)
		if !result.OK {
			t.Fatalf("%s OK=false: %#v", tt.selectorID, result.Error)
		}
		if !result.Matched {
			t.Fatalf("%s matched=false", tt.selectorID)
		}
		if result.MatchCount != 1 {
			t.Fatalf("%s match_count = %d, want 1", tt.selectorID, result.MatchCount)
		}
		match := result.Matches[0]
		if match.Path != tt.path || match.ControlType != tt.control || match.AutomationID != tt.autoID {
			t.Fatalf("%s match = %#v", tt.selectorID, match)
		}
	}
}

func TestMatchSelectorDoesNotReturnVisibleText(t *testing.T) {
	root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
	if err != nil {
		t.Fatalf("LoadDumpFile: %v", err)
	}
	result := MatchSelector(root, DefaultCatalog(), "main_window")
	if !result.OK || !result.Matched || len(result.Matches) != 1 {
		t.Fatalf("unexpected result: %#v", result)
	}
	match := result.Matches[0]
	if match.Name != "" {
		t.Fatalf("match returned visible name text %q", match.Name)
	}
	if !match.HasName {
		t.Fatalf("main_window should report has_name=true")
	}
	if match.NameLength == 0 {
		t.Fatalf("main_window should report non-zero name_length")
	}
}

func TestMatchSelectorUnknownSelectorIsStructuredFailure(t *testing.T) {
	root, err := LoadDumpFile("testdata/n12r-2026-07-09-uia-redacted.json")
	if err != nil {
		t.Fatalf("LoadDumpFile: %v", err)
	}
	result := MatchSelector(root, DefaultCatalog(), "missing_selector")
	if result.OK {
		t.Fatalf("OK=true for missing selector: %#v", result)
	}
	if result.Error.Code != "UNKNOWN_SELECTOR" {
		t.Fatalf("error code = %q, want UNKNOWN_SELECTOR", result.Error.Code)
	}
}

Step 2: Run matcher tests and verify they fail

Run:

go test ./internal/uiaselector -run TestMatchSelector -count=1

Expected: FAIL because MatchSelector and result types do not exist.

Step 3: Implement matcher

Create internal/uiaselector/matcher.go:

package uiaselector

import "strings"

type MatchResult struct {
	OK         bool        `json:"ok"`
	SelectorID string      `json:"selector_id"`
	Matched    bool        `json:"matched"`
	MatchCount int         `json:"match_count"`
	Matches    []NodeMatch `json:"matches"`
	Warnings   []string    `json:"warnings"`
	Error      MatchError  `json:"error,omitempty"`
}

type MatchError struct {
	Code    string `json:"code"`
	Message string `json:"message"`
}

type NodeMatch struct {
	Path         string `json:"path"`
	ControlType  string `json:"control_type"`
	AutomationID string `json:"automation_id"`
	ClassName    string `json:"class_name"`
	FrameworkID  string `json:"framework_id"`
	HasName      bool   `json:"has_name"`
	NameLength   int    `json:"name_length"`
	Name         string `json:"-"`
}

type nodeRef struct {
	path string
	node *Node
}

func MatchSelector(root *Node, catalog []Selector, selectorID string) MatchResult {
	selector, ok := findSelector(catalog, selectorID)
	if !ok {
		return MatchResult{OK: false, SelectorID: selectorID, Error: MatchError{Code: "UNKNOWN_SELECTOR", Message: "selector is not in catalog"}}
	}
	if selector.AllowedUse != AllowedUseReadOnlyLocate {
		return MatchResult{OK: false, SelectorID: selectorID, Error: MatchError{Code: "INVALID_SELECTOR", Message: "selector is not read-only"}}
	}
	refs := flatten(root)
	matches := matchCriteria(refs, selector.Required)
	if len(matches) == 0 {
		for _, fallback := range selector.Fallback {
			matches = matchFallback(refs, fallback)
			if len(matches) > 0 {
				break
			}
		}
	}
	result := MatchResult{OK: true, SelectorID: selectorID, Matched: len(matches) > 0, MatchCount: len(matches), Warnings: []string{}}
	for _, ref := range matches {
		result.Matches = append(result.Matches, summarizeNode(ref))
	}
	if !result.Matched {
		result.Warnings = append(result.Warnings, "selector_not_found")
	}
	return result
}

func findSelector(catalog []Selector, selectorID string) (Selector, bool) {
	for _, selector := range catalog {
		if selector.ID == selectorID {
			return selector, true
		}
	}
	return Selector{}, false
}

func flatten(root *Node) []nodeRef {
	refs := []nodeRef{}
	var walk func(node *Node, path string)
	walk = func(node *Node, path string) {
		if node == nil {
			return
		}
		refs = append(refs, nodeRef{path: path, node: node})
		for i := range node.Children {
			walk(&node.Children[i], path+"/"+itoa(i))
		}
	}
	walk(root, "root")
	return refs
}

func itoa(value int) string {
	if value == 0 {
		return "0"
	}
	digits := []byte{}
	for value > 0 {
		digits = append([]byte{byte('0' + value%10)}, digits...)
		value = value / 10
	}
	return string(digits)
}

func matchCriteria(refs []nodeRef, criteria Criteria) []nodeRef {
	matches := []nodeRef{}
	for _, ref := range refs {
		if criteria.AutomationID != "" && ref.node.AutomationID != criteria.AutomationID {
			continue
		}
		if criteria.ControlType != "" && ref.node.ControlType != criteria.ControlType {
			continue
		}
		if criteria.FrameworkID != "" && ref.node.FrameworkID != criteria.FrameworkID {
			continue
		}
		if criteria.ClassName != "" && ref.node.ClassName != criteria.ClassName {
			continue
		}
		if criteria.ClassPrefix != "" && !strings.HasPrefix(ref.node.ClassName, criteria.ClassPrefix) {
			continue
		}
		matches = append(matches, ref)
	}
	return matches
}

func matchFallback(refs []nodeRef, fallback FallbackCriteria) []nodeRef {
	matches := []nodeRef{}
	for _, ref := range refs {
		if fallback.ControlType != "" && ref.node.ControlType != fallback.ControlType {
			continue
		}
		if fallback.ClassPrefix != "" && !strings.HasPrefix(ref.node.ClassName, fallback.ClassPrefix) {
			continue
		}
		if !hasChildAutomationIDs(ref.node, fallback.ChildAutomationIDContains) {
			continue
		}
		if !hasChildControlTypes(ref.node, fallback.ChildControlTypeContains) {
			continue
		}
		matches = append(matches, ref)
	}
	return matches
}

func hasChildAutomationIDs(node *Node, ids []string) bool {
	for _, id := range ids {
		found := false
		for i := range node.Children {
			if node.Children[i].AutomationID == id {
				found = true
				break
			}
		}
		if !found {
			return false
		}
	}
	return true
}

func hasChildControlTypes(node *Node, controlTypes []string) bool {
	for _, controlType := range controlTypes {
		found := false
		for i := range node.Children {
			if node.Children[i].ControlType == controlType {
				found = true
				break
			}
		}
		if !found {
			return false
		}
	}
	return true
}

func summarizeNode(ref nodeRef) NodeMatch {
	return NodeMatch{
		Path:         ref.path,
		ControlType:  ref.node.ControlType,
		AutomationID: ref.node.AutomationID,
		ClassName:    ref.node.ClassName,
		FrameworkID:  ref.node.FrameworkID,
		HasName:      ref.node.Name != "",
		NameLength:   len(ref.node.Name),
	}
}

Step 4: Run matcher tests and verify they pass

Run:

go test ./internal/uiaselector -run TestMatchSelector -count=1

Expected: PASS.

Step 5: Commit matcher

git add internal\uiaselector\matcher.go internal\uiaselector\matcher_test.go
git commit -m "feat: match N13 UIA selectors offline"

Task 5: Add verification script

Files:

Create: scripts/verify-n13-uia-selectors.ps1
Step 1: Create verification script

Create scripts/verify-n13-uia-selectors.ps1:

$ErrorActionPreference = "Stop"
Set-StrictMode -Version Latest

$repo = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..")).Path
Set-Location -LiteralPath $repo

if (-not (Get-Command go -ErrorAction SilentlyContinue)) {
    throw "go was not found. Run this verifier in a Go 1.23.x environment."
}

Write-Host "## go test ./internal/uiaselector"
go test ./internal/uiaselector -count=1
if ($LASTEXITCODE -ne 0) {
    throw "go test ./internal/uiaselector failed with exit code $LASTEXITCODE"
}

Write-Host "## boundary grep"
$forbidden = @(
    "send_after_approval",
    "send_file_after_approval",
    "receive_file_after_approval",
    "open_conversation",
    "ValuePattern.SetValue",
    "InvokePattern"
)
$files = @(
    "internal\uiaselector\catalog.go",
    "internal\uiaselector\matcher.go"
)
foreach ($term in $forbidden) {
    $hit = Select-String -LiteralPath $files -Pattern $term -SimpleMatch -ErrorAction SilentlyContinue
    if ($hit) {
        throw "forbidden action term found in selector implementation: $term"
    }
}

Write-Host "N13 UIA selector verification passed."

Step 2: Run script and verify it passes

Run:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n13-uia-selectors.ps1

Expected in Go-enabled environment:

## go test ./internal/uiaselector
ok   isphere-ai-bridge/internal/uiaselector
## boundary grep
N13 UIA selector verification passed.

If go is unavailable, expected failure is:

go was not found. Run this verifier in a Go 1.23.x environment.

Step 3: Commit verification script

git add scripts\verify-n13-uia-selectors.ps1
git commit -m "test: add N13 selector verifier"

Task 6: Final verification and checkpoint

Files:

No new files.
Verify: internal/uiaselector/*, scripts/verify-n13-uia-selectors.ps1
Step 1: Run package tests

Run:

go test ./internal/uiaselector -count=1

Expected:

ok   isphere-ai-bridge/internal/uiaselector

Step 2: Run repository tests if Go is available

Run:

go test ./...

Expected: PASS for all packages.

If the environment lacks Go, do not mark implementation complete. Move to a Go 1.23.x environment and rerun.

Step 3: Run N13 verifier

Run:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n13-uia-selectors.ps1

Expected: PASS in Go-enabled environment.

Step 4: Check no MCP/helper action surface changed

Run:

git diff --name-only origin/main..HEAD

Expected changed implementation files are limited to:

internal/uiaselector/catalog.go
internal/uiaselector/catalog_test.go
internal/uiaselector/dump.go
internal/uiaselector/dump_test.go
internal/uiaselector/matcher.go
internal/uiaselector/matcher_test.go
internal/uiaselector/testdata/n12r-2026-07-09-uia-redacted.json
scripts/verify-n13-uia-selectors.ps1

If native/ISphereWinHelper/Program.cs, internal/tools/winhelper.go, or cmd/isphere-mcp/main.go appears, stop and review because N13 is not supposed to add live tools.

Step 5: Commit final checkpoint if needed

If any verification-only documentation changes were made, run:

git add <changed verification docs>
git commit -m "docs: record N13 selector verification"

If no files changed, do not create an empty commit.

Self-review checklist

Spec coverage: this plan covers selector catalog, fixture loading, read-only matcher, no visible text in result, and boundary verification.
Boundary: no helper op, no MCP tool, no click/input/send/file/conversation behavior.
TDD: every implementation task starts with failing tests before code.
Commit cadence: fixture, catalog, loader, matcher, verifier are separate commits.
Environment caveat: Go 1.23.x is required for implementation verification.

28 KiB Raw Blame History

N13 UIA Selector Catalog Implementation Plan

Scope and boundary

File structure

Task 1: Add N12R redacted UIA fixture

Task 2: Define selector catalog types and initial catalog

Task 3: Load redacted UIA dump fixture

Task 4: Implement read-only selector matcher

Task 5: Add verification script

Task 6: Final verification and checkpoint

Self-review checklist

28 KiB

Raw Blame History