Files

石头 d85fae0852 docs: plan N15 selector report hardening

2026-07-09 17:06:50 +08:00

13 KiB

Raw Blame History

N15 Selector Report Hardening Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Harden the N14 offline selector report tests and verifier so ambiguous strict-mode behavior and report-output safety regressions are caught before merge.

Architecture: Keep N15 as a test/verifier-only change. Add an in-memory ambiguous-match test to internal/uiaselector/report_test.go, then strengthen scripts/verify-n14-uia-selector-report.ps1 with stale-output cleanup, file/non-empty checks, and JSON match-field allowlist validation.

Tech Stack: Go 1.23.x tests, existing internal/uiaselector package, Windows PowerShell 5.1-compatible verifier script, standard go test and repository Git workflow.

Scope and boundary

N15 changes only test and verifier coverage.

Modify only:

internal/uiaselector/report_test.go
scripts/verify-n14-uia-selector-report.ps1

Do not modify:

native/ISphereWinHelper/Program.cs
internal/tools/winhelper.go
cmd/isphere-mcp/main.go
cmd/uia-selector-report/main.go
internal/uiaselector/report.go
internal/uiaselector/catalog.go
internal/uiaselector/matcher.go

Do not add selectors, helper ops, MCP tools, CLI arguments, live hwnd/process handling, UI automation calls, clicks, keyboard input, clipboard operations, conversation actions, message sending, file transfer, or visible text export.

File structure

internal/uiaselector/report_test.go
  Add one explicit ambiguous strict-mode regression test using an in-memory UIA tree.

scripts/verify-n14-uia-selector-report.ps1
  Clean stale outputs before CLI execution, require fresh non-empty report files, and validate JSON match fields with an allowlist.

Task 1: Cover ambiguous strict-mode classification

Files:

Modify: internal/uiaselector/report_test.go
Step 1: Add the ambiguous strict-mode regression test

Append this test to internal/uiaselector/report_test.go:

func TestStrictModeClassifiesAmbiguousMatches(t *testing.T) {
	root := &Node{
		ControlType:  "Window",
		AutomationID: "frmMain",
		FrameworkID:  "WinForm",
		Children: []Node{
			{ControlType: "Pane", AutomationID: "duplicate", FrameworkID: "WinForm"},
			{ControlType: "Pane", AutomationID: "duplicate", FrameworkID: "WinForm"},
		},
	}
	catalog := []Selector{
		{
			ID:             "duplicate_panel",
			Description:    "duplicate structural selector",
			AllowedUse:     AllowedUseReadOnlyLocate,
			Required:       Criteria{AutomationID: "duplicate", ControlType: "Pane", FrameworkID: "WinForm"},
			StabilityScore: 80,
			Evidence:       Evidence{CaptureID: "unit-test", UIAPath: "root/duplicate"},
		},
	}

	report := RunCatalogReport("synthetic-ambiguous.json", root, catalog, fixedReportTime())
	if !report.OK {
		t.Fatalf("report OK=false: %#v", report.Error)
	}
	if report.CatalogSize != 1 {
		t.Fatalf("catalog_size = %d, want 1", report.CatalogSize)
	}
	if report.MatchedCount != 1 {
		t.Fatalf("matched_count = %d, want 1", report.MatchedCount)
	}
	if report.UnmatchedCount != 0 {
		t.Fatalf("unmatched_count = %d, want 0", report.UnmatchedCount)
	}
	if report.AmbiguousCount != 1 {
		t.Fatalf("ambiguous_count = %d, want 1", report.AmbiguousCount)
	}
	if len(report.Selectors) != 1 {
		t.Fatalf("len(selectors) = %d, want 1", len(report.Selectors))
	}
	if got := report.Selectors[0].MatchCount; got != 2 {
		t.Fatalf("selector match_count = %d, want 2", got)
	}
	if got := StrictExitCode(report); got != StrictExitAmbiguous {
		t.Fatalf("StrictExitCode = %d, want %d", got, StrictExitAmbiguous)
	}
}

Step 2: Prove the test catches a broken ambiguous branch

Temporarily change internal/uiaselector/report.go in StrictExitCode from:

	if report.AmbiguousCount > 0 {
		return StrictExitAmbiguous
	}

to:

	if report.AmbiguousCount > 0 {
		return StrictExitOK
	}

Run:

gofmt -w internal\uiaselector\report_test.go
go test ./internal/uiaselector -run "TestStrictModeClassifiesAmbiguousMatches" -count=1

Expected: FAIL with:

StrictExitCode = 0, want 3

Restore internal/uiaselector/report.go immediately after this command so production code again returns StrictExitAmbiguous for ambiguous reports. Do not commit the temporary mutation.

Step 3: Run the ambiguous test with production code restored

Run:

gofmt -w internal\uiaselector\report_test.go internal\uiaselector\report.go
go test ./internal/uiaselector -run "TestStrictModeClassifiesAmbiguousMatches" -count=1

Expected: PASS.

Step 4: Run package tests

Run:

go test ./internal/uiaselector -count=1

Expected: PASS.

Step 5: Commit ambiguous strict-mode test

Run:

git add internal\uiaselector\report_test.go
git diff --cached --name-only

Expected staged file list:

internal/uiaselector/report_test.go

Then commit:

git commit -m "test: cover N15 ambiguous strict exit"

Task 2: Harden N14 selector report verifier

Files:

Modify: scripts/verify-n14-uia-selector-report.ps1
Step 1: Run a failing pre-change verifier hardening check

Run this PowerShell check before editing the verifier:

$scriptPath = "scripts\verify-n14-uia-selector-report.ps1"
$scriptText = Get-Content -LiteralPath $scriptPath -Raw -Encoding UTF8
$requiredSnippets = @(
    'Remove-Item -LiteralPath $jsonPath, $mdPath -Force -ErrorAction SilentlyContinue',
    '$jsonItem = Get-Item -LiteralPath $jsonPath -ErrorAction Stop',
    'if ($jsonItem.PSIsContainer)',
    'if ($jsonItem.Length -le 0)',
    '$mdItem = Get-Item -LiteralPath $mdPath -ErrorAction Stop',
    '$allowedMatchFields = @(',
    '$match.PSObject.Properties',
    'JSON match leaks visible text field'
)
foreach ($snippet in $requiredSnippets) {
    if (-not $scriptText.Contains($snippet)) {
        throw "missing verifier hardening snippet: $snippet"
    }
}

Expected: FAIL, with the first missing snippet indicating stale-output cleanup is not implemented yet.

Step 2: Insert stale-output cleanup before CLI execution

In scripts/verify-n14-uia-selector-report.ps1, after:

$jsonPath = Join-Path $outDir "n12r-selector-report.json"
$mdPath = Join-Path $outDir "n12r-selector-report.md"

insert:

Remove-Item -LiteralPath $jsonPath, $mdPath -Force -ErrorAction SilentlyContinue

Step 3: Replace output file existence checks with file and non-empty checks

Replace this block:

if (-not (Test-Path -LiteralPath $jsonPath)) {
    throw "JSON report was not created: $jsonPath"
}
if (-not (Test-Path -LiteralPath $mdPath)) {
    throw "Markdown report was not created: $mdPath"
}

with:

$jsonItem = Get-Item -LiteralPath $jsonPath -ErrorAction Stop
if ($jsonItem.PSIsContainer) {
    throw "JSON report path is not a file: $jsonPath"
}
if ($jsonItem.Length -le 0) {
    throw "JSON report is empty: $jsonPath"
}
$mdItem = Get-Item -LiteralPath $mdPath -ErrorAction Stop
if ($mdItem.PSIsContainer) {
    throw "Markdown report path is not a file: $mdPath"
}
if ($mdItem.Length -le 0) {
    throw "Markdown report is empty: $mdPath"
}

Step 4: Add JSON match-field allowlist validation after report summary checks

After this block:

if ($report.ambiguous_count -ne 0) {
    throw "ambiguous_count=$($report.ambiguous_count), want 0"
}

insert:

Write-Host "## JSON match field allowlist"
$allowedMatchFields = @(
    "path",
    "control_type",
    "automation_id",
    "class_name",
    "framework_id",
    "has_name",
    "name_length"
)
$allowedMatchFieldLookup = @{}
foreach ($field in $allowedMatchFields) {
    $allowedMatchFieldLookup[$field] = $true
}
foreach ($selector in @($report.selectors)) {
    foreach ($match in @($selector.matches)) {
        if ($null -eq $match) {
            continue
        }
        foreach ($property in @($match.PSObject.Properties)) {
            $fieldName = $property.Name
            if ($fieldName -eq "name" -or $fieldName -eq "value") {
                throw "JSON match leaks visible text field: $fieldName"
            }
            if (-not $allowedMatchFieldLookup.ContainsKey($fieldName)) {
                throw "JSON match contains disallowed field: $fieldName"
            }
        }
    }
}

Step 5: Run the hardening check again

Run:

$scriptPath = "scripts\verify-n14-uia-selector-report.ps1"
$scriptText = Get-Content -LiteralPath $scriptPath -Raw -Encoding UTF8
$requiredSnippets = @(
    'Remove-Item -LiteralPath $jsonPath, $mdPath -Force -ErrorAction SilentlyContinue',
    '$jsonItem = Get-Item -LiteralPath $jsonPath -ErrorAction Stop',
    'if ($jsonItem.PSIsContainer)',
    'if ($jsonItem.Length -le 0)',
    '$mdItem = Get-Item -LiteralPath $mdPath -ErrorAction Stop',
    '$allowedMatchFields = @(',
    '$match.PSObject.Properties',
    'JSON match leaks visible text field'
)
foreach ($snippet in $requiredSnippets) {
    if (-not $scriptText.Contains($snippet)) {
        throw "missing verifier hardening snippet: $snippet"
    }
}
Write-Host "VERIFIER_HARDENING_SNIPPETS_OK"

Expected:

VERIFIER_HARDENING_SNIPPETS_OK

Step 6: Run the hardened verifier

Run:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n14-uia-selector-report.ps1

Expected output includes:

## go test ./internal/uiaselector ./cmd/uia-selector-report
## go run ./cmd/uia-selector-report
N14 selector report generated: catalog=10 matched=10 unmatched=0 ambiguous=0
## JSON match field allowlist
## output safety scan
N14 UIA selector report verification passed.

Step 7: Commit verifier hardening

Run:

git add scripts\verify-n14-uia-selector-report.ps1
git diff --cached --name-only

Expected staged file list:

scripts/verify-n14-uia-selector-report.ps1

Then commit:

git commit -m "test: harden N15 selector report verifier"

Task 3: Final verification and boundary check

Files:

No new files.
Verify: internal/uiaselector/report_test.go, scripts/verify-n14-uia-selector-report.ps1, and unchanged helper/MCP action surface.
Step 1: Run targeted tests

Run:

go test ./internal/uiaselector -run "TestStrictModeClassifiesAmbiguousMatches" -count=1
go test ./internal/uiaselector ./cmd/uia-selector-report -count=1

Expected: both commands PASS.

Step 2: Run repository tests

Run:

go test ./...

Expected: all packages PASS.

If dependency download fails with a transient proxy error, retry once with:

$env:GOPROXY = "https://goproxy.cn,direct"
go test ./...

Step 3: Run the hardened verifier

Run:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n14-uia-selector-report.ps1

Expected: verifier PASS and output includes ## JSON match field allowlist.

Step 4: Check helper/MCP/live action boundary

Run:

git diff --name-only main..HEAD -- native/ISphereWinHelper/Program.cs internal/tools/winhelper.go cmd/isphere-mcp/main.go cmd/uia-selector-report/main.go internal/uiaselector/report.go internal/uiaselector/catalog.go internal/uiaselector/matcher.go

Expected: no output.

If any file appears, stop and inspect because N15 is test/verifier-only.

Step 5: Check changed files

Run:

git diff --name-only main..HEAD

Expected changed files are limited to:

docs/superpowers/specs/2026-07-09-n15-selector-report-hardening-design.md
docs/superpowers/plans/2026-07-09-n15-selector-report-hardening-implementation.md
internal/uiaselector/report_test.go
scripts/verify-n14-uia-selector-report.ps1

Step 6: Final status

Run:

git status --short --branch
git log --oneline --max-count=6

Expected: working tree clean on codex/n15-report-hardening. Do not push unless the user explicitly requests pushing this branch or merging to main.

Self-review checklist

Spec coverage: ambiguous strict-mode test, stale-output cleanup, file/non-empty checks, JSON match-field allowlist, retained forbidden string scan, and helper/MCP boundary check are all covered.
Scope: only internal/uiaselector/report_test.go and scripts/verify-n14-uia-selector-report.ps1 are implementation targets; design and plan docs are documentation targets.
Boundary: no helper op, no MCP tool, no live hwnd/process behavior, no UI action, no report format change, no CLI argument change.
Verification: final commands include targeted tests, all repository tests, hardened verifier, boundary diff check, and changed-file check.
Commit cadence: test and verifier hardening are separate commits.

13 KiB Raw Blame History

N15 Selector Report Hardening Implementation Plan

Scope and boundary

File structure

Task 1: Cover ambiguous strict-mode classification

Task 2: Harden N14 selector report verifier

Task 3: Final verification and boundary check

Self-review checklist

13 KiB

Raw Blame History