docs: plan N15 selector report hardening
This commit is contained in:
石头
@@ -0,0 +1,466 @@
|
||||
# N15 Selector Report Hardening Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Harden the N14 offline selector report tests and verifier so ambiguous strict-mode behavior and report-output safety regressions are caught before merge.
|
||||
|
||||
**Architecture:** Keep N15 as a test/verifier-only change. Add an in-memory ambiguous-match test to `internal/uiaselector/report_test.go`, then strengthen `scripts/verify-n14-uia-selector-report.ps1` with stale-output cleanup, file/non-empty checks, and JSON match-field allowlist validation.
|
||||
|
||||
**Tech Stack:** Go 1.23.x tests, existing `internal/uiaselector` package, Windows PowerShell 5.1-compatible verifier script, standard `go test` and repository Git workflow.
|
||||
|
||||
---
|
||||
|
||||
## Scope and boundary
|
||||
|
||||
N15 changes only test and verifier coverage.
|
||||
|
||||
Modify only:
|
||||
|
||||
```text
|
||||
internal/uiaselector/report_test.go
|
||||
scripts/verify-n14-uia-selector-report.ps1
|
||||
```
|
||||
|
||||
Do not modify:
|
||||
|
||||
```text
|
||||
native/ISphereWinHelper/Program.cs
|
||||
internal/tools/winhelper.go
|
||||
cmd/isphere-mcp/main.go
|
||||
cmd/uia-selector-report/main.go
|
||||
internal/uiaselector/report.go
|
||||
internal/uiaselector/catalog.go
|
||||
internal/uiaselector/matcher.go
|
||||
```
|
||||
|
||||
Do not add selectors, helper ops, MCP tools, CLI arguments, live hwnd/process handling, UI automation calls, clicks, keyboard input, clipboard operations, conversation actions, message sending, file transfer, or visible text export.
|
||||
|
||||
## File structure
|
||||
|
||||
```text
|
||||
internal/uiaselector/report_test.go
|
||||
Add one explicit ambiguous strict-mode regression test using an in-memory UIA tree.
|
||||
|
||||
scripts/verify-n14-uia-selector-report.ps1
|
||||
Clean stale outputs before CLI execution, require fresh non-empty report files, and validate JSON match fields with an allowlist.
|
||||
```
|
||||
|
||||
## Task 1: Cover ambiguous strict-mode classification
|
||||
|
||||
**Files:**
|
||||
- Modify: `internal/uiaselector/report_test.go`
|
||||
|
||||
- [ ] **Step 1: Add the ambiguous strict-mode regression test**
|
||||
|
||||
Append this test to `internal/uiaselector/report_test.go`:
|
||||
|
||||
```go
|
||||
func TestStrictModeClassifiesAmbiguousMatches(t *testing.T) {
|
||||
root := &Node{
|
||||
ControlType: "Window",
|
||||
AutomationID: "frmMain",
|
||||
FrameworkID: "WinForm",
|
||||
Children: []Node{
|
||||
{ControlType: "Pane", AutomationID: "duplicate", FrameworkID: "WinForm"},
|
||||
{ControlType: "Pane", AutomationID: "duplicate", FrameworkID: "WinForm"},
|
||||
},
|
||||
}
|
||||
catalog := []Selector{
|
||||
{
|
||||
ID: "duplicate_panel",
|
||||
Description: "duplicate structural selector",
|
||||
AllowedUse: AllowedUseReadOnlyLocate,
|
||||
Required: Criteria{AutomationID: "duplicate", ControlType: "Pane", FrameworkID: "WinForm"},
|
||||
StabilityScore: 80,
|
||||
Evidence: Evidence{CaptureID: "unit-test", UIAPath: "root/duplicate"},
|
||||
},
|
||||
}
|
||||
|
||||
report := RunCatalogReport("synthetic-ambiguous.json", root, catalog, fixedReportTime())
|
||||
if !report.OK {
|
||||
t.Fatalf("report OK=false: %#v", report.Error)
|
||||
}
|
||||
if report.CatalogSize != 1 {
|
||||
t.Fatalf("catalog_size = %d, want 1", report.CatalogSize)
|
||||
}
|
||||
if report.MatchedCount != 1 {
|
||||
t.Fatalf("matched_count = %d, want 1", report.MatchedCount)
|
||||
}
|
||||
if report.UnmatchedCount != 0 {
|
||||
t.Fatalf("unmatched_count = %d, want 0", report.UnmatchedCount)
|
||||
}
|
||||
if report.AmbiguousCount != 1 {
|
||||
t.Fatalf("ambiguous_count = %d, want 1", report.AmbiguousCount)
|
||||
}
|
||||
if len(report.Selectors) != 1 {
|
||||
t.Fatalf("len(selectors) = %d, want 1", len(report.Selectors))
|
||||
}
|
||||
if got := report.Selectors[0].MatchCount; got != 2 {
|
||||
t.Fatalf("selector match_count = %d, want 2", got)
|
||||
}
|
||||
if got := StrictExitCode(report); got != StrictExitAmbiguous {
|
||||
t.Fatalf("StrictExitCode = %d, want %d", got, StrictExitAmbiguous)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Prove the test catches a broken ambiguous branch**
|
||||
|
||||
Temporarily change `internal/uiaselector/report.go` in `StrictExitCode` from:
|
||||
|
||||
```go
|
||||
if report.AmbiguousCount > 0 {
|
||||
return StrictExitAmbiguous
|
||||
}
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```go
|
||||
if report.AmbiguousCount > 0 {
|
||||
return StrictExitOK
|
||||
}
|
||||
```
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
gofmt -w internal\uiaselector\report_test.go
|
||||
go test ./internal/uiaselector -run "TestStrictModeClassifiesAmbiguousMatches" -count=1
|
||||
```
|
||||
|
||||
Expected: FAIL with:
|
||||
|
||||
```text
|
||||
StrictExitCode = 0, want 3
|
||||
```
|
||||
|
||||
Restore `internal/uiaselector/report.go` immediately after this command so production code again returns `StrictExitAmbiguous` for ambiguous reports. Do not commit the temporary mutation.
|
||||
|
||||
- [ ] **Step 3: Run the ambiguous test with production code restored**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
gofmt -w internal\uiaselector\report_test.go internal\uiaselector\report.go
|
||||
go test ./internal/uiaselector -run "TestStrictModeClassifiesAmbiguousMatches" -count=1
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 4: Run package tests**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/uiaselector -count=1
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit ambiguous strict-mode test**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
git add internal\uiaselector\report_test.go
|
||||
git diff --cached --name-only
|
||||
```
|
||||
|
||||
Expected staged file list:
|
||||
|
||||
```text
|
||||
internal/uiaselector/report_test.go
|
||||
```
|
||||
|
||||
Then commit:
|
||||
|
||||
```powershell
|
||||
git commit -m "test: cover N15 ambiguous strict exit"
|
||||
```
|
||||
|
||||
## Task 2: Harden N14 selector report verifier
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/verify-n14-uia-selector-report.ps1`
|
||||
|
||||
- [ ] **Step 1: Run a failing pre-change verifier hardening check**
|
||||
|
||||
Run this PowerShell check before editing the verifier:
|
||||
|
||||
```powershell
|
||||
$scriptPath = "scripts\verify-n14-uia-selector-report.ps1"
|
||||
$scriptText = Get-Content -LiteralPath $scriptPath -Raw -Encoding UTF8
|
||||
$requiredSnippets = @(
|
||||
'Remove-Item -LiteralPath $jsonPath, $mdPath -Force -ErrorAction SilentlyContinue',
|
||||
'$jsonItem = Get-Item -LiteralPath $jsonPath -ErrorAction Stop',
|
||||
'if ($jsonItem.PSIsContainer)',
|
||||
'if ($jsonItem.Length -le 0)',
|
||||
'$mdItem = Get-Item -LiteralPath $mdPath -ErrorAction Stop',
|
||||
'$allowedMatchFields = @(',
|
||||
'$match.PSObject.Properties',
|
||||
'JSON match leaks visible text field'
|
||||
)
|
||||
foreach ($snippet in $requiredSnippets) {
|
||||
if (-not $scriptText.Contains($snippet)) {
|
||||
throw "missing verifier hardening snippet: $snippet"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Expected: FAIL, with the first missing snippet indicating stale-output cleanup is not implemented yet.
|
||||
|
||||
- [ ] **Step 2: Insert stale-output cleanup before CLI execution**
|
||||
|
||||
In `scripts/verify-n14-uia-selector-report.ps1`, after:
|
||||
|
||||
```powershell
|
||||
$jsonPath = Join-Path $outDir "n12r-selector-report.json"
|
||||
$mdPath = Join-Path $outDir "n12r-selector-report.md"
|
||||
```
|
||||
|
||||
insert:
|
||||
|
||||
```powershell
|
||||
Remove-Item -LiteralPath $jsonPath, $mdPath -Force -ErrorAction SilentlyContinue
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Replace output file existence checks with file and non-empty checks**
|
||||
|
||||
Replace this block:
|
||||
|
||||
```powershell
|
||||
if (-not (Test-Path -LiteralPath $jsonPath)) {
|
||||
throw "JSON report was not created: $jsonPath"
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $mdPath)) {
|
||||
throw "Markdown report was not created: $mdPath"
|
||||
}
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```powershell
|
||||
$jsonItem = Get-Item -LiteralPath $jsonPath -ErrorAction Stop
|
||||
if ($jsonItem.PSIsContainer) {
|
||||
throw "JSON report path is not a file: $jsonPath"
|
||||
}
|
||||
if ($jsonItem.Length -le 0) {
|
||||
throw "JSON report is empty: $jsonPath"
|
||||
}
|
||||
$mdItem = Get-Item -LiteralPath $mdPath -ErrorAction Stop
|
||||
if ($mdItem.PSIsContainer) {
|
||||
throw "Markdown report path is not a file: $mdPath"
|
||||
}
|
||||
if ($mdItem.Length -le 0) {
|
||||
throw "Markdown report is empty: $mdPath"
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add JSON match-field allowlist validation after report summary checks**
|
||||
|
||||
After this block:
|
||||
|
||||
```powershell
|
||||
if ($report.ambiguous_count -ne 0) {
|
||||
throw "ambiguous_count=$($report.ambiguous_count), want 0"
|
||||
}
|
||||
```
|
||||
|
||||
insert:
|
||||
|
||||
```powershell
|
||||
Write-Host "## JSON match field allowlist"
|
||||
$allowedMatchFields = @(
|
||||
"path",
|
||||
"control_type",
|
||||
"automation_id",
|
||||
"class_name",
|
||||
"framework_id",
|
||||
"has_name",
|
||||
"name_length"
|
||||
)
|
||||
$allowedMatchFieldLookup = @{}
|
||||
foreach ($field in $allowedMatchFields) {
|
||||
$allowedMatchFieldLookup[$field] = $true
|
||||
}
|
||||
foreach ($selector in @($report.selectors)) {
|
||||
foreach ($match in @($selector.matches)) {
|
||||
if ($null -eq $match) {
|
||||
continue
|
||||
}
|
||||
foreach ($property in @($match.PSObject.Properties)) {
|
||||
$fieldName = $property.Name
|
||||
if ($fieldName -eq "name" -or $fieldName -eq "value") {
|
||||
throw "JSON match leaks visible text field: $fieldName"
|
||||
}
|
||||
if (-not $allowedMatchFieldLookup.ContainsKey($fieldName)) {
|
||||
throw "JSON match contains disallowed field: $fieldName"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the hardening check again**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
$scriptPath = "scripts\verify-n14-uia-selector-report.ps1"
|
||||
$scriptText = Get-Content -LiteralPath $scriptPath -Raw -Encoding UTF8
|
||||
$requiredSnippets = @(
|
||||
'Remove-Item -LiteralPath $jsonPath, $mdPath -Force -ErrorAction SilentlyContinue',
|
||||
'$jsonItem = Get-Item -LiteralPath $jsonPath -ErrorAction Stop',
|
||||
'if ($jsonItem.PSIsContainer)',
|
||||
'if ($jsonItem.Length -le 0)',
|
||||
'$mdItem = Get-Item -LiteralPath $mdPath -ErrorAction Stop',
|
||||
'$allowedMatchFields = @(',
|
||||
'$match.PSObject.Properties',
|
||||
'JSON match leaks visible text field'
|
||||
)
|
||||
foreach ($snippet in $requiredSnippets) {
|
||||
if (-not $scriptText.Contains($snippet)) {
|
||||
throw "missing verifier hardening snippet: $snippet"
|
||||
}
|
||||
}
|
||||
Write-Host "VERIFIER_HARDENING_SNIPPETS_OK"
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```text
|
||||
VERIFIER_HARDENING_SNIPPETS_OK
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the hardened verifier**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n14-uia-selector-report.ps1
|
||||
```
|
||||
|
||||
Expected output includes:
|
||||
|
||||
```text
|
||||
## go test ./internal/uiaselector ./cmd/uia-selector-report
|
||||
## go run ./cmd/uia-selector-report
|
||||
N14 selector report generated: catalog=10 matched=10 unmatched=0 ambiguous=0
|
||||
## JSON match field allowlist
|
||||
## output safety scan
|
||||
N14 UIA selector report verification passed.
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Commit verifier hardening**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
git add scripts\verify-n14-uia-selector-report.ps1
|
||||
git diff --cached --name-only
|
||||
```
|
||||
|
||||
Expected staged file list:
|
||||
|
||||
```text
|
||||
scripts/verify-n14-uia-selector-report.ps1
|
||||
```
|
||||
|
||||
Then commit:
|
||||
|
||||
```powershell
|
||||
git commit -m "test: harden N15 selector report verifier"
|
||||
```
|
||||
|
||||
## Task 3: Final verification and boundary check
|
||||
|
||||
**Files:**
|
||||
- No new files.
|
||||
- Verify: `internal/uiaselector/report_test.go`, `scripts/verify-n14-uia-selector-report.ps1`, and unchanged helper/MCP action surface.
|
||||
|
||||
- [ ] **Step 1: Run targeted tests**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/uiaselector -run "TestStrictModeClassifiesAmbiguousMatches" -count=1
|
||||
go test ./internal/uiaselector ./cmd/uia-selector-report -count=1
|
||||
```
|
||||
|
||||
Expected: both commands PASS.
|
||||
|
||||
- [ ] **Step 2: Run repository tests**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Expected: all packages PASS.
|
||||
|
||||
If dependency download fails with a transient proxy error, retry once with:
|
||||
|
||||
```powershell
|
||||
$env:GOPROXY = "https://goproxy.cn,direct"
|
||||
go test ./...
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run the hardened verifier**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-n14-uia-selector-report.ps1
|
||||
```
|
||||
|
||||
Expected: verifier PASS and output includes `## JSON match field allowlist`.
|
||||
|
||||
- [ ] **Step 4: Check helper/MCP/live action boundary**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
git diff --name-only main..HEAD -- native/ISphereWinHelper/Program.cs internal/tools/winhelper.go cmd/isphere-mcp/main.go cmd/uia-selector-report/main.go internal/uiaselector/report.go internal/uiaselector/catalog.go internal/uiaselector/matcher.go
|
||||
```
|
||||
|
||||
Expected: no output.
|
||||
|
||||
If any file appears, stop and inspect because N15 is test/verifier-only.
|
||||
|
||||
- [ ] **Step 5: Check changed files**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
git diff --name-only main..HEAD
|
||||
```
|
||||
|
||||
Expected changed files are limited to:
|
||||
|
||||
```text
|
||||
docs/superpowers/specs/2026-07-09-n15-selector-report-hardening-design.md
|
||||
docs/superpowers/plans/2026-07-09-n15-selector-report-hardening-implementation.md
|
||||
internal/uiaselector/report_test.go
|
||||
scripts/verify-n14-uia-selector-report.ps1
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Final status**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
git status --short --branch
|
||||
git log --oneline --max-count=6
|
||||
```
|
||||
|
||||
Expected: working tree clean on `codex/n15-report-hardening`. Do not push unless the user explicitly requests pushing this branch or merging to main.
|
||||
|
||||
## Self-review checklist
|
||||
|
||||
- Spec coverage: ambiguous strict-mode test, stale-output cleanup, file/non-empty checks, JSON match-field allowlist, retained forbidden string scan, and helper/MCP boundary check are all covered.
|
||||
- Scope: only `internal/uiaselector/report_test.go` and `scripts/verify-n14-uia-selector-report.ps1` are implementation targets; design and plan docs are documentation targets.
|
||||
- Boundary: no helper op, no MCP tool, no live hwnd/process behavior, no UI action, no report format change, no CLI argument change.
|
||||
- Verification: final commands include targeted tests, all repository tests, hardened verifier, boundary diff check, and changed-file check.
|
||||
- Commit cadence: test and verifier hardening are separate commits.
|
||||
Reference in New Issue
Block a user