Compare commits
10 Commits
853404c67a
...
2f12fe8cda
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2f12fe8cda | ||
|
|
a7458265c0 | ||
|
|
1ea4c52ad1 | ||
|
|
347d68626e | ||
|
|
c41f260e51 | ||
|
|
581724ea42 | ||
|
|
1324f1882c | ||
|
|
5ea50e7288 | ||
|
|
ff122226b4 | ||
|
|
18923730a2 |
44
README.md
44
README.md
@@ -7,18 +7,26 @@ The project intentionally does not include an LLM, a generic browser agent, or a
|
||||
## Current Scope
|
||||
|
||||
- Open a URL or local HTML file with Playwright.
|
||||
- Capture reproducibility metadata: browser, viewport, locale, auth mode, crawl session id, input fingerprint, and policy/scoring versions.
|
||||
- Capture one page state with DOM and visible-text fingerprints.
|
||||
- Extract actionable elements: buttons, links, inputs, selects, textareas, roles, tab stops, and click handlers.
|
||||
- Generate locator candidates with scores.
|
||||
- Optionally explore low-risk click actions from the entry state.
|
||||
- Record state nodes, elements, locators, action edges, and basic effects in JSON.
|
||||
- Generate locator candidates with scores, context scopes, identity checks, and verification evidence.
|
||||
- Compile scoped locator descriptors for row/dialog/form/list-item contexts when they can make duplicated targets executable.
|
||||
- Optionally run low-risk one-hop click exploration from a clean entry state.
|
||||
- Replay actions from the entry state and record observed state transitions with snapshot-diff effects.
|
||||
- Record navigation failures as failed observations when a graph can still be emitted.
|
||||
- Record state-local skipped default-exploration actions when risk policy blocks them.
|
||||
- Record failed observations instead of turning failed locators or failed actions into successful edges.
|
||||
- Emit state nodes, elements, locators, action edges, skipped actions, failed observations, metadata, and summary counts in JSON.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No LLM decision making.
|
||||
- No natural-language task planner.
|
||||
- No high-risk automatic actions.
|
||||
- No attempt to fully recover frontend business functions in the first version.
|
||||
- No full crawler or breadth-first site exploration.
|
||||
- No general browser agent or workflow orchestration platform.
|
||||
- No attempt to fully recover frontend business functions in this engine.
|
||||
|
||||
## Install
|
||||
|
||||
@@ -34,12 +42,24 @@ Static scan:
|
||||
npm run scan -- --url ./examples/simple.html --out artifacts/simple-scan.json
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
```bash
|
||||
npm run test
|
||||
```
|
||||
|
||||
Entry-state low-risk exploration:
|
||||
|
||||
```bash
|
||||
npm run example
|
||||
```
|
||||
|
||||
Timeout tuning for slow sites:
|
||||
|
||||
```bash
|
||||
npm run explore -- --url https://www.gov.cn/ --out artifacts/govcn.json --max-actions 2 --navigation-timeout-ms 60000 --action-timeout-ms 3000
|
||||
```
|
||||
|
||||
Build:
|
||||
|
||||
```bash
|
||||
@@ -48,16 +68,20 @@ npm run build
|
||||
|
||||
## Output Model
|
||||
|
||||
The output JSON follows the schema described in [schemas/topology-graph.schema.md](schemas/topology-graph.schema.md).
|
||||
The output JSON follows the schema described in [schemas/topology-graph.schema.md](schemas/topology-graph.schema.md). A machine-checkable JSON Schema is available at [schemas/topology-graph.schema.json](schemas/topology-graph.schema.json).
|
||||
|
||||
Key objects:
|
||||
|
||||
- `StateNode`: URL, route, title, state hashes, modal stack, and entry metadata.
|
||||
- `ActionableElement`: element semantics, visibility, bounding box, and locator candidates.
|
||||
- `LocatorCandidate`: Playwright-friendly locator descriptors plus score and verification status.
|
||||
- `ActionEdge`: action from one state to another through an element, with effects and risk.
|
||||
- `TopologyGraph.metadata`: artifact version, input URL, entry URL, generated timestamp, engine version, mode, non-deterministic crawl session id, deterministic input fingerprint, runtime context, site, and policy/scoring versions.
|
||||
- `StateNode`: URL, route, title, canonical key, language, viewport, state hashes, modal stack, lifecycle, evidence, and first observed `openedByEdgeId`.
|
||||
- `ActionableElement`: state-local element semantics, visibility/interactability, bounding box, context anchors, parameter surface, evidence, and locator candidates.
|
||||
- `LocatorCandidate`: Playwright-friendly locator descriptors plus score, verification status, context `scopes`, identity evidence, and failure count.
|
||||
- `ActionEdge`: low-risk replayed action from one state to another through an element, with `riskCategory`, `riskLevel`, and snapshot-diff `effects`.
|
||||
- `EffectRecord`: snapshot-diff record for `url`, `dom`, `visibleText`, or `actionableInventory`, with `before` and `after` values.
|
||||
- `SkippedAction`: risk-policy default exploration refusal with risk category, reason, state id, element id, and evidence.
|
||||
- `FailedObservation`: locator, actionability, action, or effect failure with machine-readable reason and diagnostic evidence.
|
||||
- `summary`: count totals and grouped counts for edges, skipped actions, and failed observations.
|
||||
|
||||
## Source Notes
|
||||
|
||||
The originating ChatGPT share conversation is preserved in [docs/source/chatgpt-share-6a30e583.md](docs/source/chatgpt-share-6a30e583.md).
|
||||
|
||||
|
||||
253
docs/design/action-topology-engine-v0.2.md
Normal file
253
docs/design/action-topology-engine-v0.2.md
Normal file
@@ -0,0 +1,253 @@
|
||||
# 网站可操作拓扑引擎设计文档 v0.2
|
||||
|
||||
## 0. 文档读法
|
||||
|
||||
本文同时保留两类内容:
|
||||
|
||||
- **v0.2.1 当前输出契约**:已经由当前代码、`README.md`、`schemas/topology-graph.schema.md` 和 `src/types.ts` 支持的 artifact 形态。
|
||||
- **未来设计目标**:拓扑引擎后续可以演进的方向,但不代表 v0.2.1 已经发出或支持。
|
||||
|
||||
没有明确标注为 v0.2.1 当前输出契约的扩展能力,都应按未来设计目标理解,不能作为当前 artifact 消费合同。
|
||||
|
||||
## 1. 项目定位
|
||||
|
||||
本项目是一个确定性的 website action topology engine。它的职责不是自动完成任务,也不是理解自然语言指令,而是为一个网站建立可被机器消费的“可操作拓扑图”。
|
||||
|
||||
它输出的是底层地图:页面有哪些状态,状态中有哪些可操作项,这些可操作项如何稳定定位,执行动作后产生什么可观察变化。
|
||||
|
||||
## 2. 明确边界
|
||||
|
||||
本项目只负责拓扑图。
|
||||
|
||||
不负责 rrweb 录制,不负责将 rrweb 对齐到拓扑,不负责生成浏览器自动化脚本,不负责生成 skill,不负责 LLM 规划,不负责通用浏览器 Agent。
|
||||
|
||||
这些都可以是下游消费者,但不能反向污染本项目的模型边界。
|
||||
|
||||
## 3. 核心世界观
|
||||
|
||||
网站不是一组 DOM 节点,而是一组可操作状态。
|
||||
|
||||
基本认知单元是:
|
||||
|
||||
```text
|
||||
State -- Action(Actionable Element) --> State / Observable Effects
|
||||
```
|
||||
|
||||
真正有价值的不是“页面上有一个按钮”,而是“在某个状态下,有一个可定位、可验证、可执行的动作;执行后,浏览器世界发生了某些可观察变化”。
|
||||
|
||||
### 3.1 确定性的含义
|
||||
|
||||
本项目所谓确定性,不是指同一个网站每次探索都会得到完全相同的图。真实网站会受到账号、权限、数据、时间、A/B 实验、feature flag、语言、viewport、网络和服务端状态影响。
|
||||
|
||||
这里的确定性指:拓扑事实的来源、生成规则和验证方式是可复验的。引擎只依据可观察、可记录、可复验的浏览器事实生成拓扑,不依赖不可验证的推测。
|
||||
|
||||
因此,每份拓扑 artifact 都必须带有生成上下文:入口 URL、账号角色、环境、viewport、locale、浏览器类型、登录状态、采集时间、站点版本或数据特征。同一网站在不同上下文下可以生成不同拓扑,引擎不强行合并。
|
||||
|
||||
### 3.2 网站状态不是页面截图
|
||||
|
||||
State 不是截图,也不是完整 DOM 快照。State 是在特定环境上下文下,页面在某一时刻呈现出的可操作上下文。
|
||||
|
||||
拓扑关心的是:哪些动作可见、可定位、可执行;执行后世界如何变化。像素差异、随机节点、埋点、广告、时间戳,不应轻易制造新状态。
|
||||
|
||||
### 3.3 v0.2.1 当前输出契约
|
||||
|
||||
v0.2.1 当前 artifact 输出以下事实:
|
||||
|
||||
- `TopologyGraph` metadata、state、element、locator、edge、skipped action、failed observation。
|
||||
- state 的 URL、route、title、language、viewport、DOM hash、visible text hash、actionable signature hash、state hash、modal stack 和 lifecycle。
|
||||
- element 的语义字段、bbox、interactability、context anchors、parameter surface、locator candidates 和 evidence。
|
||||
- locator candidate 的 Playwright/CSS descriptor、score、verification metadata、identity evidence、failure count 和 `scopes`。
|
||||
- 默认探索只从干净 entry-state context 执行低风险 one-hop click replay。
|
||||
- risk policy 拒绝的默认探索动作记录为 `SkippedAction`。
|
||||
- locator、actionability、action 或 effect 失败记录为 `FailedObservation`。
|
||||
- effect 只发出 snapshot-diff 类型:`url`、`dom`、`visibleText`、`actionableInventory`;这些 v0.2.1 effect 会填充 `before` 和 `after`。
|
||||
|
||||
v0.2.1 当前不发出 network、download、dialog、storage effect buffer;不捕获真实 frame/shadow path;不发出可执行的 row/card action template;`frameContext`、`framePath`、`shadowPath` 字段存在,但当前实现为空路径。locator `scopes` 只是证据和排序/消歧线索,不是可执行 scoped descriptor。
|
||||
|
||||
## 4. 设计目标
|
||||
|
||||
拓扑图面向机器消费,目标是稳定、可复验、可解释。
|
||||
|
||||
它应该让下游系统判断:当前状态是否已知,某动作是否存在,目标元素如何定位,动作前需要什么状态,动作后应该观察什么结果,动作风险如何,失败时原因是什么。
|
||||
|
||||
## 5. 核心概念
|
||||
|
||||
`State`:页面的可操作上下文。URL 只是其中一部分;弹窗、菜单、抽屉、tab、筛选条件、可见文本、可操作项集合都会影响状态。frame 和 shadow DOM 是未来设计目标中的状态边界;v0.2.1 当前只保留空路径字段,不捕获真实 frame/shadow 层级。
|
||||
|
||||
`Actionable Element`:某个状态下可由浏览器执行动作的对象。它不能脱离状态全局存在。同样一个“保存”按钮,在不同弹窗或表格行中是不同元素。
|
||||
|
||||
`Locator Candidate`:定位该元素的一组候选方法及其证据,不是单个 selector,也不是对 DOM 结构的永久承诺。
|
||||
|
||||
`Action Edge`:从一个状态经过一个动作到另一个状态的边。边的价值在于记录动作和结果,而不是只记录“点击过”。
|
||||
|
||||
`Effect`:动作后的可观察变化。v0.2.1 当前只发出 URL、DOM hash、visible text hash 和 actionable inventory hash 的 snapshot diff。网络请求、弹窗事件、下载、storage、history、focus、validation 等属于未来设计目标;没有对应代码前,不能作为当前输出契约。
|
||||
|
||||
`Parameter Surface`:动作或元素可接受的参数形态。拓扑引擎不负责生成业务参数,但应记录输入槽位、约束和来源证据。
|
||||
|
||||
`Risk Classification`:对动作探索风险的保守分类。它不是业务授权,也不是最终执行许可,只用于指导拓扑探索和下游消费时的风险处理。
|
||||
|
||||
`Evidence Level`:拓扑字段的证据等级,区分直接观察、规则推导、复验结论、启发式猜测和外部标注。
|
||||
|
||||
### 5.1 事实与证据等级
|
||||
|
||||
拓扑 artifact 中的信息必须区分证据等级。
|
||||
|
||||
`Observed Fact` 是直接来自浏览器的事实,例如 URL、DOM 属性、AX role、bounding box、visibility、enabled 状态。network request 观察属于未来设计目标,不是 v0.2.1 当前 artifact 字段。
|
||||
|
||||
`Derived Fact` 是由确定性规则推导出的信息,例如状态签名、元素上下文、locator 候选评分、重复元素分组。
|
||||
|
||||
`Verified Claim` 是经过重新加载、重新定位、重新执行或效果复验后确认的信息。
|
||||
|
||||
`Heuristic Guess` 是启发式判断,例如某动作可能高风险、某输入框可能是客户编号。
|
||||
|
||||
`External Annotation` 是人工或下游系统补充的信息,不属于拓扑引擎原生事实。
|
||||
|
||||
拓扑引擎可以存储启发式信息,但不能把启发式信息伪装成已验证事实。
|
||||
|
||||
### 5.2 Risk Classification
|
||||
|
||||
风险分类回答的是:这个动作是否适合默认探索,是否可能造成副作用,是否需要人工确认或沙盒环境,是否应该只记录不执行。
|
||||
|
||||
建议风险类型包括:`read_only`、`navigation`、`ui_reveal`、`input_edit`、`form_submit`、`data_mutation`、`destructive`、`auth_sensitive`、`payment_sensitive`、`unknown`。
|
||||
|
||||
`read_only`、`navigation`、`ui_reveal` 通常可作为默认探索候选。`input_edit` 可以记录但默认不提交。`form_submit`、`data_mutation` 默认不探索。`destructive`、`auth_sensitive`、`payment_sensitive` 默认只记录不执行。`unknown` 按高风险处理。
|
||||
|
||||
### 5.3 Parameter Surface
|
||||
|
||||
很多动作不是简单 click,而是填写、选择、上传、筛选、提交、选择表格行或对业务对象执行动作。
|
||||
|
||||
拓扑不需要知道“应该填什么”,但需要知道参数面:输入框的 type、name、label、placeholder、required、pattern、maxlength、validation;选择控件的 options、selected value、disabled options;日期控件的格式、范围和边界;表单的字段、必填项、提交动作和校验行为。网络 body 线索属于未来设计目标,不是 v0.2.1 当前输出字段。
|
||||
|
||||
它的目标是让下游知道:这个动作需要哪些输入,哪些是必填,约束是什么,输入和后续 effect 是否有关联。
|
||||
|
||||
## 6. 状态建模原则
|
||||
|
||||
状态必须从机器能复验的事实出发,而不是从页面名称出发。
|
||||
|
||||
同一个 URL 可以有多个状态,不同 URL 也可能落到相似状态。状态判断不能只靠路由。v0.2.1 当前状态签名由 URL、title、DOM hash、visible text hash 和 actionable signature hash 派生,并记录 modal stack。frame/shadow 边界和更细的业务上下文归并是未来设计目标。
|
||||
|
||||
### 6.1 State Lifecycle
|
||||
|
||||
State 在拓扑中应有生命周期。
|
||||
|
||||
`observed` 表示首次观察到。`fingerprinted` 表示已生成状态签名。`verified` 表示经过重新进入或复验,关键可操作项仍成立。`stale` 表示曾经有效但近期复验失败或证据过期。`deprecated` 表示确认不再可达或被新状态替代。`conflicted` 表示不同页面上下文被错误归并,或同一签名下出现不兼容事实。
|
||||
|
||||
下游应优先消费 verified 状态,对 observed 或 stale 状态保持谨慎。
|
||||
|
||||
### 6.2 状态等价与归并
|
||||
|
||||
状态归并应服务动作判断,而不是追求像素级一致。
|
||||
|
||||
时间戳、随机 id、埋点节点、广告、非目标列表数据轻微变化、loading 到 loaded 的瞬态变化,通常不应单独构成新状态。
|
||||
|
||||
可操作项集合变化、modal/dialog/drawer/menu 打开或关闭、tab/stepper 改变、form schema 改变、权限导致动作可见性改变、关键筛选条件改变、目标业务对象上下文改变,通常应构成新状态。frame 上下文改变应构成新状态是未来设计目标;v0.2.1 当前不捕获真实 frame context。
|
||||
|
||||
未来状态签名应优先基于 URL canonical form、可见可操作项签名、关键 heading/landmark、modal stack、selected tab、form signature、frame path、shadow boundary 和业务对象上下文,而不是完整 DOM hash。v0.2.1 当前状态 hash 仍包含 DOM hash、visible text hash 和 actionable signature hash;这不是 frame/shadow/state-signature 完整捕获能力。
|
||||
|
||||
## 7. 元素建模原则
|
||||
|
||||
元素应按用户感知和浏览器语义识别,而不是按 DOM 结构识别。
|
||||
|
||||
重要证据包括角色、可访问名称、标签、占位符、可见文本、ARIA 信息、输入类型、是否可见、是否启用、是否接收事件、所在上下文、附近标题、所属表格行、所属弹窗或表单。
|
||||
|
||||
框架特征只作为线索,不作为地基。React、Vue、Angular、AntD、MUI 等只说明页面可能有动态 id、portal、虚拟列表、隐藏副本等风险。
|
||||
|
||||
## 8. 定位哲学
|
||||
|
||||
定位器以可复现为目标,而不是以当下能找到为目标。
|
||||
|
||||
优先级应是用户视角和显式契约:role、label、placeholder、text、test id、稳定业务属性。CSS 和 XPath 只能作为 fallback。
|
||||
|
||||
locator 是否可信,不能只看它是否匹配一个节点,还要看它是否匹配正确语义,是否重载后仍成立,是否命中隐藏副本,是否依赖脆弱结构,是否需要上下文消歧。
|
||||
|
||||
## 9. 上下文消歧
|
||||
|
||||
真实页面里“编辑”“查看”“删除”“保存”“确定”会大量重复。v0.2.1 当前记录 dialog、form、section、row、landmark、listitem 等 context anchors。card 专用 anchor 属于未来设计目标。
|
||||
|
||||
机器需要的是“客户 A 这一行的编辑按钮”,不是“第 3 个编辑按钮”。
|
||||
|
||||
### 9.1 重复结构与动作模板
|
||||
|
||||
v0.2.1 当前不会发出可执行的 row/item/card action template,也不会把 locator `scopes` 编译成 scoped execution descriptor。当前 artifact 只记录 context anchors 和 locator scopes 作为证据、排序和消歧线索。
|
||||
|
||||
未来设计目标中,重复元素不应全部平铺为互不相关的独立元素。
|
||||
|
||||
表格每行的“编辑”、订单卡片的“查看详情”、文件列表的“下载”,未来应尽量表达为 row/item/card action template。
|
||||
|
||||
模板会记录重复容器类型、候选 identity 字段、行或卡片上下文、动作元素相对位置、动作语义、风险等级和 locator 消歧方式。下游执行时,应先定位目标 row/card,再在该上下文内定位动作。只有在新增可执行 scoped descriptor 支持后,这一段才可作为当前执行合同。
|
||||
|
||||
## 10. 动作边原则
|
||||
|
||||
动作边表达一个可验证的行为关系。
|
||||
|
||||
它应说明:从哪个状态出发,作用于哪个元素,动作类型是什么,是否需要参数,风险如何,动作后出现什么可观察 effect,结果状态是否可识别。
|
||||
|
||||
### 10.1 Handler Trace 不是核心事实
|
||||
|
||||
未来如果运行环境允许,引擎可以记录事件监听器、调用栈、source map 映射和框架组件线索。
|
||||
|
||||
但 Handler Trace 只能作为辅助证据,不应成为动作边成立的前提。动作边的核心事实来自动作前状态、目标元素、动作类型、可观察 effect、结果状态、定位复验和风险分类。
|
||||
|
||||
### 10.2 失败观察
|
||||
|
||||
失败不是噪声。失败应记录为 Failed Observation,而不是伪装成成功动作边。
|
||||
|
||||
典型失败包括:`locator_not_found`、`locator_not_unique`、`element_hidden`、`element_disabled`、`element_obscured`、`state_precondition_missing`、`frame_inaccessible`、`shadow_root_closed`、`navigation_timeout`、`network_error`、`permission_denied`、`auth_required`、`captcha_or_challenge`、`risk_policy_blocked`、`effect_not_observed`、`unexpected_state`。
|
||||
|
||||
失败观察要帮助下游判断:是 locator 坏了,状态没到,页面变化了,权限不够,风险策略拦截,还是目标本身不存在。
|
||||
|
||||
## 11. 探索策略原则
|
||||
|
||||
默认探索必须保守。少而准,比广而脆更符合本项目价值。
|
||||
|
||||
低风险动作可以探索;提交、确认、支付、授权、删除、注销等动作应记录为高风险,不默认执行。探索目标不是点遍所有东西,而是建立可靠、可复验的拓扑边。
|
||||
|
||||
## 12. 输出契约
|
||||
|
||||
拓扑 artifact 应该是机器可读、稳定演进、可版本化的事实集合。
|
||||
|
||||
它不包含下游意图,不描述“如何完成某个业务任务”,不内置 skill 语义。它只提供事实:状态、元素、定位候选、动作边、效果、风险、失败原因和证据等级。
|
||||
|
||||
### 12.1 Artifact 版本化
|
||||
|
||||
每次生成或更新拓扑,应记录 artifact version、schema version、site identifier、base URL、crawl session id、browser engine、viewport、locale、auth context、created time、input seed、risk policy version、normalization rule version、locator scoring version。
|
||||
|
||||
拓扑更新不应简单覆盖旧结果。系统应支持比较状态差异、locator 健康度变化、新增或消失动作、风险等级变化、effect 变化,并能标记 stale 或 deprecated 状态。
|
||||
|
||||
### 12.2 拓扑事实不等于任务语义
|
||||
|
||||
拓扑引擎未来可以记录某按钮文本是“导出”,某请求路径包含 `/export`,某动作产生下载事件。v0.2.1 当前不记录网络请求路径,也不发出下载事件 effect。
|
||||
|
||||
但它不应在内部上升为“完成导出销售报表任务”的 skill。
|
||||
|
||||
拓扑图描述的是:在什么状态下,什么元素可被定位,可以执行什么动作,动作后出现什么可观察效果,这个动作有什么风险。
|
||||
|
||||
它不描述用户真正想完成什么业务目标,不选择业务路径,不决定业务参数,不包装 skill,不根据自然语言规划动作序列。
|
||||
|
||||
## 13. 成功标准
|
||||
|
||||
好的拓扑图应该满足:状态不是靠 URL 粗暴判断;元素绑定到具体状态;locator 有候选和可信理由;重复元素可以通过上下文区分;动作边有可观察 effect;高风险动作不被默认执行;失败原因能被机器理解;下游系统不需要重新猜测页面结构。
|
||||
|
||||
frame/shadow/modal 边界都不应被忽略是未来质量目标。v0.2.1 当前只实现 modal/context 相关记录;frame/shadow 字段为空路径,不能视为已实现捕获能力。
|
||||
|
||||
### 13.1 质量指标
|
||||
|
||||
拓扑质量应能被度量。
|
||||
|
||||
核心指标包括:`state_match_accuracy`、`locator_unique_rate`、`locator_stability_rate`、`action_effect_observed_rate`、`duplicate_disambiguation_success_rate`、`risk_false_negative_rate`、`risk_skip_rate`、`failed_observation_explainability_rate`、`stale_detection_rate`、`downstream_reguess_rate`。
|
||||
|
||||
其中最重要的反向指标是 `downstream_reguess_rate`。如果下游仍要重新猜页面结构,说明拓扑没有履行职责。
|
||||
|
||||
## 14. 设计底线
|
||||
|
||||
本项目必须保持底层拓扑引擎的克制。
|
||||
|
||||
一旦它开始理解用户任务、生成脚本、包装 skill、绕过风控、或者做智能规划,它就越界了。
|
||||
|
||||
它应该像地图测绘,不像司机;像浏览器行为地形图,不像自动驾驶系统。
|
||||
|
||||
## 15. 一句话总结
|
||||
|
||||
本项目要建立的是:**网站可操作世界的确定性拓扑地图**。
|
||||
|
||||
它记录浏览器事实,抽象页面状态,识别可操作项,生成可验证定位候选,观察动作效果,并把这些组织成下游机器可以信任的图。
|
||||
479
docs/source/2026-06-16-browser-automation-crawling-research.md
Normal file
479
docs/source/2026-06-16-browser-automation-crawling-research.md
Normal file
@@ -0,0 +1,479 @@
|
||||
# Browser Automation and Crawling Research Notes
|
||||
|
||||
- Date: 2026-06-16
|
||||
- Purpose: early research material for this repository's deterministic action topology engine.
|
||||
- Repository boundary: this document is about what to observe, verify, and record. It is not a proposal to add LLM planning, generic task execution, or a full browser-agent product.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Browser automation and web crawling fail less because an individual selector is "wrong" and more because the system did not record enough context: page state, frame/shadow boundary, accessibility semantics, timing, network effects, duplicate URL policy, crawl scope, and risk. A topology engine should therefore collect evidence first and generate actions second.
|
||||
|
||||
The strongest common pattern across Playwright, Selenium, Scrapy, and Crawlee is:
|
||||
|
||||
1. Prefer user-facing and explicit contracts for element identity: role, label, placeholder, text, alt/title, and test IDs.
|
||||
2. Re-resolve locators at action time instead of caching DOM nodes.
|
||||
3. Treat every actionable element as state-local, not globally meaningful.
|
||||
4. Make waits signal-based, not time-based.
|
||||
5. Keep crawling bounded by robots policy, rate limits, duplicate filters, canonicalization rules, and explicit scope.
|
||||
6. Preserve diagnostic evidence: locator match counts, actionability checks, screenshots, DOM/text hashes, network events, dialogs, storage changes, and trace/HAR-like records.
|
||||
7. Treat anti-bot systems and authentication flows as compliance and authorization boundaries, not as targets for stealth bypass.
|
||||
|
||||
For this project, the next useful slice is a richer "capture evidence layer": framework hints, context anchors, frame/shadow paths, dynamic-stability observations, locator verification details, and crawl/explore policy metadata.
|
||||
|
||||
## Source Map
|
||||
|
||||
Primary and high-signal sources consulted:
|
||||
|
||||
- Playwright: [Best Practices](https://playwright.dev/docs/best-practices), [Locators](https://playwright.dev/docs/locators), [Auto-waiting and actionability](https://playwright.dev/docs/actionability), [Frames](https://playwright.dev/docs/frames), [Network](https://playwright.dev/docs/network), [Trace Viewer](https://playwright.dev/docs/trace-viewer), [Test generator](https://playwright.dev/docs/codegen), [Browser contexts](https://playwright.dev/docs/browser-contexts), [Dialogs](https://playwright.dev/docs/dialogs), [ElementHandle API](https://playwright.dev/docs/api/class-elementhandle), [FrameLocator API](https://playwright.dev/docs/api/class-framelocator).
|
||||
- Selenium: [Waiting Strategies](https://www.selenium.dev/documentation/webdriver/waits/), [Understanding Common Errors](https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/), [Locator strategies](https://www.selenium.dev/documentation/webdriver/elements/locators/), [Page Object Models](https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/).
|
||||
- Scrapy: [Broad Crawls](https://docs.scrapy.org/en/latest/topics/broad-crawls.html), [AutoThrottle](https://docs.scrapy.org/en/latest/topics/autothrottle.html), [Dynamic content](https://docs.scrapy.org/en/latest/topics/dynamic-content.html), [Requests and Responses](https://docs.scrapy.org/en/latest/topics/request-response.html), [Jobs: pausing and resuming crawls](https://docs.scrapy.org/en/latest/topics/jobs.html), [Link Extractors](https://docs.scrapy.org/en/latest/topics/link-extractors.html).
|
||||
- Crawlee: [Session management](https://crawlee.dev/python/docs/guides/session-management), [Proxy management](https://crawlee.dev/js/docs/3.14/guides/proxy-management), [PlaywrightCrawler for Python](https://crawlee.dev/python/docs/guides/playwright-crawler), [PlaywrightCrawler API for JavaScript](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler), [Crawling introduction](https://crawlee.dev/js/docs/introduction/crawling).
|
||||
- Web standards and browser platform: [RFC 9309 Robots Exclusion Protocol](https://datatracker.ietf.org/doc/html/rfc9309), [Google robots.txt guide](https://developers.google.com/search/docs/crawling-indexing/robots/intro), [Google sitemaps overview](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview), [Google canonical URL guidance](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls), [MDN ShadowRoot mode](https://developer.mozilla.org/en-US/docs/Web/API/ShadowRoot/mode), [MDN iframe element](https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/iframe), [MDN Same-origin policy](https://developer.mozilla.org/en-US/docs/Web/Security/Defenses/Same-origin_policy), [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/), [Chrome DevTools accessibility reference](https://developer.chrome.com/docs/devtools/accessibility/reference).
|
||||
- Bot and abuse context: [Cloudflare bot detection engines](https://developers.cloudflare.com/bots/concepts/bot-detection-engines/), [OWASP Automated Threats to Web Applications](https://owasp.org/www-project-automated-threats-to-web-applications/).
|
||||
|
||||
## Beginner Pitfalls
|
||||
|
||||
### Locator Pitfalls
|
||||
|
||||
1. Using long CSS/XPath paths as the primary locator.
|
||||
Playwright explicitly recommends user-facing locators and says CSS/XPath tend to be less resilient when DOM structure changes. For this project, CSS should remain a fallback candidate, not the source of truth.
|
||||
|
||||
2. Treating `id` as automatically stable.
|
||||
Many frameworks and component libraries generate IDs for hydration, ARIA wiring, or internal control relationships. An ID that looks unique in one run can change across reloads, locales, builds, or user sessions.
|
||||
|
||||
3. Ignoring accessible names.
|
||||
Role alone is rarely enough. `button` plus accessible name is much stronger than `button`. Accessible names may come from visible text, `aria-label`, `aria-labelledby`, `title`, `alt`, or associated labels.
|
||||
|
||||
4. Matching text too broadly.
|
||||
Text can appear in multiple places: hidden mobile menus, desktop menus, offscreen carousel slides, modal templates, body text, and cloned rows. Locator strictness failures are often a sign that context anchors are missing.
|
||||
|
||||
5. Assuming generated code is final.
|
||||
Playwright codegen is useful because it prioritizes role, text, and test ID locators, but generated locators still need validation against repeated components, hidden duplicates, localization, and dynamic state.
|
||||
|
||||
6. Using coordinates as if they are locators.
|
||||
Coordinates are viewport-, zoom-, device-, scroll-, and layout-dependent. They are useful only as last-resort diagnostic evidence or replay hints.
|
||||
|
||||
7. Reusing DOM element handles too long.
|
||||
Selenium's stale element errors and Playwright's discouraged `ElementHandle` usage both point to the same rule: do not cache a DOM node and expect it to survive rerendering.
|
||||
|
||||
### Timing and Actionability Pitfalls
|
||||
|
||||
1. Using hard sleeps.
|
||||
Playwright discourages production `waitForTimeout` because time-based waits are flaky. Selenium similarly emphasizes explicit waits for concrete conditions.
|
||||
|
||||
2. Waiting for the wrong signal.
|
||||
`networkidle` is not always the same as "business state ready". Modern apps may keep polling, use websockets, lazy-load sections, or rerender after hydration.
|
||||
|
||||
3. Bypassing actionability with `force`.
|
||||
Playwright's `force` disables non-essential actionability checks, including whether an element receives events. This can hide the real problem: overlay, animation, disabled state, wrong target, or stale state.
|
||||
|
||||
4. Confusing visible with clickable.
|
||||
An element can be visible yet covered by a transparent overlay, animating, disabled, outside the viewport, or not receiving pointer events.
|
||||
|
||||
5. Treating auto-wait as complete synchronization.
|
||||
Auto-wait helps with actionability, not with every application-level condition. A locator may be actionable before data has finished loading or before a dependent request has completed.
|
||||
|
||||
### State and DOM Lifecycle Pitfalls
|
||||
|
||||
1. Assuming the same URL means the same state.
|
||||
SPAs often change modals, tabs, accordions, filters, drawers, and virtualized content without changing URL.
|
||||
|
||||
2. Assuming the same DOM node means the same semantic element.
|
||||
Virtual lists reuse DOM nodes for different rows. A table row's DOM position can remain while its business record changes.
|
||||
|
||||
3. Ignoring modal and overlay stacks.
|
||||
Many actions target a menu item or dialog button that only exists after an opening action. The precondition path matters as much as the target locator.
|
||||
|
||||
4. Missing portal-rendered components.
|
||||
React, MUI, Ant Design, Angular Material, and similar libraries often render dropdowns, tooltips, dialogs, and menus under `body`, outside the triggering component subtree.
|
||||
|
||||
5. Ignoring viewport variants.
|
||||
Desktop and mobile markup can coexist, with one branch hidden. A locator that is unique at 1440px may not be unique at 390px.
|
||||
|
||||
### Frame, Shadow DOM, and Browser Boundary Pitfalls
|
||||
|
||||
1. Looking only in the main document.
|
||||
Payment widgets, embedded apps, captchas, login flows, rich editors, maps, and support widgets often live in iframes. Playwright requires frame-aware location through `frameLocator` or frame APIs.
|
||||
|
||||
2. Assuming shadow DOM is transparent.
|
||||
Playwright locators can pierce open shadow roots, but XPath does not pierce shadow roots and closed shadow roots are not accessible in the same way. The engine must record the shadow boundary.
|
||||
|
||||
3. Confusing iframe DOM access with browser security policy.
|
||||
Same-origin policy restricts cross-origin document/script interaction. The engine should record iframe origin and accessibility instead of assuming every frame can be inspected.
|
||||
|
||||
4. Forgetting dialogs, popups, downloads, and beforeunload prompts.
|
||||
Browser-level events are not normal DOM changes. They must be captured as effects or risk blockers.
|
||||
|
||||
### Crawling Pitfalls
|
||||
|
||||
1. Crawling without scope.
|
||||
Broad crawls need allowed domains, depth limits, URL filters, and maximum queue sizes. Otherwise calendars, faceted search, tracking parameters, and sort/filter combinations create infinite URL spaces.
|
||||
|
||||
2. Misusing canonicalization.
|
||||
URL canonicalization is useful for duplicate checking, but Scrapy notes that canonicalized URLs can differ from the URL visible to the server. The crawler should keep raw URL, canonical key, and request fingerprint separate.
|
||||
|
||||
3. Ignoring duplicate request policy.
|
||||
Crawlers need request fingerprints and duplicate filters. A topology engine also needs state hashes because different URLs can lead to the same state and one URL can lead to different states.
|
||||
|
||||
4. Treating robots.txt as security or permission to bypass.
|
||||
RFC 9309 frames robots rules as requested crawler behavior, not access authorization. Google also notes robots.txt is mainly for managing crawler traffic, not hiding private content.
|
||||
|
||||
5. Overloading sites.
|
||||
New crawlers often set concurrency too high. Scrapy's AutoThrottle exists because responsible crawl speed depends on crawler load and target-site load.
|
||||
|
||||
6. Assuming HTML contains the data.
|
||||
Scrapy's dynamic-content guidance starts by finding the data source: HTML source, embedded JavaScript, XHR/fetch response, API endpoint, or browser-rendered DOM. Browser automation is expensive and should be used when raw HTTP/API reproduction is insufficient.
|
||||
|
||||
7. Not preserving job state.
|
||||
Large crawls need durable queues, seen fingerprints, and clean pause/resume. Abrupt shutdown can corrupt state in crawler job directories.
|
||||
|
||||
### Anti-Bot and Compliance Pitfalls
|
||||
|
||||
1. Treating blocks as merely technical obstacles.
|
||||
Bot defenses classify automated traffic using heuristics, fingerprints, behavior, JavaScript challenges, request patterns, and reputation signals. For this repository, the right response is to record block states and stop or require authorization, not to build stealth bypass.
|
||||
|
||||
2. Scraping behind login, paywalls, or private areas without authority.
|
||||
This creates legal, contractual, and ethical risk. The engine should support storage state for authorized sessions, but it should not normalize unauthorized access.
|
||||
|
||||
3. Ignoring personally identifiable information.
|
||||
Extracted content can contain personal data, tokens, account identifiers, or internal URLs. Artifacts need redaction policy before being shared outside a controlled environment.
|
||||
|
||||
4. Missing high-risk action terms.
|
||||
Delete, submit, confirm, pay, authorize, purchase, and logout should remain gated. Chinese equivalents and product-specific destructive verbs need the same treatment.
|
||||
|
||||
## Advanced Techniques
|
||||
|
||||
### Locator Engineering
|
||||
|
||||
1. Generate multiple locator candidates per element.
|
||||
Store role/name, label, placeholder, text, test ID, stable attributes, local CSS fallback, frame path, and shadow path. Keep scores, reasons, verification results, and failure counts.
|
||||
|
||||
2. Verify locator identity, not just uniqueness.
|
||||
`count() === 1` is necessary but not sufficient. Also compare accessible name, tag, role, bounding box, visibility, enabled state, and a local semantic signature.
|
||||
|
||||
3. Use context anchors for repeated components.
|
||||
A robust locator often needs a container: dialog title, table row text, card heading, form label group, nav landmark, section heading, or list item.
|
||||
|
||||
4. Track hidden duplicates.
|
||||
Record how many matches are hidden, visible, disabled, or offscreen. Hidden duplicates explain strictness failures and viewport-specific bugs.
|
||||
|
||||
5. Keep locator descriptors structured.
|
||||
Store `{ kind: "role", role, name }`, not only string expressions. This lets consumers regenerate Playwright, Selenium, or other framework locators later.
|
||||
|
||||
6. Treat localization as a first-class stability concern.
|
||||
Text locators can be excellent in one locale and brittle across locales. Capture document language, locale hints, and alternate stable attributes when present.
|
||||
|
||||
### Actionability and Replay
|
||||
|
||||
1. Probe actionability before acting.
|
||||
Playwright supports trial-like checks and actionability behavior through locator actions. For this project, the graph should record visible, stable, enabled, editable, receives-events, and attached-like observations where possible.
|
||||
|
||||
2. Record precondition paths.
|
||||
Dropdown menu items, tabs, dialogs, accordions, and nested drawers require opening actions. Store the edge path that made the element visible.
|
||||
|
||||
3. Record browser effects as separate evidence.
|
||||
URL changes, visible text changes, DOM hash changes, network events, downloads, dialogs, storage changes, console errors, and history changes are different effect types.
|
||||
|
||||
4. Use isolated browser contexts.
|
||||
Playwright browser contexts isolate cookies, local storage, IndexedDB, cache, permissions, and auth state. This matters for reproducible scans and for comparing anonymous vs authenticated topology.
|
||||
|
||||
5. Prefer replay from the entry state.
|
||||
To verify an edge, reopen or reset the context, execute preconditions, then re-resolve the target locator. This catches stale references and hidden state coupling.
|
||||
|
||||
### Dynamic Content and Network-Aware Extraction
|
||||
|
||||
1. Inspect network before rendering everything.
|
||||
Scrapy recommends finding the data source for dynamic content. If data comes from a JSON endpoint, direct HTTP collection can be faster and more reliable than DOM scraping.
|
||||
|
||||
2. Capture XHR/fetch/document responses during exploration.
|
||||
Playwright network APIs can observe and modify traffic. The topology graph can record methods, URLs, statuses, content types, and request initiator context without storing sensitive payloads by default.
|
||||
|
||||
3. Separate DOM state from API state.
|
||||
A click can change text through local state without network, through XHR without URL change, through navigation, or through storage. These should not collapse into one "changed" flag.
|
||||
|
||||
4. Use HAR/trace-style diagnostics for failures.
|
||||
Playwright traces help debug failed runs after the fact. This project does not need to emit full Playwright traces in v1, but should keep enough small evidence to explain failures.
|
||||
|
||||
### Crawling Architecture
|
||||
|
||||
1. Maintain a durable request queue.
|
||||
Crawlee and Scrapy both emphasize queues/request providers for recursive crawling. A topology engine can stay small while still recording queue metadata for reproducibility.
|
||||
|
||||
2. Use request fingerprints and state fingerprints separately.
|
||||
Request fingerprint: dedupe crawl work. State fingerprint: dedupe observed browser states. Element signature: dedupe actionable inventory within a state.
|
||||
|
||||
3. Respect robots and crawl-delay-like policies where applicable.
|
||||
Robots rules are not an authorization layer, but they are the standard signal for crawler politeness. Store robots outcome and policy decisions in metadata.
|
||||
|
||||
4. Use adaptive throttling.
|
||||
Fixed concurrency is fragile. Scrapy AutoThrottle adjusts crawl speed based on load. For this repository, even a simpler per-host delay and max-concurrency policy would prevent many early mistakes.
|
||||
|
||||
5. Detect crawler traps.
|
||||
Watch for URL patterns that expand indefinitely: calendars, repeated path segments, session IDs, sort/filter permutations, page numbers with no terminal page, and tracking parameters.
|
||||
|
||||
6. Use sitemaps as seed hints, not truth.
|
||||
Sitemaps tell crawlers which pages the site considers important, but they do not guarantee completeness or validity. Store sitemap origin if used.
|
||||
|
||||
### Observability
|
||||
|
||||
1. Store why an element was considered actionable.
|
||||
Include selector source, role inference, event handler hints, tabindex, href/form attributes, ARIA states, contenteditable, and hit-test result.
|
||||
|
||||
2. Store why an action was skipped.
|
||||
Risk term, hidden, disabled, blocked by overlay, no verified locator, duplicate locator, outside scope, robots policy, auth required, or anti-bot block.
|
||||
|
||||
3. Keep compact artifacts, not raw everything.
|
||||
Full DOM and screenshots can be large and sensitive. Hashes plus selected evidence are usually enough for topology, with optional debug artifacts in controlled runs.
|
||||
|
||||
4. Make every score explainable.
|
||||
Locator scores should be inspectable: "role plus accessible name", "explicit test id", "CSS id fallback", "matched 3 elements", "visible duplicate exists".
|
||||
|
||||
## Framework and Component-Library Notes
|
||||
|
||||
The engine should avoid framework-specific hard dependencies, but it should collect framework hints that help rank locators and explain instability.
|
||||
|
||||
### React and Next.js
|
||||
|
||||
Common observations:
|
||||
|
||||
- Hydration can replace or modify nodes after initial HTML appears.
|
||||
- Portals render menus/dialogs outside the component subtree.
|
||||
- Component libraries may generate dynamic IDs for ARIA relationships.
|
||||
- Suspense/loading states can show temporary buttons or skeleton content.
|
||||
- Virtual lists reuse row DOM nodes.
|
||||
|
||||
Useful evidence:
|
||||
|
||||
- React/Next markers and root containers.
|
||||
- Hydration timing and post-load mutation bursts.
|
||||
- Portal containers under `body`.
|
||||
- Stable app-level data attributes.
|
||||
- Nearby row/card/section text for repeated controls.
|
||||
|
||||
### Vue and Nuxt
|
||||
|
||||
Common observations:
|
||||
|
||||
- Conditional rendering and transitions can leave entering/leaving elements in the DOM.
|
||||
- Slots make component ownership hard to infer from DOM ancestry.
|
||||
- Generated attributes/classes are not always stable across builds.
|
||||
|
||||
Useful evidence:
|
||||
|
||||
- App root markers.
|
||||
- Visibility and transition state.
|
||||
- Context anchors from labels, headings, and list items.
|
||||
- Explicit test or QA attributes when available.
|
||||
|
||||
### Angular and Angular Material
|
||||
|
||||
Common observations:
|
||||
|
||||
- Angular adds framework attributes and comment anchors that should not become primary locators.
|
||||
- Material overlays often render outside local component hierarchy.
|
||||
- Form fields may expose strong label and ARIA relationships.
|
||||
|
||||
Useful evidence:
|
||||
|
||||
- Overlay containers.
|
||||
- Label/control relationships.
|
||||
- `aria-controls`, `aria-expanded`, `role`, and selected/checked states.
|
||||
- Component-library class hints only as diagnostics, not primary locators.
|
||||
|
||||
### Ant Design, MUI, Element, Naive UI, and Similar Libraries
|
||||
|
||||
Common observations:
|
||||
|
||||
- Dropdowns, selects, date pickers, autocomplete panels, modals, and tooltips often render in shared overlay roots.
|
||||
- Visible labels may be split across nested spans.
|
||||
- Some components expose good ARIA roles; others require text/context anchors.
|
||||
- Duplicate hidden markup is common for responsive variants and transitions.
|
||||
|
||||
Useful evidence:
|
||||
|
||||
- Overlay root path.
|
||||
- Trigger-to-popup relationship through `aria-controls`, `aria-haspopup`, and `aria-expanded`.
|
||||
- Option lists and enum values.
|
||||
- Open/closed state before and after click.
|
||||
|
||||
### Web Components
|
||||
|
||||
Common observations:
|
||||
|
||||
- Open shadow roots can usually be traversed by Playwright locators.
|
||||
- Closed shadow roots intentionally hide internals.
|
||||
- XPath does not pierce shadow roots.
|
||||
|
||||
Useful evidence:
|
||||
|
||||
- Custom element tag name.
|
||||
- Shadow root mode if observable.
|
||||
- Host attributes and accessible role/name.
|
||||
- Slot text and public ARIA surface.
|
||||
|
||||
### Server-Rendered or Traditional Sites
|
||||
|
||||
Common observations:
|
||||
|
||||
- HTML forms, anchors, and submit buttons expose strong native semantics.
|
||||
- CSS locators may be more stable than in SPAs, but still should be fallback.
|
||||
- Navigation effects are clearer, but forms can trigger high-risk writes.
|
||||
|
||||
Useful evidence:
|
||||
|
||||
- Form action/method/enctype.
|
||||
- Input name/type/required/pattern.
|
||||
- Link href and target.
|
||||
- Submit/reset button type.
|
||||
|
||||
## Recommended Evidence to Collect
|
||||
|
||||
### State Evidence
|
||||
|
||||
- URL, route, title, language, viewport, user agent family.
|
||||
- DOM hash, visible text hash, actionable signature hash.
|
||||
- Optional accessibility-tree or accessibility-signature hash.
|
||||
- Modal, drawer, menu, popover, and overlay stack.
|
||||
- Frame tree: URL, name, title, origin, sandbox/allow hints where available.
|
||||
- Framework and component-library hints.
|
||||
- Network summary: document URL, XHR/fetch endpoints, status families, redirects.
|
||||
- Storage summary: cookie/localStorage/sessionStorage/IndexedDB presence counts, not raw sensitive values by default.
|
||||
- Mutation summary: whether actionable inventory changes after domcontentloaded/load/networkidle.
|
||||
|
||||
### Element Evidence
|
||||
|
||||
- State-local `elementId`.
|
||||
- Tag, inferred kind, role, accessible name, text, label, placeholder, title, alt.
|
||||
- Selected attributes: `id`, `name`, `type`, `href`, `value`, `for`, `role`, ARIA fields, test IDs, QA attributes, automation IDs.
|
||||
- Form context: enclosing form action/method, fieldset/legend, required/disabled/readonly/pattern/min/max.
|
||||
- Component context: nearest dialog/menu/list/table/card/form/section/nav/main, heading text, row text, column header, list item text.
|
||||
- Frame path and shadow path.
|
||||
- Bounding box, viewport relation, z-index-ish overlay observation, center hit-test result.
|
||||
- Interactability: visible, enabled, editable, checked/selected, expanded, pressed, receives events.
|
||||
- Event/action hints: href navigation, submit/reset, file upload, contenteditable, onclick, keyboard role, pointer cursor.
|
||||
- Risk terms and risk classification.
|
||||
|
||||
### Locator Candidate Evidence
|
||||
|
||||
- Structured descriptor and generated expression.
|
||||
- Strategy: role, label, placeholder, text, test ID, alt/title, stable attribute, local CSS, XPath, coordinate fallback.
|
||||
- Score and reason.
|
||||
- Match count.
|
||||
- Visible match count.
|
||||
- Enabled/editable match count.
|
||||
- Whether the match points to the same semantic signature.
|
||||
- Whether the locator survives reload and precondition replay.
|
||||
- Failure count and last failure reason.
|
||||
|
||||
### Action Edge Evidence
|
||||
|
||||
- From state and to state.
|
||||
- Target element and chosen locator.
|
||||
- Action type: click, fill, select, check, uncheck, upload, hover, keypress, navigate.
|
||||
- Parameters or parameter schema.
|
||||
- Preconditions and opening path.
|
||||
- Effects: URL, DOM hash, visible text hash, network, dialog, popup, download, storage, console error.
|
||||
- Risk decision: executed, skipped, requires approval, blocked by policy.
|
||||
- Confidence and replay verification status.
|
||||
|
||||
### Crawl Policy Evidence
|
||||
|
||||
- Entry URLs and seed source: explicit, sitemap, in-page links, recorded trace.
|
||||
- Allowed domains and path filters.
|
||||
- Robots fetch result and policy outcome.
|
||||
- Rate limits, concurrency, backoff, retry policy.
|
||||
- Duplicate policy: raw URL, normalized URL, canonical URL, request fingerprint.
|
||||
- Trap controls: max depth, max pages per host, query parameter allowlist/denylist, calendar/facet detection.
|
||||
- Auth mode: anonymous, supplied storage state, or manually authenticated context.
|
||||
|
||||
## Suggested Near-Term Design Direction for This Repository
|
||||
|
||||
### 1. Add a Capture Evidence Layer
|
||||
|
||||
Introduce richer internal raw evidence before changing graph behavior too much. The graph can expose selected fields, while the engine keeps implementation detail behind stable types.
|
||||
|
||||
Candidate additions:
|
||||
|
||||
- `frameworkHints` on `StateNode` or metadata.
|
||||
- `context` on `ActionableElement`: nearest landmark, dialog, form, table row, card, section heading.
|
||||
- Real `framePath` and `shadowPath`.
|
||||
- Locator verification details beyond `matchCount`.
|
||||
- `skipReason` or exploration decision records for skipped elements.
|
||||
|
||||
### 2. Add Locator Context Ranking
|
||||
|
||||
Keep current priority:
|
||||
|
||||
1. role plus accessible name
|
||||
2. label
|
||||
3. placeholder
|
||||
4. test ID or explicit automation attribute
|
||||
5. text
|
||||
6. stable local CSS
|
||||
|
||||
Then add context-aware variants:
|
||||
|
||||
- `dialog[name].getByRole(...)`
|
||||
- `row[hasText].getByRole(...)`
|
||||
- `form[has label/legend].getByLabel(...)`
|
||||
- `section[heading].getByRole(...)`
|
||||
- frame/shadow-aware variants
|
||||
|
||||
### 3. Add Replay-Based Verification
|
||||
|
||||
A locator should have at least two levels of verification:
|
||||
|
||||
- Snapshot verification: resolves uniquely now.
|
||||
- Replay verification: after reload or precondition replay, resolves to the same semantic target.
|
||||
|
||||
This will catch framework rerendering, transient DOM, and hidden duplicate issues much earlier.
|
||||
|
||||
### 4. Add Conservative Crawl/Explore Policy
|
||||
|
||||
Before expanding exploration:
|
||||
|
||||
- Per-host max pages and max actions.
|
||||
- Explicit allowlist for domains and paths.
|
||||
- Robots policy record.
|
||||
- Query parameter normalization policy.
|
||||
- Risk-gated default action set.
|
||||
- No form submission or destructive clicks by default.
|
||||
|
||||
### 5. Keep Anti-Bot Handling Non-Evasive
|
||||
|
||||
For this repo, anti-bot detection should become observable state, not a bypass feature:
|
||||
|
||||
- detect challenge/block pages,
|
||||
- record response status and visible challenge text,
|
||||
- mark edge/state as blocked or requires authorization,
|
||||
- stop exploration unless the user supplied an authorized session and policy allows it.
|
||||
|
||||
## What Not To Build Here
|
||||
|
||||
To preserve the repository boundary:
|
||||
|
||||
- Do not add LLM task planning.
|
||||
- Do not add natural-language instruction execution.
|
||||
- Do not add stealth CAPTCHA or paywall bypass.
|
||||
- Do not add a generic browser-agent orchestration layer.
|
||||
- Do not turn generated topology into autonomous high-risk actions.
|
||||
- Do not store raw sensitive cookies, tokens, request bodies, or personal data in default artifacts.
|
||||
|
||||
## Practical Checklist for Future Implementation
|
||||
|
||||
Before a locator or action edge is considered reliable, the engine should be able to answer:
|
||||
|
||||
1. Which page state owned this element?
|
||||
2. Which frame and shadow boundary contained it?
|
||||
3. Which user-facing semantics identified it?
|
||||
4. Which nearby context disambiguated it from similar elements?
|
||||
5. Which locator candidates were generated?
|
||||
6. Which candidates were verified, and how many elements did each match?
|
||||
7. Was the target visible, enabled, stable, editable if needed, and receiving events?
|
||||
8. Did replay from entry/preconditions find the same target?
|
||||
9. What happened after action: URL, DOM, text, network, storage, dialog, download?
|
||||
10. Was the action skipped or gated by risk, robots, scope, auth, or anti-bot policy?
|
||||
|
||||
## Bottom Line
|
||||
|
||||
The high-leverage research conclusion is simple: reliable browser automation is mostly evidence management. The engine should capture enough state-local, semantic, contextual, timing, and policy evidence that downstream consumers can choose actions quickly without guessing why a locator worked once.
|
||||
17
docs/source/2026-06-17-markitdown-reference-notes.md
Normal file
17
docs/source/2026-06-17-markitdown-reference-notes.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# MarkItDown Reference Notes
|
||||
|
||||
Source reviewed: Microsoft `markitdown`, official repository and README.
|
||||
|
||||
## Useful Patterns
|
||||
|
||||
- Keep conversion and extraction units narrow. MarkItDown separates `accepts()` from `convert()`, which maps well to this engine's separation between candidate selection, locator verification, replay, and failure records.
|
||||
- Prefer a registry-style extraction pipeline when multiple strategies may apply. MarkItDown registers specific converters before generic HTML/plain-text fallbacks; a future topology engine can use the same pattern for snapshot extraction strategies without becoming a generic crawler.
|
||||
- Preserve machine-friendly output over high-fidelity rendering. MarkItDown targets Markdown for LLM/text pipelines, not perfect visual reproduction. This matches this project's graph-artifact boundary.
|
||||
- Normalize output centrally. MarkItDown trims trailing line whitespace and collapses excessive blank lines after converter execution; this project now uses graph summaries and JSON Schema validation as artifact-level normalization checks.
|
||||
- Keep security boundaries explicit. MarkItDown warns that URI/file conversion performs I/O with current process privileges and recommends narrow conversion APIs. This project should keep navigation scopes, timeouts, and risk policy explicit rather than adding broad fetch/convert behavior.
|
||||
|
||||
## Non-Adopted Items
|
||||
|
||||
- Do not add MarkItDown as a runtime dependency for v0.2.2.
|
||||
- Do not emit Markdown as the primary artifact. Markdown can be future evidence or summaries, but the project contract remains `TopologyGraph`.
|
||||
- Do not add LLM/OCR/cloud conversion plugins to this repository.
|
||||
1955
docs/superpowers/plans/2026-06-16-v0.2.1-evidence-layer.md
Normal file
1955
docs/superpowers/plans/2026-06-16-v0.2.1-evidence-layer.md
Normal file
File diff suppressed because it is too large
Load Diff
59
package-lock.json
generated
59
package-lock.json
generated
@@ -15,6 +15,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.15.30",
|
||||
"ajv": "^8.20.0",
|
||||
"tsx": "^4.20.3",
|
||||
"typescript": "^5.8.3"
|
||||
}
|
||||
@@ -471,6 +472,23 @@
|
||||
"undici-types": "~6.21.0"
|
||||
}
|
||||
},
|
||||
"node_modules/ajv": {
|
||||
"version": "8.20.0",
|
||||
"resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz",
|
||||
"integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"fast-deep-equal": "^3.1.3",
|
||||
"fast-uri": "^3.0.1",
|
||||
"json-schema-traverse": "^1.0.0",
|
||||
"require-from-string": "^2.0.2"
|
||||
},
|
||||
"funding": {
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/epoberezkin"
|
||||
}
|
||||
},
|
||||
"node_modules/esbuild": {
|
||||
"version": "0.28.1",
|
||||
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.28.1.tgz",
|
||||
@@ -513,6 +531,30 @@
|
||||
"@esbuild/win32-x64": "0.28.1"
|
||||
}
|
||||
},
|
||||
"node_modules/fast-deep-equal": {
|
||||
"version": "3.1.3",
|
||||
"resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz",
|
||||
"integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/fast-uri": {
|
||||
"version": "3.1.2",
|
||||
"resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.2.tgz",
|
||||
"integrity": "sha512-rVjf7ArG3LTk+FS6Yw81V1DLuZl1bRbNrev6Tmd/9RaroeeRRJhAt7jg/6YFxbvAQXUCavSoZhPPj6oOx+5KjQ==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/fastify"
|
||||
},
|
||||
{
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/fastify"
|
||||
}
|
||||
],
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/fsevents": {
|
||||
"version": "2.3.2",
|
||||
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
|
||||
@@ -527,6 +569,13 @@
|
||||
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/json-schema-traverse": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz",
|
||||
"integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/playwright": {
|
||||
"version": "1.61.0",
|
||||
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.61.0.tgz",
|
||||
@@ -557,6 +606,16 @@
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/require-from-string": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
|
||||
"integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/tsx": {
|
||||
"version": "4.22.4",
|
||||
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.22.4.tgz",
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
"scripts": {
|
||||
"build": "tsc -p tsconfig.json",
|
||||
"typecheck": "tsc --noEmit -p tsconfig.json",
|
||||
"test": "tsx --test tests/**/*.test.ts",
|
||||
"scan": "tsx src/cli.ts scan",
|
||||
"explore": "tsx src/cli.ts explore",
|
||||
"example": "tsx src/cli.ts explore --url ./examples/simple.html --out artifacts/simple-graph.json --max-actions 8"
|
||||
@@ -19,8 +20,8 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.15.30",
|
||||
"ajv": "^8.20.0",
|
||||
"tsx": "^4.20.3",
|
||||
"typescript": "^5.8.3"
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
552
schemas/topology-graph.schema.json
Normal file
552
schemas/topology-graph.schema.json
Normal file
@@ -0,0 +1,552 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"$id": "https://action-topology-engine.local/schemas/topology-graph.schema.json",
|
||||
"title": "TopologyGraph",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"schemaVersion",
|
||||
"metadata",
|
||||
"states",
|
||||
"elements",
|
||||
"edges",
|
||||
"skippedActions",
|
||||
"failedObservations",
|
||||
"summary"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"schemaVersion": { "const": "0.2" },
|
||||
"metadata": { "$ref": "#/$defs/metadata" },
|
||||
"states": { "type": "array", "items": { "$ref": "#/$defs/stateNode" } },
|
||||
"elements": { "type": "array", "items": { "$ref": "#/$defs/actionableElement" } },
|
||||
"edges": { "type": "array", "items": { "$ref": "#/$defs/actionEdge" } },
|
||||
"skippedActions": { "type": "array", "items": { "$ref": "#/$defs/skippedAction" } },
|
||||
"failedObservations": { "type": "array", "items": { "$ref": "#/$defs/failedObservation" } },
|
||||
"summary": { "$ref": "#/$defs/graphSummary" }
|
||||
},
|
||||
"$defs": {
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"artifactVersion",
|
||||
"inputUrl",
|
||||
"entryUrl",
|
||||
"generatedAt",
|
||||
"engineVersion",
|
||||
"mode",
|
||||
"crawlSessionId",
|
||||
"inputSeed",
|
||||
"inputFingerprint",
|
||||
"runtimeContext",
|
||||
"site",
|
||||
"riskPolicyVersion",
|
||||
"normalizationRuleVersion",
|
||||
"locatorScoringVersion"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"artifactVersion": { "const": "topology-graph-v0.2" },
|
||||
"inputUrl": { "type": "string" },
|
||||
"entryUrl": { "type": "string" },
|
||||
"generatedAt": { "type": "string" },
|
||||
"engineVersion": { "const": "0.2.2" },
|
||||
"mode": { "enum": ["scan", "explore"] },
|
||||
"crawlSessionId": { "type": "string", "pattern": "^crawl_" },
|
||||
"inputSeed": { "type": "string" },
|
||||
"inputFingerprint": { "type": "string", "pattern": "^input_" },
|
||||
"runtimeContext": {
|
||||
"type": "object",
|
||||
"required": ["browserName", "viewport", "locale", "timezoneId", "authContext"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"browserName": { "const": "chromium" },
|
||||
"viewport": { "$ref": "#/$defs/viewport" },
|
||||
"locale": { "type": "string" },
|
||||
"timezoneId": { "type": ["string", "null"] },
|
||||
"authContext": { "enum": ["anonymous", "storage_state"] }
|
||||
}
|
||||
},
|
||||
"site": {
|
||||
"type": "object",
|
||||
"required": ["baseUrl", "entryUrl"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"baseUrl": { "type": "string" },
|
||||
"entryUrl": { "type": "string" }
|
||||
}
|
||||
},
|
||||
"riskPolicyVersion": { "const": "risk-policy-v0.2" },
|
||||
"normalizationRuleVersion": { "const": "normalization-v0.2" },
|
||||
"locatorScoringVersion": { "const": "locator-scoring-v0.2" }
|
||||
}
|
||||
},
|
||||
"stateNode": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"stateId",
|
||||
"url",
|
||||
"route",
|
||||
"title",
|
||||
"frameContext",
|
||||
"stateHash",
|
||||
"domHash",
|
||||
"visibleTextHash",
|
||||
"actionableSignatureHash",
|
||||
"modalStack",
|
||||
"openedByEdgeId",
|
||||
"createdAt",
|
||||
"lifecycle",
|
||||
"urlCanonicalKey",
|
||||
"language",
|
||||
"viewport",
|
||||
"evidence"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"stateId": { "type": "string", "pattern": "^state_" },
|
||||
"url": { "type": "string" },
|
||||
"route": { "type": "string" },
|
||||
"title": { "type": "string" },
|
||||
"frameContext": { "type": "array", "items": { "type": "string" } },
|
||||
"stateHash": { "type": "string" },
|
||||
"domHash": { "type": "string" },
|
||||
"visibleTextHash": { "type": "string" },
|
||||
"actionableSignatureHash": { "type": "string" },
|
||||
"modalStack": { "type": "array", "items": { "type": "string" } },
|
||||
"openedByEdgeId": { "type": ["string", "null"] },
|
||||
"createdAt": { "type": "string" },
|
||||
"lifecycle": { "enum": ["observed", "fingerprinted", "verified", "stale", "deprecated", "conflicted"] },
|
||||
"urlCanonicalKey": { "type": "string" },
|
||||
"language": { "type": ["string", "null"] },
|
||||
"viewport": { "$ref": "#/$defs/viewport" },
|
||||
"evidence": { "$ref": "#/$defs/evidenceList" }
|
||||
}
|
||||
},
|
||||
"actionableElement": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"elementId",
|
||||
"stateId",
|
||||
"tag",
|
||||
"role",
|
||||
"accessibleName",
|
||||
"text",
|
||||
"label",
|
||||
"placeholder",
|
||||
"attributes",
|
||||
"bbox",
|
||||
"framePath",
|
||||
"shadowPath",
|
||||
"interactability",
|
||||
"locators",
|
||||
"context",
|
||||
"parameterSurface",
|
||||
"evidence"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"elementId": { "type": "string", "pattern": "^el_" },
|
||||
"stateId": { "type": "string", "pattern": "^state_" },
|
||||
"tag": { "type": "string" },
|
||||
"role": { "type": ["string", "null"] },
|
||||
"accessibleName": { "type": ["string", "null"] },
|
||||
"text": { "type": ["string", "null"] },
|
||||
"label": { "type": ["string", "null"] },
|
||||
"placeholder": { "type": ["string", "null"] },
|
||||
"attributes": { "type": "object", "additionalProperties": { "type": "string" } },
|
||||
"bbox": { "anyOf": [{ "$ref": "#/$defs/boundingBox" }, { "type": "null" }] },
|
||||
"framePath": { "type": "array", "items": { "type": "string" } },
|
||||
"shadowPath": { "type": "array", "items": { "type": "string" } },
|
||||
"interactability": { "$ref": "#/$defs/interactability" },
|
||||
"locators": { "type": "array", "items": { "$ref": "#/$defs/locatorCandidate" } },
|
||||
"context": { "$ref": "#/$defs/elementContext" },
|
||||
"parameterSurface": { "anyOf": [{ "$ref": "#/$defs/parameterSurface" }, { "type": "null" }] },
|
||||
"evidence": { "$ref": "#/$defs/evidenceList" }
|
||||
}
|
||||
},
|
||||
"locatorCandidate": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"locatorId",
|
||||
"elementId",
|
||||
"framework",
|
||||
"strategy",
|
||||
"descriptor",
|
||||
"expression",
|
||||
"score",
|
||||
"reason",
|
||||
"verified",
|
||||
"scopes",
|
||||
"identity",
|
||||
"evidence",
|
||||
"failureCount"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"locatorId": { "type": "string", "pattern": "^loc_" },
|
||||
"elementId": { "type": "string", "pattern": "^el_" },
|
||||
"framework": { "enum": ["playwright", "css"] },
|
||||
"strategy": { "enum": ["role", "label", "placeholder", "text", "testId", "css", "scoped"] },
|
||||
"descriptor": { "$ref": "#/$defs/locatorDescriptor" },
|
||||
"expression": { "type": "string" },
|
||||
"score": { "type": "number" },
|
||||
"reason": { "type": "string" },
|
||||
"verified": { "type": "boolean" },
|
||||
"lastVerifiedAt": { "type": "string" },
|
||||
"matchCount": { "type": "integer", "minimum": 0 },
|
||||
"scopes": { "type": "array", "items": { "$ref": "#/$defs/locatorScope" } },
|
||||
"identity": { "anyOf": [{ "$ref": "#/$defs/locatorIdentity" }, { "type": "null" }] },
|
||||
"evidence": { "$ref": "#/$defs/evidenceList" },
|
||||
"failureCount": { "type": "integer", "minimum": 0 }
|
||||
}
|
||||
},
|
||||
"locatorDescriptor": {
|
||||
"oneOf": [
|
||||
{ "$ref": "#/$defs/baseLocatorDescriptor" },
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["kind", "scope", "target"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"kind": { "const": "scoped" },
|
||||
"scope": { "$ref": "#/$defs/locatorScope" },
|
||||
"target": { "$ref": "#/$defs/baseLocatorDescriptor" }
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"baseLocatorDescriptor": {
|
||||
"oneOf": [
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["kind", "role"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"kind": { "const": "role" },
|
||||
"role": { "type": "string" },
|
||||
"name": { "type": "string" }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["kind", "label"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"kind": { "const": "label" },
|
||||
"label": { "type": "string" }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["kind", "placeholder"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"kind": { "const": "placeholder" },
|
||||
"placeholder": { "type": "string" }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["kind", "text", "exact"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"kind": { "const": "text" },
|
||||
"text": { "type": "string" },
|
||||
"exact": { "type": "boolean" }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["kind", "testId"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"kind": { "const": "testId" },
|
||||
"testId": { "type": "string" }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["kind", "selector"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"kind": { "const": "css" },
|
||||
"selector": { "type": "string" }
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"actionEdge": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"edgeId",
|
||||
"fromStateId",
|
||||
"toStateId",
|
||||
"elementId",
|
||||
"actionType",
|
||||
"actionParams",
|
||||
"preconditions",
|
||||
"effects",
|
||||
"parameterSpecs",
|
||||
"riskLevel",
|
||||
"riskCategory",
|
||||
"confidence",
|
||||
"createdAt"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"edgeId": { "type": "string", "pattern": "^edge_" },
|
||||
"fromStateId": { "type": "string", "pattern": "^state_" },
|
||||
"toStateId": { "type": ["string", "null"] },
|
||||
"elementId": { "type": "string", "pattern": "^el_" },
|
||||
"actionType": { "$ref": "#/$defs/actionType" },
|
||||
"actionParams": { "type": "object" },
|
||||
"preconditions": { "type": "array", "items": { "type": "string" } },
|
||||
"effects": { "type": "array", "items": { "$ref": "#/$defs/effectRecord" } },
|
||||
"parameterSpecs": { "type": "array", "items": { "$ref": "#/$defs/parameterSpec" } },
|
||||
"riskLevel": { "$ref": "#/$defs/riskLevel" },
|
||||
"riskCategory": { "$ref": "#/$defs/riskCategory" },
|
||||
"confidence": { "type": "number" },
|
||||
"createdAt": { "type": "string" }
|
||||
}
|
||||
},
|
||||
"effectRecord": {
|
||||
"type": "object",
|
||||
"required": ["type", "description"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"type": { "enum": ["url", "dom", "visibleText", "actionableInventory"] },
|
||||
"before": { "type": "string" },
|
||||
"after": { "type": "string" },
|
||||
"description": { "type": "string" }
|
||||
}
|
||||
},
|
||||
"skippedAction": {
|
||||
"type": "object",
|
||||
"required": ["skippedActionId", "stateId", "elementId", "actionType", "riskLevel", "riskCategory", "reason", "evidence", "createdAt"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"skippedActionId": { "type": "string", "pattern": "^skip_" },
|
||||
"stateId": { "type": "string", "pattern": "^state_" },
|
||||
"elementId": { "type": "string", "pattern": "^el_" },
|
||||
"actionType": { "$ref": "#/$defs/actionType" },
|
||||
"riskLevel": { "$ref": "#/$defs/riskLevel" },
|
||||
"riskCategory": { "$ref": "#/$defs/riskCategory" },
|
||||
"reason": { "type": "string" },
|
||||
"evidence": { "$ref": "#/$defs/evidenceList" },
|
||||
"createdAt": { "type": "string" }
|
||||
}
|
||||
},
|
||||
"failedObservation": {
|
||||
"type": "object",
|
||||
"required": ["failedObservationId", "stateId", "elementId", "actionType", "reason", "message", "evidence", "createdAt"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"failedObservationId": { "type": "string", "pattern": "^fail_" },
|
||||
"stateId": { "type": ["string", "null"] },
|
||||
"elementId": { "type": ["string", "null"] },
|
||||
"actionType": { "anyOf": [{ "$ref": "#/$defs/actionType" }, { "type": "null" }] },
|
||||
"reason": { "$ref": "#/$defs/failedObservationReason" },
|
||||
"message": { "type": "string" },
|
||||
"evidence": { "$ref": "#/$defs/evidenceList" },
|
||||
"createdAt": { "type": "string" }
|
||||
}
|
||||
},
|
||||
"graphSummary": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"states",
|
||||
"elements",
|
||||
"edges",
|
||||
"skippedActions",
|
||||
"failedObservations",
|
||||
"failedObservationsByReason",
|
||||
"skippedActionsByRiskCategory",
|
||||
"edgesByRiskCategory"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"states": { "type": "integer", "minimum": 0 },
|
||||
"elements": { "type": "integer", "minimum": 0 },
|
||||
"edges": { "type": "integer", "minimum": 0 },
|
||||
"skippedActions": { "type": "integer", "minimum": 0 },
|
||||
"failedObservations": { "type": "integer", "minimum": 0 },
|
||||
"failedObservationsByReason": { "$ref": "#/$defs/countMap" },
|
||||
"skippedActionsByRiskCategory": { "$ref": "#/$defs/countMap" },
|
||||
"edgesByRiskCategory": { "$ref": "#/$defs/countMap" }
|
||||
}
|
||||
},
|
||||
"parameterSpec": {
|
||||
"type": "object",
|
||||
"required": ["name", "type", "required"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": { "type": "string" },
|
||||
"type": { "enum": ["string", "number", "boolean", "enum", "date", "unknown"] },
|
||||
"required": { "type": "boolean" },
|
||||
"enumValues": { "type": "array", "items": { "type": "string" } },
|
||||
"pattern": { "type": "string" },
|
||||
"examples": { "type": "array", "items": { "type": "string" } }
|
||||
}
|
||||
},
|
||||
"parameterSurface": {
|
||||
"type": "object",
|
||||
"required": ["name", "inputType", "required", "readonly", "disabled", "pattern", "min", "max", "maxLength", "options"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": { "type": ["string", "null"] },
|
||||
"inputType": { "type": ["string", "null"] },
|
||||
"required": { "type": "boolean" },
|
||||
"readonly": { "type": "boolean" },
|
||||
"disabled": { "type": "boolean" },
|
||||
"pattern": { "type": ["string", "null"] },
|
||||
"min": { "type": ["string", "null"] },
|
||||
"max": { "type": ["string", "null"] },
|
||||
"maxLength": { "type": ["integer", "null"] },
|
||||
"options": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["label", "value", "disabled", "selected"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"label": { "type": "string" },
|
||||
"value": { "type": "string" },
|
||||
"disabled": { "type": "boolean" },
|
||||
"selected": { "type": "boolean" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"elementContext": {
|
||||
"type": "object",
|
||||
"required": ["dialog", "form", "section", "row", "landmark", "listitem"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"dialog": { "anyOf": [{ "$ref": "#/$defs/contextAnchor" }, { "type": "null" }] },
|
||||
"form": { "anyOf": [{ "$ref": "#/$defs/contextAnchor" }, { "type": "null" }] },
|
||||
"section": { "anyOf": [{ "$ref": "#/$defs/contextAnchor" }, { "type": "null" }] },
|
||||
"row": { "anyOf": [{ "$ref": "#/$defs/contextAnchor" }, { "type": "null" }] },
|
||||
"landmark": { "anyOf": [{ "$ref": "#/$defs/contextAnchor" }, { "type": "null" }] },
|
||||
"listitem": { "anyOf": [{ "$ref": "#/$defs/contextAnchor" }, { "type": "null" }] }
|
||||
}
|
||||
},
|
||||
"contextAnchor": {
|
||||
"type": "object",
|
||||
"required": ["kind", "name", "text"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"kind": { "enum": ["dialog", "form", "section", "row", "landmark", "listitem"] },
|
||||
"name": { "type": ["string", "null"] },
|
||||
"text": { "type": ["string", "null"] }
|
||||
}
|
||||
},
|
||||
"locatorScope": {
|
||||
"type": "object",
|
||||
"required": ["kind", "name", "text"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"kind": { "enum": ["dialog", "form", "section", "row", "landmark", "listitem"] },
|
||||
"name": { "type": ["string", "null"] },
|
||||
"text": { "type": ["string", "null"] }
|
||||
}
|
||||
},
|
||||
"locatorIdentity": {
|
||||
"type": "object",
|
||||
"required": ["sameTag", "sameRole", "sameAccessibleName", "sameText", "sameVisibility"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"sameTag": { "type": "boolean" },
|
||||
"sameRole": { "type": "boolean" },
|
||||
"sameAccessibleName": { "type": "boolean" },
|
||||
"sameText": { "type": "boolean" },
|
||||
"sameVisibility": { "type": "boolean" }
|
||||
}
|
||||
},
|
||||
"interactability": {
|
||||
"type": "object",
|
||||
"required": ["visible", "enabled", "editable", "receivesEvents"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"visible": { "type": "boolean" },
|
||||
"enabled": { "type": "boolean" },
|
||||
"editable": { "type": "boolean" },
|
||||
"receivesEvents": { "type": "boolean" }
|
||||
}
|
||||
},
|
||||
"boundingBox": {
|
||||
"type": "object",
|
||||
"required": ["x", "y", "width", "height"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"x": { "type": "number" },
|
||||
"y": { "type": "number" },
|
||||
"width": { "type": "number" },
|
||||
"height": { "type": "number" }
|
||||
}
|
||||
},
|
||||
"viewport": {
|
||||
"type": "object",
|
||||
"required": ["width", "height"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"width": { "type": "integer" },
|
||||
"height": { "type": "integer" }
|
||||
}
|
||||
},
|
||||
"evidenceList": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/$defs/evidenceRecord" }
|
||||
},
|
||||
"evidenceRecord": {
|
||||
"type": "object",
|
||||
"required": ["level", "source", "description"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"level": { "enum": ["observed", "derived", "verified", "heuristic", "external"] },
|
||||
"source": { "type": "string" },
|
||||
"description": { "type": "string" }
|
||||
}
|
||||
},
|
||||
"actionType": { "enum": ["click", "fill", "select", "check", "uncheck", "navigate"] },
|
||||
"riskLevel": { "enum": ["low", "medium", "high"] },
|
||||
"riskCategory": {
|
||||
"enum": [
|
||||
"read_only",
|
||||
"navigation",
|
||||
"ui_reveal",
|
||||
"input_edit",
|
||||
"form_submit",
|
||||
"data_mutation",
|
||||
"destructive",
|
||||
"auth_sensitive",
|
||||
"payment_sensitive",
|
||||
"unknown"
|
||||
]
|
||||
},
|
||||
"failedObservationReason": {
|
||||
"enum": [
|
||||
"locator_not_found",
|
||||
"locator_not_unique",
|
||||
"element_hidden",
|
||||
"element_disabled",
|
||||
"element_obscured",
|
||||
"state_precondition_missing",
|
||||
"frame_inaccessible",
|
||||
"shadow_root_closed",
|
||||
"navigation_timeout",
|
||||
"network_error",
|
||||
"permission_denied",
|
||||
"auth_required",
|
||||
"captcha_or_challenge",
|
||||
"risk_policy_blocked",
|
||||
"effect_not_observed",
|
||||
"unexpected_state"
|
||||
]
|
||||
},
|
||||
"countMap": {
|
||||
"type": "object",
|
||||
"additionalProperties": { "type": "integer", "minimum": 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,51 +1,121 @@
|
||||
# Topology Graph Schema
|
||||
|
||||
This document describes the JSON shape emitted by the engine. It is a practical schema note, not yet a formal JSON Schema file.
|
||||
This document describes the JSON shape emitted by the engine. A formal JSON Schema is also provided in `schemas/topology-graph.schema.json`.
|
||||
|
||||
## `TopologyGraph`
|
||||
|
||||
- `schemaVersion`: currently `0.1`.
|
||||
- `schemaVersion`: currently `0.2`.
|
||||
- `metadata.artifactVersion`: currently `topology-graph-v0.2`.
|
||||
- `metadata.inputUrl`: raw URL or file path supplied by the caller.
|
||||
- `metadata.entryUrl`: entry URL after Playwright navigation.
|
||||
- `metadata.generatedAt`: ISO timestamp for this artifact generation.
|
||||
- `metadata.engineVersion`: engine implementation version that emitted the artifact.
|
||||
- `metadata.mode`: `scan` or `explore`.
|
||||
- `metadata.crawlSessionId`: unique non-deterministic identifier for this graph generation session. It is intentionally different across runs.
|
||||
- `metadata.inputSeed`: caller-supplied input seed. In v0.2 this is the raw `inputUrl`.
|
||||
- `metadata.inputFingerprint`: deterministic hash of input URL, mode, and policy/scoring versions.
|
||||
- `metadata.runtimeContext`: browser engine, viewport, locale, timezone, and auth context.
|
||||
- `metadata.site`: base URL and entry URL.
|
||||
- `metadata.riskPolicyVersion`: currently `risk-policy-v0.2`.
|
||||
- `metadata.normalizationRuleVersion`: currently `normalization-v0.2`.
|
||||
- `metadata.locatorScoringVersion`: currently `locator-scoring-v0.2`.
|
||||
- `states`: page states.
|
||||
- `elements`: actionable elements bound to a state.
|
||||
- `edges`: observed actions from one state to another.
|
||||
- `skippedActions`: state-local default-explorer actions that risk policy refused to execute.
|
||||
- `failedObservations`: failed locator, actionability, action, or effect observations recorded as evidence.
|
||||
- `summary`: machine-readable artifact counts and grouped counts.
|
||||
|
||||
## Evidence Levels
|
||||
|
||||
- `observed`: directly captured browser fact.
|
||||
- `derived`: deterministic rule output from observed facts.
|
||||
- `verified`: re-resolved or replay-verified claim.
|
||||
- `heuristic`: conservative inference such as risk classification.
|
||||
- `external`: human or downstream annotation.
|
||||
|
||||
## `StateNode`
|
||||
|
||||
- `stateId`: stable id derived from URL, DOM hash, visible text hash, and actionable signature hash.
|
||||
- `url`, `route`, `title`: browser-visible state identity.
|
||||
- `domHash`: hash of full `document.documentElement.outerHTML`.
|
||||
- `visibleTextHash`: hash of normalized visible body text.
|
||||
- `actionableSignatureHash`: hash of actionable element signatures.
|
||||
- `modalStack`: visible dialogs or modal-like surfaces.
|
||||
- `openedByEdgeId`: edge that opened this state, if known.
|
||||
- `stateId`: stable id derived from URL, normalized state hashes, and actionable signatures.
|
||||
- `lifecycle`: `observed`, `fingerprinted`, `verified`, `stale`, `deprecated`, or `conflicted`.
|
||||
- `url`, `route`, `urlCanonicalKey`, `title`, `language`, `viewport`: browser-visible and normalized state identity.
|
||||
- `stateHash`, `domHash`, `visibleTextHash`, `actionableSignatureHash`: deterministic state fingerprints.
|
||||
- `modalStack`: visible modal context.
|
||||
- `frameContext`: reserved frame context path. In v0.2.1 this is emitted as an empty array.
|
||||
- `openedByEdgeId`: first edge that observed this state in the current run. It is not the only possible predecessor.
|
||||
- `createdAt`: ISO timestamp for state capture.
|
||||
- `evidence`: observed and derived state facts.
|
||||
|
||||
## `ActionableElement`
|
||||
|
||||
- `elementId`: stable id within the captured state.
|
||||
- `elementId`: state-local element id. It encodes state identity and must not be treated as a global DOM node id.
|
||||
- `stateId`: owning state.
|
||||
- `tag`, `role`, `accessibleName`, `text`, `label`, `placeholder`: semantic identity.
|
||||
- `attributes`: selected stable attributes such as `data-testid`, `id`, `name`, `type`, `href`, and ARIA fields.
|
||||
- `bbox`: page-space bounding box.
|
||||
- `attributes`: selected stable attributes such as test id, id, name, type, href, and ARIA fields.
|
||||
- `bbox`: geometry.
|
||||
- `framePath`, `shadowPath`: reserved containment paths. In v0.2.1 these are emitted as empty arrays.
|
||||
- `interactability`: visibility, enabled/editable status, and whether the element receives events.
|
||||
- `context`: nearest dialog, form, section, row, landmark, and list item anchors.
|
||||
- `parameterSurface`: input/select/textarea constraints, options, required state, pattern, min/max, and maxLength.
|
||||
- `locators`: scored locator candidates.
|
||||
- `evidence`: observed and derived element facts.
|
||||
|
||||
## `LocatorCandidate`
|
||||
|
||||
- `descriptor`: structured Playwright locator descriptor.
|
||||
- `locatorId`: locator candidate id.
|
||||
- `elementId`: parent element id. This must equal the containing `ActionableElement.elementId`.
|
||||
- `framework`: `playwright` or `css`.
|
||||
- `strategy`: locator strategy such as role, label, placeholder, text, test id, CSS, or scoped.
|
||||
- `descriptor`: structured locator descriptor.
|
||||
- `expression`: human-readable locator expression.
|
||||
- `score`: ranking score.
|
||||
- `verified`: whether it resolved to exactly one element during capture.
|
||||
- `matchCount`: observed locator match count.
|
||||
- `score`, `reason`, `verified`, `matchCount`, `failureCount`, `lastVerifiedAt`: ranking and verification metadata.
|
||||
- `scopes`: context-scope evidence such as row, dialog, form, or section anchors.
|
||||
- `descriptor.kind: "scoped"`: executable scoped locator descriptor. In v0.2.2 this contains a scope descriptor plus a base target descriptor, with row, dialog, form, and list-item scopes used for execution when available.
|
||||
- `identity`: semantic identity checks for tag, role, accessible name, text, and visibility.
|
||||
- `evidence`: generation and verification facts.
|
||||
|
||||
## `ActionEdge`
|
||||
|
||||
- `fromStateId`, `toStateId`: source and resulting states.
|
||||
- `elementId`: action target.
|
||||
- `actionType`: currently focused on `click`, with type support for future `fill`, `select`, and navigation actions.
|
||||
- `preconditions`: reserved for stored open paths.
|
||||
- `effects`: URL, visible text, network, download, dialog, storage, or DOM effects.
|
||||
- `riskLevel`: `low`, `medium`, or `high`.
|
||||
- `edgeId`: action edge id.
|
||||
- `fromStateId`, `toStateId`, `elementId`, `actionType`: source state, resulting state if observed, target element, and action kind.
|
||||
- `actionParams`, `preconditions`, `parameterSpecs`: action payload and future replay metadata.
|
||||
- `effects`: snapshot-diff effects observed after replay.
|
||||
- `riskLevel`, `riskCategory`: risk classification used by default exploration.
|
||||
- `confidence`: confidence in observed edge replayability.
|
||||
- `createdAt`: ISO timestamp for edge capture.
|
||||
- In v0.2.1 default exploration replays each edge from a clean entry-state context.
|
||||
- Navigation timeout during entry or replay can be recorded as `FailedObservation` instead of aborting artifact generation.
|
||||
|
||||
## `EffectRecord`
|
||||
|
||||
- `type`: exactly one of `url`, `dom`, `visibleText`, or `actionableInventory`.
|
||||
- `before`, `after`: populated for v0.2.1 snapshot-diff effects.
|
||||
- `description`: human-readable effect summary.
|
||||
- v0.2.1 emits snapshot-diff effects only: URL, DOM hash, visible text hash, and actionable inventory hash.
|
||||
- Network, storage, download, and dialog event buffers are not emitted in v0.2.1 and are future work, not current contract.
|
||||
|
||||
## `SkippedAction`
|
||||
|
||||
- `skippedActionId`, `stateId`, `elementId`, `actionType`: state-local skipped action identity.
|
||||
- `riskLevel`, `riskCategory`: risk-policy classification.
|
||||
- `reason`: policy decision string such as `blocked:destructive`.
|
||||
- `evidence`: risk-policy evidence explaining why the action was skipped.
|
||||
- `createdAt`: ISO timestamp for the skip record.
|
||||
- `SkippedAction` is only for risk-policy default-explorer refusals. Hidden, disabled, obscured, duplicate-locator, locator-not-found, action, and effect failures are `FailedObservation`.
|
||||
|
||||
## `FailedObservation`
|
||||
|
||||
- `failedObservationId`, `stateId`, optional `elementId`, optional `actionType`: failed observation identity.
|
||||
- `stateId` may be `null` for graph-level failures such as entry navigation timeout before any state exists.
|
||||
- `reason`: machine-readable failure code.
|
||||
- `message`: diagnostic message.
|
||||
- `evidence`: observed facts explaining the failure.
|
||||
- `createdAt`: ISO timestamp for the failure record.
|
||||
- `FailedObservation` covers hidden, disabled, obscured, locator, action, and effect failures. Risk-policy default-explorer refusals belong in `SkippedAction`.
|
||||
|
||||
## `summary`
|
||||
|
||||
- `states`, `elements`, `edges`, `skippedActions`, `failedObservations`: total counts for the artifact.
|
||||
- `failedObservationsByReason`: count map keyed by `FailedObservation.reason`.
|
||||
- `skippedActionsByRiskCategory`: count map keyed by skipped action risk category.
|
||||
- `edgesByRiskCategory`: count map keyed by executed edge risk category.
|
||||
|
||||
@@ -33,6 +33,12 @@ function parseArgs(argv: string[]): ParsedArgs {
|
||||
case "--max-actions":
|
||||
options.maxActions = Number(requireValue(argv, ++index, "--max-actions"));
|
||||
break;
|
||||
case "--navigation-timeout-ms":
|
||||
options.navigationTimeoutMs = Number(requireValue(argv, ++index, "--navigation-timeout-ms"));
|
||||
break;
|
||||
case "--action-timeout-ms":
|
||||
options.actionTimeoutMs = Number(requireValue(argv, ++index, "--action-timeout-ms"));
|
||||
break;
|
||||
case "--headed":
|
||||
options.headed = true;
|
||||
break;
|
||||
@@ -54,7 +60,7 @@ function requireValue(argv: string[], index: number, flag: string): string {
|
||||
function usage(): never {
|
||||
console.error(`Usage:
|
||||
action-topology scan --url <url-or-file> [--out graph.json] [--headed] [--storage-state state.json]
|
||||
action-topology explore --url <url-or-file> [--out graph.json] [--max-actions 10] [--headed] [--storage-state state.json]
|
||||
action-topology explore --url <url-or-file> [--out graph.json] [--max-actions 10] [--navigation-timeout-ms 30000] [--action-timeout-ms 2000] [--headed] [--storage-state state.json]
|
||||
`);
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -63,4 +69,3 @@ main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.stack || error.message : String(error));
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
|
||||
41
src/effects.ts
Normal file
41
src/effects.ts
Normal file
@@ -0,0 +1,41 @@
|
||||
import type { EffectRecord, SnapshotResult } from "./types.js";
|
||||
|
||||
export function diffSnapshotEffects(args: {
|
||||
beforeUrl: string;
|
||||
afterUrl: string;
|
||||
beforeSnapshot: SnapshotResult;
|
||||
afterSnapshot: SnapshotResult;
|
||||
}): EffectRecord[] {
|
||||
const effects: EffectRecord[] = [];
|
||||
if (args.beforeUrl !== args.afterUrl) {
|
||||
effects.push({ type: "url", before: args.beforeUrl, after: args.afterUrl, description: "URL changed after action" });
|
||||
}
|
||||
if (args.beforeSnapshot.state.domHash !== args.afterSnapshot.state.domHash) {
|
||||
effects.push({
|
||||
type: "dom",
|
||||
before: args.beforeSnapshot.state.domHash,
|
||||
after: args.afterSnapshot.state.domHash,
|
||||
description: "DOM hash changed after action",
|
||||
});
|
||||
}
|
||||
|
||||
const beforeVisibleTextHash = args.beforeSnapshot.state.visibleTextHash;
|
||||
const afterVisibleTextHash = args.afterSnapshot.state.visibleTextHash;
|
||||
if (beforeVisibleTextHash !== afterVisibleTextHash) {
|
||||
effects.push({
|
||||
type: "visibleText",
|
||||
before: beforeVisibleTextHash,
|
||||
after: afterVisibleTextHash,
|
||||
description: "Visible text changed after action",
|
||||
});
|
||||
}
|
||||
if (args.beforeSnapshot.state.actionableSignatureHash !== args.afterSnapshot.state.actionableSignatureHash) {
|
||||
effects.push({
|
||||
type: "actionableInventory",
|
||||
before: args.beforeSnapshot.state.actionableSignatureHash,
|
||||
after: args.afterSnapshot.state.actionableSignatureHash,
|
||||
description: "Actionable inventory changed after action",
|
||||
});
|
||||
}
|
||||
return effects;
|
||||
}
|
||||
480
src/engine.ts
480
src/engine.ts
@@ -1,12 +1,15 @@
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { chromium, type Browser, type BrowserContext, type Page, type Response } from "playwright";
|
||||
import { chromium, type Browser, type BrowserContext, type Page } from "playwright";
|
||||
import { diffSnapshotEffects } from "./effects.js";
|
||||
import { shortHash } from "./hash.js";
|
||||
import { resolveLocator } from "./locator.js";
|
||||
import { isLowRiskClickable, classifyClickRisk } from "./risk.js";
|
||||
import { buildEmptyGraph, defaultViewport } from "./metadata.js";
|
||||
import { defaultExploreDecision } from "./risk.js";
|
||||
import { snapshotPage } from "./snapshot.js";
|
||||
import type { ActionEdge, EffectRecord, TopologyGraph } from "./types.js";
|
||||
import { refreshGraphSummary } from "./summary.js";
|
||||
import type { ActionableElement, ActionEdge, FailedObservation, LocatorCandidate, SnapshotResult, TopologyGraph } from "./types.js";
|
||||
|
||||
export interface EngineOptions {
|
||||
url: string;
|
||||
@@ -14,31 +17,52 @@ export interface EngineOptions {
|
||||
headed?: boolean;
|
||||
storageState?: string;
|
||||
maxActions?: number;
|
||||
navigationTimeoutMs?: number;
|
||||
actionTimeoutMs?: number;
|
||||
}
|
||||
|
||||
interface ExploreCandidate {
|
||||
element: ActionableElement;
|
||||
locator: LocatorCandidate;
|
||||
}
|
||||
|
||||
export async function scanUrl(options: EngineOptions): Promise<TopologyGraph> {
|
||||
const browser = await chromium.launch({ headless: !options.headed });
|
||||
try {
|
||||
const requestedEntryUrl = toNavigableUrl(options.url);
|
||||
const context = await newContext(browser, options);
|
||||
const page = await context.newPage();
|
||||
await page.goto(toNavigableUrl(options.url), { waitUntil: "domcontentloaded" });
|
||||
await page.waitForLoadState("networkidle").catch(() => undefined);
|
||||
const snapshot = await snapshotPage(page);
|
||||
const graph: TopologyGraph = {
|
||||
schemaVersion: "0.1",
|
||||
metadata: {
|
||||
try {
|
||||
const page = await createReplayPage(context, requestedEntryUrl, options);
|
||||
const snapshot = await snapshotPage(page);
|
||||
const graph = buildEmptyGraph({
|
||||
inputUrl: options.url,
|
||||
entryUrl: page.url(),
|
||||
generatedAt: new Date().toISOString(),
|
||||
engineVersion: "0.1.0",
|
||||
mode: "scan",
|
||||
},
|
||||
states: [snapshot.state],
|
||||
elements: snapshot.elements,
|
||||
edges: [],
|
||||
};
|
||||
await maybeWriteGraph(graph, options.out);
|
||||
await context.close();
|
||||
return graph;
|
||||
storageState: options.storageState,
|
||||
});
|
||||
graph.states.push(snapshot.state);
|
||||
graph.elements.push(...snapshot.elements);
|
||||
await maybeWriteGraph(graph, options.out);
|
||||
return graph;
|
||||
} catch (error) {
|
||||
const graph = buildEmptyGraph({
|
||||
inputUrl: options.url,
|
||||
entryUrl: requestedEntryUrl,
|
||||
mode: "scan",
|
||||
storageState: options.storageState,
|
||||
});
|
||||
recordNavigationFailure(graph, {
|
||||
stateId: null,
|
||||
elementId: null,
|
||||
actionType: "navigate",
|
||||
url: requestedEntryUrl,
|
||||
error,
|
||||
});
|
||||
await maybeWriteGraph(graph, options.out);
|
||||
return graph;
|
||||
} finally {
|
||||
await context.close();
|
||||
}
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
@@ -47,82 +71,132 @@ export async function scanUrl(options: EngineOptions): Promise<TopologyGraph> {
|
||||
export async function exploreUrl(options: EngineOptions): Promise<TopologyGraph> {
|
||||
const browser = await chromium.launch({ headless: !options.headed });
|
||||
try {
|
||||
const context = await newContext(browser, options);
|
||||
const entryUrl = toNavigableUrl(options.url);
|
||||
const page = await context.newPage();
|
||||
await page.goto(entryUrl, { waitUntil: "domcontentloaded" });
|
||||
await page.waitForLoadState("networkidle").catch(() => undefined);
|
||||
|
||||
const entrySnapshot = await snapshotPage(page);
|
||||
const graph: TopologyGraph = {
|
||||
schemaVersion: "0.1",
|
||||
metadata: {
|
||||
entryUrl: page.url(),
|
||||
generatedAt: new Date().toISOString(),
|
||||
engineVersion: "0.1.0",
|
||||
const requestedEntryUrl = toNavigableUrl(options.url);
|
||||
const entryContext = await newContext(browser, options);
|
||||
let entrySnapshot: SnapshotResult;
|
||||
let entryUrl: string;
|
||||
try {
|
||||
const page = await createReplayPage(entryContext, requestedEntryUrl, options);
|
||||
entrySnapshot = await snapshotPage(page);
|
||||
entryUrl = page.url();
|
||||
} catch (error) {
|
||||
const graph = buildEmptyGraph({
|
||||
inputUrl: options.url,
|
||||
entryUrl: requestedEntryUrl,
|
||||
mode: "explore",
|
||||
},
|
||||
states: [entrySnapshot.state],
|
||||
elements: [...entrySnapshot.elements],
|
||||
edges: [],
|
||||
};
|
||||
|
||||
const candidates = entrySnapshot.elements.filter(isLowRiskClickable).slice(0, options.maxActions ?? 10);
|
||||
await page.close();
|
||||
|
||||
for (const element of candidates) {
|
||||
const actionPage = await context.newPage();
|
||||
const networkEvents: EffectRecord[] = [];
|
||||
actionPage.on("response", (response) => {
|
||||
const record = responseToEffect(response);
|
||||
if (record) networkEvents.push(record);
|
||||
storageState: options.storageState,
|
||||
});
|
||||
recordNavigationFailure(graph, {
|
||||
stateId: null,
|
||||
elementId: null,
|
||||
actionType: "navigate",
|
||||
url: requestedEntryUrl,
|
||||
error,
|
||||
});
|
||||
await maybeWriteGraph(graph, options.out);
|
||||
return graph;
|
||||
} finally {
|
||||
await entryContext.close();
|
||||
}
|
||||
|
||||
await actionPage.goto(entryUrl, { waitUntil: "domcontentloaded" });
|
||||
await actionPage.waitForLoadState("networkidle").catch(() => undefined);
|
||||
const beforeUrl = actionPage.url();
|
||||
const beforeSnapshot = await snapshotPage(actionPage);
|
||||
const locator = element.locators.find((candidate) => candidate.verified) ?? element.locators[0];
|
||||
const graph = buildEmptyGraph({
|
||||
inputUrl: options.url,
|
||||
entryUrl,
|
||||
mode: "explore",
|
||||
storageState: options.storageState,
|
||||
});
|
||||
graph.states.push(entrySnapshot.state);
|
||||
graph.elements.push(...entrySnapshot.elements);
|
||||
|
||||
let toStateId: string | null = null;
|
||||
const effects: EffectRecord[] = [];
|
||||
if (locator) {
|
||||
try {
|
||||
await resolveLocator(actionPage, locator.descriptor).first().click({ timeout: 2000 });
|
||||
await actionPage.waitForLoadState("networkidle", { timeout: 2000 }).catch(() => undefined);
|
||||
const afterSnapshot = await snapshotPage(actionPage);
|
||||
graph.states = upsertById(graph.states, afterSnapshot.state, "stateId");
|
||||
graph.elements.push(...afterSnapshot.elements);
|
||||
toStateId = afterSnapshot.state.stateId;
|
||||
effects.push(...diffEffects(beforeUrl, beforeSnapshot.visibleText, actionPage.url(), afterSnapshot.visibleText));
|
||||
} catch (error) {
|
||||
effects.push({
|
||||
type: "dom",
|
||||
description: `Action failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||
});
|
||||
}
|
||||
recordSkippedActionsForState(graph, entrySnapshot);
|
||||
|
||||
const lowRiskCandidates: ExploreCandidate[] = [];
|
||||
for (const element of entrySnapshot.elements) {
|
||||
if (!isDefaultClickCandidate(element)) continue;
|
||||
if (!isActionableForClick(element)) {
|
||||
recordActionabilityFailure(graph, entrySnapshot, element);
|
||||
continue;
|
||||
}
|
||||
const decision = defaultExploreDecision(element, "click");
|
||||
if (!decision.allowed) continue;
|
||||
const locator = findVerifiedUniqueLocator(element);
|
||||
if (!locator) {
|
||||
recordLocatorFailure(graph, entrySnapshot, element, locatorFailureReason(element));
|
||||
continue;
|
||||
}
|
||||
lowRiskCandidates.push({ element, locator });
|
||||
}
|
||||
lowRiskCandidates.sort((left, right) => compareExploreCandidatePriority(left.element, right.element));
|
||||
const candidates = lowRiskCandidates.slice(0, options.maxActions ?? 10);
|
||||
|
||||
const edge: ActionEdge = {
|
||||
edgeId: `edge_${shortHash([entrySnapshot.state.stateId, element.elementId, "click"])}`,
|
||||
fromStateId: entrySnapshot.state.stateId,
|
||||
toStateId,
|
||||
elementId: element.elementId,
|
||||
actionType: "click",
|
||||
actionParams: {},
|
||||
preconditions: [],
|
||||
effects: [...effects, ...networkEvents],
|
||||
parameterSpecs: [],
|
||||
riskLevel: classifyClickRisk(element),
|
||||
confidence: locator?.verified ? 0.82 : 0.55,
|
||||
createdAt: new Date().toISOString(),
|
||||
};
|
||||
graph.edges = upsertById(graph.edges, edge, "edgeId");
|
||||
await actionPage.close();
|
||||
for (const { element, locator } of candidates) {
|
||||
const decision = defaultExploreDecision(element, "click");
|
||||
const actionContext = await newContext(browser, options);
|
||||
try {
|
||||
let actionPage: Page;
|
||||
try {
|
||||
actionPage = await createReplayPage(actionContext, entryUrl, options);
|
||||
} catch (error) {
|
||||
recordNavigationFailure(graph, {
|
||||
stateId: entrySnapshot.state.stateId,
|
||||
elementId: element.elementId,
|
||||
actionType: "click",
|
||||
url: entryUrl,
|
||||
error,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const beforeUrl = actionPage.url();
|
||||
const beforeSnapshot = await snapshotPage(actionPage);
|
||||
const edgeId = `edge_${shortHash([beforeSnapshot.state.stateId, element.elementId, "click"])}`;
|
||||
const target = resolveLocator(actionPage, locator.descriptor);
|
||||
const matchCount = await target.count();
|
||||
if (matchCount !== 1) {
|
||||
recordLocatorFailure(graph, beforeSnapshot, element, matchCount === 0 ? "locator_not_found" : "locator_not_unique", matchCount);
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
await target.click({ timeout: actionTimeout(options), trial: true });
|
||||
await target.click({ timeout: actionTimeout(options) });
|
||||
} catch (error) {
|
||||
recordClickFailure(graph, beforeSnapshot, element, error);
|
||||
continue;
|
||||
}
|
||||
|
||||
await actionPage.waitForLoadState("networkidle", { timeout: actionTimeout(options) }).catch(() => undefined);
|
||||
const afterSnapshot = await snapshotPage(actionPage, edgeId);
|
||||
const discoveredState = addStateIfNew(graph, afterSnapshot.state);
|
||||
graph.elements = upsertManyById(graph.elements, afterSnapshot.elements, "elementId");
|
||||
if (discoveredState) recordSkippedActionsForState(graph, afterSnapshot);
|
||||
|
||||
const edge: ActionEdge = {
|
||||
edgeId,
|
||||
fromStateId: beforeSnapshot.state.stateId,
|
||||
toStateId: afterSnapshot.state.stateId,
|
||||
elementId: element.elementId,
|
||||
actionType: "click",
|
||||
actionParams: {},
|
||||
preconditions: [],
|
||||
effects: diffSnapshotEffects({
|
||||
beforeUrl,
|
||||
afterUrl: actionPage.url(),
|
||||
beforeSnapshot,
|
||||
afterSnapshot,
|
||||
}),
|
||||
parameterSpecs: [],
|
||||
riskLevel: decision.risk.level,
|
||||
riskCategory: decision.risk.category,
|
||||
confidence: 0.82,
|
||||
createdAt: new Date().toISOString(),
|
||||
};
|
||||
graph.edges = upsertById(graph.edges, edge, "edgeId");
|
||||
} finally {
|
||||
await actionContext.close();
|
||||
}
|
||||
}
|
||||
|
||||
await maybeWriteGraph(graph, options.out);
|
||||
await context.close();
|
||||
return graph;
|
||||
} finally {
|
||||
await browser.close();
|
||||
@@ -132,11 +206,93 @@ export async function exploreUrl(options: EngineOptions): Promise<TopologyGraph>
|
||||
async function newContext(browser: Browser, options: EngineOptions): Promise<BrowserContext> {
|
||||
return browser.newContext({
|
||||
storageState: options.storageState,
|
||||
viewport: { width: 1440, height: 1000 },
|
||||
viewport: defaultViewport(),
|
||||
locale: "en-US",
|
||||
});
|
||||
}
|
||||
|
||||
async function createReplayPage(context: BrowserContext, entryUrl: string, options: EngineOptions): Promise<Page> {
|
||||
const page = await context.newPage();
|
||||
await page.goto(entryUrl, { waitUntil: "domcontentloaded", timeout: navigationTimeout(options) });
|
||||
await page.waitForLoadState("networkidle", { timeout: actionTimeout(options) }).catch(() => undefined);
|
||||
return page;
|
||||
}
|
||||
|
||||
function addStateIfNew(graph: TopologyGraph, state: SnapshotResult["state"]): boolean {
|
||||
if (graph.states.some((existing) => existing.stateId === state.stateId)) return false;
|
||||
graph.states.push(state);
|
||||
return true;
|
||||
}
|
||||
|
||||
function findVerifiedUniqueLocator(element: ActionableElement): LocatorCandidate | null {
|
||||
return element.locators.find((locator) => locator.verified && locator.matchCount === 1) ?? null;
|
||||
}
|
||||
|
||||
function locatorFailureReason(element: ActionableElement): "locator_not_found" | "locator_not_unique" {
|
||||
if (element.locators.some((locator) => (locator.matchCount ?? 0) > 1)) return "locator_not_unique";
|
||||
return "locator_not_found";
|
||||
}
|
||||
|
||||
function recordLocatorFailure(
|
||||
graph: TopologyGraph,
|
||||
snapshot: SnapshotResult,
|
||||
element: ActionableElement,
|
||||
reason: "locator_not_found" | "locator_not_unique",
|
||||
matchCount?: number,
|
||||
): void {
|
||||
const message =
|
||||
reason === "locator_not_unique"
|
||||
? `Locator matched ${matchCount ?? "multiple"} elements and cannot be clicked safely`
|
||||
: matchCount === 0
|
||||
? "Locator matched no elements during replay"
|
||||
: "No verified locator candidate was available for default exploration";
|
||||
recordFailedObservation(graph, {
|
||||
failedObservationId: `fail_${shortHash([snapshot.state.stateId, element.elementId, "click", reason, matchCount ?? "candidate"])}`,
|
||||
stateId: snapshot.state.stateId,
|
||||
elementId: element.elementId,
|
||||
actionType: "click",
|
||||
reason,
|
||||
message,
|
||||
evidence: [
|
||||
{
|
||||
level: "observed",
|
||||
source: "locator.verify",
|
||||
description:
|
||||
matchCount === undefined
|
||||
? "Entry-state locator candidates did not contain a verified unique match"
|
||||
: `Replay locator matched ${matchCount} element(s)`,
|
||||
},
|
||||
],
|
||||
createdAt: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
|
||||
function recordClickFailure(graph: TopologyGraph, snapshot: SnapshotResult, element: ActionableElement, error: unknown): void {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
recordFailedObservation(graph, {
|
||||
failedObservationId: `fail_${shortHash([snapshot.state.stateId, element.elementId, "click", "unexpected_state", message])}`,
|
||||
stateId: snapshot.state.stateId,
|
||||
elementId: element.elementId,
|
||||
actionType: "click",
|
||||
reason: "unexpected_state",
|
||||
message: `Action failed: ${message}`,
|
||||
evidence: [
|
||||
{
|
||||
level: "observed",
|
||||
source: "engine.action",
|
||||
description: "Click action raised before effects could be fully observed",
|
||||
},
|
||||
],
|
||||
createdAt: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
|
||||
function recordFailedObservation(graph: TopologyGraph, observation: FailedObservation): void {
|
||||
graph.failedObservations = upsertById(graph.failedObservations, observation, "failedObservationId");
|
||||
}
|
||||
|
||||
async function maybeWriteGraph(graph: TopologyGraph, out?: string): Promise<void> {
|
||||
refreshGraphSummary(graph);
|
||||
if (!out) {
|
||||
console.log(JSON.stringify(graph, null, 2));
|
||||
return;
|
||||
@@ -145,38 +301,137 @@ async function maybeWriteGraph(graph: TopologyGraph, out?: string): Promise<void
|
||||
await writeFile(out, `${JSON.stringify(graph, null, 2)}\n`, "utf8");
|
||||
}
|
||||
|
||||
function navigationTimeout(options: EngineOptions): number {
|
||||
return options.navigationTimeoutMs ?? 30_000;
|
||||
}
|
||||
|
||||
function actionTimeout(options: EngineOptions): number {
|
||||
return options.actionTimeoutMs ?? 2_000;
|
||||
}
|
||||
|
||||
function recordNavigationFailure(
|
||||
graph: TopologyGraph,
|
||||
args: {
|
||||
stateId: string | null;
|
||||
elementId: string | null;
|
||||
actionType: FailedObservation["actionType"];
|
||||
url: string;
|
||||
error: unknown;
|
||||
},
|
||||
): void {
|
||||
const message = args.error instanceof Error ? args.error.message : String(args.error);
|
||||
const reason: FailedObservation["reason"] = /timeout/i.test(message) ? "navigation_timeout" : "network_error";
|
||||
recordFailedObservation(graph, {
|
||||
failedObservationId: `fail_${shortHash([args.stateId ?? "graph", args.elementId ?? "none", args.actionType, reason, args.url])}`,
|
||||
stateId: args.stateId,
|
||||
elementId: args.elementId,
|
||||
actionType: args.actionType,
|
||||
reason,
|
||||
message: `Navigation failed for ${args.url}: ${message}`,
|
||||
evidence: [
|
||||
{
|
||||
level: "observed",
|
||||
source: "engine.navigation",
|
||||
description: "Navigation failure was recorded without converting it into an observed edge",
|
||||
},
|
||||
],
|
||||
createdAt: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
|
||||
function toNavigableUrl(value: string): string {
|
||||
if (/^https?:\/\//.test(value) || value.startsWith("file://")) return value;
|
||||
return pathToFileURL(path.resolve(value)).toString();
|
||||
}
|
||||
|
||||
function diffEffects(beforeUrl: string, beforeText: string, afterUrl: string, afterText: string): EffectRecord[] {
|
||||
const effects: EffectRecord[] = [];
|
||||
if (beforeUrl !== afterUrl) {
|
||||
effects.push({ type: "url", before: beforeUrl, after: afterUrl, description: "URL changed after action" });
|
||||
function recordSkippedActionsForState(graph: TopologyGraph, snapshot: SnapshotResult): void {
|
||||
for (const element of snapshot.elements) {
|
||||
if (!isDefaultClickCandidate(element)) continue;
|
||||
if (!isActionableForClick(element)) {
|
||||
recordActionabilityFailure(graph, snapshot, element);
|
||||
continue;
|
||||
}
|
||||
const decision = defaultExploreDecision(element, "click");
|
||||
if (decision.allowed) continue;
|
||||
graph.skippedActions = upsertById(
|
||||
graph.skippedActions,
|
||||
{
|
||||
skippedActionId: `skip_${shortHash([snapshot.state.stateId, element.elementId, "click", decision.risk.category])}`,
|
||||
stateId: snapshot.state.stateId,
|
||||
elementId: element.elementId,
|
||||
actionType: "click" as const,
|
||||
riskLevel: decision.risk.level,
|
||||
riskCategory: decision.risk.category,
|
||||
reason: decision.reason,
|
||||
evidence: decision.risk.evidence,
|
||||
createdAt: new Date().toISOString(),
|
||||
},
|
||||
"skippedActionId",
|
||||
);
|
||||
}
|
||||
if (shortHash(beforeText) !== shortHash(afterText)) {
|
||||
effects.push({
|
||||
type: "visibleText",
|
||||
before: shortHash(beforeText),
|
||||
after: shortHash(afterText),
|
||||
description: "Visible text changed after action",
|
||||
});
|
||||
}
|
||||
return effects;
|
||||
}
|
||||
|
||||
function responseToEffect(response: Response): EffectRecord | null {
|
||||
const request = response.request();
|
||||
const resourceType = request.resourceType();
|
||||
if (!["fetch", "xhr", "document"].includes(resourceType)) return null;
|
||||
return {
|
||||
type: "network",
|
||||
method: request.method(),
|
||||
url: response.url(),
|
||||
status: response.status(),
|
||||
description: `${request.method()} ${response.url()} -> ${response.status()}`,
|
||||
function recordActionabilityFailure(graph: TopologyGraph, snapshot: SnapshotResult, element: ActionableElement): void {
|
||||
const reason = actionabilityFailureReason(element);
|
||||
if (!reason) return;
|
||||
const messages = {
|
||||
element_hidden: "Element is hidden and cannot be clicked",
|
||||
element_disabled: "Element is disabled and cannot be clicked",
|
||||
element_obscured: "Element does not receive pointer events at its center point",
|
||||
};
|
||||
graph.failedObservations = upsertById(
|
||||
graph.failedObservations,
|
||||
{
|
||||
failedObservationId: `fail_${shortHash([snapshot.state.stateId, element.elementId, "click", reason])}`,
|
||||
stateId: snapshot.state.stateId,
|
||||
elementId: element.elementId,
|
||||
actionType: "click" as const,
|
||||
reason,
|
||||
message: messages[reason],
|
||||
evidence: [
|
||||
{
|
||||
level: "observed",
|
||||
source: "engine.actionability",
|
||||
description: `visible=${element.interactability.visible}; enabled=${element.interactability.enabled}; receivesEvents=${element.interactability.receivesEvents}`,
|
||||
},
|
||||
],
|
||||
createdAt: new Date().toISOString(),
|
||||
},
|
||||
"failedObservationId",
|
||||
);
|
||||
}
|
||||
|
||||
function isDefaultClickCandidate(element: ActionableElement): boolean {
|
||||
const role = element.role;
|
||||
return (
|
||||
element.tag === "button" ||
|
||||
element.tag === "a" ||
|
||||
role === "button" ||
|
||||
role === "link" ||
|
||||
role === "tab" ||
|
||||
role === "menuitem"
|
||||
);
|
||||
}
|
||||
|
||||
function isActionableForClick(element: ActionableElement): boolean {
|
||||
if (!element.interactability.visible || !element.interactability.enabled || !element.interactability.receivesEvents) return false;
|
||||
return isDefaultClickCandidate(element);
|
||||
}
|
||||
|
||||
function compareExploreCandidatePriority(left: ActionableElement, right: ActionableElement): number {
|
||||
return candidatePriority(left) - candidatePriority(right);
|
||||
}
|
||||
|
||||
function candidatePriority(element: ActionableElement): number {
|
||||
// Keep initially visible dialog controls from starving non-dialog entry controls under maxActions.
|
||||
return element.context.dialog ? 1 : 0;
|
||||
}
|
||||
|
||||
function actionabilityFailureReason(element: ActionableElement): "element_hidden" | "element_disabled" | "element_obscured" | null {
|
||||
if (!element.interactability.visible) return "element_hidden";
|
||||
if (!element.interactability.enabled) return "element_disabled";
|
||||
if (!element.interactability.receivesEvents) return "element_obscured";
|
||||
return null;
|
||||
}
|
||||
|
||||
function upsertById<T extends Record<K, string>, K extends keyof T>(items: T[], item: T, key: K): T[] {
|
||||
@@ -187,3 +442,6 @@ function upsertById<T extends Record<K, string>, K extends keyof T>(items: T[],
|
||||
return next;
|
||||
}
|
||||
|
||||
function upsertManyById<T extends Record<K, string>, K extends keyof T>(items: T[], incoming: T[], key: K): T[] {
|
||||
return incoming.reduce((current, item) => upsertById(current, item, key), items);
|
||||
}
|
||||
|
||||
242
src/locator.ts
242
src/locator.ts
@@ -1,9 +1,11 @@
|
||||
import type { Locator, Page } from "playwright";
|
||||
import { shortHash } from "./hash.js";
|
||||
import type { LocatorCandidate, LocatorDescriptor, RawActionableElement } from "./types.js";
|
||||
import type { BaseLocatorDescriptor, LocatorCandidate, LocatorDescriptor, LocatorScopeDescriptor, RawActionableElement } from "./types.js";
|
||||
|
||||
type LocatorRoot = Page | Locator;
|
||||
|
||||
export function generateLocators(elementId: string, raw: RawActionableElement): LocatorCandidate[] {
|
||||
const candidates: Array<Omit<LocatorCandidate, "locatorId" | "elementId" | "verified" | "failureCount">> = [];
|
||||
const candidates: Array<Omit<LocatorCandidate, "locatorId" | "elementId" | "verified" | "failureCount" | "identity">> = [];
|
||||
const name = clean(raw.accessibleName);
|
||||
const label = clean(raw.label);
|
||||
const placeholder = clean(raw.placeholder);
|
||||
@@ -11,6 +13,31 @@ export function generateLocators(elementId: string, raw: RawActionableElement):
|
||||
const testId = raw.attributes["data-testid"] || raw.attributes["data-test-id"] || raw.attributes["data-test"];
|
||||
const id = raw.attributes.id;
|
||||
const nameAttr = raw.attributes.name;
|
||||
const scopes = contextScopes(raw);
|
||||
const evidence = [
|
||||
{
|
||||
level: "derived" as const,
|
||||
source: "locator.generator",
|
||||
description: "Locator candidate generated from state-local element semantics",
|
||||
},
|
||||
];
|
||||
|
||||
if (raw.role && name) {
|
||||
const scope = executableScope(scopes, name);
|
||||
if (scope) {
|
||||
const target: BaseLocatorDescriptor = { kind: "role", role: raw.role, name };
|
||||
candidates.push({
|
||||
framework: "playwright",
|
||||
strategy: "scoped",
|
||||
descriptor: { kind: "scoped", scope, target },
|
||||
expression: `${scopeExpression(scope)}.getByRole(${JSON.stringify(raw.role)}, { name: ${JSON.stringify(name)} })`,
|
||||
score: 0.97,
|
||||
reason: "context-scoped role plus accessible name",
|
||||
scopes,
|
||||
evidence,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (raw.role && name) {
|
||||
candidates.push({
|
||||
@@ -20,6 +47,8 @@ export function generateLocators(elementId: string, raw: RawActionableElement):
|
||||
expression: `page.getByRole(${JSON.stringify(raw.role)}, { name: ${JSON.stringify(name)} })`,
|
||||
score: 0.95,
|
||||
reason: "role plus accessible name",
|
||||
scopes,
|
||||
evidence,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -31,6 +60,8 @@ export function generateLocators(elementId: string, raw: RawActionableElement):
|
||||
expression: `page.getByTestId(${JSON.stringify(testId)})`,
|
||||
score: 0.92,
|
||||
reason: "explicit test id",
|
||||
scopes,
|
||||
evidence,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -42,6 +73,8 @@ export function generateLocators(elementId: string, raw: RawActionableElement):
|
||||
expression: `page.getByLabel(${JSON.stringify(label)})`,
|
||||
score: 0.9,
|
||||
reason: "associated label",
|
||||
scopes,
|
||||
evidence,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -53,6 +86,8 @@ export function generateLocators(elementId: string, raw: RawActionableElement):
|
||||
expression: `page.getByPlaceholder(${JSON.stringify(placeholder)})`,
|
||||
score: 0.86,
|
||||
reason: "placeholder text",
|
||||
scopes,
|
||||
evidence,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -64,6 +99,8 @@ export function generateLocators(elementId: string, raw: RawActionableElement):
|
||||
expression: `page.getByText(${JSON.stringify(text)}, { exact: true })`,
|
||||
score: 0.75,
|
||||
reason: "visible text fallback",
|
||||
scopes,
|
||||
evidence,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -75,6 +112,8 @@ export function generateLocators(elementId: string, raw: RawActionableElement):
|
||||
expression: `page.locator(${JSON.stringify(`#${cssEscape(id)}`)})`,
|
||||
score: 0.72,
|
||||
reason: "id selector fallback",
|
||||
scopes,
|
||||
evidence,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -86,6 +125,8 @@ export function generateLocators(elementId: string, raw: RawActionableElement):
|
||||
expression: `page.locator(${JSON.stringify(`${raw.tag}[name="${cssString(nameAttr)}"]`)})`,
|
||||
score: 0.68,
|
||||
reason: "name attribute fallback",
|
||||
scopes,
|
||||
evidence,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -94,23 +135,126 @@ export function generateLocators(elementId: string, raw: RawActionableElement):
|
||||
locatorId: `loc_${shortHash([elementId, candidate.descriptor])}`,
|
||||
elementId,
|
||||
verified: false,
|
||||
identity: null,
|
||||
failureCount: 0,
|
||||
}));
|
||||
}
|
||||
|
||||
export async function verifyLocators(page: Page, locators: LocatorCandidate[]): Promise<LocatorCandidate[]> {
|
||||
export async function verifyLocators(page: Page, raw: RawActionableElement, locators: LocatorCandidate[]): Promise<LocatorCandidate[]> {
|
||||
const verifiedAt = new Date().toISOString();
|
||||
const verified: LocatorCandidate[] = [];
|
||||
|
||||
for (const locator of locators) {
|
||||
try {
|
||||
const matchCount = await resolveLocator(page, locator.descriptor).count();
|
||||
const resolved = resolveLocator(page, locator.descriptor);
|
||||
const matchCount = await resolved.count();
|
||||
const identity =
|
||||
matchCount === 1
|
||||
? await resolved.first().evaluate(
|
||||
(element, expected) => {
|
||||
const html = element as HTMLElement;
|
||||
const tag = html.tagName.toLowerCase();
|
||||
const text = normalizeNullable(html.innerText || html.textContent || "");
|
||||
const label = getLabel(html);
|
||||
const placeholder = "placeholder" in html ? normalizeNullable(String((html as HTMLInputElement).placeholder || "")) : null;
|
||||
const accessibleName = normalizeNullable(getAccessibleName(html) || label || placeholder || text || "");
|
||||
const inferredRole = html.getAttribute("role") || inferRole(html);
|
||||
const visible = isVisible(html);
|
||||
return {
|
||||
sameTag: tag === expected.tag,
|
||||
sameRole: expected.role ? inferredRole === expected.role : true,
|
||||
sameAccessibleName: expected.accessibleName ? accessibleName === expected.accessibleName : true,
|
||||
sameText: expected.text ? text === expected.text : true,
|
||||
sameVisibility: visible === expected.visible,
|
||||
};
|
||||
|
||||
function inferRole(item: HTMLElement): string | null {
|
||||
const itemTag = item.tagName.toLowerCase();
|
||||
if (itemTag === "button") return "button";
|
||||
if (itemTag === "a" && item.hasAttribute("href")) return "link";
|
||||
if (itemTag === "select") return "combobox";
|
||||
if (itemTag === "textarea") return "textbox";
|
||||
if (itemTag === "input") {
|
||||
const type = ((item as HTMLInputElement).type || "text").toLowerCase();
|
||||
if (["button", "submit", "reset"].includes(type)) return "button";
|
||||
if (type === "checkbox") return "checkbox";
|
||||
if (type === "radio") return "radio";
|
||||
return "textbox";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function getAccessibleName(item: HTMLElement): string | null {
|
||||
const ariaLabel = item.getAttribute("aria-label");
|
||||
if (ariaLabel) return normalizeNullable(ariaLabel);
|
||||
const labelledBy = item.getAttribute("aria-labelledby");
|
||||
if (labelledBy) {
|
||||
const labelledText = labelledBy
|
||||
.split(/\s+/)
|
||||
.map((id) => document.getElementById(id)?.innerText || document.getElementById(id)?.textContent || "")
|
||||
.join(" ");
|
||||
if (labelledText.trim()) return normalizeNullable(labelledText);
|
||||
}
|
||||
const title = item.getAttribute("title");
|
||||
if (title) return normalizeNullable(title);
|
||||
const alt = item.getAttribute("alt");
|
||||
if (alt) return normalizeNullable(alt);
|
||||
return normalizeNullable(item.innerText || item.textContent || "");
|
||||
}
|
||||
|
||||
function getLabel(item: HTMLElement): string | null {
|
||||
const id = item.getAttribute("id");
|
||||
if (id) {
|
||||
const explicit = document.querySelector<HTMLLabelElement>(`label[for="${CSS.escape(id)}"]`);
|
||||
if (explicit) return normalizeNullable(explicit.innerText || explicit.textContent || "");
|
||||
}
|
||||
const implicit = item.closest("label");
|
||||
if (implicit) return normalizeNullable(implicit.innerText || implicit.textContent || "");
|
||||
return null;
|
||||
}
|
||||
|
||||
function isVisible(item: HTMLElement): boolean {
|
||||
const style = getComputedStyle(item);
|
||||
if (style.visibility === "hidden" || style.display === "none" || Number(style.opacity) === 0) return false;
|
||||
const rect = item.getBoundingClientRect();
|
||||
return rect.width > 0 && rect.height > 0;
|
||||
}
|
||||
|
||||
function normalizeText(value: string): string {
|
||||
return value.replace(/\s+/g, " ").trim();
|
||||
}
|
||||
|
||||
function normalizeNullable(value: string): string | null {
|
||||
const normalized = normalizeText(value);
|
||||
return normalized || null;
|
||||
}
|
||||
},
|
||||
{
|
||||
tag: raw.tag,
|
||||
role: raw.role,
|
||||
accessibleName: raw.accessibleName,
|
||||
text: raw.text,
|
||||
visible: raw.visibility.visible,
|
||||
},
|
||||
)
|
||||
: null;
|
||||
const identityMatches = identity ? Object.values(identity).every(Boolean) : false;
|
||||
const locatorVerified = matchCount === 1 && identityMatches;
|
||||
verified.push({
|
||||
...locator,
|
||||
verified: matchCount === 1,
|
||||
verified: locatorVerified,
|
||||
matchCount,
|
||||
lastVerifiedAt: verifiedAt,
|
||||
score: matchCount === 1 ? locator.score : Math.max(0.1, locator.score - 0.35),
|
||||
score: locatorVerified ? locator.score : Math.max(0.1, locator.score - (matchCount === 1 ? 0.2 : 0.35)),
|
||||
identity,
|
||||
evidence: [
|
||||
...locator.evidence,
|
||||
{
|
||||
level: locatorVerified ? "verified" : "observed",
|
||||
source: "locator.verify",
|
||||
description: `Locator matched ${matchCount} element(s)`,
|
||||
},
|
||||
],
|
||||
});
|
||||
} catch {
|
||||
verified.push({
|
||||
@@ -119,6 +263,11 @@ export async function verifyLocators(page: Page, locators: LocatorCandidate[]):
|
||||
matchCount: 0,
|
||||
lastVerifiedAt: verifiedAt,
|
||||
score: Math.max(0.1, locator.score - 0.45),
|
||||
identity: null,
|
||||
evidence: [
|
||||
...locator.evidence,
|
||||
{ level: "observed", source: "locator.verify", description: "Locator verification failed before matching elements" },
|
||||
],
|
||||
failureCount: locator.failureCount + 1,
|
||||
});
|
||||
}
|
||||
@@ -128,19 +277,44 @@ export async function verifyLocators(page: Page, locators: LocatorCandidate[]):
|
||||
}
|
||||
|
||||
export function resolveLocator(page: Page, descriptor: LocatorDescriptor): Locator {
|
||||
if (descriptor.kind === "scoped") {
|
||||
return resolveBaseLocator(resolveScopeLocator(page, descriptor.scope), descriptor.target);
|
||||
}
|
||||
return resolveBaseLocator(page, descriptor);
|
||||
}
|
||||
|
||||
function resolveBaseLocator(root: LocatorRoot, descriptor: BaseLocatorDescriptor): Locator {
|
||||
switch (descriptor.kind) {
|
||||
case "role":
|
||||
return page.getByRole(descriptor.role as never, descriptor.name ? { name: descriptor.name } : undefined);
|
||||
return root.getByRole(descriptor.role as never, descriptor.name ? { name: descriptor.name } : undefined);
|
||||
case "label":
|
||||
return page.getByLabel(descriptor.label);
|
||||
return root.getByLabel(descriptor.label);
|
||||
case "placeholder":
|
||||
return page.getByPlaceholder(descriptor.placeholder);
|
||||
return root.getByPlaceholder(descriptor.placeholder);
|
||||
case "text":
|
||||
return page.getByText(descriptor.text, { exact: descriptor.exact });
|
||||
return root.getByText(descriptor.text, { exact: descriptor.exact });
|
||||
case "testId":
|
||||
return page.getByTestId(descriptor.testId);
|
||||
return root.getByTestId(descriptor.testId);
|
||||
case "css":
|
||||
return page.locator(descriptor.selector);
|
||||
return root.locator(descriptor.selector);
|
||||
}
|
||||
}
|
||||
|
||||
function resolveScopeLocator(page: Page, scope: LocatorScopeDescriptor): Locator {
|
||||
const text = scope.name ?? scope.text ?? "";
|
||||
switch (scope.kind) {
|
||||
case "row":
|
||||
return page.getByRole("row").filter({ hasText: text });
|
||||
case "dialog":
|
||||
return scope.name ? page.getByRole("dialog", { name: scope.name }) : page.getByRole("dialog").filter({ hasText: text });
|
||||
case "form":
|
||||
return page.locator("form").filter({ hasText: text });
|
||||
case "section":
|
||||
return page.locator("section, article").filter({ hasText: text });
|
||||
case "landmark":
|
||||
return page.locator("nav, aside, header, footer, main, [role='navigation'], [role='main'], [role='complementary']").filter({ hasText: text });
|
||||
case "listitem":
|
||||
return page.getByRole("listitem").filter({ hasText: text });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -155,6 +329,49 @@ function dedupeCandidates<T extends { expression: string }>(candidates: T[]): T[
|
||||
return deduped;
|
||||
}
|
||||
|
||||
function contextScopes(raw: RawActionableElement): LocatorScopeDescriptor[] {
|
||||
const contexts = [raw.context.dialog, raw.context.row, raw.context.form, raw.context.section, raw.context.landmark, raw.context.listitem];
|
||||
return contexts
|
||||
.filter((context): context is LocatorScopeDescriptor => Boolean(context && (context.name || context.text)))
|
||||
.map((context) => ({ kind: context.kind, name: clean(context.name) ?? null, text: clean(context.text) ?? null }));
|
||||
}
|
||||
|
||||
function executableScope(scopes: LocatorScopeDescriptor[], targetName: string): LocatorScopeDescriptor | null {
|
||||
const scope = scopes.find((item) => ["row", "dialog", "form", "listitem"].includes(item.kind) && Boolean(item.name || item.text));
|
||||
if (!scope) return null;
|
||||
return {
|
||||
...scope,
|
||||
text: shortenScopeText(scope.text, targetName),
|
||||
};
|
||||
}
|
||||
|
||||
function shortenScopeText(scopeText: string | null, targetName: string): string | null {
|
||||
if (!scopeText) return null;
|
||||
const escapedTarget = targetName.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
const shortened = scopeText.replace(new RegExp(`\\s*${escapedTarget}\\s*$`), "").trim();
|
||||
return clean(shortened) ?? scopeText;
|
||||
}
|
||||
|
||||
function scopeExpression(scope: LocatorScopeDescriptor): string {
|
||||
const text = scope.name ?? scope.text ?? "";
|
||||
switch (scope.kind) {
|
||||
case "row":
|
||||
return `page.getByRole("row").filter({ hasText: ${JSON.stringify(text)} })`;
|
||||
case "dialog":
|
||||
return scope.name
|
||||
? `page.getByRole("dialog", { name: ${JSON.stringify(scope.name)} })`
|
||||
: `page.getByRole("dialog").filter({ hasText: ${JSON.stringify(text)} })`;
|
||||
case "form":
|
||||
return `page.locator("form").filter({ hasText: ${JSON.stringify(text)} })`;
|
||||
case "listitem":
|
||||
return `page.getByRole("listitem").filter({ hasText: ${JSON.stringify(text)} })`;
|
||||
case "section":
|
||||
return `page.locator("section, article").filter({ hasText: ${JSON.stringify(text)} })`;
|
||||
case "landmark":
|
||||
return `page.locator("nav, aside, header, footer, main").filter({ hasText: ${JSON.stringify(text)} })`;
|
||||
}
|
||||
}
|
||||
|
||||
function clean(value: string | null | undefined): string | undefined {
|
||||
const cleaned = value?.replace(/\s+/g, " ").trim();
|
||||
return cleaned || undefined;
|
||||
@@ -167,4 +384,3 @@ function cssEscape(value: string): string {
|
||||
function cssString(value: string): string {
|
||||
return value.replace(/\\/g, "\\\\").replace(/"/g, '\\"');
|
||||
}
|
||||
|
||||
|
||||
86
src/metadata.ts
Normal file
86
src/metadata.ts
Normal file
@@ -0,0 +1,86 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { ENGINE_VERSION, SCHEMA_VERSION, type GraphMode, type RuntimeContext, type TopologyGraph } from "./types.js";
|
||||
import { shortHash } from "./hash.js";
|
||||
import { summarizeGraph } from "./summary.js";
|
||||
|
||||
const DEFAULT_VIEWPORT = { width: 1440, height: 1000 } as const;
|
||||
|
||||
export function defaultViewport(): { width: number; height: number } {
|
||||
return { ...DEFAULT_VIEWPORT };
|
||||
}
|
||||
|
||||
export function buildRuntimeContext(storageState?: string): RuntimeContext {
|
||||
return {
|
||||
browserName: "chromium",
|
||||
viewport: defaultViewport(),
|
||||
locale: "en-US",
|
||||
timezoneId: null,
|
||||
authContext: storageState ? "storage_state" : "anonymous",
|
||||
};
|
||||
}
|
||||
|
||||
export function buildEmptyGraph(args: {
|
||||
inputUrl: string;
|
||||
entryUrl: string;
|
||||
mode: GraphMode;
|
||||
storageState?: string;
|
||||
}): TopologyGraph {
|
||||
const generatedAt = new Date().toISOString();
|
||||
const crawlSessionId = `crawl_${randomUUID()}`;
|
||||
const inputFingerprint = `input_${shortHash([
|
||||
args.inputUrl,
|
||||
args.mode,
|
||||
"risk-policy-v0.2",
|
||||
"normalization-v0.2",
|
||||
"locator-scoring-v0.2",
|
||||
])}`;
|
||||
const graph: TopologyGraph = {
|
||||
schemaVersion: SCHEMA_VERSION,
|
||||
metadata: {
|
||||
artifactVersion: "topology-graph-v0.2",
|
||||
inputUrl: args.inputUrl,
|
||||
entryUrl: args.entryUrl,
|
||||
generatedAt,
|
||||
engineVersion: ENGINE_VERSION,
|
||||
mode: args.mode,
|
||||
crawlSessionId,
|
||||
inputSeed: args.inputUrl,
|
||||
inputFingerprint,
|
||||
runtimeContext: buildRuntimeContext(args.storageState),
|
||||
site: {
|
||||
baseUrl: baseUrl(args.entryUrl),
|
||||
entryUrl: args.entryUrl,
|
||||
},
|
||||
riskPolicyVersion: "risk-policy-v0.2",
|
||||
normalizationRuleVersion: "normalization-v0.2",
|
||||
locatorScoringVersion: "locator-scoring-v0.2",
|
||||
},
|
||||
states: [],
|
||||
elements: [],
|
||||
edges: [],
|
||||
skippedActions: [],
|
||||
failedObservations: [],
|
||||
summary: {
|
||||
states: 0,
|
||||
elements: 0,
|
||||
edges: 0,
|
||||
skippedActions: 0,
|
||||
failedObservations: 0,
|
||||
failedObservationsByReason: {},
|
||||
skippedActionsByRiskCategory: {},
|
||||
edgesByRiskCategory: {},
|
||||
},
|
||||
};
|
||||
graph.summary = summarizeGraph(graph);
|
||||
return graph;
|
||||
}
|
||||
|
||||
function baseUrl(value: string): string {
|
||||
try {
|
||||
const url = new URL(value);
|
||||
if (url.protocol === "file:") return url.protocol;
|
||||
return `${url.protocol}//${url.host}`;
|
||||
} catch {
|
||||
return "unknown";
|
||||
}
|
||||
}
|
||||
106
src/risk.ts
106
src/risk.ts
@@ -1,4 +1,4 @@
|
||||
import type { ActionableElement } from "./types.js";
|
||||
import type { ActionableElement, ActionType, RiskClassification, RiskPolicyDecision } from "./types.js";
|
||||
|
||||
const HIGH_RISK_TERMS = [
|
||||
"delete",
|
||||
@@ -6,10 +6,15 @@ const HIGH_RISK_TERMS = [
|
||||
"destroy",
|
||||
"drop",
|
||||
"logout",
|
||||
"log out",
|
||||
"sign out",
|
||||
"signin",
|
||||
"sign in",
|
||||
"pay",
|
||||
"payment",
|
||||
"purchase",
|
||||
"checkout",
|
||||
"check out",
|
||||
"submit",
|
||||
"confirm",
|
||||
"authorize",
|
||||
@@ -24,28 +29,93 @@ const HIGH_RISK_TERMS = [
|
||||
"授权",
|
||||
];
|
||||
|
||||
export function classifyClickRisk(element: ActionableElement): "low" | "medium" | "high" {
|
||||
const joined = [
|
||||
element.accessibleName,
|
||||
element.text,
|
||||
element.label,
|
||||
element.placeholder,
|
||||
element.attributes.href,
|
||||
element.attributes.type,
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(" ")
|
||||
.toLowerCase();
|
||||
const DESTRUCTIVE_TERMS = ["delete", "remove", "destroy", "drop", "删除", "移除"];
|
||||
const PAYMENT_TERMS = ["pay", "payment", "purchase", "checkout", "check out", "支付", "购买"];
|
||||
const AUTH_TERMS = ["authorize", "login", "log in", "logout", "log out", "signin", "sign in", "sign out", "授权", "注销", "退出"];
|
||||
const SUBMIT_TERMS = ["submit", "confirm", "提交", "确认"];
|
||||
|
||||
if (HIGH_RISK_TERMS.some((term) => joined.includes(term.toLowerCase()))) return "high";
|
||||
if (element.tag === "input" || element.tag === "textarea" || element.tag === "select") return "medium";
|
||||
return "low";
|
||||
export function classifyActionRisk(element: ActionableElement, actionType: ActionType): RiskClassification {
|
||||
const joined = riskText(element);
|
||||
const reasons: string[] = [];
|
||||
if (containsAny(joined, DESTRUCTIVE_TERMS)) {
|
||||
reasons.push("matched destructive action term");
|
||||
return risk("high", "destructive", reasons);
|
||||
}
|
||||
if (containsAny(joined, PAYMENT_TERMS)) {
|
||||
reasons.push("matched payment-sensitive action term");
|
||||
return risk("high", "payment_sensitive", reasons);
|
||||
}
|
||||
if (containsAny(joined, AUTH_TERMS)) {
|
||||
reasons.push("matched auth-sensitive action term");
|
||||
return risk("high", "auth_sensitive", reasons);
|
||||
}
|
||||
if (containsAny(joined, SUBMIT_TERMS) || element.attributes.type === "submit") {
|
||||
reasons.push("matched form submission action term or submit type");
|
||||
return risk("high", "form_submit", reasons);
|
||||
}
|
||||
if (HIGH_RISK_TERMS.some((term) => joined.includes(term.toLowerCase()))) {
|
||||
reasons.push("matched high-risk fallback term");
|
||||
return risk("high", "unknown", reasons);
|
||||
}
|
||||
if (actionType === "fill" || element.tag === "input" || element.tag === "textarea" || element.tag === "select") {
|
||||
reasons.push("editable control captured but not default-explored");
|
||||
return risk("medium", "input_edit", reasons);
|
||||
}
|
||||
if (element.tag === "a" || element.role === "link" || element.attributes.href) {
|
||||
reasons.push("link or href navigation");
|
||||
return risk("low", "navigation", reasons);
|
||||
}
|
||||
if (
|
||||
element.role === "button" ||
|
||||
element.role === "tab" ||
|
||||
element.role === "menuitem" ||
|
||||
element.attributes["aria-haspopup"] ||
|
||||
element.attributes["aria-expanded"]
|
||||
) {
|
||||
reasons.push("button-like UI reveal candidate");
|
||||
return risk("low", "ui_reveal", reasons);
|
||||
}
|
||||
reasons.push("no low-risk semantic evidence");
|
||||
return risk("high", "unknown", reasons);
|
||||
}
|
||||
|
||||
export function defaultExploreDecision(element: ActionableElement, actionType: ActionType): RiskPolicyDecision {
|
||||
const risk = classifyActionRisk(element, actionType);
|
||||
const allowed = risk.level === "low" && ["navigation", "ui_reveal", "read_only"].includes(risk.category);
|
||||
return {
|
||||
allowed,
|
||||
risk,
|
||||
reason: allowed ? `allowed:${risk.category}` : `blocked:${risk.category}`,
|
||||
};
|
||||
}
|
||||
|
||||
export function classifyClickRisk(element: ActionableElement): "low" | "medium" | "high" {
|
||||
return classifyActionRisk(element, "click").level;
|
||||
}
|
||||
|
||||
export function isLowRiskClickable(element: ActionableElement): boolean {
|
||||
if (!element.interactability.visible || !element.interactability.enabled || !element.interactability.receivesEvents) return false;
|
||||
if (classifyClickRisk(element) !== "low") return false;
|
||||
const role = element.role;
|
||||
return element.tag === "button" || element.tag === "a" || role === "button" || role === "link" || role === "tab" || role === "menuitem";
|
||||
const clickable = element.tag === "button" || element.tag === "a" || role === "button" || role === "link" || role === "tab" || role === "menuitem";
|
||||
return clickable && defaultExploreDecision(element, "click").allowed;
|
||||
}
|
||||
|
||||
function risk(level: RiskClassification["level"], category: RiskClassification["category"], reasons: string[]): RiskClassification {
|
||||
return {
|
||||
level,
|
||||
category,
|
||||
reasons,
|
||||
evidence: [{ level: "heuristic", source: "risk.policy", description: reasons.join("; ") }],
|
||||
};
|
||||
}
|
||||
|
||||
function riskText(element: ActionableElement): string {
|
||||
return [element.accessibleName, element.text, element.label, element.placeholder, element.attributes.href, element.attributes.type]
|
||||
.filter(Boolean)
|
||||
.join(" ")
|
||||
.toLowerCase();
|
||||
}
|
||||
|
||||
function containsAny(value: string, terms: string[]): boolean {
|
||||
return terms.some((term) => value.includes(term.toLowerCase()));
|
||||
}
|
||||
|
||||
210
src/snapshot.ts
210
src/snapshot.ts
@@ -1,7 +1,15 @@
|
||||
import type { Page } from "playwright";
|
||||
import { shortHash, stableHash } from "./hash.js";
|
||||
import { generateLocators, verifyLocators } from "./locator.js";
|
||||
import type { ActionableElement, RawActionableElement, SnapshotResult, StateNode } from "./types.js";
|
||||
import type {
|
||||
ActionableElement,
|
||||
ContextAnchor,
|
||||
ElementContext,
|
||||
ParameterSurface,
|
||||
RawActionableElement,
|
||||
SnapshotResult,
|
||||
StateNode,
|
||||
} from "./types.js";
|
||||
|
||||
export async function snapshotPage(page: Page, openedByEdgeId: string | null = null): Promise<SnapshotResult> {
|
||||
const raw = await page.evaluate(collectPageSnapshot);
|
||||
@@ -13,6 +21,7 @@ export async function snapshotPage(page: Page, openedByEdgeId: string | null = n
|
||||
visibleTextHash: stableHash(raw.visibleText),
|
||||
actionableSignatureHash: stableHash(raw.elements.map(actionableSignature)),
|
||||
});
|
||||
const viewport = raw.viewport;
|
||||
|
||||
const state: StateNode = {
|
||||
stateId: `state_${shortHash(stateHash)}`,
|
||||
@@ -27,12 +36,39 @@ export async function snapshotPage(page: Page, openedByEdgeId: string | null = n
|
||||
modalStack: raw.modalStack,
|
||||
openedByEdgeId,
|
||||
createdAt,
|
||||
lifecycle: "fingerprinted",
|
||||
urlCanonicalKey: raw.urlCanonicalKey,
|
||||
language: raw.language,
|
||||
viewport,
|
||||
evidence: [
|
||||
{ level: "observed", source: "browser.location", description: "URL and route captured from the page" },
|
||||
{
|
||||
level: "derived",
|
||||
source: "snapshot.hash",
|
||||
description: "State hashes derived from DOM, visible text, and actionable signatures",
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const elements: ActionableElement[] = [];
|
||||
for (const rawElement of raw.elements) {
|
||||
const elementId = `el_${shortHash([state.stateId, rawElement.uid, actionableSignature(rawElement)])}`;
|
||||
const locators = await verifyLocators(page, generateLocators(elementId, rawElement));
|
||||
const elementId = `el_${shortHash([
|
||||
state.stateId,
|
||||
rawElement.candidateIndex,
|
||||
rawElement.tag,
|
||||
rawElement.role,
|
||||
rawElement.attributes["data-testid"] || rawElement.attributes["data-test-id"] || rawElement.attributes["data-test"] || null,
|
||||
rawElement.attributes.id || null,
|
||||
rawElement.attributes.name || null,
|
||||
rawElement.identityText,
|
||||
rawElement.context.dialog?.name ?? null,
|
||||
rawElement.context.form?.name ?? null,
|
||||
rawElement.context.section?.name ?? null,
|
||||
rawElement.context.landmark?.name ?? null,
|
||||
rawElement.context.listitem?.name ?? null,
|
||||
null,
|
||||
])}`;
|
||||
const locators = await verifyLocators(page, rawElement, generateLocators(elementId, rawElement));
|
||||
elements.push({
|
||||
elementId,
|
||||
stateId: state.stateId,
|
||||
@@ -48,18 +84,30 @@ export async function snapshotPage(page: Page, openedByEdgeId: string | null = n
|
||||
shadowPath: [],
|
||||
interactability: rawElement.visibility,
|
||||
locators,
|
||||
context: rawElement.context,
|
||||
parameterSurface: rawElement.parameterSurface,
|
||||
evidence: rawElement.evidence,
|
||||
});
|
||||
}
|
||||
|
||||
return { state, elements, visibleText: raw.visibleText };
|
||||
}
|
||||
|
||||
function boundedIdentityText(value: string | null | undefined): string | null {
|
||||
const normalized = value?.replace(/\s+/g, " ").trim() ?? "";
|
||||
if (!normalized || normalized.length > 120) return null;
|
||||
return normalized;
|
||||
}
|
||||
|
||||
interface BrowserSnapshot {
|
||||
url: string;
|
||||
route: string;
|
||||
title: string;
|
||||
dom: string;
|
||||
visibleText: string;
|
||||
language: string | null;
|
||||
viewport: { width: number; height: number };
|
||||
urlCanonicalKey: string;
|
||||
modalStack: string[];
|
||||
elements: RawActionableElement[];
|
||||
}
|
||||
@@ -102,6 +150,9 @@ function collectPageSnapshot(): BrowserSnapshot {
|
||||
title: document.title,
|
||||
dom: document.documentElement.outerHTML,
|
||||
visibleText: normalizeText(document.body?.innerText || ""),
|
||||
language: normalizeNullable(document.documentElement.lang || navigator.language || ""),
|
||||
viewport: { width: window.innerWidth, height: window.innerHeight },
|
||||
urlCanonicalKey: canonicalUrl(location.href),
|
||||
modalStack: Array.from(document.querySelectorAll<HTMLElement>("[role='dialog'], dialog[open], [aria-modal='true']"))
|
||||
.filter((element) => isVisible(element))
|
||||
.map((element) => getAccessibleName(element) || element.id || element.tagName.toLowerCase()),
|
||||
@@ -125,15 +176,28 @@ function collectPageSnapshot(): BrowserSnapshot {
|
||||
const label = getLabel(element);
|
||||
const placeholder = "placeholder" in element ? normalizeNullable(String((element as HTMLInputElement).placeholder || "")) : null;
|
||||
const accessibleName = normalizeNullable(getAccessibleName(element) || label || placeholder || text);
|
||||
const context = getElementContext(element);
|
||||
const parameterSurface = getParameterSurface(element);
|
||||
const identityText = getElementIdentityText(element, label, placeholder, text, accessibleName);
|
||||
const centerX = rect.left + rect.width / 2;
|
||||
const centerY = rect.top + rect.height / 2;
|
||||
const topAtCenter = document.elementFromPoint(centerX, centerY);
|
||||
|
||||
return {
|
||||
uid: `${tag}_${index}_${attributes.id || attributes.name || accessibleName || text || ""}`,
|
||||
uid: `${tag}_${index}_${
|
||||
attributes.id ||
|
||||
attributes.name ||
|
||||
attributes["data-testid"] ||
|
||||
attributes["data-test-id"] ||
|
||||
attributes["data-test"] ||
|
||||
identityText ||
|
||||
""
|
||||
}`,
|
||||
candidateIndex: index,
|
||||
tag,
|
||||
role,
|
||||
accessibleName,
|
||||
identityText,
|
||||
text: normalizeNullable(text),
|
||||
label,
|
||||
placeholder,
|
||||
@@ -150,6 +214,132 @@ function collectPageSnapshot(): BrowserSnapshot {
|
||||
editable: isEditable(element),
|
||||
receivesEvents: topAtCenter === element || Boolean(topAtCenter && element.contains(topAtCenter)),
|
||||
},
|
||||
context,
|
||||
parameterSurface,
|
||||
evidence: [
|
||||
{ level: "observed", source: "dom", description: "Element was visible and matched actionable selector" },
|
||||
{
|
||||
level: "derived",
|
||||
source: "accessibility",
|
||||
description: "Role, name, label, and placeholder were normalized from DOM evidence",
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
function getElementContext(element: HTMLElement): ElementContext {
|
||||
return {
|
||||
dialog: anchor(element.closest<HTMLElement>("[role='dialog'], dialog[open], [aria-modal='true']"), "dialog"),
|
||||
form: anchor(element.closest<HTMLElement>("form"), "form"),
|
||||
section: anchor(element.closest<HTMLElement>("section, article"), "section"),
|
||||
row: anchor(element.closest<HTMLElement>("tr, [role='row']"), "row"),
|
||||
landmark: anchor(
|
||||
element.closest<HTMLElement>("nav, aside, header, footer, main, [role='navigation'], [role='main'], [role='complementary']"),
|
||||
"landmark",
|
||||
),
|
||||
listitem: anchor(element.closest<HTMLElement>("li, [role='listitem']"), "listitem"),
|
||||
};
|
||||
}
|
||||
|
||||
function anchor(element: HTMLElement | null, kind: ContextAnchor["kind"]): ContextAnchor | null {
|
||||
if (!element) return null;
|
||||
return {
|
||||
kind,
|
||||
name: getAnchorName(element),
|
||||
text: normalizeNullable((element.innerText || element.textContent || "").slice(0, 240)),
|
||||
};
|
||||
}
|
||||
|
||||
function getAnchorName(element: HTMLElement): string | null {
|
||||
return (
|
||||
boundedAnchorText(element.getAttribute("aria-label")) ??
|
||||
getAriaLabelledByAnchorName(element) ??
|
||||
boundedAnchorText(element.getAttribute("title")) ??
|
||||
boundedAnchorText(element.id) ??
|
||||
getHeadingAnchorName(element)
|
||||
);
|
||||
}
|
||||
|
||||
function getAriaLabelledByAnchorName(element: HTMLElement): string | null {
|
||||
const labelledBy = element.getAttribute("aria-labelledby");
|
||||
if (!labelledBy) return null;
|
||||
const text = labelledBy
|
||||
.split(/\s+/)
|
||||
.map((id) => document.getElementById(id)?.innerText || document.getElementById(id)?.textContent || "")
|
||||
.join(" ");
|
||||
return boundedAnchorText(text);
|
||||
}
|
||||
|
||||
function getHeadingAnchorName(element: HTMLElement): string | null {
|
||||
const heading = element.querySelector<HTMLElement>("h1, h2, h3, h4, h5, h6, [role='heading']");
|
||||
if (!heading) return null;
|
||||
return boundedAnchorText(heading.innerText || heading.textContent || "");
|
||||
}
|
||||
|
||||
function boundedAnchorText(value: string | null | undefined): string | null {
|
||||
const normalized = normalizeText(value || "");
|
||||
if (!normalized || normalized.length > 120) return null;
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function boundedIdentityText(value: string | null | undefined): string | null {
|
||||
const normalized = normalizeText(value || "");
|
||||
if (!normalized || normalized.length > 120) return null;
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function getElementIdentityText(
|
||||
element: HTMLElement,
|
||||
label: string | null,
|
||||
placeholder: string | null,
|
||||
text: string,
|
||||
accessibleName: string | null,
|
||||
): string | null {
|
||||
return (
|
||||
boundedIdentityText(element.getAttribute("aria-label")) ??
|
||||
getAriaLabelledByAnchorName(element) ??
|
||||
boundedIdentityText(element.getAttribute("title")) ??
|
||||
boundedIdentityText(label) ??
|
||||
boundedIdentityText(placeholder) ??
|
||||
(isContainerLike(element) ? null : boundedIdentityText(accessibleName) ?? boundedIdentityText(text))
|
||||
);
|
||||
}
|
||||
|
||||
function isContainerLike(element: HTMLElement): boolean {
|
||||
const tag = element.tagName.toLowerCase();
|
||||
const role = element.getAttribute("role")?.toLowerCase() || "";
|
||||
return (
|
||||
["article", "aside", "dialog", "div", "footer", "form", "header", "li", "main", "nav", "ol", "section", "table", "tbody", "thead", "tr", "ul"].includes(
|
||||
tag,
|
||||
) ||
|
||||
["complementary", "dialog", "form", "grid", "list", "listitem", "main", "navigation", "region", "row", "table"].includes(role)
|
||||
);
|
||||
}
|
||||
|
||||
function getParameterSurface(element: HTMLElement): ParameterSurface | null {
|
||||
const tag = element.tagName.toLowerCase();
|
||||
if (!["input", "select", "textarea"].includes(tag) && !element.isContentEditable) return null;
|
||||
const input = element as HTMLInputElement;
|
||||
const select = element as HTMLSelectElement;
|
||||
return {
|
||||
name: input.name || element.getAttribute("name"),
|
||||
inputType: tag === "input" ? (input.type || "text").toLowerCase() : tag,
|
||||
required: Boolean((input as HTMLInputElement).required),
|
||||
readonly: Boolean((input as HTMLInputElement).readOnly),
|
||||
disabled: Boolean((input as HTMLInputElement).disabled),
|
||||
pattern: input.getAttribute("pattern"),
|
||||
min: input.getAttribute("min"),
|
||||
max: input.getAttribute("max"),
|
||||
maxLength: Number.isFinite(input.maxLength) && input.maxLength >= 0 ? input.maxLength : null,
|
||||
options:
|
||||
tag === "select"
|
||||
? Array.from(select.options).map((option) => ({
|
||||
label: normalizeText(option.label || option.textContent || ""),
|
||||
value: option.value,
|
||||
disabled: option.disabled,
|
||||
selected: option.selected,
|
||||
}))
|
||||
: [],
|
||||
};
|
||||
}
|
||||
|
||||
@@ -247,5 +437,15 @@ function collectPageSnapshot(): BrowserSnapshot {
|
||||
const normalized = normalizeText(value);
|
||||
return normalized || null;
|
||||
}
|
||||
}
|
||||
|
||||
function canonicalUrl(value: string): string {
|
||||
const url = new URL(value);
|
||||
url.hash = "";
|
||||
const params = Array.from(url.searchParams.entries())
|
||||
.filter(([key]) => !/^utm_|^fbclid$|^gclid$/.test(key))
|
||||
.sort(([left], [right]) => left.localeCompare(right));
|
||||
url.search = "";
|
||||
for (const [key, item] of params) url.searchParams.append(key, item);
|
||||
return url.toString();
|
||||
}
|
||||
}
|
||||
|
||||
28
src/summary.ts
Normal file
28
src/summary.ts
Normal file
@@ -0,0 +1,28 @@
|
||||
import type { GraphSummary, TopologyGraph } from "./types.js";
|
||||
|
||||
export function summarizeGraph(graph: TopologyGraph): GraphSummary {
|
||||
return {
|
||||
states: graph.states.length,
|
||||
elements: graph.elements.length,
|
||||
edges: graph.edges.length,
|
||||
skippedActions: graph.skippedActions.length,
|
||||
failedObservations: graph.failedObservations.length,
|
||||
failedObservationsByReason: countBy(graph.failedObservations, (failure) => failure.reason),
|
||||
skippedActionsByRiskCategory: countBy(graph.skippedActions, (skip) => skip.riskCategory),
|
||||
edgesByRiskCategory: countBy(graph.edges, (edge) => edge.riskCategory),
|
||||
};
|
||||
}
|
||||
|
||||
export function refreshGraphSummary(graph: TopologyGraph): TopologyGraph {
|
||||
graph.summary = summarizeGraph(graph);
|
||||
return graph;
|
||||
}
|
||||
|
||||
function countBy<T>(items: T[], keyFor: (item: T) => string): Record<string, number> {
|
||||
const counts: Record<string, number> = {};
|
||||
for (const item of items) {
|
||||
const key = keyFor(item);
|
||||
counts[key] = (counts[key] ?? 0) + 1;
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
192
src/types.ts
192
src/types.ts
@@ -1,4 +1,79 @@
|
||||
export type LocatorDescriptor =
|
||||
export const SCHEMA_VERSION = "0.2" as const;
|
||||
export const ENGINE_VERSION = "0.2.2" as const;
|
||||
|
||||
export type GraphMode = "scan" | "explore";
|
||||
export type EvidenceLevel = "observed" | "derived" | "verified" | "heuristic" | "external";
|
||||
export type StateLifecycle = "observed" | "fingerprinted" | "verified" | "stale" | "deprecated" | "conflicted";
|
||||
|
||||
export interface EvidenceRecord {
|
||||
level: EvidenceLevel;
|
||||
source: string;
|
||||
description: string;
|
||||
}
|
||||
|
||||
export type RiskLevel = "low" | "medium" | "high";
|
||||
export type RiskCategory =
|
||||
| "read_only"
|
||||
| "navigation"
|
||||
| "ui_reveal"
|
||||
| "input_edit"
|
||||
| "form_submit"
|
||||
| "data_mutation"
|
||||
| "destructive"
|
||||
| "auth_sensitive"
|
||||
| "payment_sensitive"
|
||||
| "unknown";
|
||||
|
||||
export interface RiskClassification {
|
||||
level: RiskLevel;
|
||||
category: RiskCategory;
|
||||
reasons: string[];
|
||||
evidence: EvidenceRecord[];
|
||||
}
|
||||
|
||||
export interface RiskPolicyDecision {
|
||||
allowed: boolean;
|
||||
risk: RiskClassification;
|
||||
reason: string;
|
||||
}
|
||||
|
||||
export interface ContextAnchor {
|
||||
kind: "dialog" | "form" | "section" | "row" | "landmark" | "listitem";
|
||||
name: string | null;
|
||||
text: string | null;
|
||||
}
|
||||
|
||||
export interface ElementContext {
|
||||
dialog: ContextAnchor | null;
|
||||
form: ContextAnchor | null;
|
||||
section: ContextAnchor | null;
|
||||
row: ContextAnchor | null;
|
||||
landmark: ContextAnchor | null;
|
||||
listitem: ContextAnchor | null;
|
||||
}
|
||||
|
||||
export interface ParameterSurface {
|
||||
name: string | null;
|
||||
inputType: string | null;
|
||||
required: boolean;
|
||||
readonly: boolean;
|
||||
disabled: boolean;
|
||||
pattern: string | null;
|
||||
min: string | null;
|
||||
max: string | null;
|
||||
maxLength: number | null;
|
||||
options: Array<{ label: string; value: string; disabled: boolean; selected: boolean }>;
|
||||
}
|
||||
|
||||
export interface RuntimeContext {
|
||||
browserName: "chromium";
|
||||
viewport: { width: number; height: number };
|
||||
locale: string;
|
||||
timezoneId: string | null;
|
||||
authContext: "anonymous" | "storage_state";
|
||||
}
|
||||
|
||||
export type BaseLocatorDescriptor =
|
||||
| { kind: "role"; role: string; name?: string }
|
||||
| { kind: "label"; label: string }
|
||||
| { kind: "placeholder"; placeholder: string }
|
||||
@@ -6,8 +81,26 @@ export type LocatorDescriptor =
|
||||
| { kind: "testId"; testId: string }
|
||||
| { kind: "css"; selector: string };
|
||||
|
||||
export type LocatorDescriptor =
|
||||
| BaseLocatorDescriptor
|
||||
| { kind: "scoped"; scope: LocatorScopeDescriptor; target: BaseLocatorDescriptor };
|
||||
|
||||
export type LocatorFramework = "playwright" | "css";
|
||||
|
||||
export interface LocatorScopeDescriptor {
|
||||
kind: ContextAnchor["kind"];
|
||||
name: string | null;
|
||||
text: string | null;
|
||||
}
|
||||
|
||||
export interface LocatorIdentityEvidence {
|
||||
sameTag: boolean;
|
||||
sameRole: boolean;
|
||||
sameAccessibleName: boolean;
|
||||
sameText: boolean;
|
||||
sameVisibility: boolean;
|
||||
}
|
||||
|
||||
export interface LocatorCandidate {
|
||||
locatorId: string;
|
||||
elementId: string;
|
||||
@@ -20,6 +113,9 @@ export interface LocatorCandidate {
|
||||
verified: boolean;
|
||||
lastVerifiedAt?: string;
|
||||
matchCount?: number;
|
||||
scopes: LocatorScopeDescriptor[];
|
||||
identity: LocatorIdentityEvidence | null;
|
||||
evidence: EvidenceRecord[];
|
||||
failureCount: number;
|
||||
}
|
||||
|
||||
@@ -52,6 +148,9 @@ export interface ActionableElement {
|
||||
shadowPath: string[];
|
||||
interactability: VisibilityInfo;
|
||||
locators: LocatorCandidate[];
|
||||
context: ElementContext;
|
||||
parameterSurface: ParameterSurface | null;
|
||||
evidence: EvidenceRecord[];
|
||||
}
|
||||
|
||||
export interface StateNode {
|
||||
@@ -67,17 +166,19 @@ export interface StateNode {
|
||||
modalStack: string[];
|
||||
openedByEdgeId: string | null;
|
||||
createdAt: string;
|
||||
lifecycle: StateLifecycle;
|
||||
urlCanonicalKey: string;
|
||||
language: string | null;
|
||||
viewport: { width: number; height: number };
|
||||
evidence: EvidenceRecord[];
|
||||
}
|
||||
|
||||
export type ActionType = "click" | "fill" | "select" | "check" | "uncheck" | "navigate";
|
||||
|
||||
export interface EffectRecord {
|
||||
type: "url" | "dom" | "visibleText" | "network" | "download" | "dialog" | "storage";
|
||||
type: "url" | "dom" | "visibleText" | "actionableInventory";
|
||||
before?: string;
|
||||
after?: string;
|
||||
method?: string;
|
||||
url?: string;
|
||||
status?: number;
|
||||
description: string;
|
||||
}
|
||||
|
||||
@@ -100,35 +201,107 @@ export interface ActionEdge {
|
||||
preconditions: string[];
|
||||
effects: EffectRecord[];
|
||||
parameterSpecs: ParameterSpec[];
|
||||
riskLevel: "low" | "medium" | "high";
|
||||
riskLevel: RiskLevel;
|
||||
riskCategory: RiskCategory;
|
||||
confidence: number;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export interface TopologyGraph {
|
||||
schemaVersion: "0.1";
|
||||
schemaVersion: typeof SCHEMA_VERSION;
|
||||
metadata: {
|
||||
artifactVersion: string;
|
||||
inputUrl: string;
|
||||
entryUrl: string;
|
||||
generatedAt: string;
|
||||
engineVersion: string;
|
||||
mode: "scan" | "explore";
|
||||
engineVersion: typeof ENGINE_VERSION;
|
||||
mode: GraphMode;
|
||||
crawlSessionId: string;
|
||||
inputSeed: string;
|
||||
inputFingerprint: string;
|
||||
runtimeContext: RuntimeContext;
|
||||
site: {
|
||||
baseUrl: string;
|
||||
entryUrl: string;
|
||||
};
|
||||
riskPolicyVersion: "risk-policy-v0.2";
|
||||
normalizationRuleVersion: "normalization-v0.2";
|
||||
locatorScoringVersion: "locator-scoring-v0.2";
|
||||
};
|
||||
states: StateNode[];
|
||||
elements: ActionableElement[];
|
||||
edges: ActionEdge[];
|
||||
skippedActions: SkippedAction[];
|
||||
failedObservations: FailedObservation[];
|
||||
summary: GraphSummary;
|
||||
}
|
||||
|
||||
export interface GraphSummary {
|
||||
states: number;
|
||||
elements: number;
|
||||
edges: number;
|
||||
skippedActions: number;
|
||||
failedObservations: number;
|
||||
failedObservationsByReason: Partial<Record<FailedObservation["reason"], number>>;
|
||||
skippedActionsByRiskCategory: Partial<Record<RiskCategory, number>>;
|
||||
edgesByRiskCategory: Partial<Record<RiskCategory, number>>;
|
||||
}
|
||||
|
||||
export interface SkippedAction {
|
||||
skippedActionId: string;
|
||||
stateId: string;
|
||||
elementId: string;
|
||||
actionType: ActionType;
|
||||
riskLevel: RiskLevel;
|
||||
riskCategory: RiskCategory;
|
||||
reason: string;
|
||||
evidence: EvidenceRecord[];
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export interface FailedObservation {
|
||||
failedObservationId: string;
|
||||
stateId: string | null;
|
||||
elementId: string | null;
|
||||
actionType: ActionType | null;
|
||||
reason:
|
||||
| "locator_not_found"
|
||||
| "locator_not_unique"
|
||||
| "element_hidden"
|
||||
| "element_disabled"
|
||||
| "element_obscured"
|
||||
| "state_precondition_missing"
|
||||
| "frame_inaccessible"
|
||||
| "shadow_root_closed"
|
||||
| "navigation_timeout"
|
||||
| "network_error"
|
||||
| "permission_denied"
|
||||
| "auth_required"
|
||||
| "captcha_or_challenge"
|
||||
| "risk_policy_blocked"
|
||||
| "effect_not_observed"
|
||||
| "unexpected_state";
|
||||
message: string;
|
||||
evidence: EvidenceRecord[];
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export interface RawActionableElement {
|
||||
uid: string;
|
||||
candidateIndex: number;
|
||||
tag: string;
|
||||
role: string | null;
|
||||
accessibleName: string | null;
|
||||
identityText: string | null;
|
||||
text: string | null;
|
||||
label: string | null;
|
||||
placeholder: string | null;
|
||||
attributes: Record<string, string>;
|
||||
bbox: BoundingBox | null;
|
||||
visibility: VisibilityInfo;
|
||||
context: ElementContext;
|
||||
parameterSurface: ParameterSurface | null;
|
||||
evidence: EvidenceRecord[];
|
||||
}
|
||||
|
||||
export interface SnapshotResult {
|
||||
@@ -136,4 +309,3 @@ export interface SnapshotResult {
|
||||
elements: ActionableElement[];
|
||||
visibleText: string;
|
||||
}
|
||||
|
||||
|
||||
133
tests/fixtures/context-risk.html
vendored
Normal file
133
tests/fixtures/context-risk.html
vendored
Normal file
@@ -0,0 +1,133 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Topology Context Risk Fixture</title>
|
||||
<style>
|
||||
body { font-family: system-ui, sans-serif; margin: 24px; }
|
||||
table { border-collapse: collapse; margin: 16px 0; }
|
||||
th, td { border: 1px solid #ccc; padding: 8px 10px; }
|
||||
button, input, select, a { margin: 4px; padding: 6px 10px; }
|
||||
[role="dialog"] { display: none; border: 1px solid #999; padding: 16px; margin-top: 16px; }
|
||||
[role="dialog"].open { display: block; }
|
||||
#settings-dialog.open { position: fixed; inset: 24px auto auto 24px; background: white; z-index: 10; }
|
||||
#toast { margin-top: 16px; min-height: 24px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Accounts</h1>
|
||||
<button id="storage-leak-marker" hidden>Replay storage leaked</button>
|
||||
<button data-testid="remember-filter" onclick="localStorage.setItem('ateReplayLeak', '1')">Remember filter</button>
|
||||
<nav aria-label="Primary">
|
||||
<a href="#overview" data-testid="overview-link">Overview</a>
|
||||
<a href="#reports" data-testid="reports-link">Reports</a>
|
||||
</nav>
|
||||
|
||||
<section aria-labelledby="customers-heading">
|
||||
<h2 id="customers-heading">Customers</h2>
|
||||
<table aria-label="Customer accounts">
|
||||
<thead>
|
||||
<tr><th>Name</th><th>Status</th><th>Action</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Acme Co</td>
|
||||
<td>Active</td>
|
||||
<td><button data-testid="edit-acme" onclick="openEditor('Acme Co')">Edit</button></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Beta LLC</td>
|
||||
<td>Trial</td>
|
||||
<td><button data-testid="edit-beta" onclick="openEditor('Beta LLC')">Edit</button></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</section>
|
||||
|
||||
<form aria-label="Search customers" action="/search" method="get">
|
||||
<label>
|
||||
Query
|
||||
<input name="q" placeholder="Search customers" required maxlength="40" />
|
||||
</label>
|
||||
<label>
|
||||
Segment
|
||||
<select name="segment">
|
||||
<option value="">Any</option>
|
||||
<option value="active">Active</option>
|
||||
<option value="trial">Trial</option>
|
||||
</select>
|
||||
</label>
|
||||
<button type="submit">Submit search</button>
|
||||
</form>
|
||||
|
||||
<form>
|
||||
<p>
|
||||
This unnamed form deliberately contains a long body of descendant text that should remain available as bounded
|
||||
context evidence but must not be promoted into the anchor name used for element identity. It describes internal
|
||||
notes, operator workflow details, historical context, and extra explanatory copy that would make a noisy and
|
||||
unstable identifier if copied into the context anchor name.
|
||||
</p>
|
||||
<label>
|
||||
Unbounded Probe
|
||||
<input name="unboundedProbe" />
|
||||
</label>
|
||||
</form>
|
||||
|
||||
<main aria-label="Operations">
|
||||
<button data-testid="main-action">Main action</button>
|
||||
</main>
|
||||
|
||||
<section aria-label="Duplicate controls">
|
||||
<button>Duplicate</button>
|
||||
<button>Duplicate</button>
|
||||
<button><span aria-hidden="true"></span></button>
|
||||
</section>
|
||||
|
||||
<section role="dialog" aria-modal="true" class="open">
|
||||
<span>Short volatile copy</span>
|
||||
<button data-testid="short-dialog-action">Short dialog action</button>
|
||||
</section>
|
||||
|
||||
<section role="dialog" aria-modal="true" class="open">
|
||||
<p>
|
||||
This unnamed dialog carries noisy descendant copy about multi-step operator notes, escalation background,
|
||||
support annotations, internal timestamps, and other volatile details that must stay out of the dialog anchor
|
||||
name while remaining available as bounded context text evidence for nearby actionable controls.
|
||||
</p>
|
||||
<button data-testid="dialog-probe">Dialog probe</button>
|
||||
</section>
|
||||
|
||||
<button disabled data-testid="disabled-open">Open disabled panel</button>
|
||||
<button disabled data-testid="disabled-delete">Delete disabled account</button>
|
||||
<button data-testid="open-settings" onclick="document.getElementById('settings-dialog').classList.add('open')">Open settings</button>
|
||||
<section id="settings-dialog" role="dialog" aria-modal="true" aria-label="Settings dialog">
|
||||
<h2>Settings</h2>
|
||||
<button data-testid="save-settings" onclick="document.getElementById('toast').textContent='Saved settings'">Save</button>
|
||||
<button data-testid="delete-account">Delete account</button>
|
||||
<button data-testid="close-settings" onclick="document.getElementById('settings-dialog').classList.remove('open')">Close</button>
|
||||
</section>
|
||||
|
||||
<section id="editor" role="dialog" aria-modal="true" aria-label="Customer editor">
|
||||
<h2 id="editor-title">Customer editor</h2>
|
||||
<label>
|
||||
Customer name
|
||||
<input id="customer-name" name="customerName" required />
|
||||
</label>
|
||||
<button data-testid="close-editor" onclick="document.getElementById('editor').classList.remove('open')">Close editor</button>
|
||||
</section>
|
||||
|
||||
<p id="toast" role="status"></p>
|
||||
<script>
|
||||
if (localStorage.getItem("ateReplayLeak")) {
|
||||
document.getElementById("storage-leak-marker").hidden = false;
|
||||
}
|
||||
|
||||
function openEditor(name) {
|
||||
document.getElementById("customer-name").value = name;
|
||||
document.getElementById("editor-title").textContent = "Editing " + name;
|
||||
document.getElementById("editor").classList.add("open");
|
||||
}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
489
tests/integration.test.ts
Normal file
489
tests/integration.test.ts
Normal file
@@ -0,0 +1,489 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { createServer, type Server } from "node:http";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { exploreUrl, scanUrl } from "../src/engine.js";
|
||||
import { shortHash } from "../src/hash.js";
|
||||
|
||||
const fixturePath = path.resolve("tests/fixtures/context-risk.html");
|
||||
const fixtureFileUrl = pathToFileURL(fixturePath).href;
|
||||
const examplePath = path.resolve("examples/simple.html");
|
||||
const snapshotDiffEffectTypes = ["actionableInventory", "dom", "url", "visibleText"];
|
||||
|
||||
async function withTempOutput<T>(name: string, callback: (out: string) => Promise<T>): Promise<T> {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "action-topology-engine-test-"));
|
||||
try {
|
||||
return await callback(path.join(dir, `${name}.json`));
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function withServer<T>(handler: Parameters<typeof createServer>[0], callback: (url: string) => Promise<T>): Promise<T> {
|
||||
const server = createServer(handler);
|
||||
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
|
||||
const address = server.address();
|
||||
assert.ok(address && typeof address === "object");
|
||||
try {
|
||||
return await callback(`http://127.0.0.1:${address.port}/`);
|
||||
} finally {
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
(server as Server).close((error) => (error ? reject(error) : resolve()));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function hasEdgeForAccessibleName(
|
||||
graph: {
|
||||
edges: Array<{ elementId: string }>;
|
||||
elements: Array<{ elementId: string; accessibleName: string | null }>;
|
||||
},
|
||||
accessibleName: string,
|
||||
): boolean {
|
||||
return graph.edges.some((edge) => {
|
||||
const element = graph.elements.find((item) => item.elementId === edge.elementId);
|
||||
return element?.accessibleName === accessibleName;
|
||||
});
|
||||
}
|
||||
|
||||
function edgeForAccessibleName(
|
||||
graph: {
|
||||
edges: Array<{
|
||||
edgeId: string;
|
||||
elementId: string;
|
||||
toStateId: string | null;
|
||||
effects: Array<{ type: string; before?: string; after?: string; description: string }>;
|
||||
}>;
|
||||
elements: Array<{ elementId: string; accessibleName: string | null }>;
|
||||
},
|
||||
accessibleName: string,
|
||||
) {
|
||||
return graph.edges.find((edge) => {
|
||||
const element = graph.elements.find((item) => item.elementId === edge.elementId);
|
||||
return element?.accessibleName === accessibleName;
|
||||
});
|
||||
}
|
||||
|
||||
test("scan captures a state-local inventory without clicking high-risk actions", async () => {
|
||||
const graph = await withTempOutput("scan", (out) => scanUrl({ url: fixturePath, out }));
|
||||
assert.equal(graph.states.length, 1);
|
||||
assert.equal(graph.edges.length, 0);
|
||||
assert.equal(graph.elements.some((element) => element.accessibleName === "Open settings"), true);
|
||||
assert.equal(graph.elements.some((element) => element.accessibleName === "Delete account"), false);
|
||||
});
|
||||
|
||||
test("duplicated row actions are verified with semantic locator evidence", async () => {
|
||||
const graph = await withTempOutput("duplicate-locators", (out) => scanUrl({ url: fixturePath, out }));
|
||||
const editButtons = graph.elements.filter((element) => element.accessibleName === "Edit");
|
||||
assert.equal(editButtons.length, 2);
|
||||
for (const edit of editButtons) {
|
||||
assert.equal(edit.locators.some((locator) => locator.verified && locator.identity?.sameAccessibleName), true);
|
||||
assert.equal(edit.locators.some((locator) => locator.scopes.some((scope) => scope.kind === "row")), true);
|
||||
}
|
||||
});
|
||||
|
||||
test("graph metadata records generation context and policy versions", async () => {
|
||||
const graph = await withTempOutput("metadata", (out) => scanUrl({ url: fixturePath, out }));
|
||||
assert.equal(graph.schemaVersion, "0.2");
|
||||
assert.equal(graph.metadata.engineVersion, "0.2.2");
|
||||
assert.equal(graph.metadata.runtimeContext.browserName, "chromium");
|
||||
assert.equal(graph.metadata.runtimeContext.viewport.width, 1440);
|
||||
assert.equal(graph.metadata.runtimeContext.viewport.height, 1000);
|
||||
assert.equal(graph.metadata.riskPolicyVersion, "risk-policy-v0.2");
|
||||
assert.equal(graph.metadata.locatorScoringVersion, "locator-scoring-v0.2");
|
||||
assert.equal(typeof graph.metadata.crawlSessionId, "string");
|
||||
assert.equal(graph.metadata.inputUrl, fixturePath);
|
||||
assert.equal(typeof graph.metadata.inputFingerprint, "string");
|
||||
});
|
||||
|
||||
test("scan records state lifecycle, language, context anchors, and parameter surfaces", async () => {
|
||||
const graph = await withTempOutput("evidence", (out) =>
|
||||
scanUrl({ url: `${fixtureFileUrl}?utm_source=codex&gclid=ad-click&keep=1#customers`, out }),
|
||||
);
|
||||
const state = graph.states[0];
|
||||
assert.equal(state.lifecycle, "fingerprinted");
|
||||
assert.equal(state.language, "en");
|
||||
assert.equal(state.viewport.width, 1440);
|
||||
assert.equal(state.urlCanonicalKey, `${fixtureFileUrl}?keep=1`);
|
||||
assert.equal(state.evidence.some((item) => item.level === "observed" && item.source === "browser.location"), true);
|
||||
assert.equal(state.evidence.some((item) => item.level === "derived" && item.source === "snapshot.hash"), true);
|
||||
|
||||
const query = graph.elements.find((element) => element.accessibleName === "Query");
|
||||
assert.ok(query);
|
||||
assert.equal(query.parameterSurface?.name, "q");
|
||||
assert.equal(query.parameterSurface?.required, true);
|
||||
assert.equal(query.parameterSurface?.maxLength, 40);
|
||||
assert.equal(query.context.form?.name, "Search customers");
|
||||
assert.equal(query.evidence.some((item) => item.level === "observed"), true);
|
||||
|
||||
const dialogProbe = graph.elements.find((element) => element.accessibleName === "Dialog probe");
|
||||
assert.ok(dialogProbe);
|
||||
assert.equal(dialogProbe.context.dialog?.name, null);
|
||||
assert.equal(dialogProbe.context.dialog?.text?.includes("noisy descendant copy"), true);
|
||||
|
||||
const shortUnnamedDialog = graph.elements.find(
|
||||
(element) => element.tag === "section" && element.role === "dialog" && element.text?.includes("Short volatile copy"),
|
||||
);
|
||||
assert.ok(shortUnnamedDialog);
|
||||
assert.equal(shortUnnamedDialog.context.dialog?.name, null);
|
||||
assert.equal(shortUnnamedDialog.context.dialog?.text?.includes("Short volatile copy"), true);
|
||||
const shortUnnamedDialogIndex = graph.elements.findIndex((element) => element === shortUnnamedDialog);
|
||||
assert.notEqual(shortUnnamedDialogIndex, -1);
|
||||
assert.equal(
|
||||
shortUnnamedDialog.elementId,
|
||||
`el_${shortHash([
|
||||
state.stateId,
|
||||
shortUnnamedDialogIndex,
|
||||
shortUnnamedDialog.tag,
|
||||
shortUnnamedDialog.role,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
shortUnnamedDialog.context.dialog?.name,
|
||||
shortUnnamedDialog.context.form?.name,
|
||||
shortUnnamedDialog.context.section?.name,
|
||||
shortUnnamedDialog.context.landmark?.name,
|
||||
shortUnnamedDialog.context.listitem?.name,
|
||||
null,
|
||||
])}`,
|
||||
);
|
||||
|
||||
const unnamedDialog = graph.elements.find(
|
||||
(element) => element.tag === "section" && element.role === "dialog" && element.text?.includes("noisy descendant copy"),
|
||||
);
|
||||
assert.ok(unnamedDialog);
|
||||
const unnamedDialogIndex = graph.elements.findIndex((element) => element === unnamedDialog);
|
||||
assert.notEqual(unnamedDialogIndex, -1);
|
||||
assert.equal(
|
||||
unnamedDialog.elementId,
|
||||
`el_${shortHash([
|
||||
state.stateId,
|
||||
unnamedDialogIndex,
|
||||
unnamedDialog.tag,
|
||||
unnamedDialog.role,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
unnamedDialog.context.dialog?.name,
|
||||
unnamedDialog.context.form?.name,
|
||||
unnamedDialog.context.section?.name,
|
||||
unnamedDialog.context.landmark?.name,
|
||||
unnamedDialog.context.listitem?.name,
|
||||
null,
|
||||
])}`,
|
||||
);
|
||||
|
||||
const duplicates = graph.elements.filter((element) => element.accessibleName === "Duplicate");
|
||||
assert.equal(duplicates.length, 2);
|
||||
assert.notEqual(duplicates[0]?.elementId, duplicates[1]?.elementId);
|
||||
|
||||
const unboundedProbe = graph.elements.find((element) => element.accessibleName === "Unbounded Probe");
|
||||
assert.ok(unboundedProbe);
|
||||
assert.equal(unboundedProbe.context.form?.name, null);
|
||||
assert.equal(unboundedProbe.context.form?.text?.includes("long body of descendant text"), true);
|
||||
|
||||
const mainAction = graph.elements.find((element) => element.accessibleName === "Main action");
|
||||
assert.ok(mainAction);
|
||||
assert.equal(mainAction.context.section, null);
|
||||
assert.equal(mainAction.context.landmark?.name, "Operations");
|
||||
|
||||
const edit = graph.elements.find((element) => element.accessibleName === "Edit");
|
||||
assert.ok(edit);
|
||||
assert.equal(edit.context.row?.text.includes("Acme Co") || edit.context.row?.text.includes("Beta LLC"), true);
|
||||
const edits = graph.elements.filter((element) => element.accessibleName === "Edit");
|
||||
assert.equal(edits.length, 2);
|
||||
assert.notEqual(edits[0]?.elementId, edits[1]?.elementId);
|
||||
assert.notEqual(edits[0]?.context.row?.text, edits[1]?.context.row?.text);
|
||||
for (const item of edits) {
|
||||
const index = graph.elements.findIndex((element) => element === item);
|
||||
assert.notEqual(index, -1);
|
||||
assert.equal(
|
||||
item.elementId,
|
||||
`el_${shortHash([
|
||||
state.stateId,
|
||||
index,
|
||||
item.tag,
|
||||
item.role,
|
||||
item.attributes["data-testid"] || item.attributes["data-test-id"] || item.attributes["data-test"] || null,
|
||||
item.attributes.id || null,
|
||||
item.attributes.name || null,
|
||||
item.accessibleName,
|
||||
item.context.dialog?.name,
|
||||
item.context.form?.name,
|
||||
item.context.section?.name,
|
||||
item.context.landmark?.name,
|
||||
item.context.listitem?.name,
|
||||
null,
|
||||
])}`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test("explore records low-risk edges and keeps high-risk submit actions out of default execution", async () => {
|
||||
const graph = await withTempOutput("explore", (out) => exploreUrl({ url: fixturePath, maxActions: 8, out }));
|
||||
assert.equal(graph.edges.some((edge) => edge.riskLevel === "low" && edge.toStateId), true);
|
||||
assert.equal(hasEdgeForAccessibleName(graph, "Delete account"), false);
|
||||
assert.equal(hasEdgeForAccessibleName(graph, "Submit search"), false);
|
||||
assert.equal(graph.states.length >= 2, true);
|
||||
});
|
||||
|
||||
test("explore records edge effects and opened-by state relationships", async () => {
|
||||
const graph = await withTempOutput("explore-effects", (out) => exploreUrl({ url: fixturePath, maxActions: 12, out }));
|
||||
const entryState = graph.states[0];
|
||||
const openSettingsEdge = edgeForAccessibleName(graph, "Open settings");
|
||||
|
||||
assert.ok(entryState);
|
||||
assert.ok(openSettingsEdge);
|
||||
assert.ok(openSettingsEdge.toStateId);
|
||||
|
||||
const openedState = graph.states.find((state) => state.stateId === openSettingsEdge.toStateId);
|
||||
assert.ok(openedState);
|
||||
assert.equal(entryState.openedByEdgeId, null);
|
||||
assert.equal(openedState.openedByEdgeId, openSettingsEdge.edgeId);
|
||||
|
||||
const visibleTextEffect = openSettingsEdge.effects.find((effect) => effect.type === "visibleText");
|
||||
assert.ok(visibleTextEffect);
|
||||
assert.equal(visibleTextEffect.before, entryState.visibleTextHash);
|
||||
assert.equal(visibleTextEffect.after, openedState.visibleTextHash);
|
||||
assert.equal(openSettingsEdge.effects.some((effect) => effect.type === "actionableInventory"), true);
|
||||
assert.equal(openSettingsEdge.effects.some((effect) => effect.description === "Actionable inventory changed after action" && effect.type === "dom"), false);
|
||||
});
|
||||
|
||||
test("explore artifact preserves referential integrity and snapshot-diff effect records", async () => {
|
||||
const graph = await withTempOutput("explore-integrity", (out) => exploreUrl({ url: fixturePath, maxActions: 12, out }));
|
||||
const stateIds = new Set(graph.states.map((state) => state.stateId));
|
||||
const elementIds = new Set(graph.elements.map((element) => element.elementId));
|
||||
const edgeIds = new Set(graph.edges.map((edge) => edge.edgeId));
|
||||
const locatorIds = new Set<string>();
|
||||
const skippedActionIds = new Set(graph.skippedActions.map((skip) => skip.skippedActionId));
|
||||
const failedObservationIds = new Set(graph.failedObservations.map((failure) => failure.failedObservationId));
|
||||
const elementById = new Map(graph.elements.map((element) => [element.elementId, element]));
|
||||
const observedEffectTypes = new Set<string>();
|
||||
|
||||
assert.equal(stateIds.size, graph.states.length);
|
||||
assert.equal(elementIds.size, graph.elements.length);
|
||||
assert.equal(edgeIds.size, graph.edges.length);
|
||||
assert.equal(skippedActionIds.size, graph.skippedActions.length);
|
||||
assert.equal(failedObservationIds.size, graph.failedObservations.length);
|
||||
|
||||
for (const element of graph.elements) {
|
||||
assert.equal(stateIds.has(element.stateId), true);
|
||||
for (const locator of element.locators) {
|
||||
assert.equal(locator.elementId, element.elementId);
|
||||
assert.equal(locatorIds.has(locator.locatorId), false);
|
||||
locatorIds.add(locator.locatorId);
|
||||
}
|
||||
}
|
||||
|
||||
for (const edge of graph.edges) {
|
||||
const targetElement = elementById.get(edge.elementId);
|
||||
assert.equal(stateIds.has(edge.fromStateId), true);
|
||||
if (edge.toStateId !== null) assert.equal(stateIds.has(edge.toStateId), true);
|
||||
assert.ok(targetElement);
|
||||
assert.equal(targetElement.stateId, edge.fromStateId);
|
||||
for (const effect of edge.effects) {
|
||||
assert.equal(snapshotDiffEffectTypes.includes(effect.type), true);
|
||||
assert.equal(typeof effect.before, "string");
|
||||
assert.equal(typeof effect.after, "string");
|
||||
observedEffectTypes.add(effect.type);
|
||||
}
|
||||
}
|
||||
|
||||
assert.deepEqual([...observedEffectTypes].sort(), snapshotDiffEffectTypes);
|
||||
|
||||
for (const state of graph.states) {
|
||||
if (state.openedByEdgeId === null) continue;
|
||||
const openingEdge = graph.edges.find((edge) => edge.edgeId === state.openedByEdgeId);
|
||||
assert.ok(openingEdge);
|
||||
assert.equal(edgeIds.has(state.openedByEdgeId), true);
|
||||
assert.equal(openingEdge.toStateId, state.stateId);
|
||||
}
|
||||
|
||||
for (const skip of graph.skippedActions) {
|
||||
const element = graph.elements.find((item) => item.elementId === skip.elementId);
|
||||
assert.equal(stateIds.has(skip.stateId), true);
|
||||
assert.ok(element);
|
||||
assert.equal(skip.stateId, element.stateId);
|
||||
}
|
||||
|
||||
for (const failure of graph.failedObservations) {
|
||||
if (failure.stateId === null) continue;
|
||||
assert.equal(stateIds.has(failure.stateId), true);
|
||||
if (failure.elementId === null) continue;
|
||||
const element = graph.elements.find((item) => item.elementId === failure.elementId);
|
||||
assert.ok(element);
|
||||
assert.equal(failure.stateId, element.stateId);
|
||||
}
|
||||
});
|
||||
|
||||
test("explore replays each v0.2.1 edge from the entry state", async () => {
|
||||
const graph = await withTempOutput("explore-entry-replay", (out) => exploreUrl({ url: fixturePath, maxActions: 8, out }));
|
||||
const entryState = graph.states[0];
|
||||
|
||||
assert.ok(entryState);
|
||||
assert.equal(graph.edges.length > 0, true);
|
||||
assert.equal(graph.edges.every((edge) => edge.fromStateId === entryState.stateId), true);
|
||||
});
|
||||
|
||||
test("explore isolates replay contexts for storage-writing actions", async () => {
|
||||
const graph = await withTempOutput("explore-storage-isolation", (out) => exploreUrl({ url: fixturePath, maxActions: 8, out }));
|
||||
const entryState = graph.states[0];
|
||||
const rememberFilterEdge = edgeForAccessibleName(graph, "Remember filter");
|
||||
const leakText = "Replay storage leaked";
|
||||
|
||||
assert.ok(entryState);
|
||||
assert.ok(rememberFilterEdge);
|
||||
assert.equal(graph.edges.every((edge) => edge.fromStateId === entryState.stateId), true);
|
||||
assert.equal(
|
||||
graph.states.some(
|
||||
(state) =>
|
||||
state.title.includes(leakText) ||
|
||||
state.url.includes(leakText) ||
|
||||
state.route.includes(leakText) ||
|
||||
state.evidence.some((item) => item.description.includes(leakText)),
|
||||
),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
graph.elements.some(
|
||||
(element) =>
|
||||
element.accessibleName === leakText ||
|
||||
element.text?.includes(leakText) ||
|
||||
element.evidence.some((item) => item.description.includes(leakText)),
|
||||
),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
graph.edges.some((edge) =>
|
||||
edge.effects.some(
|
||||
(effect) =>
|
||||
effect.description.includes(leakText) || effect.before?.includes(leakText) || effect.after?.includes(leakText),
|
||||
),
|
||||
),
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test("explore records locator failures instead of edges or fake effects for unsafe low-risk candidates", async () => {
|
||||
const graph = await withTempOutput("explore-locator-failures", (out) => exploreUrl({ url: fixturePath, maxActions: 10, out }));
|
||||
const entryState = graph.states[0];
|
||||
assert.ok(entryState);
|
||||
|
||||
const duplicates = graph.elements.filter((element) => element.stateId === entryState.stateId && element.accessibleName === "Duplicate");
|
||||
const namelessButton = graph.elements.find((element) => element.stateId === entryState.stateId && element.tag === "button" && element.accessibleName === null);
|
||||
|
||||
assert.equal(duplicates.length, 2);
|
||||
assert.equal(hasEdgeForAccessibleName(graph, "Duplicate"), false);
|
||||
assert.ok(namelessButton);
|
||||
assert.equal(graph.edges.some((edge) => edge.elementId === namelessButton.elementId), false);
|
||||
for (const duplicate of duplicates) {
|
||||
assert.equal(
|
||||
graph.failedObservations.some(
|
||||
(failure) => failure.elementId === duplicate.elementId && failure.reason === "locator_not_unique",
|
||||
),
|
||||
true,
|
||||
);
|
||||
}
|
||||
assert.equal(
|
||||
graph.failedObservations.some(
|
||||
(failure) => failure.elementId === namelessButton.elementId && failure.reason === "locator_not_found",
|
||||
),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
graph.edges.some((edge) => edge.effects.some((effect) => effect.description.includes("Action failed:"))),
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test("explore records replay navigation timeout as a failed observation", async () => {
|
||||
let requestCount = 0;
|
||||
await withServer(
|
||||
(_request, response) => {
|
||||
requestCount += 1;
|
||||
if (requestCount === 1) {
|
||||
response.writeHead(200, { "content-type": "text/html; charset=utf-8" });
|
||||
response.end("<!doctype html><button>Open panel</button>");
|
||||
return;
|
||||
}
|
||||
// Leave replay navigation hanging so the engine must record a timeout.
|
||||
},
|
||||
async (url) => {
|
||||
const graph = await withTempOutput("explore-navigation-timeout", (out) =>
|
||||
exploreUrl({ url, maxActions: 1, navigationTimeoutMs: 50, out }),
|
||||
);
|
||||
const button = graph.elements.find((element) => element.accessibleName === "Open panel");
|
||||
assert.ok(button);
|
||||
assert.equal(graph.edges.length, 0);
|
||||
assert.equal(
|
||||
graph.failedObservations.some(
|
||||
(failure) => failure.elementId === button.elementId && failure.reason === "navigation_timeout",
|
||||
),
|
||||
true,
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test("explore records state-local skipped high-risk actions with evidence", async () => {
|
||||
const graph = await withTempOutput("explore-skips", (out) => exploreUrl({ url: fixturePath, maxActions: 12, out }));
|
||||
assert.equal(graph.skippedActions.some((skip) => skip.reason.includes("form_submit")), true);
|
||||
|
||||
const deleteElement = graph.elements.find((element) => element.accessibleName === "Delete account");
|
||||
assert.ok(deleteElement);
|
||||
const deleteSkip = graph.skippedActions.find((skip) => skip.elementId === deleteElement.elementId);
|
||||
assert.ok(deleteSkip);
|
||||
assert.equal(deleteSkip.stateId, deleteElement.stateId);
|
||||
assert.equal(deleteSkip.riskCategory, "destructive");
|
||||
assert.equal(deleteSkip.riskLevel, "high");
|
||||
|
||||
assert.equal(graph.skippedActions.some((skip) => skip.reason === "allowed:ui_reveal"), false);
|
||||
assert.equal(graph.failedObservations.every((failure) => failure.message.length > 0), true);
|
||||
});
|
||||
|
||||
test("disabled low-risk controls are failed observations, not risk skips", async () => {
|
||||
const graph = await withTempOutput("explore-disabled", (out) => exploreUrl({ url: fixturePath, maxActions: 8, out }));
|
||||
const disabled = graph.elements.find((element) => element.accessibleName === "Open disabled panel");
|
||||
assert.ok(disabled);
|
||||
assert.equal(graph.skippedActions.some((skip) => skip.elementId === disabled.elementId), false);
|
||||
assert.equal(
|
||||
graph.failedObservations.some(
|
||||
(failure) => failure.elementId === disabled.elementId && failure.reason === "element_disabled",
|
||||
),
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
test("explore does not record non-click elements as skipped click actions", async () => {
|
||||
const graph = await withTempOutput("explore-example-skips", (out) => exploreUrl({ url: examplePath, maxActions: 8, out }));
|
||||
|
||||
const searchInput = graph.elements.find((element) => element.accessibleName === "Search");
|
||||
assert.ok(searchInput);
|
||||
assert.equal(graph.skippedActions.some((skip) => skip.elementId === searchInput.elementId), false);
|
||||
|
||||
const settingsDialog = graph.elements.find((element) => element.accessibleName === "Settings dialog");
|
||||
assert.ok(settingsDialog);
|
||||
assert.equal(graph.skippedActions.some((skip) => skip.elementId === settingsDialog.elementId), false);
|
||||
|
||||
const deleteElement = graph.elements.find((element) => element.accessibleName === "Delete account");
|
||||
assert.ok(deleteElement);
|
||||
assert.equal(graph.skippedActions.some((skip) => skip.elementId === deleteElement.elementId && skip.riskCategory === "destructive"), true);
|
||||
});
|
||||
|
||||
test("disabled high-risk click candidates are failed observations, not risk skips", async () => {
|
||||
const graph = await withTempOutput("explore-disabled-high-risk", (out) => exploreUrl({ url: fixturePath, maxActions: 8, out }));
|
||||
const disabled = graph.elements.find((element) => element.accessibleName === "Delete disabled account");
|
||||
assert.ok(disabled);
|
||||
assert.equal(graph.skippedActions.some((skip) => skip.elementId === disabled.elementId), false);
|
||||
assert.equal(
|
||||
graph.failedObservations.some(
|
||||
(failure) => failure.elementId === disabled.elementId && failure.reason === "element_disabled",
|
||||
),
|
||||
true,
|
||||
);
|
||||
});
|
||||
205
tests/locator.test.ts
Normal file
205
tests/locator.test.ts
Normal file
@@ -0,0 +1,205 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { chromium, type Page } from "playwright";
|
||||
import { generateLocators, verifyLocators } from "../src/locator.js";
|
||||
import type { RawActionableElement } from "../src/types.js";
|
||||
|
||||
function raw(overrides: Partial<RawActionableElement>): RawActionableElement {
|
||||
return {
|
||||
uid: "button_0_Open settings",
|
||||
candidateIndex: 0,
|
||||
tag: "button",
|
||||
role: "button",
|
||||
accessibleName: "Open settings",
|
||||
identityText: "Open settings",
|
||||
text: "Open settings",
|
||||
label: null,
|
||||
placeholder: null,
|
||||
attributes: { "data-testid": "open-settings", id: "openSettings" },
|
||||
bbox: { x: 0, y: 0, width: 100, height: 30 },
|
||||
visibility: { visible: true, enabled: true, editable: false, receivesEvents: true },
|
||||
context: {
|
||||
dialog: null,
|
||||
form: null,
|
||||
section: null,
|
||||
row: null,
|
||||
landmark: null,
|
||||
listitem: null,
|
||||
},
|
||||
parameterSurface: null,
|
||||
evidence: [],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
async function withPage<T>(html: string, callback: (page: Page) => Promise<T>): Promise<T> {
|
||||
const browser = await chromium.launch();
|
||||
const page = await browser.newPage();
|
||||
try {
|
||||
await page.setContent(html);
|
||||
return await callback(page);
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
}
|
||||
|
||||
test("locator generation prioritizes user-facing Playwright locators before CSS fallbacks", () => {
|
||||
const candidates = generateLocators("el_test", raw({}));
|
||||
assert.equal(candidates[0].strategy, "role");
|
||||
assert.equal(candidates[0].framework, "playwright");
|
||||
assert.equal(candidates.some((candidate) => candidate.strategy === "testId"), true);
|
||||
assert.equal(candidates.at(-1)?.strategy, "css");
|
||||
});
|
||||
|
||||
test("locator generation keeps CSS as fallback for id and name", () => {
|
||||
const candidates = generateLocators(
|
||||
"el_input",
|
||||
raw({
|
||||
tag: "input",
|
||||
role: "textbox",
|
||||
accessibleName: "Query",
|
||||
text: null,
|
||||
label: "Query",
|
||||
placeholder: "Search customers",
|
||||
attributes: { id: "queryInput", name: "q" },
|
||||
}),
|
||||
);
|
||||
assert.equal(candidates.some((candidate) => candidate.expression.includes("#queryInput")), true);
|
||||
assert.equal(candidates.some((candidate) => candidate.expression.includes("[name=")), true);
|
||||
});
|
||||
|
||||
test("locator candidates carry context-scope evidence for duplicated actions", () => {
|
||||
const candidates = generateLocators(
|
||||
"el_edit",
|
||||
raw({
|
||||
uid: "button_4_Edit",
|
||||
tag: "button",
|
||||
role: "button",
|
||||
accessibleName: "Edit",
|
||||
text: "Edit",
|
||||
attributes: {},
|
||||
context: {
|
||||
dialog: null,
|
||||
form: null,
|
||||
section: { kind: "section", name: "Customers", text: "Customers Acme Co Active Edit" },
|
||||
row: { kind: "row", name: null, text: "Acme Co Active Edit" },
|
||||
landmark: null,
|
||||
listitem: null,
|
||||
},
|
||||
}),
|
||||
);
|
||||
assert.equal(candidates[0].scopes.some((scope) => scope.kind === "row"), true);
|
||||
});
|
||||
|
||||
test("scoped row locator resolves duplicated row actions to one target", async () => {
|
||||
await withPage(
|
||||
`
|
||||
<table>
|
||||
<tr><td>Acme Co</td><td><button>Edit</button></td></tr>
|
||||
<tr><td>Beta LLC</td><td><button>Edit</button></td></tr>
|
||||
</table>
|
||||
`,
|
||||
async (page) => {
|
||||
const rawElement = raw({
|
||||
uid: "button_1_Edit",
|
||||
candidateIndex: 1,
|
||||
tag: "button",
|
||||
role: "button",
|
||||
accessibleName: "Edit",
|
||||
identityText: "Edit",
|
||||
text: "Edit",
|
||||
attributes: {},
|
||||
context: {
|
||||
dialog: null,
|
||||
form: null,
|
||||
section: null,
|
||||
row: { kind: "row", name: null, text: "Acme Co Edit" },
|
||||
landmark: null,
|
||||
listitem: null,
|
||||
},
|
||||
});
|
||||
const scoped = generateLocators("el_scoped_edit", rawElement).find((candidate) => candidate.strategy === "scoped");
|
||||
assert.ok(scoped);
|
||||
|
||||
const [verified] = await verifyLocators(page, rawElement, [scoped]);
|
||||
|
||||
assert.equal(verified.verified, true);
|
||||
assert.equal(verified.matchCount, 1);
|
||||
assert.equal(verified.identity?.sameAccessibleName, true);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test("placeholder-only input locator verifies identity from raw placeholder evidence", async () => {
|
||||
await withPage(`<input placeholder="Search customers" />`, async (page) => {
|
||||
const rawElement = raw({
|
||||
uid: "input_0_Search customers",
|
||||
tag: "input",
|
||||
role: "textbox",
|
||||
accessibleName: "Search customers",
|
||||
identityText: "Search customers",
|
||||
text: null,
|
||||
label: null,
|
||||
placeholder: "Search customers",
|
||||
attributes: {},
|
||||
visibility: { visible: true, enabled: true, editable: true, receivesEvents: true },
|
||||
});
|
||||
const placeholder = generateLocators("el_placeholder", rawElement).find((candidate) => candidate.strategy === "placeholder");
|
||||
assert.ok(placeholder);
|
||||
|
||||
const [verified] = await verifyLocators(page, rawElement, [placeholder]);
|
||||
|
||||
assert.equal(verified.verified, true);
|
||||
assert.equal(verified.matchCount, 1);
|
||||
assert.equal(verified.identity?.sameAccessibleName, true);
|
||||
});
|
||||
});
|
||||
|
||||
test("label-derived input locator verifies identity from raw label evidence", async () => {
|
||||
await withPage(`<label for="query">Query</label><input id="query" />`, async (page) => {
|
||||
const rawElement = raw({
|
||||
uid: "input_0_Query",
|
||||
tag: "input",
|
||||
role: "textbox",
|
||||
accessibleName: "Query",
|
||||
identityText: "Query",
|
||||
text: null,
|
||||
label: "Query",
|
||||
placeholder: null,
|
||||
attributes: { id: "query" },
|
||||
visibility: { visible: true, enabled: true, editable: true, receivesEvents: true },
|
||||
});
|
||||
const label = generateLocators("el_label", rawElement).find((candidate) => candidate.strategy === "label");
|
||||
assert.ok(label);
|
||||
|
||||
const [verified] = await verifyLocators(page, rawElement, [label]);
|
||||
|
||||
assert.equal(verified.verified, true);
|
||||
assert.equal(verified.matchCount, 1);
|
||||
assert.equal(verified.identity?.sameAccessibleName, true);
|
||||
});
|
||||
});
|
||||
|
||||
test("duplicate Edit locators are unverified and omit non-unique identity evidence", async () => {
|
||||
await withPage(`<button>Edit</button><button>Edit</button>`, async (page) => {
|
||||
const rawElement = raw({
|
||||
uid: "button_0_Edit",
|
||||
tag: "button",
|
||||
role: "button",
|
||||
accessibleName: "Edit",
|
||||
identityText: "Edit",
|
||||
text: "Edit",
|
||||
attributes: {},
|
||||
});
|
||||
const duplicateLocators = generateLocators("el_edit", rawElement).filter((candidate) => ["role", "text"].includes(candidate.strategy));
|
||||
assert.equal(duplicateLocators.length, 2);
|
||||
|
||||
const verified = await verifyLocators(page, rawElement, duplicateLocators);
|
||||
|
||||
for (const locator of verified) {
|
||||
assert.equal(locator.verified, false);
|
||||
assert.equal(locator.matchCount, 2);
|
||||
assert.equal(locator.identity, null);
|
||||
}
|
||||
});
|
||||
});
|
||||
60
tests/metadata.test.ts
Normal file
60
tests/metadata.test.ts
Normal file
@@ -0,0 +1,60 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { buildEmptyGraph } from "../src/metadata.js";
|
||||
|
||||
const graphArgs = {
|
||||
inputUrl: "./examples/simple.html",
|
||||
entryUrl: "file:///examples/simple.html",
|
||||
mode: "scan" as const,
|
||||
};
|
||||
|
||||
test("input fingerprint is stable for the same input and mode", () => {
|
||||
const first = buildEmptyGraph(graphArgs);
|
||||
const second = buildEmptyGraph(graphArgs);
|
||||
|
||||
assert.equal(first.metadata.inputFingerprint, second.metadata.inputFingerprint);
|
||||
});
|
||||
|
||||
test("crawl session id is unique for each generated graph", () => {
|
||||
const first = withFixedDate(() => buildEmptyGraph(graphArgs));
|
||||
const second = withFixedDate(() => buildEmptyGraph(graphArgs));
|
||||
|
||||
assert.notEqual(first.metadata.crawlSessionId, second.metadata.crawlSessionId);
|
||||
});
|
||||
|
||||
test("empty graph starts with a machine-readable summary", () => {
|
||||
const graph = buildEmptyGraph(graphArgs);
|
||||
|
||||
assert.deepEqual(graph.summary, {
|
||||
states: 0,
|
||||
elements: 0,
|
||||
edges: 0,
|
||||
skippedActions: 0,
|
||||
failedObservations: 0,
|
||||
failedObservationsByReason: {},
|
||||
skippedActionsByRiskCategory: {},
|
||||
edgesByRiskCategory: {},
|
||||
});
|
||||
});
|
||||
|
||||
function withFixedDate<T>(callback: () => T): T {
|
||||
const RealDate = Date;
|
||||
const fixedTime = new RealDate("2026-06-16T00:00:00.000Z").getTime();
|
||||
|
||||
class FixedDate extends RealDate {
|
||||
constructor(value?: string | number | Date) {
|
||||
super(value ?? fixedTime);
|
||||
}
|
||||
|
||||
static now(): number {
|
||||
return fixedTime;
|
||||
}
|
||||
}
|
||||
|
||||
globalThis.Date = FixedDate as DateConstructor;
|
||||
try {
|
||||
return callback();
|
||||
} finally {
|
||||
globalThis.Date = RealDate;
|
||||
}
|
||||
}
|
||||
77
tests/risk.test.ts
Normal file
77
tests/risk.test.ts
Normal file
@@ -0,0 +1,77 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { classifyActionRisk, classifyClickRisk, defaultExploreDecision, isLowRiskClickable } from "../src/risk.js";
|
||||
import type { ActionableElement } from "../src/types.js";
|
||||
|
||||
function element(overrides: Partial<ActionableElement>): ActionableElement {
|
||||
return {
|
||||
elementId: "el_test",
|
||||
stateId: "state_test",
|
||||
tag: "button",
|
||||
role: "button",
|
||||
accessibleName: "Open settings",
|
||||
text: "Open settings",
|
||||
label: null,
|
||||
placeholder: null,
|
||||
attributes: {},
|
||||
bbox: { x: 0, y: 0, width: 100, height: 30 },
|
||||
framePath: [],
|
||||
shadowPath: [],
|
||||
interactability: { visible: true, enabled: true, editable: false, receivesEvents: true },
|
||||
locators: [],
|
||||
context: {
|
||||
dialog: null,
|
||||
form: null,
|
||||
section: null,
|
||||
row: null,
|
||||
landmark: null,
|
||||
listitem: null,
|
||||
},
|
||||
parameterSurface: null,
|
||||
evidence: [],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
test("risk classifier blocks destructive and submission terms", () => {
|
||||
assert.equal(classifyClickRisk(element({ accessibleName: "Delete account" })), "high");
|
||||
assert.equal(classifyClickRisk(element({ accessibleName: "提交订单" })), "high");
|
||||
assert.equal(classifyClickRisk(element({ attributes: { type: "submit" }, text: "Submit search" })), "high");
|
||||
});
|
||||
|
||||
test("risk classifier treats editable controls as medium", () => {
|
||||
assert.equal(classifyClickRisk(element({ tag: "input", role: "textbox", accessibleName: "Query" })), "medium");
|
||||
assert.equal(classifyClickRisk(element({ tag: "select", role: "combobox", accessibleName: "Segment" })), "medium");
|
||||
});
|
||||
|
||||
test("default explorer only clicks low-risk visible clickable elements", () => {
|
||||
assert.equal(isLowRiskClickable(element({ accessibleName: "Open settings" })), true);
|
||||
assert.equal(isLowRiskClickable(element({ accessibleName: "Delete account" })), false);
|
||||
assert.equal(isLowRiskClickable(element({ interactability: { visible: true, enabled: false, editable: false, receivesEvents: true } })), false);
|
||||
});
|
||||
|
||||
test("risk policy exposes category, level, and reasons", () => {
|
||||
const decision = defaultExploreDecision(element({ accessibleName: "Authorize payment" }), "click");
|
||||
assert.equal(decision.allowed, false);
|
||||
assert.equal(decision.risk.category, "payment_sensitive");
|
||||
assert.equal(decision.risk.level, "high");
|
||||
assert.equal(decision.reason.includes("payment_sensitive"), true);
|
||||
});
|
||||
|
||||
test("risk policy treats UI reveal as default explorable", () => {
|
||||
const risk = classifyActionRisk(element({ accessibleName: "Open settings" }), "click");
|
||||
assert.equal(risk.category, "ui_reveal");
|
||||
assert.equal(defaultExploreDecision(element({ accessibleName: "Open settings" }), "click").allowed, true);
|
||||
});
|
||||
|
||||
test("risk policy blocks spaced auth and payment phrases", () => {
|
||||
const logOut = defaultExploreDecision(element({ accessibleName: "Log out" }), "click");
|
||||
assert.equal(logOut.allowed, false);
|
||||
assert.equal(logOut.risk.category, "auth_sensitive");
|
||||
assert.equal(logOut.risk.level, "high");
|
||||
|
||||
const checkOut = defaultExploreDecision(element({ accessibleName: "Check out" }), "click");
|
||||
assert.equal(checkOut.allowed, false);
|
||||
assert.equal(checkOut.risk.category, "payment_sensitive");
|
||||
assert.equal(checkOut.risk.level, "high");
|
||||
});
|
||||
21
tests/schema.test.ts
Normal file
21
tests/schema.test.ts
Normal file
@@ -0,0 +1,21 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { test } from "node:test";
|
||||
import Ajv from "ajv";
|
||||
import { scanUrl } from "../src/engine.js";
|
||||
|
||||
test("topology graph JSON schema validates emitted scan artifacts", async () => {
|
||||
const schema = JSON.parse(await readFile("schemas/topology-graph.schema.json", "utf8"));
|
||||
const ajv = new Ajv({ allErrors: true });
|
||||
const validate = ajv.compile(schema);
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "action-topology-schema-test-"));
|
||||
try {
|
||||
const graph = await scanUrl({ url: "examples/simple.html", out: path.join(dir, "graph.json") });
|
||||
|
||||
assert.equal(validate(graph), true, JSON.stringify(validate.errors, null, 2));
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user