diff --git a/docs/design/action-topology-engine-v0.2.md b/docs/design/action-topology-engine-v0.2.md
new file mode 100644
index 0000000..4bf8cb5
--- /dev/null
+++ b/docs/design/action-topology-engine-v0.2.md
@@ -0,0 +1,225 @@
+# 网站可操作拓扑引擎设计文档 v0.2
+
+## 1. 项目定位
+
+本项目是一个确定性的 website action topology engine。它的职责不是自动完成任务，也不是理解自然语言指令，而是为一个网站建立可被机器消费的“可操作拓扑图”。
+
+它输出的是底层地图：页面有哪些状态，状态中有哪些可操作项，这些可操作项如何稳定定位，执行动作后产生什么可观察变化。
+
+## 2. 明确边界
+
+本项目只负责拓扑图。
+
+不负责 rrweb 录制，不负责将 rrweb 对齐到拓扑，不负责生成浏览器自动化脚本，不负责生成 skill，不负责 LLM 规划，不负责通用浏览器 Agent。
+
+这些都可以是下游消费者，但不能反向污染本项目的模型边界。
+
+## 3. 核心世界观
+
+网站不是一组 DOM 节点，而是一组可操作状态。
+
+基本认知单元是：
+
+```text
+State -- Action(Actionable Element) --> State / Observable Effects
+```
+
+真正有价值的不是“页面上有一个按钮”，而是“在某个状态下，有一个可定位、可验证、可执行的动作；执行后，浏览器世界发生了某些可观察变化”。
+
+### 3.1 确定性的含义
+
+本项目所谓确定性，不是指同一个网站每次探索都会得到完全相同的图。真实网站会受到账号、权限、数据、时间、A/B 实验、feature flag、语言、viewport、网络和服务端状态影响。
+
+这里的确定性指：拓扑事实的来源、生成规则和验证方式是可复验的。引擎只依据可观察、可记录、可复验的浏览器事实生成拓扑，不依赖不可验证的推测。
+
+因此，每份拓扑 artifact 都必须带有生成上下文：入口 URL、账号角色、环境、viewport、locale、浏览器类型、登录状态、采集时间、站点版本或数据特征。同一网站在不同上下文下可以生成不同拓扑，引擎不强行合并。
+
+### 3.2 网站状态不是页面截图
+
+State 不是截图，也不是完整 DOM 快照。State 是在特定环境上下文下，页面在某一时刻呈现出的可操作上下文。
+
+拓扑关心的是：哪些动作可见、可定位、可执行；执行后世界如何变化。像素差异、随机节点、埋点、广告、时间戳，不应轻易制造新状态。
+
+## 4. 设计目标
+
+拓扑图面向机器消费，目标是稳定、可复验、可解释。
+
+它应该让下游系统判断：当前状态是否已知，某动作是否存在，目标元素如何定位，动作前需要什么状态，动作后应该观察什么结果，动作风险如何，失败时原因是什么。
+
+## 5. 核心概念
+
+`State`：页面的可操作上下文。URL 只是其中一部分；弹窗、菜单、抽屉、tab、筛选条件、frame、shadow DOM、可见文本、可操作项集合都会影响状态。
+
+`Actionable Element`：某个状态下可由浏览器执行动作的对象。它不能脱离状态全局存在。同样一个“保存”按钮，在不同弹窗或表格行中是不同元素。
+
+`Locator Candidate`：定位该元素的一组候选方法及其证据，不是单个 selector，也不是对 DOM 结构的永久承诺。
+
+`Action Edge`：从一个状态经过一个动作到另一个状态的边。边的价值在于记录动作和结果，而不是只记录“点击过”。
+
+`Effect`：动作后的可观察变化，包括 URL、DOM、可见文本、网络请求、弹窗、下载、storage、history、focus、validation 等。没有明显视觉变化，也可能有真实 effect。
+
+`Parameter Surface`：动作或元素可接受的参数形态。拓扑引擎不负责生成业务参数，但应记录输入槽位、约束和来源证据。
+
+`Risk Classification`：对动作探索风险的保守分类。它不是业务授权，也不是最终执行许可，只用于指导拓扑探索和下游消费时的风险处理。
+
+`Evidence Level`：拓扑字段的证据等级，区分直接观察、规则推导、复验结论、启发式猜测和外部标注。
+
+### 5.1 事实与证据等级
+
+拓扑 artifact 中的信息必须区分证据等级。
+
+`Observed Fact` 是直接来自浏览器的事实，例如 URL、DOM 属性、AX role、network request、bounding box、visibility、enabled 状态。
+
+`Derived Fact` 是由确定性规则推导出的信息，例如状态签名、元素上下文、locator 候选评分、重复元素分组。
+
+`Verified Claim` 是经过重新加载、重新定位、重新执行或效果复验后确认的信息。
+
+`Heuristic Guess` 是启发式判断，例如某动作可能高风险、某输入框可能是客户编号。
+
+`External Annotation` 是人工或下游系统补充的信息，不属于拓扑引擎原生事实。
+
+拓扑引擎可以存储启发式信息，但不能把启发式信息伪装成已验证事实。
+
+### 5.2 Risk Classification
+
+风险分类回答的是：这个动作是否适合默认探索，是否可能造成副作用，是否需要人工确认或沙盒环境，是否应该只记录不执行。
+
+建议风险类型包括：`read_only`、`navigation`、`ui_reveal`、`input_edit`、`form_submit`、`data_mutation`、`destructive`、`auth_sensitive`、`payment_sensitive`、`unknown`。
+
+`read_only`、`navigation`、`ui_reveal` 通常可作为默认探索候选。`input_edit` 可以记录但默认不提交。`form_submit`、`data_mutation` 默认不探索。`destructive`、`auth_sensitive`、`payment_sensitive` 默认只记录不执行。`unknown` 按高风险处理。
+
+### 5.3 Parameter Surface
+
+很多动作不是简单 click，而是填写、选择、上传、筛选、提交、选择表格行或对业务对象执行动作。
+
+拓扑不需要知道“应该填什么”，但需要知道参数面：输入框的 type、name、label、placeholder、required、pattern、maxlength、validation；选择控件的 options、selected value、disabled options；日期控件的格式、范围和边界；表单的字段、必填项、提交动作、校验行为和网络 body 线索。
+
+它的目标是让下游知道：这个动作需要哪些输入，哪些是必填，约束是什么，输入和后续 effect 是否有关联。
+
+## 6. 状态建模原则
+
+状态必须从机器能复验的事实出发，而不是从页面名称出发。
+
+同一个 URL 可以有多个状态，不同 URL 也可能落到相似状态。状态判断不能只靠路由，必须结合可见内容、可操作项签名、模态层、frame/shadow 边界和关键上下文。
+
+### 6.1 State Lifecycle
+
+State 在拓扑中应有生命周期。
+
+`observed` 表示首次观察到。`fingerprinted` 表示已生成状态签名。`verified` 表示经过重新进入或复验，关键可操作项仍成立。`stale` 表示曾经有效但近期复验失败或证据过期。`deprecated` 表示确认不再可达或被新状态替代。`conflicted` 表示不同页面上下文被错误归并，或同一签名下出现不兼容事实。
+
+下游应优先消费 verified 状态，对 observed 或 stale 状态保持谨慎。
+
+### 6.2 状态等价与归并
+
+状态归并应服务动作判断，而不是追求像素级一致。
+
+时间戳、随机 id、埋点节点、广告、非目标列表数据轻微变化、loading 到 loaded 的瞬态变化，通常不应单独构成新状态。
+
+可操作项集合变化、modal/dialog/drawer/menu 打开或关闭、tab/stepper 改变、form schema 改变、frame 上下文改变、权限导致动作可见性改变、关键筛选条件改变、目标业务对象上下文改变，通常应构成新状态。
+
+状态签名应优先基于 URL canonical form、可见可操作项签名、关键 heading/landmark、modal stack、selected tab、form signature、frame path、shadow boundary 和业务对象上下文，而不是完整 DOM hash。
+
+## 7. 元素建模原则
+
+元素应按用户感知和浏览器语义识别，而不是按 DOM 结构识别。
+
+重要证据包括角色、可访问名称、标签、占位符、可见文本、ARIA 信息、输入类型、是否可见、是否启用、是否接收事件、所在上下文、附近标题、所属表格行、所属弹窗或表单。
+
+框架特征只作为线索，不作为地基。React、Vue、Angular、AntD、MUI 等只说明页面可能有动态 id、portal、虚拟列表、隐藏副本等风险。
+
+## 8. 定位哲学
+
+定位器以可复现为目标，而不是以当下能找到为目标。
+
+优先级应是用户视角和显式契约：role、label、placeholder、text、test id、稳定业务属性。CSS 和 XPath 只能作为 fallback。
+
+locator 是否可信，不能只看它是否匹配一个节点，还要看它是否匹配正确语义，是否重载后仍成立，是否命中隐藏副本，是否依赖脆弱结构，是否需要上下文消歧。
+
+## 9. 上下文消歧
+
+真实页面里“编辑”“查看”“删除”“保存”“确定”会大量重复。拓扑必须记录弹窗标题、表单标题、section heading、表格行文本、卡片标题、导航区域、列表项、父级 landmark。
+
+机器需要的是“客户 A 这一行的编辑按钮”，不是“第 3 个编辑按钮”。
+
+### 9.1 重复结构与动作模板
+
+重复元素不应全部平铺为互不相关的独立元素。
+
+表格每行的“编辑”、订单卡片的“查看详情”、文件列表的“下载”，应尽量表达为 row/item/card action template。
+
+模板记录重复容器类型、候选 identity 字段、行或卡片上下文、动作元素相对位置、动作语义、风险等级和 locator 消歧方式。下游执行时，应先定位目标 row/card，再在该上下文内定位动作。
+
+## 10. 动作边原则
+
+动作边表达一个可验证的行为关系。
+
+它应说明：从哪个状态出发，作用于哪个元素，动作类型是什么，是否需要参数，风险如何，动作后出现什么可观察 effect，结果状态是否可识别。
+
+### 10.1 Handler Trace 不是核心事实
+
+如果运行环境允许，引擎可以记录事件监听器、调用栈、source map 映射和框架组件线索。
+
+但 Handler Trace 只能作为辅助证据，不应成为动作边成立的前提。动作边的核心事实来自动作前状态、目标元素、动作类型、可观察 effect、结果状态、定位复验和风险分类。
+
+### 10.2 失败观察
+
+失败不是噪声。失败应记录为 Failed Observation，而不是伪装成成功动作边。
+
+典型失败包括：`locator_not_found`、`locator_not_unique`、`element_hidden`、`element_disabled`、`element_obscured`、`state_precondition_missing`、`frame_inaccessible`、`shadow_root_closed`、`navigation_timeout`、`network_error`、`permission_denied`、`auth_required`、`captcha_or_challenge`、`risk_policy_blocked`、`effect_not_observed`、`unexpected_state`。
+
+失败观察要帮助下游判断：是 locator 坏了，状态没到，页面变化了，权限不够，风险策略拦截，还是目标本身不存在。
+
+## 11. 探索策略原则
+
+默认探索必须保守。少而准，比广而脆更符合本项目价值。
+
+低风险动作可以探索；提交、确认、支付、授权、删除、注销等动作应记录为高风险，不默认执行。探索目标不是点遍所有东西，而是建立可靠、可复验的拓扑边。
+
+## 12. 输出契约
+
+拓扑 artifact 应该是机器可读、稳定演进、可版本化的事实集合。
+
+它不包含下游意图，不描述“如何完成某个业务任务”，不内置 skill 语义。它只提供事实：状态、元素、定位候选、动作边、效果、风险、失败原因和证据等级。
+
+### 12.1 Artifact 版本化
+
+每次生成或更新拓扑，应记录 artifact version、schema version、site identifier、base URL、crawl session id、browser engine、viewport、locale、auth context、created time、input seed、risk policy version、normalization rule version、locator scoring version。
+
+拓扑更新不应简单覆盖旧结果。系统应支持比较状态差异、locator 健康度变化、新增或消失动作、风险等级变化、effect 变化，并能标记 stale 或 deprecated 状态。
+
+### 12.2 拓扑事实不等于任务语义
+
+拓扑引擎可以记录某按钮文本是“导出”，某请求路径包含 `/export`，某动作产生下载事件。
+
+但它不应在内部上升为“完成导出销售报表任务”的 skill。
+
+拓扑图描述的是：在什么状态下，什么元素可被定位，可以执行什么动作，动作后出现什么可观察效果，这个动作有什么风险。
+
+它不描述用户真正想完成什么业务目标，不选择业务路径，不决定业务参数，不包装 skill，不根据自然语言规划动作序列。
+
+## 13. 成功标准
+
+好的拓扑图应该满足：状态不是靠 URL 粗暴判断；元素绑定到具体状态；locator 有候选和可信理由；重复元素可以通过上下文区分；动作边有可观察 effect；高风险动作不被默认执行；frame/shadow/modal 边界不被忽略；失败原因能被机器理解；下游系统不需要重新猜测页面结构。
+
+### 13.1 质量指标
+
+拓扑质量应能被度量。
+
+核心指标包括：`state_match_accuracy`、`locator_unique_rate`、`locator_stability_rate`、`action_effect_observed_rate`、`duplicate_disambiguation_success_rate`、`risk_false_negative_rate`、`risk_skip_rate`、`failed_observation_explainability_rate`、`stale_detection_rate`、`downstream_reguess_rate`。
+
+其中最重要的反向指标是 `downstream_reguess_rate`。如果下游仍要重新猜页面结构，说明拓扑没有履行职责。
+
+## 14. 设计底线
+
+本项目必须保持底层拓扑引擎的克制。
+
+一旦它开始理解用户任务、生成脚本、包装 skill、绕过风控、或者做智能规划，它就越界了。
+
+它应该像地图测绘，不像司机；像浏览器行为地形图，不像自动驾驶系统。
+
+## 15. 一句话总结
+
+本项目要建立的是：**网站可操作世界的确定性拓扑地图**。
+
+它记录浏览器事实，抽象页面状态，识别可操作项，生成可验证定位候选，观察动作效果，并把这些组织成下游机器可以信任的图。
diff --git a/docs/source/2026-06-16-browser-automation-crawling-research.md b/docs/source/2026-06-16-browser-automation-crawling-research.md
new file mode 100644
index 0000000..4f1f825
--- /dev/null
+++ b/docs/source/2026-06-16-browser-automation-crawling-research.md
@@ -0,0 +1,479 @@
+# Browser Automation and Crawling Research Notes
+
+- Date: 2026-06-16
+- Purpose: early research material for this repository's deterministic action topology engine.
+- Repository boundary: this document is about what to observe, verify, and record. It is not a proposal to add LLM planning, generic task execution, or a full browser-agent product.
+
+## Executive Summary
+
+Browser automation and web crawling fail less because an individual selector is "wrong" and more because the system did not record enough context: page state, frame/shadow boundary, accessibility semantics, timing, network effects, duplicate URL policy, crawl scope, and risk. A topology engine should therefore collect evidence first and generate actions second.
+
+The strongest common pattern across Playwright, Selenium, Scrapy, and Crawlee is:
+
+1. Prefer user-facing and explicit contracts for element identity: role, label, placeholder, text, alt/title, and test IDs.
+2. Re-resolve locators at action time instead of caching DOM nodes.
+3. Treat every actionable element as state-local, not globally meaningful.
+4. Make waits signal-based, not time-based.
+5. Keep crawling bounded by robots policy, rate limits, duplicate filters, canonicalization rules, and explicit scope.
+6. Preserve diagnostic evidence: locator match counts, actionability checks, screenshots, DOM/text hashes, network events, dialogs, storage changes, and trace/HAR-like records.
+7. Treat anti-bot systems and authentication flows as compliance and authorization boundaries, not as targets for stealth bypass.
+
+For this project, the next useful slice is a richer "capture evidence layer": framework hints, context anchors, frame/shadow paths, dynamic-stability observations, locator verification details, and crawl/explore policy metadata.
+
+## Source Map
+
+Primary and high-signal sources consulted:
+
+- Playwright: [Best Practices](https://playwright.dev/docs/best-practices), [Locators](https://playwright.dev/docs/locators), [Auto-waiting and actionability](https://playwright.dev/docs/actionability), [Frames](https://playwright.dev/docs/frames), [Network](https://playwright.dev/docs/network), [Trace Viewer](https://playwright.dev/docs/trace-viewer), [Test generator](https://playwright.dev/docs/codegen), [Browser contexts](https://playwright.dev/docs/browser-contexts), [Dialogs](https://playwright.dev/docs/dialogs), [ElementHandle API](https://playwright.dev/docs/api/class-elementhandle), [FrameLocator API](https://playwright.dev/docs/api/class-framelocator).
+- Selenium: [Waiting Strategies](https://www.selenium.dev/documentation/webdriver/waits/), [Understanding Common Errors](https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/), [Locator strategies](https://www.selenium.dev/documentation/webdriver/elements/locators/), [Page Object Models](https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/).
+- Scrapy: [Broad Crawls](https://docs.scrapy.org/en/latest/topics/broad-crawls.html), [AutoThrottle](https://docs.scrapy.org/en/latest/topics/autothrottle.html), [Dynamic content](https://docs.scrapy.org/en/latest/topics/dynamic-content.html), [Requests and Responses](https://docs.scrapy.org/en/latest/topics/request-response.html), [Jobs: pausing and resuming crawls](https://docs.scrapy.org/en/latest/topics/jobs.html), [Link Extractors](https://docs.scrapy.org/en/latest/topics/link-extractors.html).
+- Crawlee: [Session management](https://crawlee.dev/python/docs/guides/session-management), [Proxy management](https://crawlee.dev/js/docs/3.14/guides/proxy-management), [PlaywrightCrawler for Python](https://crawlee.dev/python/docs/guides/playwright-crawler), [PlaywrightCrawler API for JavaScript](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler), [Crawling introduction](https://crawlee.dev/js/docs/introduction/crawling).
+- Web standards and browser platform: [RFC 9309 Robots Exclusion Protocol](https://datatracker.ietf.org/doc/html/rfc9309), [Google robots.txt guide](https://developers.google.com/search/docs/crawling-indexing/robots/intro), [Google sitemaps overview](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview), [Google canonical URL guidance](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls), [MDN ShadowRoot mode](https://developer.mozilla.org/en-US/docs/Web/API/ShadowRoot/mode), [MDN iframe element](https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/iframe), [MDN Same-origin policy](https://developer.mozilla.org/en-US/docs/Web/Security/Defenses/Same-origin_policy), [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/), [Chrome DevTools accessibility reference](https://developer.chrome.com/docs/devtools/accessibility/reference).
+- Bot and abuse context: [Cloudflare bot detection engines](https://developers.cloudflare.com/bots/concepts/bot-detection-engines/), [OWASP Automated Threats to Web Applications](https://owasp.org/www-project-automated-threats-to-web-applications/).
+
+## Beginner Pitfalls
+
+### Locator Pitfalls
+
+1. Using long CSS/XPath paths as the primary locator.
+   Playwright explicitly recommends user-facing locators and says CSS/XPath tend to be less resilient when DOM structure changes. For this project, CSS should remain a fallback candidate, not the source of truth.
+
+2. Treating `id` as automatically stable.
+   Many frameworks and component libraries generate IDs for hydration, ARIA wiring, or internal control relationships. An ID that looks unique in one run can change across reloads, locales, builds, or user sessions.
+
+3. Ignoring accessible names.
+   Role alone is rarely enough. `button` plus accessible name is much stronger than `button`. Accessible names may come from visible text, `aria-label`, `aria-labelledby`, `title`, `alt`, or associated labels.
+
+4. Matching text too broadly.
+   Text can appear in multiple places: hidden mobile menus, desktop menus, offscreen carousel slides, modal templates, body text, and cloned rows. Locator strictness failures are often a sign that context anchors are missing.
+
+5. Assuming generated code is final.
+   Playwright codegen is useful because it prioritizes role, text, and test ID locators, but generated locators still need validation against repeated components, hidden duplicates, localization, and dynamic state.
+
+6. Using coordinates as if they are locators.
+   Coordinates are viewport-, zoom-, device-, scroll-, and layout-dependent. They are useful only as last-resort diagnostic evidence or replay hints.
+
+7. Reusing DOM element handles too long.
+   Selenium's stale element errors and Playwright's discouraged `ElementHandle` usage both point to the same rule: do not cache a DOM node and expect it to survive rerendering.
+
+### Timing and Actionability Pitfalls
+
+1. Using hard sleeps.
+   Playwright discourages production `waitForTimeout` because time-based waits are flaky. Selenium similarly emphasizes explicit waits for concrete conditions.
+
+2. Waiting for the wrong signal.
+   `networkidle` is not always the same as "business state ready". Modern apps may keep polling, use websockets, lazy-load sections, or rerender after hydration.
+
+3. Bypassing actionability with `force`.
+   Playwright's `force` disables non-essential actionability checks, including whether an element receives events. This can hide the real problem: overlay, animation, disabled state, wrong target, or stale state.
+
+4. Confusing visible with clickable.
+   An element can be visible yet covered by a transparent overlay, animating, disabled, outside the viewport, or not receiving pointer events.
+
+5. Treating auto-wait as complete synchronization.
+   Auto-wait helps with actionability, not with every application-level condition. A locator may be actionable before data has finished loading or before a dependent request has completed.
+
+### State and DOM Lifecycle Pitfalls
+
+1. Assuming the same URL means the same state.
+   SPAs often change modals, tabs, accordions, filters, drawers, and virtualized content without changing URL.
+
+2. Assuming the same DOM node means the same semantic element.
+   Virtual lists reuse DOM nodes for different rows. A table row's DOM position can remain while its business record changes.
+
+3. Ignoring modal and overlay stacks.
+   Many actions target a menu item or dialog button that only exists after an opening action. The precondition path matters as much as the target locator.
+
+4. Missing portal-rendered components.
+   React, MUI, Ant Design, Angular Material, and similar libraries often render dropdowns, tooltips, dialogs, and menus under `body`, outside the triggering component subtree.
+
+5. Ignoring viewport variants.
+   Desktop and mobile markup can coexist, with one branch hidden. A locator that is unique at 1440px may not be unique at 390px.
+
+### Frame, Shadow DOM, and Browser Boundary Pitfalls
+
+1. Looking only in the main document.
+   Payment widgets, embedded apps, captchas, login flows, rich editors, maps, and support widgets often live in iframes. Playwright requires frame-aware location through `frameLocator` or frame APIs.
+
+2. Assuming shadow DOM is transparent.
+   Playwright locators can pierce open shadow roots, but XPath does not pierce shadow roots and closed shadow roots are not accessible in the same way. The engine must record the shadow boundary.
+
+3. Confusing iframe DOM access with browser security policy.
+   Same-origin policy restricts cross-origin document/script interaction. The engine should record iframe origin and accessibility instead of assuming every frame can be inspected.
+
+4. Forgetting dialogs, popups, downloads, and beforeunload prompts.
+   Browser-level events are not normal DOM changes. They must be captured as effects or risk blockers.
+
+### Crawling Pitfalls
+
+1. Crawling without scope.
+   Broad crawls need allowed domains, depth limits, URL filters, and maximum queue sizes. Otherwise calendars, faceted search, tracking parameters, and sort/filter combinations create infinite URL spaces.
+
+2. Misusing canonicalization.
+   URL canonicalization is useful for duplicate checking, but Scrapy notes that canonicalized URLs can differ from the URL visible to the server. The crawler should keep raw URL, canonical key, and request fingerprint separate.
+
+3. Ignoring duplicate request policy.
+   Crawlers need request fingerprints and duplicate filters. A topology engine also needs state hashes because different URLs can lead to the same state and one URL can lead to different states.
+
+4. Treating robots.txt as security or permission to bypass.
+   RFC 9309 frames robots rules as requested crawler behavior, not access authorization. Google also notes robots.txt is mainly for managing crawler traffic, not hiding private content.
+
+5. Overloading sites.
+   New crawlers often set concurrency too high. Scrapy's AutoThrottle exists because responsible crawl speed depends on crawler load and target-site load.
+
+6. Assuming HTML contains the data.
+   Scrapy's dynamic-content guidance starts by finding the data source: HTML source, embedded JavaScript, XHR/fetch response, API endpoint, or browser-rendered DOM. Browser automation is expensive and should be used when raw HTTP/API reproduction is insufficient.
+
+7. Not preserving job state.
+   Large crawls need durable queues, seen fingerprints, and clean pause/resume. Abrupt shutdown can corrupt state in crawler job directories.
+
+### Anti-Bot and Compliance Pitfalls
+
+1. Treating blocks as merely technical obstacles.
+   Bot defenses classify automated traffic using heuristics, fingerprints, behavior, JavaScript challenges, request patterns, and reputation signals. For this repository, the right response is to record block states and stop or require authorization, not to build stealth bypass.
+
+2. Scraping behind login, paywalls, or private areas without authority.
+   This creates legal, contractual, and ethical risk. The engine should support storage state for authorized sessions, but it should not normalize unauthorized access.
+
+3. Ignoring personally identifiable information.
+   Extracted content can contain personal data, tokens, account identifiers, or internal URLs. Artifacts need redaction policy before being shared outside a controlled environment.
+
+4. Missing high-risk action terms.
+   Delete, submit, confirm, pay, authorize, purchase, and logout should remain gated. Chinese equivalents and product-specific destructive verbs need the same treatment.
+
+## Advanced Techniques
+
+### Locator Engineering
+
+1. Generate multiple locator candidates per element.
+   Store role/name, label, placeholder, text, test ID, stable attributes, local CSS fallback, frame path, and shadow path. Keep scores, reasons, verification results, and failure counts.
+
+2. Verify locator identity, not just uniqueness.
+   `count() === 1` is necessary but not sufficient. Also compare accessible name, tag, role, bounding box, visibility, enabled state, and a local semantic signature.
+
+3. Use context anchors for repeated components.
+   A robust locator often needs a container: dialog title, table row text, card heading, form label group, nav landmark, section heading, or list item.
+
+4. Track hidden duplicates.
+   Record how many matches are hidden, visible, disabled, or offscreen. Hidden duplicates explain strictness failures and viewport-specific bugs.
+
+5. Keep locator descriptors structured.
+   Store `{ kind: "role", role, name }`, not only string expressions. This lets consumers regenerate Playwright, Selenium, or other framework locators later.
+
+6. Treat localization as a first-class stability concern.
+   Text locators can be excellent in one locale and brittle across locales. Capture document language, locale hints, and alternate stable attributes when present.
+
+### Actionability and Replay
+
+1. Probe actionability before acting.
+   Playwright supports trial-like checks and actionability behavior through locator actions. For this project, the graph should record visible, stable, enabled, editable, receives-events, and attached-like observations where possible.
+
+2. Record precondition paths.
+   Dropdown menu items, tabs, dialogs, accordions, and nested drawers require opening actions. Store the edge path that made the element visible.
+
+3. Record browser effects as separate evidence.
+   URL changes, visible text changes, DOM hash changes, network events, downloads, dialogs, storage changes, console errors, and history changes are different effect types.
+
+4. Use isolated browser contexts.
+   Playwright browser contexts isolate cookies, local storage, IndexedDB, cache, permissions, and auth state. This matters for reproducible scans and for comparing anonymous vs authenticated topology.
+
+5. Prefer replay from the entry state.
+   To verify an edge, reopen or reset the context, execute preconditions, then re-resolve the target locator. This catches stale references and hidden state coupling.
+
+### Dynamic Content and Network-Aware Extraction
+
+1. Inspect network before rendering everything.
+   Scrapy recommends finding the data source for dynamic content. If data comes from a JSON endpoint, direct HTTP collection can be faster and more reliable than DOM scraping.
+
+2. Capture XHR/fetch/document responses during exploration.
+   Playwright network APIs can observe and modify traffic. The topology graph can record methods, URLs, statuses, content types, and request initiator context without storing sensitive payloads by default.
+
+3. Separate DOM state from API state.
+   A click can change text through local state without network, through XHR without URL change, through navigation, or through storage. These should not collapse into one "changed" flag.
+
+4. Use HAR/trace-style diagnostics for failures.
+   Playwright traces help debug failed runs after the fact. This project does not need to emit full Playwright traces in v1, but should keep enough small evidence to explain failures.
+
+### Crawling Architecture
+
+1. Maintain a durable request queue.
+   Crawlee and Scrapy both emphasize queues/request providers for recursive crawling. A topology engine can stay small while still recording queue metadata for reproducibility.
+
+2. Use request fingerprints and state fingerprints separately.
+   Request fingerprint: dedupe crawl work. State fingerprint: dedupe observed browser states. Element signature: dedupe actionable inventory within a state.
+
+3. Respect robots and crawl-delay-like policies where applicable.
+   Robots rules are not an authorization layer, but they are the standard signal for crawler politeness. Store robots outcome and policy decisions in metadata.
+
+4. Use adaptive throttling.
+   Fixed concurrency is fragile. Scrapy AutoThrottle adjusts crawl speed based on load. For this repository, even a simpler per-host delay and max-concurrency policy would prevent many early mistakes.
+
+5. Detect crawler traps.
+   Watch for URL patterns that expand indefinitely: calendars, repeated path segments, session IDs, sort/filter permutations, page numbers with no terminal page, and tracking parameters.
+
+6. Use sitemaps as seed hints, not truth.
+   Sitemaps tell crawlers which pages the site considers important, but they do not guarantee completeness or validity. Store sitemap origin if used.
+
+### Observability
+
+1. Store why an element was considered actionable.
+   Include selector source, role inference, event handler hints, tabindex, href/form attributes, ARIA states, contenteditable, and hit-test result.
+
+2. Store why an action was skipped.
+   Risk term, hidden, disabled, blocked by overlay, no verified locator, duplicate locator, outside scope, robots policy, auth required, or anti-bot block.
+
+3. Keep compact artifacts, not raw everything.
+   Full DOM and screenshots can be large and sensitive. Hashes plus selected evidence are usually enough for topology, with optional debug artifacts in controlled runs.
+
+4. Make every score explainable.
+   Locator scores should be inspectable: "role plus accessible name", "explicit test id", "CSS id fallback", "matched 3 elements", "visible duplicate exists".
+
+## Framework and Component-Library Notes
+
+The engine should avoid framework-specific hard dependencies, but it should collect framework hints that help rank locators and explain instability.
+
+### React and Next.js
+
+Common observations:
+
+- Hydration can replace or modify nodes after initial HTML appears.
+- Portals render menus/dialogs outside the component subtree.
+- Component libraries may generate dynamic IDs for ARIA relationships.
+- Suspense/loading states can show temporary buttons or skeleton content.
+- Virtual lists reuse row DOM nodes.
+
+Useful evidence:
+
+- React/Next markers and root containers.
+- Hydration timing and post-load mutation bursts.
+- Portal containers under `body`.
+- Stable app-level data attributes.
+- Nearby row/card/section text for repeated controls.
+
+### Vue and Nuxt
+
+Common observations:
+
+- Conditional rendering and transitions can leave entering/leaving elements in the DOM.
+- Slots make component ownership hard to infer from DOM ancestry.
+- Generated attributes/classes are not always stable across builds.
+
+Useful evidence:
+
+- App root markers.
+- Visibility and transition state.
+- Context anchors from labels, headings, and list items.
+- Explicit test or QA attributes when available.
+
+### Angular and Angular Material
+
+Common observations:
+
+- Angular adds framework attributes and comment anchors that should not become primary locators.
+- Material overlays often render outside local component hierarchy.
+- Form fields may expose strong label and ARIA relationships.
+
+Useful evidence:
+
+- Overlay containers.
+- Label/control relationships.
+- `aria-controls`, `aria-expanded`, `role`, and selected/checked states.
+- Component-library class hints only as diagnostics, not primary locators.
+
+### Ant Design, MUI, Element, Naive UI, and Similar Libraries
+
+Common observations:
+
+- Dropdowns, selects, date pickers, autocomplete panels, modals, and tooltips often render in shared overlay roots.
+- Visible labels may be split across nested spans.
+- Some components expose good ARIA roles; others require text/context anchors.
+- Duplicate hidden markup is common for responsive variants and transitions.
+
+Useful evidence:
+
+- Overlay root path.
+- Trigger-to-popup relationship through `aria-controls`, `aria-haspopup`, and `aria-expanded`.
+- Option lists and enum values.
+- Open/closed state before and after click.
+
+### Web Components
+
+Common observations:
+
+- Open shadow roots can usually be traversed by Playwright locators.
+- Closed shadow roots intentionally hide internals.
+- XPath does not pierce shadow roots.
+
+Useful evidence:
+
+- Custom element tag name.
+- Shadow root mode if observable.
+- Host attributes and accessible role/name.
+- Slot text and public ARIA surface.
+
+### Server-Rendered or Traditional Sites
+
+Common observations:
+
+- HTML forms, anchors, and submit buttons expose strong native semantics.
+- CSS locators may be more stable than in SPAs, but still should be fallback.
+- Navigation effects are clearer, but forms can trigger high-risk writes.
+
+Useful evidence:
+
+- Form action/method/enctype.
+- Input name/type/required/pattern.
+- Link href and target.
+- Submit/reset button type.
+
+## Recommended Evidence to Collect
+
+### State Evidence
+
+- URL, route, title, language, viewport, user agent family.
+- DOM hash, visible text hash, actionable signature hash.
+- Optional accessibility-tree or accessibility-signature hash.
+- Modal, drawer, menu, popover, and overlay stack.
+- Frame tree: URL, name, title, origin, sandbox/allow hints where available.
+- Framework and component-library hints.
+- Network summary: document URL, XHR/fetch endpoints, status families, redirects.
+- Storage summary: cookie/localStorage/sessionStorage/IndexedDB presence counts, not raw sensitive values by default.
+- Mutation summary: whether actionable inventory changes after domcontentloaded/load/networkidle.
+
+### Element Evidence
+
+- State-local `elementId`.
+- Tag, inferred kind, role, accessible name, text, label, placeholder, title, alt.
+- Selected attributes: `id`, `name`, `type`, `href`, `value`, `for`, `role`, ARIA fields, test IDs, QA attributes, automation IDs.
+- Form context: enclosing form action/method, fieldset/legend, required/disabled/readonly/pattern/min/max.
+- Component context: nearest dialog/menu/list/table/card/form/section/nav/main, heading text, row text, column header, list item text.
+- Frame path and shadow path.
+- Bounding box, viewport relation, z-index-ish overlay observation, center hit-test result.
+- Interactability: visible, enabled, editable, checked/selected, expanded, pressed, receives events.
+- Event/action hints: href navigation, submit/reset, file upload, contenteditable, onclick, keyboard role, pointer cursor.
+- Risk terms and risk classification.
+
+### Locator Candidate Evidence
+
+- Structured descriptor and generated expression.
+- Strategy: role, label, placeholder, text, test ID, alt/title, stable attribute, local CSS, XPath, coordinate fallback.
+- Score and reason.
+- Match count.
+- Visible match count.
+- Enabled/editable match count.
+- Whether the match points to the same semantic signature.
+- Whether the locator survives reload and precondition replay.
+- Failure count and last failure reason.
+
+### Action Edge Evidence
+
+- From state and to state.
+- Target element and chosen locator.
+- Action type: click, fill, select, check, uncheck, upload, hover, keypress, navigate.
+- Parameters or parameter schema.
+- Preconditions and opening path.
+- Effects: URL, DOM hash, visible text hash, network, dialog, popup, download, storage, console error.
+- Risk decision: executed, skipped, requires approval, blocked by policy.
+- Confidence and replay verification status.
+
+### Crawl Policy Evidence
+
+- Entry URLs and seed source: explicit, sitemap, in-page links, recorded trace.
+- Allowed domains and path filters.
+- Robots fetch result and policy outcome.
+- Rate limits, concurrency, backoff, retry policy.
+- Duplicate policy: raw URL, normalized URL, canonical URL, request fingerprint.
+- Trap controls: max depth, max pages per host, query parameter allowlist/denylist, calendar/facet detection.
+- Auth mode: anonymous, supplied storage state, or manually authenticated context.
+
+## Suggested Near-Term Design Direction for This Repository
+
+### 1. Add a Capture Evidence Layer
+
+Introduce richer internal raw evidence before changing graph behavior too much. The graph can expose selected fields, while the engine keeps implementation detail behind stable types.
+
+Candidate additions:
+
+- `frameworkHints` on `StateNode` or metadata.
+- `context` on `ActionableElement`: nearest landmark, dialog, form, table row, card, section heading.
+- Real `framePath` and `shadowPath`.
+- Locator verification details beyond `matchCount`.
+- `skipReason` or exploration decision records for skipped elements.
+
+### 2. Add Locator Context Ranking
+
+Keep current priority:
+
+1. role plus accessible name
+2. label
+3. placeholder
+4. test ID or explicit automation attribute
+5. text
+6. stable local CSS
+
+Then add context-aware variants:
+
+- `dialog[name].getByRole(...)`
+- `row[hasText].getByRole(...)`
+- `form[has label/legend].getByLabel(...)`
+- `section[heading].getByRole(...)`
+- frame/shadow-aware variants
+
+### 3. Add Replay-Based Verification
+
+A locator should have at least two levels of verification:
+
+- Snapshot verification: resolves uniquely now.
+- Replay verification: after reload or precondition replay, resolves to the same semantic target.
+
+This will catch framework rerendering, transient DOM, and hidden duplicate issues much earlier.
+
+### 4. Add Conservative Crawl/Explore Policy
+
+Before expanding exploration:
+
+- Per-host max pages and max actions.
+- Explicit allowlist for domains and paths.
+- Robots policy record.
+- Query parameter normalization policy.
+- Risk-gated default action set.
+- No form submission or destructive clicks by default.
+
+### 5. Keep Anti-Bot Handling Non-Evasive
+
+For this repo, anti-bot detection should become observable state, not a bypass feature:
+
+- detect challenge/block pages,
+- record response status and visible challenge text,
+- mark edge/state as blocked or requires authorization,
+- stop exploration unless the user supplied an authorized session and policy allows it.
+
+## What Not To Build Here
+
+To preserve the repository boundary:
+
+- Do not add LLM task planning.
+- Do not add natural-language instruction execution.
+- Do not add stealth CAPTCHA or paywall bypass.
+- Do not add a generic browser-agent orchestration layer.
+- Do not turn generated topology into autonomous high-risk actions.
+- Do not store raw sensitive cookies, tokens, request bodies, or personal data in default artifacts.
+
+## Practical Checklist for Future Implementation
+
+Before a locator or action edge is considered reliable, the engine should be able to answer:
+
+1. Which page state owned this element?
+2. Which frame and shadow boundary contained it?
+3. Which user-facing semantics identified it?
+4. Which nearby context disambiguated it from similar elements?
+5. Which locator candidates were generated?
+6. Which candidates were verified, and how many elements did each match?
+7. Was the target visible, enabled, stable, editable if needed, and receiving events?
+8. Did replay from entry/preconditions find the same target?
+9. What happened after action: URL, DOM, text, network, storage, dialog, download?
+10. Was the action skipped or gated by risk, robots, scope, auth, or anti-bot policy?
+
+## Bottom Line
+
+The high-leverage research conclusion is simple: reliable browser automation is mostly evidence management. The engine should capture enough state-local, semantic, contextual, timing, and policy evidence that downstream consumers can choose actions quickly without guessing why a locator worked once.