action-topology-engine/docs/source/2026-06-16-browser-automation-crawling-research.md

Browser Automation and Crawling Research Notes

• Date: 2026-06-16
• Purpose: early research material for this repository's deterministic action topology engine.
• Repository boundary: this document is about what to observe, verify, and record. It is not a proposal to add LLM planning, generic task execution, or a full browser-agent product.

Executive Summary

Browser automation and web crawling fail less because an individual selector is "wrong" and more because the system did not record enough context: page state, frame/shadow boundary, accessibility semantics, timing, network effects, duplicate URL policy, crawl scope, and risk. A topology engine should therefore collect evidence first and generate actions second.

The strongest common pattern across Playwright, Selenium, Scrapy, and Crawlee is:

1. Prefer user-facing and explicit contracts for element identity: role, label, placeholder, text, alt/title, and test IDs.
2. Re-resolve locators at action time instead of caching DOM nodes.
3. Treat every actionable element as state-local, not globally meaningful.
4. Make waits signal-based, not time-based.
5. Keep crawling bounded by robots policy, rate limits, duplicate filters, canonicalization rules, and explicit scope.
6. Preserve diagnostic evidence: locator match counts, actionability checks, screenshots, DOM/text hashes, network events, dialogs, storage changes, and trace/HAR-like records.
7. Treat anti-bot systems and authentication flows as compliance and authorization boundaries, not as targets for stealth bypass.

For this project, the next useful slice is a richer "capture evidence layer": framework hints, context anchors, frame/shadow paths, dynamic-stability observations, locator verification details, and crawl/explore policy metadata.

Source Map

Primary and high-signal sources consulted:

• Playwright: Best Practices, Locators, Auto-waiting and actionability, Frames, Network, Trace Viewer, Test generator, Browser contexts, Dialogs, ElementHandle API, FrameLocator API.
• Selenium: Waiting Strategies, Understanding Common Errors, Locator strategies, Page Object Models.
• Scrapy: Broad Crawls, AutoThrottle, Dynamic content, Requests and Responses, Jobs: pausing and resuming crawls, Link Extractors.
• Crawlee: Session management, Proxy management, PlaywrightCrawler for Python, PlaywrightCrawler API for JavaScript, Crawling introduction.
• Web standards and browser platform: RFC 9309 Robots Exclusion Protocol, Google robots.txt guide, Google sitemaps overview, Google canonical URL guidance, MDN ShadowRoot mode, MDN iframe element, MDN Same-origin policy, Chrome DevTools Protocol, Chrome DevTools accessibility reference.
• Bot and abuse context: Cloudflare bot detection engines, OWASP Automated Threats to Web Applications.

Beginner Pitfalls

Locator Pitfalls

1. Using long CSS/XPath paths as the primary locator. Playwright explicitly recommends user-facing locators and says CSS/XPath tend to be less resilient when DOM structure changes. For this project, CSS should remain a fallback candidate, not the source of truth.

2. Treating id as automatically stable. Many frameworks and component libraries generate IDs for hydration, ARIA wiring, or internal control relationships. An ID that looks unique in one run can change across reloads, locales, builds, or user sessions.

3. Ignoring accessible names. Role alone is rarely enough. button plus accessible name is much stronger than button. Accessible names may come from visible text, aria-label, aria-labelledby, title, alt, or associated labels.

4. Matching text too broadly. Text can appear in multiple places: hidden mobile menus, desktop menus, offscreen carousel slides, modal templates, body text, and cloned rows. Locator strictness failures are often a sign that context anchors are missing.

5. Assuming generated code is final. Playwright codegen is useful because it prioritizes role, text, and test ID locators, but generated locators still need validation against repeated components, hidden duplicates, localization, and dynamic state.

6. Using coordinates as if they are locators. Coordinates are viewport-, zoom-, device-, scroll-, and layout-dependent. They are useful only as last-resort diagnostic evidence or replay hints.

7. Reusing DOM element handles too long. Selenium's stale element errors and Playwright's discouraged ElementHandle usage both point to the same rule: do not cache a DOM node and expect it to survive rerendering.

Timing and Actionability Pitfalls

1. Using hard sleeps. Playwright discourages production waitForTimeout because time-based waits are flaky. Selenium similarly emphasizes explicit waits for concrete conditions.

2. Waiting for the wrong signal. networkidle is not always the same as "business state ready". Modern apps may keep polling, use websockets, lazy-load sections, or rerender after hydration.

3. Bypassing actionability with force. Playwright's force disables non-essential actionability checks, including whether an element receives events. This can hide the real problem: overlay, animation, disabled state, wrong target, or stale state.

4. Confusing visible with clickable. An element can be visible yet covered by a transparent overlay, animating, disabled, outside the viewport, or not receiving pointer events.

5. Treating auto-wait as complete synchronization. Auto-wait helps with actionability, not with every application-level condition. A locator may be actionable before data has finished loading or before a dependent request has completed.

State and DOM Lifecycle Pitfalls

1. Assuming the same URL means the same state. SPAs often change modals, tabs, accordions, filters, drawers, and virtualized content without changing URL.

2. Assuming the same DOM node means the same semantic element. Virtual lists reuse DOM nodes for different rows. A table row's DOM position can remain while its business record changes.

3. Ignoring modal and overlay stacks. Many actions target a menu item or dialog button that only exists after an opening action. The precondition path matters as much as the target locator.

4. Missing portal-rendered components. React, MUI, Ant Design, Angular Material, and similar libraries often render dropdowns, tooltips, dialogs, and menus under body, outside the triggering component subtree.

5. Ignoring viewport variants. Desktop and mobile markup can coexist, with one branch hidden. A locator that is unique at 1440px may not be unique at 390px.

Frame, Shadow DOM, and Browser Boundary Pitfalls

1. Looking only in the main document. Payment widgets, embedded apps, captchas, login flows, rich editors, maps, and support widgets often live in iframes. Playwright requires frame-aware location through frameLocator or frame APIs.

2. Assuming shadow DOM is transparent. Playwright locators can pierce open shadow roots, but XPath does not pierce shadow roots and closed shadow roots are not accessible in the same way. The engine must record the shadow boundary.

3. Confusing iframe DOM access with browser security policy. Same-origin policy restricts cross-origin document/script interaction. The engine should record iframe origin and accessibility instead of assuming every frame can be inspected.

4. Forgetting dialogs, popups, downloads, and beforeunload prompts. Browser-level events are not normal DOM changes. They must be captured as effects or risk blockers.

Crawling Pitfalls

1. Crawling without scope. Broad crawls need allowed domains, depth limits, URL filters, and maximum queue sizes. Otherwise calendars, faceted search, tracking parameters, and sort/filter combinations create infinite URL spaces.

2. Misusing canonicalization. URL canonicalization is useful for duplicate checking, but Scrapy notes that canonicalized URLs can differ from the URL visible to the server. The crawler should keep raw URL, canonical key, and request fingerprint separate.

3. Ignoring duplicate request policy. Crawlers need request fingerprints and duplicate filters. A topology engine also needs state hashes because different URLs can lead to the same state and one URL can lead to different states.

4. Treating robots.txt as security or permission to bypass. RFC 9309 frames robots rules as requested crawler behavior, not access authorization. Google also notes robots.txt is mainly for managing crawler traffic, not hiding private content.

5. Overloading sites. New crawlers often set concurrency too high. Scrapy's AutoThrottle exists because responsible crawl speed depends on crawler load and target-site load.

6. Assuming HTML contains the data. Scrapy's dynamic-content guidance starts by finding the data source: HTML source, embedded JavaScript, XHR/fetch response, API endpoint, or browser-rendered DOM. Browser automation is expensive and should be used when raw HTTP/API reproduction is insufficient.

7. Not preserving job state. Large crawls need durable queues, seen fingerprints, and clean pause/resume. Abrupt shutdown can corrupt state in crawler job directories.

Anti-Bot and Compliance Pitfalls

1. Treating blocks as merely technical obstacles. Bot defenses classify automated traffic using heuristics, fingerprints, behavior, JavaScript challenges, request patterns, and reputation signals. For this repository, the right response is to record block states and stop or require authorization, not to build stealth bypass.

2. Scraping behind login, paywalls, or private areas without authority. This creates legal, contractual, and ethical risk. The engine should support storage state for authorized sessions, but it should not normalize unauthorized access.

3. Ignoring personally identifiable information. Extracted content can contain personal data, tokens, account identifiers, or internal URLs. Artifacts need redaction policy before being shared outside a controlled environment.

4. Missing high-risk action terms. Delete, submit, confirm, pay, authorize, purchase, and logout should remain gated. Chinese equivalents and product-specific destructive verbs need the same treatment.

Advanced Techniques

Locator Engineering

1. Generate multiple locator candidates per element. Store role/name, label, placeholder, text, test ID, stable attributes, local CSS fallback, frame path, and shadow path. Keep scores, reasons, verification results, and failure counts.

2. Verify locator identity, not just uniqueness. count() === 1 is necessary but not sufficient. Also compare accessible name, tag, role, bounding box, visibility, enabled state, and a local semantic signature.

3. Use context anchors for repeated components. A robust locator often needs a container: dialog title, table row text, card heading, form label group, nav landmark, section heading, or list item.

4. Track hidden duplicates. Record how many matches are hidden, visible, disabled, or offscreen. Hidden duplicates explain strictness failures and viewport-specific bugs.

5. Keep locator descriptors structured. Store { kind: "role", role, name }, not only string expressions. This lets consumers regenerate Playwright, Selenium, or other framework locators later.

6. Treat localization as a first-class stability concern. Text locators can be excellent in one locale and brittle across locales. Capture document language, locale hints, and alternate stable attributes when present.

Actionability and Replay

1. Probe actionability before acting. Playwright supports trial-like checks and actionability behavior through locator actions. For this project, the graph should record visible, stable, enabled, editable, receives-events, and attached-like observations where possible.

2. Record precondition paths. Dropdown menu items, tabs, dialogs, accordions, and nested drawers require opening actions. Store the edge path that made the element visible.

3. Record browser effects as separate evidence. URL changes, visible text changes, DOM hash changes, network events, downloads, dialogs, storage changes, console errors, and history changes are different effect types.

4. Use isolated browser contexts. Playwright browser contexts isolate cookies, local storage, IndexedDB, cache, permissions, and auth state. This matters for reproducible scans and for comparing anonymous vs authenticated topology.

5. Prefer replay from the entry state. To verify an edge, reopen or reset the context, execute preconditions, then re-resolve the target locator. This catches stale references and hidden state coupling.

Dynamic Content and Network-Aware Extraction

1. Inspect network before rendering everything. Scrapy recommends finding the data source for dynamic content. If data comes from a JSON endpoint, direct HTTP collection can be faster and more reliable than DOM scraping.

2. Capture XHR/fetch/document responses during exploration. Playwright network APIs can observe and modify traffic. The topology graph can record methods, URLs, statuses, content types, and request initiator context without storing sensitive payloads by default.

3. Separate DOM state from API state. A click can change text through local state without network, through XHR without URL change, through navigation, or through storage. These should not collapse into one "changed" flag.

4. Use HAR/trace-style diagnostics for failures. Playwright traces help debug failed runs after the fact. This project does not need to emit full Playwright traces in v1, but should keep enough small evidence to explain failures.

Crawling Architecture

1. Maintain a durable request queue. Crawlee and Scrapy both emphasize queues/request providers for recursive crawling. A topology engine can stay small while still recording queue metadata for reproducibility.

2. Use request fingerprints and state fingerprints separately. Request fingerprint: dedupe crawl work. State fingerprint: dedupe observed browser states. Element signature: dedupe actionable inventory within a state.

3. Respect robots and crawl-delay-like policies where applicable. Robots rules are not an authorization layer, but they are the standard signal for crawler politeness. Store robots outcome and policy decisions in metadata.

4. Use adaptive throttling. Fixed concurrency is fragile. Scrapy AutoThrottle adjusts crawl speed based on load. For this repository, even a simpler per-host delay and max-concurrency policy would prevent many early mistakes.

5. Detect crawler traps. Watch for URL patterns that expand indefinitely: calendars, repeated path segments, session IDs, sort/filter permutations, page numbers with no terminal page, and tracking parameters.

6. Use sitemaps as seed hints, not truth. Sitemaps tell crawlers which pages the site considers important, but they do not guarantee completeness or validity. Store sitemap origin if used.

Observability

1. Store why an element was considered actionable. Include selector source, role inference, event handler hints, tabindex, href/form attributes, ARIA states, contenteditable, and hit-test result.

2. Store why an action was skipped. Risk term, hidden, disabled, blocked by overlay, no verified locator, duplicate locator, outside scope, robots policy, auth required, or anti-bot block.

3. Keep compact artifacts, not raw everything. Full DOM and screenshots can be large and sensitive. Hashes plus selected evidence are usually enough for topology, with optional debug artifacts in controlled runs.

4. Make every score explainable. Locator scores should be inspectable: "role plus accessible name", "explicit test id", "CSS id fallback", "matched 3 elements", "visible duplicate exists".

Framework and Component-Library Notes

The engine should avoid framework-specific hard dependencies, but it should collect framework hints that help rank locators and explain instability.

React and Next.js

Common observations:

• Hydration can replace or modify nodes after initial HTML appears.
• Portals render menus/dialogs outside the component subtree.
• Component libraries may generate dynamic IDs for ARIA relationships.
• Suspense/loading states can show temporary buttons or skeleton content.
• Virtual lists reuse row DOM nodes.

Useful evidence:

• React/Next markers and root containers.
• Hydration timing and post-load mutation bursts.
• Portal containers under body.
• Stable app-level data attributes.
• Nearby row/card/section text for repeated controls.

Vue and Nuxt

Common observations:

• Conditional rendering and transitions can leave entering/leaving elements in the DOM.
• Slots make component ownership hard to infer from DOM ancestry.
• Generated attributes/classes are not always stable across builds.

Useful evidence:

• App root markers.
• Visibility and transition state.
• Context anchors from labels, headings, and list items.
• Explicit test or QA attributes when available.

Angular and Angular Material

Common observations:

• Angular adds framework attributes and comment anchors that should not become primary locators.
• Material overlays often render outside local component hierarchy.
• Form fields may expose strong label and ARIA relationships.

Useful evidence:

• Overlay containers.
• Label/control relationships.
• aria-controls, aria-expanded, role, and selected/checked states.
• Component-library class hints only as diagnostics, not primary locators.

Ant Design, MUI, Element, Naive UI, and Similar Libraries

Common observations:

• Dropdowns, selects, date pickers, autocomplete panels, modals, and tooltips often render in shared overlay roots.
• Visible labels may be split across nested spans.
• Some components expose good ARIA roles; others require text/context anchors.
• Duplicate hidden markup is common for responsive variants and transitions.

Useful evidence:

• Overlay root path.
• Trigger-to-popup relationship through aria-controls, aria-haspopup, and aria-expanded.
• Option lists and enum values.
• Open/closed state before and after click.

Web Components

Common observations:

• Open shadow roots can usually be traversed by Playwright locators.
• Closed shadow roots intentionally hide internals.
• XPath does not pierce shadow roots.

Useful evidence:

• Custom element tag name.
• Shadow root mode if observable.
• Host attributes and accessible role/name.
• Slot text and public ARIA surface.

Server-Rendered or Traditional Sites

Common observations:

• HTML forms, anchors, and submit buttons expose strong native semantics.
• CSS locators may be more stable than in SPAs, but still should be fallback.
• Navigation effects are clearer, but forms can trigger high-risk writes.

Useful evidence:

• Form action/method/enctype.
• Input name/type/required/pattern.
• Link href and target.
• Submit/reset button type.

Recommended Evidence to Collect

State Evidence

• URL, route, title, language, viewport, user agent family.
• DOM hash, visible text hash, actionable signature hash.
• Optional accessibility-tree or accessibility-signature hash.
• Modal, drawer, menu, popover, and overlay stack.
• Frame tree: URL, name, title, origin, sandbox/allow hints where available.
• Framework and component-library hints.
• Network summary: document URL, XHR/fetch endpoints, status families, redirects.
• Storage summary: cookie/localStorage/sessionStorage/IndexedDB presence counts, not raw sensitive values by default.
• Mutation summary: whether actionable inventory changes after domcontentloaded/load/networkidle.

Element Evidence

• State-local elementId.
• Tag, inferred kind, role, accessible name, text, label, placeholder, title, alt.
• Selected attributes: id, name, type, href, value, for, role, ARIA fields, test IDs, QA attributes, automation IDs.
• Form context: enclosing form action/method, fieldset/legend, required/disabled/readonly/pattern/min/max.
• Component context: nearest dialog/menu/list/table/card/form/section/nav/main, heading text, row text, column header, list item text.
• Frame path and shadow path.
• Bounding box, viewport relation, z-index-ish overlay observation, center hit-test result.
• Interactability: visible, enabled, editable, checked/selected, expanded, pressed, receives events.
• Event/action hints: href navigation, submit/reset, file upload, contenteditable, onclick, keyboard role, pointer cursor.
• Risk terms and risk classification.

Locator Candidate Evidence

• Structured descriptor and generated expression.
• Strategy: role, label, placeholder, text, test ID, alt/title, stable attribute, local CSS, XPath, coordinate fallback.
• Score and reason.
• Match count.
• Visible match count.
• Enabled/editable match count.
• Whether the match points to the same semantic signature.
• Whether the locator survives reload and precondition replay.
• Failure count and last failure reason.

Action Edge Evidence

• From state and to state.
• Target element and chosen locator.
• Action type: click, fill, select, check, uncheck, upload, hover, keypress, navigate.
• Parameters or parameter schema.
• Preconditions and opening path.
• Effects: URL, DOM hash, visible text hash, network, dialog, popup, download, storage, console error.
• Risk decision: executed, skipped, requires approval, blocked by policy.
• Confidence and replay verification status.

Crawl Policy Evidence

• Entry URLs and seed source: explicit, sitemap, in-page links, recorded trace.
• Allowed domains and path filters.
• Robots fetch result and policy outcome.
• Rate limits, concurrency, backoff, retry policy.
• Duplicate policy: raw URL, normalized URL, canonical URL, request fingerprint.
• Trap controls: max depth, max pages per host, query parameter allowlist/denylist, calendar/facet detection.
• Auth mode: anonymous, supplied storage state, or manually authenticated context.

Suggested Near-Term Design Direction for This Repository

1. Add a Capture Evidence Layer

Introduce richer internal raw evidence before changing graph behavior too much. The graph can expose selected fields, while the engine keeps implementation detail behind stable types.

Candidate additions:

• frameworkHints on StateNode or metadata.
• context on ActionableElement: nearest landmark, dialog, form, table row, card, section heading.
• Real framePath and shadowPath.
• Locator verification details beyond matchCount.
• skipReason or exploration decision records for skipped elements.

2. Add Locator Context Ranking

Keep current priority:

1. role plus accessible name
2. label
3. placeholder
4. test ID or explicit automation attribute
5. text
6. stable local CSS

Then add context-aware variants:

• dialog[name].getByRole(...)
• row[hasText].getByRole(...)
• form[has label/legend].getByLabel(...)
• section[heading].getByRole(...)
• frame/shadow-aware variants

3. Add Replay-Based Verification

A locator should have at least two levels of verification:

• Snapshot verification: resolves uniquely now.
• Replay verification: after reload or precondition replay, resolves to the same semantic target.

This will catch framework rerendering, transient DOM, and hidden duplicate issues much earlier.

4. Add Conservative Crawl/Explore Policy

Before expanding exploration:

• Per-host max pages and max actions.
• Explicit allowlist for domains and paths.
• Robots policy record.
• Query parameter normalization policy.
• Risk-gated default action set.
• No form submission or destructive clicks by default.

5. Keep Anti-Bot Handling Non-Evasive

For this repo, anti-bot detection should become observable state, not a bypass feature:

• detect challenge/block pages,
• record response status and visible challenge text,
• mark edge/state as blocked or requires authorization,
• stop exploration unless the user supplied an authorized session and policy allows it.

What Not To Build Here

To preserve the repository boundary:

• Do not add LLM task planning.
• Do not add natural-language instruction execution.
• Do not add stealth CAPTCHA or paywall bypass.
• Do not add a generic browser-agent orchestration layer.
• Do not turn generated topology into autonomous high-risk actions.
• Do not store raw sensitive cookies, tokens, request bodies, or personal data in default artifacts.

Practical Checklist for Future Implementation

Before a locator or action edge is considered reliable, the engine should be able to answer:

1. Which page state owned this element?
2. Which frame and shadow boundary contained it?
3. Which user-facing semantics identified it?
4. Which nearby context disambiguated it from similar elements?
5. Which locator candidates were generated?
6. Which candidates were verified, and how many elements did each match?
7. Was the target visible, enabled, stable, editable if needed, and receiving events?
8. Did replay from entry/preconditions find the same target?
9. What happened after action: URL, DOM, text, network, storage, dialog, download?
10. Was the action skipped or gated by risk, robots, scope, auth, or anti-bot policy?

Bottom Line

The high-leverage research conclusion is simple: reliable browser automation is mostly evidence management. The engine should capture enough state-local, semantic, contextual, timing, and policy evidence that downstream consumers can choose actions quickly without guessing why a locator worked once.