Commit Graph

135 Commits

Author SHA1 Message Date
木炎
2cab25952e fix(generator): add processData to jQuery ajax for form-urlencoded requests
jQuery default processData:true re-serializes string bodies, causing
double-encoding for form-urlencoded payloads. Set processData:false
when contentType is application/x-www-form-urlencoded.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 18:37:20 +08:00
木炎
6fee4e2083 fix(generator): handle PageStateEval with no endpoints, optimize clones
- Guard against empty modes falling through to compile_multi_mode_request
  by falling back to compile_simple_request_script for edge cases
- Optimize ensure_modes_populated to clone only when needed
- Reduces dead-code warnings from 3 to 2

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 18:33:09 +08:00
木炎
46005d9718 fix: add legacy comment to compile_page_state_eval 2026-04-17 18:30:41 +08:00
木炎
d1711a3db3 feat(generator): unify all scene types through multi-mode path
Single-mode and page-state-eval scenes now get auto-wrapped into a
default mode and compiled through compile_multi_mode_request. This
eliminates the old browser_script_with_business_logic code path and
ensures all scenes get responsePath extraction, requestTemplate, and
contentType support.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 18:28:47 +08:00
木炎
b1647cd865 docs: add detailed implementation plan for scene generator quality improvement
8 tasks across 3 phases with exact file paths, step-by-step instructions,
test commands, and commit messages for each task.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 18:19:37 +08:00
木炎
1c964c3e70 docs: add scene generator quality improvement design spec
🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 18:13:45 +08:00
木炎
a6aa18c6d9 fix: replace eval() with safe template resolver and add empty MODES guard
Issue #1 (Critical): Replaced eval() in buildModeRequest with a safe
resolveTemplateValue function that only supports args.fieldName and
args['fieldName'] patterns, eliminating XSS/injection vulnerability.

Issue #2 (Critical): Added defensive guard in detectMode that throws a
clear error when MODES array is empty, preventing undefined property access.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 13:20:55 +08:00
木炎
7420af8f31 fix: add serde rename for NormalizeRulesJson fields to match JS camelCase
The NormalizeRulesJson struct was serializing required_fields and filter_null
as snake_case, but the JavaScript template accesses them as requiredFields
and filterNull. Add #[serde(rename = "requiredFields")] and #[serde(rename = "filterNull")]
to fix the systematic snake_case vs camelCase mismatch in multi-mode templates.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 13:16:14 +08:00
木炎
42eb716b7e feat(generator): add multi-mode template for mode-aware script generation
Add browser_script_with_modes function that generates JavaScript with:
- detectMode() to select mode based on args
- buildModeRequest() with content-type handling (JSON/form-urlencoded)
- normalizeModeRows() with validation rules
- queryModeData() with jQuery + fetch dual HTTP client

Modify browser_script() to check for modes first before falling back
to business_logic or skeleton templates.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 13:10:21 +08:00
木炎
a325add167 fix(llm): add defaults for multi-mode fields in analyzeSceneDeep
🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 13:07:10 +08:00
木炎
d95b8aaf26 feat(llm): enhance DEEP_SYSTEM_PROMPT for multi-mode detection
🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 13:02:36 +08:00
木炎
847f2c62c6 feat(generator): add multi-mode schema structs for enhanced LLM extraction
Add ModeConditionJson, NormalizeRulesJson, ApiEndpointEnhancedJson, and
ModeConfigJson structs to support multi-mode business logic in generated
scripts. Enhance SceneInfoJson with modes, default_mode, and mode_switch_field.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 12:56:46 +08:00
木炎
ea9147defb docs: add implementation plan for enhanced LLM extraction schema 2026-04-17 12:53:36 +08:00
木炎
5ff6e05911 docs: add enhanced LLM extraction schema design for multi-mode business logic 2026-04-17 12:51:09 +08:00
木炎
eead8f7802 feat(generator): enhance buildBrowserEntrypointResult with validation flow
- Add page context validation step with deps.validatePageContext
- Change validation failure status from 'error' to 'blocked'
- Add row normalization partial detection
- Use buildArtifact for all return paths consistently
- Add COLUMNS and REPORT_NAME constants for buildArtifact
- Export determineArtifactStatus, COLUMNS, REPORT_NAME in module.exports

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 12:28:35 +08:00
木炎
d123ee0aca feat(generator): enhance buildArtifact with determineArtifactStatus integration
- Change function signature to accept destructured object parameter
- Integrate determineArtifactStatus for automatic status calculation
- Use REPORT_NAME and COLUMNS constants (to be defined in Task 6)
- Add proper filtering for partial_reasons (exclude api_/validation_ prefixes)
- Add deduplication for reasons array
- Update call site in buildBrowserEntrypointResult to use new signature

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 12:24:17 +08:00
木炎
d26d96ef64 feat(generator): add determineArtifactStatus for complete status determination
Add a helper function to determine artifact status based on business
logic conditions:
- blocked: blockedReason exists (validation failed, wrong page context)
- error: fatalError exists (API call failed)
- partial: reasons array non-empty (some data issues)
- empty: no rows returned
- ok: rows exist with no issues

This is Task 3 of 8 in the progressive template enhancement.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 12:21:13 +08:00
木炎
d996b511f6 feat(generator): add jQuery + fetch dual HTTP client support in template
Enhance defaultDeps with:
- validatePageContext: Checks hostname against expected_domain for page context validation
- queryData: Dual HTTP client with jQuery $.ajax as primary and fetch as fallback
  - jQuery preferred for internal pages that typically have it
  - Clear error message if neither client is available

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 12:19:02 +08:00
木炎
b8d2eb9faa fix(generator): use endpoint.url directly in buildRequest to fix URL construction bug
The previous implementation used `new URL(endpoint.url, window.location.origin)`
which incorrectly constructed URLs based on the current page's origin. This broke
when endpoint URLs were already complete URLs (e.g., http://20.76.57.61:18080/...).

The fix uses endpoint.url directly since LLM extraction already provides complete
URLs. Also changed from GET with query params to POST with JSON body to match
typical API expectations.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 11:34:16 +08:00
木炎
4167639231 docs: add progressive template enhancement implementation plan 2026-04-17 11:26:34 +08:00
木炎
78a36a73b4 docs: add progressive template enhancement design spec 2026-04-17 11:16:11 +08:00
木炎
74c42af717 fix: handle apiEndpoints/columnDefs objects in preview, add string type checks 2026-04-17 10:55:58 +08:00
木炎
bb15d14749 feat(runner): pass sceneInfoJson to Rust CLI for enhanced template rendering 2026-04-17 10:39:36 +08:00
木炎
7289cc5779 feat(ui): add deep extraction preview panel with API/column/static-params display 2026-04-17 10:37:50 +08:00
木炎
689abf08ec feat(rust): add --scene-info-json parameter for LLM extraction results
- Add SceneInfoJson struct with serde rename for camelCase compatibility
- Add ApiEndpointJson and BusinessLogicJson support structs
- Add scene_info_json field to GenerateSceneRequest (backward compatible)
- Rename browser_script to browser_script_skeleton
- Add new browser_script that delegates based on scene_info presence
- Add browser_script_with_business_logic for enhanced script generation
- Update CLI to accept --scene-info-json parameter
- Update usage string to document new parameter

Generated with [Qoder][https://qoder.com]
2026-04-17 10:27:27 +08:00
木炎
2ffb42c181 feat(server): add /analyze-deep endpoint for deep extraction
🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 10:20:04 +08:00
木炎
614e9a3a45 feat(generator-runner): read index.html in readDirectory()
🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 10:16:17 +08:00
木炎
517ac6bf39 feat(llm-client): add deep extraction with apiEndpoints, staticParams, columnDefs
Add analyzeSceneDeep() function that extracts complete SceneInfo from
index.html content including API endpoints, static parameters, column
definitions, and business logic. Also adds http module import to support
non-HTTPS LLM endpoints.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 10:13:29 +08:00
木炎
dd7b3c582a docs: add LLM-driven skill generation design spec
Design for enhancing sg_scene_generate to produce complete,
runnable skill packages with:

- Deep LLM extraction from index.html (API endpoints, params, columns)
- Enhanced Rust template rendering with business logic
- Web UI preview of extracted results

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 09:51:59 +08:00
木炎
f268668713 feat: add folder picker and target_url input to Web UI
- Add /select-folder and /select-file APIs using PowerShell dialogs
- Add --target-url parameter to CLI for explicit target URL override
- Redesign Web UI with folder browse buttons for all path inputs
- Add target_url optional input field for specifying target page URL
- Auto-fill scene-id from selected folder name

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 00:23:09 +08:00
木炎
ce072c2ebe feat: auto-extract expected_domain from external script URLs
When HTML has no sgclaw-expected-domain meta tag, analyzer now scans
for external script URLs (http:// or https://) and extracts the
domain (host:port) as expected_domain.

Example:
  <script src="http://25.215.213.128:18080/a_js/YPTAPI.js"></script>
  → expected_domain = "25.215.213.128:18080"

This reduces manual editing required for third-party scenes.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 00:14:05 +08:00
木炎
464f18c672 fix: remove mandatory validation for ReportCollection fields
Remove the validation that required target_url, expected_domain, and
entry_script meta tags for ReportCollection scenes. Third-party scenes
do not have these meta tags, so the generator now uses empty values
that users can manually edit in scene.toml.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 00:09:12 +08:00
木炎
b5131c858a feat: add scene kind dropdown to Web UI
Add scene type selector to the generator form:
- CSS styles for select element matching existing input style
- Dropdown with "报表收集类" (report_collection) and "监测类" (monitoring) options
- Pass sceneKind to /generate API endpoint

🤖 Generated with [Qoder][https://qoder.com]
2026-04-17 00:02:17 +08:00
木炎
2e69fa7239 feat: pass sceneKind from /generate request to generator
🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 23:58:57 +08:00
木炎
f84e11c631 feat: add sceneKind param to generator-runner
Pass sceneKind to sg_scene_generate CLI when specified,
allowing generation of different scene kinds (report_collection,
monitoring, etc.).

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 23:57:30 +08:00
木炎
73edf1e5cf feat: add monitoring template support to generator
- Add scene_kind field to GenerateSceneRequest
- Update generate_scene_package to use analyze_scene_source_with_hint
- Implement scene_toml_monitoring for simplified monitoring scene manifests
- Add scene_toml router to dispatch to different templates based on SceneKind
- Add --scene-kind CLI option to sg_scene_generate binary
- Add generator_emits_monitoring_template test
- Create monitoring fixture with index.html (no meta tag, tests hint param)

Monitoring templates have simplified scene.toml without org/period resolvers.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 23:51:11 +08:00
木炎
87cee36173 fix: delegate analyze_scene_source to analyze_scene_source_with_hint
Remove duplicate implementation in analyze_scene_source() and replace
with simple delegation to analyze_scene_source_with_hint(source_dir, None).

This eliminates ~50 lines of duplicated logic that could drift apart
from the main implementation. Updated the test to verify the new
behavior where sources without scene-kind meta tag default to
ReportCollection instead of being rejected.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 23:42:46 +08:00
木炎
67fe17302e feat: add SceneKind::Monitoring and scene_kind_hint param to analyzer
- Add SceneKind::Monitoring enum variant with from_str/as_str helpers
- Add analyze_scene_source_with_hint function accepting optional scene kind hint
- User hint takes priority over meta tag, defaults to ReportCollection
- ReportCollection requires target_url, expected_domain, entry_script
- Monitoring type has optional fields
- Add test cases for hint parameter behavior
- Update non_report fixture with required meta tags for ReportCollection

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 23:33:24 +08:00
木炎
45b54ab007 docs: add multi-scene-kind generator implementation plan
7 tasks covering:
- Task 1: Extend SceneKind enum and analyzer function
- Task 2: Add multi-template support to generator
- Task 3: Add --scene-kind CLI parameter
- Task 4-6: Pass sceneKind through Node.js stack to Web UI
- Task 7: E2E testing and verification

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 23:26:01 +08:00
木炎
af8f261b79 docs: add multi-scene-kind generator design spec
Design for extending sg_scene_generate to support multiple scene kinds
(report_collection, monitoring) with manual type selection in Web UI,
relaxing the requirement for meta tags in third-party scene directories.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 23:22:08 +08:00
木炎
f168f9f375 fix: auto-find correct sgclaw_config.json in parent directory
Add parent sgClaw directory to config search candidates so Node.js
server automatically finds the correct config with DeepSeek API
endpoint instead of the test config in claw-new directory.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 22:47:39 +08:00
木炎
23845413c5 fix: patch path traversal and baseUrl normalization in scene generator
- server.js: sanitize static file paths to prevent directory traversal
  (GET /../../sgclaw_config.json would expose API key)
- config-loader.js: fix normalizeBaseUrl to strip /v1 before appending,
  preventing double /v1 for non-standard base URLs

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 22:30:35 +08:00
木炎
ea6be128e7 feat: scene skill generator — complete implementation
Adds a web-based UI for generating scene skill packages:
- Node.js HTTP server (zero npm dependencies) on port 3210
- HTML page with glass-morphism UI, dual-panel layout, settings modal
- LLM-powered scene-id/scene-name auto-extraction from directory contents
- Real-time SSE progress streaming during skill generation
- Spawns sg_scene_generate CLI with configurable parameters
- Windows-compatible startup scripts (serve.sh + serve.cmd)
- Rust integration tests for server files and HTML structure

Architecture:
  Browser (HTML/JS) → Node.js server → LLM API + cargo run → sg_scene_generate

Files:
  frontend/scene-generator/{server.js,config-loader.js,llm-client.js,generator-runner.js,sg_scene_generator.html,serve.sh,serve.cmd}
  tests/{scene_generator_server_test.rs,scene_generator_html_test.rs,scene_generator_llm_test.js}
  docs/superpowers/{plans,specs}/2026-04-16-scene-skill-generator*

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 22:27:41 +08:00
木炎
6c1865eb1c feat: add serve.sh and serve.cmd startup scripts for Scene Skill Generator
🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 22:25:34 +08:00
木炎
d00086a70b feat: add sg_scene_generator.html with dual-panel UI and settings modal
Add self-contained HTML page for the Scene Skill Generator frontend:
- Dual-column glass-morphism layout matching service-console style
- Left sidebar: status card, sourceDir input with analyze button,
  sceneId/sceneName inputs, settings button, generate button
- Right panel: streaming log display with SSE event rendering
- Settings modal: outputRoot, lessons, llmBaseUrl, llmModel fields
- JavaScript: connects to http://127.0.0.1:3210, implements analyze()
  via fetch POST /analyze, generate() via fetch POST /generate with
  SSE stream reading, settings modal open/close
- Rust test verifying HTML file exists and contains required elements

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 22:23:33 +08:00
木炎
e7a4179513 feat: add server.js with /health, /analyze, /generate routes 2026-04-16 22:18:32 +08:00
木炎
15d4b0dcc1 feat: add generator-runner.js for cargo subprocess + SSE streaming 2026-04-16 22:15:52 +08:00
木炎
294426ced9 feat: add llm-client.js with prompt builder and JSON extractor
🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 22:13:11 +08:00
木炎
ead9ea76fa feat: add config-loader.js and initial server test
Add config-loader.js module for loading sgclaw_config.json credentials
and resolving project root directory. Add initial Rust source-guard test
to verify server file paths exist.

🤖 Generated with [Qoder][https://qoder.com]
2026-04-16 22:10:48 +08:00
木炎
e8d7d6b796 test: lock request URL resolution precedence
Align the service task flow callback-host regression with the hidden helper close/open bootstrap sequence uncovered during final request-url verification.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 20:27:09 +08:00