feat: load packet logs from directory source
This commit is contained in:
zhaoyilun
@@ -24,13 +24,13 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor
|
||||
- `isphere_search_groups`
|
||||
- `isphere_receive_files`
|
||||
- C# `ISphereWinHelper` exists under `native/ISphereWinHelper` and uses the `isphere.helper.v1` stdin/stdout JSON contract.
|
||||
- `scripts/verify-win-helper.ps1` and `scripts/verify-go-mcp.ps1` provide repeatable local checks for the helper, eight-tool MCP surface, default empty-source receive/contact/group/file calls, and synthetic configured-source receive/contact/group/file smoke.
|
||||
- `scripts/verify-win-helper.ps1` and `scripts/verify-go-mcp.ps1` provide repeatable local checks for the helper, eight-tool MCP surface, default empty-source receive/contact/group/file calls, synthetic configured-file receive/contact/group/file smoke, and synthetic configured-directory receive/contact/group/file smoke.
|
||||
- N12-pre and N12R documents define safe offline evidence intake and internal-sandbox live UIA capture procedures.
|
||||
- N13/N14/N15 produced selector catalog/report validation infrastructure, but those are supporting validation assets only.
|
||||
|
||||
## What cannot be claimed externally
|
||||
|
||||
- The project has not yet implemented business MCP tools for message sending, file download, or file sending; `isphere_receive_messages`, `isphere_search_contacts`, `isphere_search_groups`, and `isphere_receive_files` list mode have operator-local encrypted log-file support but not a production ingestion path.
|
||||
- The project has not yet implemented business MCP tools for message sending, file download, or file sending; `isphere_receive_messages`, `isphere_search_contacts`, `isphere_search_groups`, and `isphere_receive_files` list mode have operator-local encrypted log-file/log-directory support but not a production ingestion path.
|
||||
- The project has not completed production iSphere integration.
|
||||
- The project has not proven a stable real message source for the full business workflow beyond decrypted PacketReader log evidence and synthetic/redacted tests.
|
||||
- The project has not implemented real message sending or real file upload.
|
||||
@@ -43,7 +43,7 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor
|
||||
- The helper protocol boundary is explicit and keeps Windows/UIA work separate from the Go MCP service layer.
|
||||
- The current code can support future MCP business tools by routing low-level Windows/UIA fallback work through a bounded helper layer.
|
||||
- The project has repeatable verification scripts for the helper/MCP foundation and the current eight-tool surface.
|
||||
- The project now has a corrected business contract and plan for core MCP communication tools, plus registered receive/contact/group/file-list read tools that can use `ISPHERE_PACKET_LOG_FILE` for operator-local encrypted PacketReader log lines.
|
||||
- The project now has a corrected business contract and plan for core MCP communication tools, plus registered receive/contact/group/file-list read tools that can use `ISPHERE_PACKET_LOG_FILE` for one operator-local encrypted PacketReader log-line file or `ISPHERE_PACKET_LOG_DIR` for an operator-local directory of encrypted `.log`/`.txt` PacketReader files.
|
||||
|
||||
## N13/N14/N15 positioning
|
||||
|
||||
@@ -51,7 +51,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme
|
||||
|
||||
## Current loop
|
||||
|
||||
Current loop: `Stage C / Loop C16 - configured PacketReader source hardening`.
|
||||
Current loop: `Stage C / Loop C17 - read-tool contract alignment precheck`.
|
||||
|
||||
Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.
|
||||
|
||||
@@ -79,10 +79,11 @@ Current loop output so far:
|
||||
13. C13: `internal/isphere.ListFilesFromMessages` extracts synthetic/redacted file metadata candidates from normalized messages.
|
||||
14. C14: `isphere_receive_files` MCP list mode registered and verify-go-mcp now covers eight tools plus default/configured file-list smoke.
|
||||
15. C15: `isphere_send_message` source precheck found no validated write connector; send implementation is blocked pending bridge/API, UIA action, or protocol evidence.
|
||||
16. C16: configured PacketReader source hardening added `ISPHERE_PACKET_LOG_DIR`, deterministic recursive `.log`/`.txt` directory loading, ambiguous file/dir config failure, and configured-directory receive/contact/group/file smoke verification.
|
||||
|
||||
## Next business mainline
|
||||
|
||||
1. Run Stage C Loop C16: harden configured PacketReader source loading so existing read tools can consume directory/multi-log operator evidence.
|
||||
1. Run Stage C Loop C17: audit and align the registered read-tool outputs against `docs/mcp-core-tools-contract.md`, starting with `isphere_receive_messages` because it is currently the least contract-shaped output.
|
||||
2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated.
|
||||
3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated.
|
||||
4. Keep UIA selector/report work as fallback support, not as the primary roadmap.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
This runbook is for the first Go MCP phase of `isphere-ai-bridge`. It lets an operator build and verify the Windows helper and the Go MCP server without reading the source code.
|
||||
|
||||
Current scope: expose four WinHelper observation operations plus four log-backed business read tools through Go MCP: `isphere_receive_messages`, `isphere_search_contacts`, `isphere_search_groups`, and `isphere_receive_files` list mode. These business read tools use an empty default source unless `ISPHERE_PACKET_LOG_FILE` points to an operator-local encrypted PacketReader log-line file.
|
||||
Current scope: expose four WinHelper observation operations plus four log-backed business read tools through Go MCP: `isphere_receive_messages`, `isphere_search_contacts`, `isphere_search_groups`, and `isphere_receive_files` list mode. These business read tools use an empty default source unless `ISPHERE_PACKET_LOG_FILE` points to an operator-local encrypted PacketReader log-line file or `ISPHERE_PACKET_LOG_DIR` points to an operator-local directory of encrypted PacketReader `.log`/`.txt` files.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
@@ -94,6 +94,7 @@ The verification confirms:
|
||||
- `win_helper_version` returns `ISphereWinHelper`.
|
||||
- `isphere_receive_messages` is callable and returns an empty `messages` array from the default empty source.
|
||||
- A synthetic configured-source smoke check proves `ISPHERE_PACKET_LOG_FILE` can drive `isphere_receive_messages` end-to-end with one redacted fixture message.
|
||||
- A synthetic configured-directory smoke check proves `ISPHERE_PACKET_LOG_DIR` can drive `isphere_receive_messages`, contact search, group search, and file-list extraction from a temporary directory fixture.
|
||||
- `isphere_search_contacts` is callable and returns JID-derived contact candidates from the same configured synthetic message fixture.
|
||||
- `isphere_search_groups` is callable and returns JID-derived group candidates from the configured synthetic groupchat fixture.
|
||||
- `isphere_receive_files` is callable in list mode and returns one file metadata candidate from the configured synthetic file-transfer fixture.
|
||||
@@ -116,7 +117,15 @@ $env:ISPHERE_PACKET_LOG_FILE = "E:\coding\codex\isphere-ai-bridge\runs\offline-e
|
||||
|
||||
The file must contain one encrypted Base64 log line per line. Blank lines are ignored. Keep this file under ignored `runs/` paths or another operator-local location; do not commit raw logs or decrypted message content.
|
||||
|
||||
Then start the MCP server normally. The command entry point uses `ISPHERE_PACKET_LOG_FILE` if it is set. The repeatable verification script checks both paths: it clears the variable for the default empty-source smoke test, then creates a temporary synthetic encrypted fixture file and proves the configured-source path returns one redacted message.
|
||||
To point the server at a directory containing multiple local encrypted PacketReader files, set:
|
||||
|
||||
```powershell
|
||||
$env:ISPHERE_PACKET_LOG_DIR = "E:\coding\codex\isphere-ai-bridge\runs\offline-evidence-intake\<evidence-id>\raw\temp\PacketReader"
|
||||
```
|
||||
|
||||
The directory loader recursively reads `.log` and `.txt` files, sorts paths deterministically, and ignores blank lines. Set only one source variable at a time: if both `ISPHERE_PACKET_LOG_FILE` and `ISPHERE_PACKET_LOG_DIR` are set, server startup fails fast as an ambiguous configuration.
|
||||
|
||||
Then start the MCP server normally. The command entry point uses `ISPHERE_PACKET_LOG_FILE` when the file variable is set, otherwise `ISPHERE_PACKET_LOG_DIR` when the directory variable is set. The repeatable verification script checks three paths: default empty source, a temporary synthetic encrypted fixture file, and a temporary synthetic encrypted fixture directory.
|
||||
|
||||
## 7. Configure MCP client command
|
||||
|
||||
@@ -163,7 +172,7 @@ These eight tools are currently allowed:
|
||||
| `win_helper_self_check` | helper `self_check` | Read desktop/UIA availability status. |
|
||||
| `win_helper_scan_windows` | helper `scan_windows` | Read visible window metadata or likely iSphere candidates. |
|
||||
| `win_helper_dump_uia` | helper `dump_uia` | Read a UI Automation tree for a specified window handle. |
|
||||
| `isphere_receive_messages` | log-backed `EncryptedPacketLogSource` | Read normalized messages from the configured PacketReader log source. The default server source is empty until a loader is wired. |
|
||||
| `isphere_receive_messages` | log-backed `EncryptedPacketLogSource` | Read normalized messages from the configured PacketReader log source. The default server source is empty unless `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` is configured. |
|
||||
| `isphere_search_contacts` | log-backed message JIDs | Search contact candidates extracted from sender/receiver bare JIDs in the configured message source. |
|
||||
| `isphere_search_groups` | log-backed groupchat JIDs | Search group candidates extracted from groupchat/conference/MUC bare JIDs in the configured message source. |
|
||||
| `isphere_receive_files` | log-backed file-transfer messages | List file metadata candidates extracted from file-transfer message bodies. Download/cache mapping is not implemented yet. |
|
||||
@@ -192,7 +201,7 @@ The current runbook verifies eight tools:
|
||||
- `isphere_search_groups`
|
||||
- `isphere_receive_files`
|
||||
|
||||
The current log-backed business read tools have parser/source/tool registration and an empty default server source. Raw N12-pre evidence remains under ignored `runs/` paths and is not committed. The current command entry point can read an operator-local encrypted PacketReader log-line file via `ISPHERE_PACKET_LOG_FILE`; this is still not a production ingestion claim because raw evidence remains local and uncommitted. `isphere_receive_files` currently supports list mode only; download/cache mapping is deferred.
|
||||
The current log-backed business read tools have parser/source/tool registration and an empty default server source. Raw N12-pre evidence remains under ignored `runs/` paths and is not committed. The current command entry point can read an operator-local encrypted PacketReader log-line file via `ISPHERE_PACKET_LOG_FILE` or a directory of `.log`/`.txt` files via `ISPHERE_PACKET_LOG_DIR`; this is still not a production ingestion claim because raw evidence remains local and uncommitted. `isphere_receive_files` currently supports list mode only; download/cache mapping is deferred.
|
||||
|
||||
Core business tools for contacts, groups, messages, and files are defined in `docs/mcp-core-tools-contract.md`. Send capabilities enter the roadmap through that contract after source discovery, connector selection, preview metadata, execution metadata, and audit records are in place.
|
||||
|
||||
|
||||
@@ -17,10 +17,10 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
|
||||
|
||||
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; wrapped `MsgLib.db` after DB wrapper is solved | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | C5 env loader wired through ISPHERE_PACKET_LOG_FILE; default source remains empty when unset | fixture-backed configured-source operator smoke |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages | decrypted roster/contact stanzas from `Smark.SendReceive`; `MsgLib.db` contact/conversation tables after DB wrapper is solved; UIA helper source if display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#log-decryption-result` | C8 MCP tool registered and verify-go-mcp covers default/configured contact smoke; JID-only display/account fallback | enrich display names from roster/db/uia evidence |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs | `MsgLib.db` group/conversation tables after DB wrapper is solved; UIA helper source if group display names are missing | C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C11 MCP tool registered and verify-go-mcp covers default/configured group smoke; JID-only display fallback | enrich group display/member data from roster/db/uia evidence |
|
||||
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C14 MCP list mode registered and verify-go-mcp covers default/configured file-list smoke; no log-to-cache/download mapping yet | defer download mapping; run send-message source evidence precheck |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; wrapped `MsgLib.db` after DB wrapper is solved | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | C16 env loader supports one encrypted file or a deterministic `.log`/`.txt` directory source; default source remains empty when unset | read-tool output contract alignment |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | decrypted roster/contact stanzas from `Smark.SendReceive`; `MsgLib.db` contact/conversation tables after DB wrapper is solved; UIA helper source if display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#log-decryption-result` | C16 verification covers default, configured-file, and configured-directory contact smoke; JID-only display/account fallback | enrich display names from roster/db/uia evidence |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | `MsgLib.db` group/conversation tables after DB wrapper is solved; UIA helper source if group display names are missing | C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C16 verification covers default, configured-file, and configured-directory group smoke; JID-only display fallback | enrich group display/member data from roster/db/uia evidence |
|
||||
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C16 verification covers default, configured-file, and configured-directory file-list smoke; no log-to-cache/download mapping yet | defer download mapping; align read-tool contract output |
|
||||
| `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration |
|
||||
| `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated |
|
||||
|
||||
|
||||
@@ -1456,7 +1456,7 @@ Evidence precheck result:
|
||||
- Extend verification with a synthetic directory fixture proving configured receive/contact/group/file-list tools still work from the hardened source path.
|
||||
- Keep raw N12-pre evidence under ignored/operator-local paths; commit only synthetic fixtures/tests.
|
||||
|
||||
- [ ] **Step 1: Write failing directory source test**
|
||||
- [x] **Step 1: Write failing directory source test**
|
||||
|
||||
Create or extend `internal/isphere/file_source_test.go` and run:
|
||||
|
||||
@@ -1466,15 +1466,15 @@ go test ./internal/isphere -run TestLoadEncryptedPacketLogSourceFromDirectoryRea
|
||||
|
||||
Expected initial failure: directory loader does not exist.
|
||||
|
||||
- [ ] **Step 2: Implement directory/multi-file loader**
|
||||
- [x] **Step 2: Implement directory/multi-file loader**
|
||||
|
||||
Implement minimal deterministic loading over `.log`/`.txt` files without decrypting during load.
|
||||
|
||||
- [ ] **Step 3: Wire env and verification**
|
||||
- [x] **Step 3: Wire env and verification**
|
||||
|
||||
Add `ISPHERE_PACKET_LOG_DIR` handling in `mcpserver.NewServerFromEnv()` and update the verification script with a synthetic directory smoke.
|
||||
|
||||
- [ ] **Step 4: Run full verification and commit**
|
||||
- [x] **Step 4: Run full verification and commit**
|
||||
|
||||
Run:
|
||||
|
||||
@@ -1489,6 +1489,57 @@ Expected: all exit 0. Remove any generated root `isphere-mcp.exe` after build. C
|
||||
|
||||
---
|
||||
|
||||
## Loop C16 Result
|
||||
|
||||
Implemented and verified in this loop:
|
||||
|
||||
- `internal/isphere.LoadEncryptedPacketLogSourceFromDirectory(path)` loads operator-local encrypted PacketReader `.log` and `.txt` files recursively, sorts file paths deterministically, and ignores blank lines.
|
||||
- `internal/mcpserver.NewServerFromEnv()` now supports `ISPHERE_PACKET_LOG_FILE` for a single encrypted log-line file or `ISPHERE_PACKET_LOG_DIR` for a directory source. It fails fast if both variables are set.
|
||||
- `scripts/verify-go-mcp.ps1` now verifies three read-source paths: default empty source, synthetic configured file source, and synthetic configured directory source. The directory smoke covers receive messages, contact search, group search, and file-list extraction.
|
||||
- Runbook/status/matrix docs now describe the dual file/directory configuration while keeping raw N12-pre evidence under ignored/operator-local paths.
|
||||
- This is still a log-backed local evidence source, not a production ingestion connector.
|
||||
|
||||
---
|
||||
|
||||
## Loop C17: Read-tool contract alignment precheck
|
||||
|
||||
**Goal:** Audit the four registered business read tools against `docs/mcp-core-tools-contract.md`, then choose the smallest safe implementation slice that makes digital-employee consumption more stable without opening write/send behavior.
|
||||
|
||||
**Planned files:**
|
||||
- Modify: `docs/source-discovery/capability-source-matrix.md`
|
||||
- Modify: `docs/current-status-card.md`
|
||||
- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`
|
||||
- Optionally modify `internal/tools/isphere_read.go`, `internal/tools/isphere_read_test.go`, and `scripts/verify-go-mcp.ps1` if the precheck confirms a low-risk response-shape fix.
|
||||
|
||||
**Planned behavior:**
|
||||
- Compare current outputs for `isphere_receive_messages`, `isphere_search_contacts`, `isphere_search_groups`, and `isphere_receive_files` with the contract's `ok`, `next_cursor`, `audit`, and business field names.
|
||||
- Prioritize `isphere_receive_messages` because its current output remains parser-native (`id`, `text`, `subject`, etc.) instead of contract-facing (`message_id`, `content_text`, `content_type`, attachments, source/raw_ref).
|
||||
- Do not change send-message or send-file behavior in this loop.
|
||||
- If output-shape changes are backward-compatible and small, implement them with tests; if they are breaking or broad, document the compatibility decision and write the next implementation loop.
|
||||
|
||||
- [ ] **Step 1: Run contract/output precheck**
|
||||
|
||||
Inspect current tool registration code and tests, then record exact gaps against `docs/mcp-core-tools-contract.md`.
|
||||
|
||||
- [ ] **Step 2: Choose one small fix or defer**
|
||||
|
||||
If a small non-breaking fix is available, write the failing test first. Otherwise update docs with the contract gap and rewrite C18 as the implementation node.
|
||||
|
||||
- [ ] **Step 3: Verify and commit**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
git diff --check
|
||||
go test ./...
|
||||
go build ./cmd/isphere-mcp
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/verify-go-mcp.ps1
|
||||
```
|
||||
|
||||
Expected: all exit 0. Remove any generated root `isphere-mcp.exe` after build. Commit exact C17 paths only and rewrite the next loop from actual results.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
- Spec coverage: the plan follows the user's cyclic requirement: plan first, implement one loop at a time, update the next loop after each result, and ask only on blockers.
|
||||
|
||||
Reference in New Issue
Block a user