From 78f0e475474e416b7afc5251f8088018afb4584e Mon Sep 17 00:00:00 2001 From: zhaoyilun Date: Fri, 10 Jul 2026 08:27:32 +0800 Subject: [PATCH] docs: add msglib enrichment runbook --- docs/current-status-card.md | 5 +- docs/go-mcp-runbook.md | 83 +++++++++++++++++-- .../capability-source-matrix.md | 8 +- ...tage-c-log-backed-receive-messages-loop.md | 51 ++++++++++-- 4 files changed, 127 insertions(+), 20 deletions(-) diff --git a/docs/current-status-card.md b/docs/current-status-card.md index 0126062..b38b5d7 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Stage C / Loop C34 - MsgLib enrichment operator runbook and setup smoke`. +Current loop: `Stage C / Loop C35 - MsgLib message-table source precheck`. Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. @@ -102,10 +102,11 @@ Current loop output so far: 31. C31: added optional `scripts/verify-msglib-mcp-enrichment.ps1`; real copied-DB MCP smoke returned sanitized contact/group counts and MsgLib source refs without printing names, JIDs, message bodies, file paths, or raw rows. 32. C32: `isphere_receive_messages` can now use the optional MsgLib display-entity source to fill safe `sender_name` and `conversation.display_name` fields while preserving message text/content/attachment behavior. 33. C33: optional real copied-DB MCP smoke now also constructs a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verifies receive-message display enrichment through MCP; output stays sanitized to counts, booleans, and `msglib:` refs only. +34. C34: `docs/go-mcp-runbook.md` now documents the operator setup path for optional MsgLib display-name enrichment, including the required env vars, partial-config failure rule, standard no-MsgLib verification, optional provider smoke, optional MCP enrichment smoke, and sanitized output boundaries. ## Next business mainline -1. Run Stage C Loop C34: document and verify the operator setup path for enabling optional MsgLib enrichment in MCP without exposing local names, JIDs, file paths, message bodies, or raw rows. +1. Run Stage C Loop C35: precheck whether copied `MsgLib.db` message tables can support a future DB-backed `isphere_receive_messages` source, starting with schema/column mapping only and no message-body or raw-row output. 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated. 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated. 4. Keep UIA selector/report work as fallback support, not as the primary roadmap. diff --git a/docs/go-mcp-runbook.md b/docs/go-mcp-runbook.md index f616957..aa7cdbf 100644 --- a/docs/go-mcp-runbook.md +++ b/docs/go-mcp-runbook.md @@ -2,7 +2,7 @@ This runbook is for the first Go MCP phase of `isphere-ai-bridge`. It lets an operator build and verify the Windows helper and the Go MCP server without reading the source code. -Current scope: expose four WinHelper observation operations plus four log-backed business read tools through Go MCP: `isphere_receive_messages`, `isphere_search_contacts`, `isphere_search_groups`, and `isphere_receive_files` list mode. These business read tools use an empty default source unless `ISPHERE_PACKET_LOG_FILE` points to an operator-local encrypted PacketReader log-line file or `ISPHERE_PACKET_LOG_DIR` points to an operator-local directory of encrypted PacketReader `.log`/`.txt` files. +Current scope: expose four WinHelper observation operations plus four business read tools through Go MCP: `isphere_receive_messages`, `isphere_search_contacts`, `isphere_search_groups`, and `isphere_receive_files` list mode. These business read tools use an empty default message source unless `ISPHERE_PACKET_LOG_FILE` points to an operator-local encrypted PacketReader log-line file or `ISPHERE_PACKET_LOG_DIR` points to an operator-local directory of encrypted PacketReader `.log`/`.txt` files. Contact/group display names can also be optionally enriched from an operator-local copied `MsgLib.db` through the bounded read-only `MsgLibReadSidecar`. ## 1. Prerequisites @@ -100,6 +100,7 @@ The verification confirms: - `isphere_search_groups` is callable and returns JID-derived group candidates from the configured synthetic groupchat fixture. - `isphere_receive_files` is callable in list mode and returns one file metadata candidate from the configured synthetic file-transfer fixture. - Verification uses synthetic/local helper checks and does not load raw N12-pre evidence. +- The deterministic verification path clears MsgLib env variables, so it proves the default no-MsgLib behavior remains stable. - File download/cache mapping and send business tools remain later-stage work. ## 6. Configure optional message source @@ -128,7 +129,73 @@ The directory loader recursively reads `.log` and `.txt` files, sorts paths dete Then start the MCP server normally. The command entry point uses `ISPHERE_PACKET_LOG_FILE` when the file variable is set, otherwise `ISPHERE_PACKET_LOG_DIR` when the directory variable is set. The repeatable verification script checks three paths: default empty source, a temporary synthetic encrypted fixture file, and a temporary synthetic encrypted fixture directory. -## 7. Configure MCP client command +## 7. Configure optional MsgLib display-name enrichment + +This is optional. It enriches contact/group display metadata; it does not turn `MsgLib.db` into the primary message source. `isphere_receive_messages` still needs a PacketReader source to return messages, and MsgLib only fills safe display fields such as `sender_name` and `conversation.display_name`. + +Build the bounded x86 sidecar first: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build-msglib-sidecar.ps1 +``` + +Then configure all three MsgLib variables before starting the MCP server: + +```powershell +$env:ISPHERE_MSGLIB_SIDECAR_EXE = "E:\coding\codex\isphere-ai-bridge\runs\msglib-sidecar\MsgLibReadSidecar.exe" +$env:ISPHERE_MSGLIB_SQLITE_DLL = "" +$env:ISPHERE_MSGLIB_DB = "" +``` + +Optional password override: + +```powershell +$env:ISPHERE_MSGLIB_PASSWORD = "123" +``` + +Rules: + +- Set all of `ISPHERE_MSGLIB_SIDECAR_EXE`, `ISPHERE_MSGLIB_SQLITE_DLL`, and `ISPHERE_MSGLIB_DB`, or set none of them. Partial MsgLib config fails server startup. +- Keep copied DB/provider files under ignored `runs/` paths or another operator-local location. Do not commit them. +- `MsgLibReadSidecar` is read-only and bounded to schema/display metadata operations. It does not send messages, download files, mutate DB rows, log in, hook processes, or return raw rows. +- Standard `scripts\verify-go-mcp.ps1` intentionally clears MsgLib variables to keep the default smoke deterministic. + +Verify the optional provider path: + +```powershell +$env:ISPHERE_MSGLIB_SQLITE_DLL = "" +$env:ISPHERE_MSGLIB_DB = "" +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1 +``` + +Expected provider evidence contains `provider_checked=true`, a `display_source_count`, and sanitized `display_entity_summaries` with counts/source table names only. + +Verify MCP-level optional enrichment, including receive-message display fields: + +```powershell +$env:ISPHERE_MSGLIB_SQLITE_DLL = "" +$env:ISPHERE_MSGLIB_DB = "" +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-enrichment.ps1 +``` + +Expected MCP evidence is sanitized and should include: + +```json +{ + "ok": true, + "receive_display_smoke": true, + "receive_sender_name_populated": true, + "receive_conversation_display_populated": true, + "printed_entity_values": false, + "message_bodies_returned": false, + "file_paths_returned": false, + "raw_rows_returned": false +} +``` + +The optional MCP smoke internally uses copied-DB display metadata to build a synthetic encrypted PacketReader fixture, but the printed output must stay limited to counts, booleans, and `msglib:` refs. It must not print entity values, JIDs, message bodies, local DB paths, attachment paths, or raw rows. + +## 8. Configure MCP client command After building a stable binary, configure your MCP client to run the Go MCP executable as a stdio server. @@ -163,7 +230,7 @@ If you used `go build ./cmd/isphere-mcp` without `-o`, point the command to: E:\coding\codex\isphere-ai-bridge\isphere-mcp.exe ``` -## 8. Allowed tools +## 9. Allowed tools These eight tools are currently allowed: @@ -189,7 +256,7 @@ Allowed parameters: - `isphere_search_groups`: `query`, `cursor`, `source_preference`, `include_archived`, `limit` only. `source_preference` currently supports empty/`auto`/`local_readonly`; `cursor` must be empty until pagination is implemented. - `isphere_receive_files`: `conversation_id`, `message_id`, `file_id`, `name_contains`, `mode`, `output_dir`, `cursor`, `source_preference`, `preview`, `limit` only. C22 supports `mode` empty or `list`; `source_preference` currently supports empty/`auto`/`local_readonly`; `cursor` and `output_dir` must be empty in current list-only mode; download is deferred. -## 9. Current tool surface and next-stage route +## 10. Current tool surface and next-stage route The current runbook verifies eight tools: @@ -202,11 +269,11 @@ The current runbook verifies eight tools: - `isphere_search_groups` - `isphere_receive_files` -The current log-backed business read tools have parser/source/tool registration and an empty default server source. Raw N12-pre evidence remains under ignored `runs/` paths and is not committed. The current command entry point can read an operator-local encrypted PacketReader log-line file via `ISPHERE_PACKET_LOG_FILE` or a directory of `.log`/`.txt` files via `ISPHERE_PACKET_LOG_DIR`; this is still not a production ingestion claim because raw evidence remains local and uncommitted. `isphere_receive_messages` now exposes contract-facing fields for digital-employee consumption while preserving older aliases for compatibility, supports `conversation_id`, keyword `query`, and RFC3339 `since` filtering, and validates the remaining safe contract args without pretending unsupported connectors/pagination exist. Contact and group search also accept the safe local-readonly contract args. `isphere_receive_files` supports list mode with safe contract arg validation; download/cache mapping remains deferred. +The current log-backed business read tools have parser/source/tool registration and an empty default server source. Raw N12-pre evidence remains under ignored `runs/` paths and is not committed. The current command entry point can read an operator-local encrypted PacketReader log-line file via `ISPHERE_PACKET_LOG_FILE` or a directory of `.log`/`.txt` files via `ISPHERE_PACKET_LOG_DIR`; this is still not a production ingestion claim because raw evidence remains local and uncommitted. `isphere_receive_messages` now exposes contract-facing fields for digital-employee consumption while preserving older aliases for compatibility, supports `conversation_id`, keyword `query`, and RFC3339 `since` filtering, and validates the remaining safe contract args without pretending unsupported connectors/pagination exist. Contact and group search also accept the safe local-readonly contract args and can optionally enrich display metadata from `MsgLibReadSidecar`; receive-message `sender_name` and `conversation.display_name` can use that same optional metadata when MsgLib env is configured. `isphere_receive_files` supports list mode with safe contract arg validation; download/cache mapping remains deferred. Core business tools for contacts, groups, messages, and files are defined in `docs/mcp-core-tools-contract.md`. Send capabilities enter the roadmap through that contract after source discovery, connector selection, preview metadata, execution metadata, and audit records are in place. -## 10. Real-login UIA capture procedure +## 11. Real-login UIA capture procedure This is a later real-login capture gate used to collect live UI structure for connector work. @@ -247,7 +314,7 @@ Procedure: 8. Review and redact any sensitive content before sharing reports. -## 11. If this outer-network environment cannot log in +## 12. If this outer-network environment cannot log in iSphere is internal-network only. If the current repository environment is outside that network, use two separate routes: @@ -278,7 +345,7 @@ powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\validate-n12r-retu Keep the validation report outside `$captureRoot`; the returned package is evidence input and should remain unchanged. -## 12. Troubleshooting +## 13. Troubleshooting - If C# helper build fails, run `scripts\build-win-helper.ps1` directly and check for missing .NET Framework reference assemblies. - If `win_helper_version` fails, rerun `powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-win-helper.ps1` first. diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index 58d0884..68d13cb 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | --- | --- | --- | --- | --- | --- | -| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; DB-backed message listing is still not implemented | C34 operator runbook/setup smoke | -| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C33 proves real copied-DB contact display metadata can flow into receive-message `sender_name` through MCP while output remains sanitized; search enrichment remains available | C34 operator runbook/setup smoke | -| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C33 proves real copied-DB group display metadata can flow into receive-message `conversation.display_name` through MCP while output remains sanitized; search enrichment remains available | C34 operator runbook/setup smoke | +| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; DB-backed message listing is still not implemented | C35 MsgLib message-table source precheck | +| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C35 MsgLib message-table source precheck | +| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C35 MsgLib message-table source precheck | | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence | | `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration | | `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated | @@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac ## Deferred source work -`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, and C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values. The next step is an operator runbook/setup smoke for the optional enrichment path. +`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, and C34 documents the operator setup/verification path. The next step is a MsgLib message-table source precheck for future DB-backed receive-message ingestion. diff --git a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md index a8f9a3a..cf915f5 100644 --- a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md +++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md @@ -2521,17 +2521,56 @@ Optional real receive-message display smoke result: - Optional MsgLib enrichment remains opt-in through `ISPHERE_MSGLIB_SIDECAR_EXE`, `ISPHERE_MSGLIB_SQLITE_DLL`, and `ISPHERE_MSGLIB_DB`. - Operator-facing docs must explain expected sanitized evidence and no-value output boundaries. -- [ ] **Step 1: Audit current runbook for MsgLib setup gaps** +- [x] **Step 1: Audit current runbook for MsgLib setup gaps** -Review `docs/go-mcp-runbook.md` against the C27-C33 implementation and note missing operator steps. +Confirmed `docs/go-mcp-runbook.md` only documented PacketReader source setup and deterministic `verify-go-mcp.ps1`; it did not document the optional MsgLib env shape, sidecar build, partial-config failure behavior, provider smoke, MCP enrichment smoke, or sanitized output expectations. -- [ ] **Step 2: Add operator setup and verification section** +- [x] **Step 2: Add operator setup and verification section** -Document build, env config, default deterministic verification, optional copied-DB smoke, and expected sanitized output. +Added `Configure optional MsgLib display-name enrichment` to `docs/go-mcp-runbook.md`, with sidecar build, three required env vars, optional password override, copied-file handling rules, standard no-MsgLib verification note, provider smoke command, MCP enrichment smoke command, and expected sanitized JSON booleans. -- [ ] **Step 3: Run verification and update next loop** +- [x] **Step 3: Run verification and update next loop** -Run the full gate, record C34 outcome, and choose the next business slice. +Verification passed: `git diff --check`, `go test ./...`, `scripts\verify-go-mcp.ps1`, evidence-backed `scripts\verify-msglib-sidecar.ps1`, and evidence-backed `scripts\verify-msglib-mcp-enrichment.ps1`. Optional MCP output included `receive_display_smoke=true`, `receive_sender_name_populated=true`, and `receive_conversation_display_populated=true` with no entity values, message bodies, file paths, or raw rows. + +--- + +## Loop C34 Result + +MsgLib enrichment operator runbook result: + +- `docs/go-mcp-runbook.md` now gives operators the exact optional MsgLib display-name enrichment setup path. +- It separates default deterministic verification from optional copied-DB verification. +- It documents that partial MsgLib env config fails startup and that copied DB/provider files stay local/uncommitted. +- It records expected sanitized provider/MCP evidence and no-value output boundaries. + +--- + +## Loop C35: MsgLib message-table source precheck + +**Goal:** Determine whether copied `MsgLib.db` message tables can become a future DB-backed source for `isphere_receive_messages`, starting with schema/column mapping and safe metadata-only evidence. + +**Planned files:** +- Create or modify: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md`. +- Modify: `docs/source-discovery/capability-source-matrix.md`, `docs/current-status-card.md`, and this plan. +- Do not implement message-row reads until the precheck identifies a bounded contract. + +**Planned behavior:** +- Use existing C26/C27 schema evidence and sidecar metadata operations first. +- Map candidate message tables/columns to the `isphere_receive_messages` contract fields. +- Do not print or commit message bodies, file paths, entity values, raw rows, or copied DB contents. + +- [ ] **Step 1: Collect message-table schema evidence** + +Review existing schema extraction docs and optional sidecar metadata for `tblPersonMsg`, `tblMsgGroupPersonMsg`, `tblRecent`, and related file/message tables. + +- [ ] **Step 2: Map candidate fields to receive-message contract** + +Document which fields may map to message id, conversation id/type, sender/receiver, timestamp, body/text, subject, receipt/read state, and attachment metadata; mark gaps explicitly. + +- [ ] **Step 3: Decide next implementation slice** + +Choose between a metadata-only sidecar extension, a bounded DB-backed receive source, or keeping PacketReader as primary until more evidence exists. ---