docs: add msglib enrichment runbook

2026-07-10 08:27:32 +08:00
parent 07d127c44a
commit 78f0e47547
4 changed files with 127 additions and 20 deletions
--- a/docs/current-status-card.md
+++ b/docs/current-status-card.md
@@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme

 ## Current loop

-Current loop: `Stage C / Loop C34 - MsgLib enrichment operator runbook and setup smoke`.
+Current loop: `Stage C / Loop C35 - MsgLib message-table source precheck`.

 Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.

@@ -102,10 +102,11 @@ Current loop output so far:
 31. C31: added optional `scripts/verify-msglib-mcp-enrichment.ps1`; real copied-DB MCP smoke returned sanitized contact/group counts and MsgLib source refs without printing names, JIDs, message bodies, file paths, or raw rows.
 32. C32: `isphere_receive_messages` can now use the optional MsgLib display-entity source to fill safe `sender_name` and `conversation.display_name` fields while preserving message text/content/attachment behavior.
 33. C33: optional real copied-DB MCP smoke now also constructs a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verifies receive-message display enrichment through MCP; output stays sanitized to counts, booleans, and `msglib:<source_table>` refs only.
+34. C34: `docs/go-mcp-runbook.md` now documents the operator setup path for optional MsgLib display-name enrichment, including the required env vars, partial-config failure rule, standard no-MsgLib verification, optional provider smoke, optional MCP enrichment smoke, and sanitized output boundaries.

 ## Next business mainline

-1. Run Stage C Loop C34: document and verify the operator setup path for enabling optional MsgLib enrichment in MCP without exposing local names, JIDs, file paths, message bodies, or raw rows.
+1. Run Stage C Loop C35: precheck whether copied `MsgLib.db` message tables can support a future DB-backed `isphere_receive_messages` source, starting with schema/column mapping only and no message-body or raw-row output.
 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated.
 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated.
 4. Keep UIA selector/report work as fallback support, not as the primary roadmap.
--- a/docs/go-mcp-runbook.md
+++ b/docs/go-mcp-runbook.md
@@ -2,7 +2,7 @@

 This runbook is for the first Go MCP phase of `isphere-ai-bridge`. It lets an operator build and verify the Windows helper and the Go MCP server without reading the source code.

-Current scope: expose four WinHelper observation operations plus four log-backed business read tools through Go MCP: `isphere_receive_messages`, `isphere_search_contacts`, `isphere_search_groups`, and `isphere_receive_files` list mode. These business read tools use an empty default source unless `ISPHERE_PACKET_LOG_FILE` points to an operator-local encrypted PacketReader log-line file or `ISPHERE_PACKET_LOG_DIR` points to an operator-local directory of encrypted PacketReader `.log`/`.txt` files.
+Current scope: expose four WinHelper observation operations plus four business read tools through Go MCP: `isphere_receive_messages`, `isphere_search_contacts`, `isphere_search_groups`, and `isphere_receive_files` list mode. These business read tools use an empty default message source unless `ISPHERE_PACKET_LOG_FILE` points to an operator-local encrypted PacketReader log-line file or `ISPHERE_PACKET_LOG_DIR` points to an operator-local directory of encrypted PacketReader `.log`/`.txt` files. Contact/group display names can also be optionally enriched from an operator-local copied `MsgLib.db` through the bounded read-only `MsgLibReadSidecar`.

 ## 1. Prerequisites

@@ -100,6 +100,7 @@ The verification confirms:
 - `isphere_search_groups` is callable and returns JID-derived group candidates from the configured synthetic groupchat fixture.
 - `isphere_receive_files` is callable in list mode and returns one file metadata candidate from the configured synthetic file-transfer fixture.
 - Verification uses synthetic/local helper checks and does not load raw N12-pre evidence.
+- The deterministic verification path clears MsgLib env variables, so it proves the default no-MsgLib behavior remains stable.
 - File download/cache mapping and send business tools remain later-stage work.

 ## 6. Configure optional message source
@@ -128,7 +129,73 @@ The directory loader recursively reads `.log` and `.txt` files, sorts paths dete

 Then start the MCP server normally. The command entry point uses `ISPHERE_PACKET_LOG_FILE` when the file variable is set, otherwise `ISPHERE_PACKET_LOG_DIR` when the directory variable is set. The repeatable verification script checks three paths: default empty source, a temporary synthetic encrypted fixture file, and a temporary synthetic encrypted fixture directory.

-## 7. Configure MCP client command
+## 7. Configure optional MsgLib display-name enrichment
+
+This is optional. It enriches contact/group display metadata; it does not turn `MsgLib.db` into the primary message source. `isphere_receive_messages` still needs a PacketReader source to return messages, and MsgLib only fills safe display fields such as `sender_name` and `conversation.display_name`.
+
+Build the bounded x86 sidecar first:
+
+```powershell
+powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build-msglib-sidecar.ps1
+```
+
+Then configure all three MsgLib variables before starting the MCP server:
+
+```powershell
+$env:ISPHERE_MSGLIB_SIDECAR_EXE = "E:\coding\codex\isphere-ai-bridge\runs\msglib-sidecar\MsgLibReadSidecar.exe"
+$env:ISPHERE_MSGLIB_SQLITE_DLL = "<path-to-copied-System.Data.SQLite.dll>"
+$env:ISPHERE_MSGLIB_DB = "<path-to-copied-MsgLib.db>"
+```
+
+Optional password override:
+
+```powershell
+$env:ISPHERE_MSGLIB_PASSWORD = "123"
+```
+
+Rules:
+
+- Set all of `ISPHERE_MSGLIB_SIDECAR_EXE`, `ISPHERE_MSGLIB_SQLITE_DLL`, and `ISPHERE_MSGLIB_DB`, or set none of them. Partial MsgLib config fails server startup.
+- Keep copied DB/provider files under ignored `runs/` paths or another operator-local location. Do not commit them.
+- `MsgLibReadSidecar` is read-only and bounded to schema/display metadata operations. It does not send messages, download files, mutate DB rows, log in, hook processes, or return raw rows.
+- Standard `scripts\verify-go-mcp.ps1` intentionally clears MsgLib variables to keep the default smoke deterministic.
+
+Verify the optional provider path:
+
+```powershell
+$env:ISPHERE_MSGLIB_SQLITE_DLL = "<path-to-copied-System.Data.SQLite.dll>"
+$env:ISPHERE_MSGLIB_DB = "<path-to-copied-MsgLib.db>"
+powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1
+```
+
+Expected provider evidence contains `provider_checked=true`, a `display_source_count`, and sanitized `display_entity_summaries` with counts/source table names only.
+
+Verify MCP-level optional enrichment, including receive-message display fields:
+
+```powershell
+$env:ISPHERE_MSGLIB_SQLITE_DLL = "<path-to-copied-System.Data.SQLite.dll>"
+$env:ISPHERE_MSGLIB_DB = "<path-to-copied-MsgLib.db>"
+powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-enrichment.ps1
+```
+
+Expected MCP evidence is sanitized and should include:
+
+```json
+{
+  "ok": true,
+  "receive_display_smoke": true,
+  "receive_sender_name_populated": true,
+  "receive_conversation_display_populated": true,
+  "printed_entity_values": false,
+  "message_bodies_returned": false,
+  "file_paths_returned": false,
+  "raw_rows_returned": false
+}
+```
+
+The optional MCP smoke internally uses copied-DB display metadata to build a synthetic encrypted PacketReader fixture, but the printed output must stay limited to counts, booleans, and `msglib:<source_table>` refs. It must not print entity values, JIDs, message bodies, local DB paths, attachment paths, or raw rows.
+
+## 8. Configure MCP client command

 After building a stable binary, configure your MCP client to run the Go MCP executable as a stdio server.

@@ -163,7 +230,7 @@ If you used `go build ./cmd/isphere-mcp` without `-o`, point the command to:
 E:\coding\codex\isphere-ai-bridge\isphere-mcp.exe
 ```

-## 8. Allowed tools
+## 9. Allowed tools

 These eight tools are currently allowed:

@@ -189,7 +256,7 @@ Allowed parameters:
 - `isphere_search_groups`: `query`, `cursor`, `source_preference`, `include_archived`, `limit` only. `source_preference` currently supports empty/`auto`/`local_readonly`; `cursor` must be empty until pagination is implemented.
 - `isphere_receive_files`: `conversation_id`, `message_id`, `file_id`, `name_contains`, `mode`, `output_dir`, `cursor`, `source_preference`, `preview`, `limit` only. C22 supports `mode` empty or `list`; `source_preference` currently supports empty/`auto`/`local_readonly`; `cursor` and `output_dir` must be empty in current list-only mode; download is deferred.

-## 9. Current tool surface and next-stage route
+## 10. Current tool surface and next-stage route

 The current runbook verifies eight tools:

@@ -202,11 +269,11 @@ The current runbook verifies eight tools:
 - `isphere_search_groups`
 - `isphere_receive_files`

-The current log-backed business read tools have parser/source/tool registration and an empty default server source. Raw N12-pre evidence remains under ignored `runs/` paths and is not committed. The current command entry point can read an operator-local encrypted PacketReader log-line file via `ISPHERE_PACKET_LOG_FILE` or a directory of `.log`/`.txt` files via `ISPHERE_PACKET_LOG_DIR`; this is still not a production ingestion claim because raw evidence remains local and uncommitted. `isphere_receive_messages` now exposes contract-facing fields for digital-employee consumption while preserving older aliases for compatibility, supports `conversation_id`, keyword `query`, and RFC3339 `since` filtering, and validates the remaining safe contract args without pretending unsupported connectors/pagination exist. Contact and group search also accept the safe local-readonly contract args. `isphere_receive_files` supports list mode with safe contract arg validation; download/cache mapping remains deferred.
+The current log-backed business read tools have parser/source/tool registration and an empty default server source. Raw N12-pre evidence remains under ignored `runs/` paths and is not committed. The current command entry point can read an operator-local encrypted PacketReader log-line file via `ISPHERE_PACKET_LOG_FILE` or a directory of `.log`/`.txt` files via `ISPHERE_PACKET_LOG_DIR`; this is still not a production ingestion claim because raw evidence remains local and uncommitted. `isphere_receive_messages` now exposes contract-facing fields for digital-employee consumption while preserving older aliases for compatibility, supports `conversation_id`, keyword `query`, and RFC3339 `since` filtering, and validates the remaining safe contract args without pretending unsupported connectors/pagination exist. Contact and group search also accept the safe local-readonly contract args and can optionally enrich display metadata from `MsgLibReadSidecar`; receive-message `sender_name` and `conversation.display_name` can use that same optional metadata when MsgLib env is configured. `isphere_receive_files` supports list mode with safe contract arg validation; download/cache mapping remains deferred.

 Core business tools for contacts, groups, messages, and files are defined in `docs/mcp-core-tools-contract.md`. Send capabilities enter the roadmap through that contract after source discovery, connector selection, preview metadata, execution metadata, and audit records are in place.

-## 10. Real-login UIA capture procedure
+## 11. Real-login UIA capture procedure

 This is a later real-login capture gate used to collect live UI structure for connector work.

@@ -247,7 +314,7 @@ Procedure:

 8. Review and redact any sensitive content before sharing reports.

-## 11. If this outer-network environment cannot log in
+## 12. If this outer-network environment cannot log in

 iSphere is internal-network only. If the current repository environment is outside that network, use two separate routes:

@@ -278,7 +345,7 @@ powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\validate-n12r-retu

 Keep the validation report outside `$captureRoot`; the returned package is evidence input and should remain unchanged.

-## 12. Troubleshooting
+## 13. Troubleshooting

 - If C# helper build fails, run `scripts\build-win-helper.ps1` directly and check for missing .NET Framework reference assemblies.
 - If `win_helper_version` fails, rerun `powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-win-helper.ps1` first.
--- a/docs/source-discovery/capability-source-matrix.md
+++ b/docs/source-discovery/capability-source-matrix.md
@@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`

 | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
 | --- | --- | --- | --- | --- | --- |
-| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; DB-backed message listing is still not implemented | C34 operator runbook/setup smoke |
-| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C33 proves real copied-DB contact display metadata can flow into receive-message `sender_name` through MCP while output remains sanitized; search enrichment remains available | C34 operator runbook/setup smoke |
-| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C33 proves real copied-DB group display metadata can flow into receive-message `conversation.display_name` through MCP while output remains sanitized; search enrichment remains available | C34 operator runbook/setup smoke |
+| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; DB-backed message listing is still not implemented | C35 MsgLib message-table source precheck |
+| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C35 MsgLib message-table source precheck |
+| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C35 MsgLib message-table source precheck |
 | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
 | `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration |
 | `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated |
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac

 ## Deferred source work

-`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, and C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values. The next step is an operator runbook/setup smoke for the optional enrichment path.
+`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, and C34 documents the operator setup/verification path. The next step is a MsgLib message-table source precheck for future DB-backed receive-message ingestion.
--- a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md
+++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md
@@ -2521,17 +2521,56 @@ Optional real receive-message display smoke result:
 - Optional MsgLib enrichment remains opt-in through `ISPHERE_MSGLIB_SIDECAR_EXE`, `ISPHERE_MSGLIB_SQLITE_DLL`, and `ISPHERE_MSGLIB_DB`.
 - Operator-facing docs must explain expected sanitized evidence and no-value output boundaries.

- [ ] **Step 1: Audit current runbook for MsgLib setup gaps**
+- [x] **Step 1: Audit current runbook for MsgLib setup gaps**

-Review `docs/go-mcp-runbook.md` against the C27-C33 implementation and note missing operator steps.
+Confirmed `docs/go-mcp-runbook.md` only documented PacketReader source setup and deterministic `verify-go-mcp.ps1`; it did not document the optional MsgLib env shape, sidecar build, partial-config failure behavior, provider smoke, MCP enrichment smoke, or sanitized output expectations.

- [ ] **Step 2: Add operator setup and verification section**
+- [x] **Step 2: Add operator setup and verification section**

-Document build, env config, default deterministic verification, optional copied-DB smoke, and expected sanitized output.
+Added `Configure optional MsgLib display-name enrichment` to `docs/go-mcp-runbook.md`, with sidecar build, three required env vars, optional password override, copied-file handling rules, standard no-MsgLib verification note, provider smoke command, MCP enrichment smoke command, and expected sanitized JSON booleans.

- [ ] **Step 3: Run verification and update next loop**
+- [x] **Step 3: Run verification and update next loop**

-Run the full gate, record C34 outcome, and choose the next business slice.
+Verification passed: `git diff --check`, `go test ./...`, `scripts\verify-go-mcp.ps1`, evidence-backed `scripts\verify-msglib-sidecar.ps1`, and evidence-backed `scripts\verify-msglib-mcp-enrichment.ps1`. Optional MCP output included `receive_display_smoke=true`, `receive_sender_name_populated=true`, and `receive_conversation_display_populated=true` with no entity values, message bodies, file paths, or raw rows.
+
+---
+
+## Loop C34 Result
+
+MsgLib enrichment operator runbook result:
+
+- `docs/go-mcp-runbook.md` now gives operators the exact optional MsgLib display-name enrichment setup path.
+- It separates default deterministic verification from optional copied-DB verification.
+- It documents that partial MsgLib env config fails startup and that copied DB/provider files stay local/uncommitted.
+- It records expected sanitized provider/MCP evidence and no-value output boundaries.
+
+---
+
+## Loop C35: MsgLib message-table source precheck
+
+**Goal:** Determine whether copied `MsgLib.db` message tables can become a future DB-backed source for `isphere_receive_messages`, starting with schema/column mapping and safe metadata-only evidence.
+
+**Planned files:**
+- Create or modify: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md`.
+- Modify: `docs/source-discovery/capability-source-matrix.md`, `docs/current-status-card.md`, and this plan.
+- Do not implement message-row reads until the precheck identifies a bounded contract.
+
+**Planned behavior:**
+- Use existing C26/C27 schema evidence and sidecar metadata operations first.
+- Map candidate message tables/columns to the `isphere_receive_messages` contract fields.
+- Do not print or commit message bodies, file paths, entity values, raw rows, or copied DB contents.
+
+- [ ] **Step 1: Collect message-table schema evidence**
+
+Review existing schema extraction docs and optional sidecar metadata for `tblPersonMsg`, `tblMsgGroupPersonMsg`, `tblRecent`, and related file/message tables.
+
+- [ ] **Step 2: Map candidate fields to receive-message contract**
+
+Document which fields may map to message id, conversation id/type, sender/receiver, timestamp, body/text, subject, receipt/read state, and attachment metadata; mark gaps explicitly.
+
+- [ ] **Step 3: Decide next implementation slice**
+
+Choose between a metadata-only sidecar extension, a bounded DB-backed receive source, or keeping PacketReader as primary until more evidence exists.

 ---