feat: add msglib bounded message listing

2026-07-10 08:54:11 +08:00
parent c0287fa2e6
commit 44d27879f6
8 changed files with 922 additions and 17 deletions
--- a/docs/current-status-card.md
+++ b/docs/current-status-card.md
@@ -29,7 +29,7 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor
 - `isphere_receive_messages` accepts the read-contract argument set for the current local-readonly path: `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, `source_preference`, `preview`, and empty `cursor`.
 - `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path and can optionally enrich results from the env-configured C29 `MsgLib.db` display-entity reader; default behavior remains log/JID-backed when MsgLib env vars are absent.
 - `isphere_receive_files` list mode accepts and validates the safe file-list contract args for the current local-readonly path while keeping download blocked.
- `native/MsgLibReadSidecar` provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks and bounded contact/group display-entity reads; `internal/msglib` wraps it from Go, and C30 wires it as an optional MCP contact/group enrichment source.
+- `native/MsgLibReadSidecar` provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks, bounded contact/group display-entity reads, message-source metadata, and sidecar-level bounded message listing; `internal/msglib` wraps it from Go, and C30 wires display metadata as an optional MCP contact/group enrichment source.
 - N12-pre and N12R documents define safe offline evidence intake and internal-sandbox live UIA capture procedures.
 - N13/N14/N15 produced selector catalog/report validation infrastructure, but those are supporting validation assets only.

@@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme

 ## Current loop

-Current loop: `Stage C / Loop C38 - sidecar bounded DB message listing`.
+Current loop: `Stage C / Loop C39 - DB-backed receive source adapter test slice`.

 Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.

@@ -106,10 +106,11 @@ Current loop output so far:
 35. C35: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` maps `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, recent, and file-record schema columns to the future `isphere_receive_messages` contract; decision is metadata-only `message_sources` sidecar first, not body-returning DB listing yet.
 36. C36: added metadata-only MsgLib `message_sources` across the x86 sidecar, Go wrapper, and sidecar verification; evidence-backed smoke returned six message-source summaries with source/table/role/availability/row-count metadata only.
 37. C37: `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md` defines the bounded DB-backed receive-message design: sidecar op `list_messages`, hard limit, no cursor initially, explicit `include_body`, no raw rows, no file path values, PacketReader remains default until reconciliation.
+38. C38: added sidecar/Go-wrapper `list_messages`, clamped to `1..50`, cursor rejected, optional attachment metadata without path/download refs, and copied-DB verification with `include_body=false`; printed evidence is only count/source-table/safety booleans.

 ## Next business mainline

-1. Run Stage C Loop C38: implement sidecar-level bounded `list_messages` plus Go wrapper tests and sanitized verification; do not wire DB rows into MCP yet.
+1. Run Stage C Loop C39: design and test a DB-backed receive source adapter around `msglib.ListMessages`, with source selection explicit and PacketReader remaining the default until reconciliation is verified.
 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated.
 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated.
 4. Keep UIA selector/report work as fallback support, not as the primary roadmap.
--- a/docs/msglib-read-sidecar-contract.md
+++ b/docs/msglib-read-sidecar-contract.md
@@ -14,8 +14,8 @@ It exists because C26 proved the database opens with the bundled 32-bit SQLite p
 - Transport: one JSON request on stdin, one JSON response on stdout.
 - Process architecture: x86.
 - Database mode: read-only only.
- Raw rows: not returned by the first contract.
- Message bodies: not returned by the first contract.
+- Raw rows: never returned.
+- Message bodies: returned only when a future caller explicitly sets `list_messages.include_body=true`; current verification and MCP defaults keep it `false`.
 - DB mutation: never allowed.

 Request envelope:
@@ -58,7 +58,7 @@ Key response fields:
 - `process_bits`
 - `requires_process_bits`
 - `db_mutation_allowed=false`
- `message_body_reads_allowed=false`
+- `message_body_reads_allowed=true` for explicit `list_messages.include_body` opt-in; verification keeps it disabled.

 ### `schema_summary`

@@ -183,6 +183,58 @@ Initial message source candidates:
 - `received_file_metadata` from `TD_ReceiveFileRecord`.
 - `recent_conversation_index` from `tblRecent`.

+### `list_messages`
+
+Same safe provider/DB validation as `message_sources`, but returns a bounded normalized message list for future DB-backed receive-message work. C38 keeps this at sidecar/Go-wrapper level only; it is not wired into MCP receive by default.
+
+Required args:
+
+- `sqlite_dll_path`
+- `db_path`
+
+Optional args:
+
+- `password`: default `123`.
+- `conversation_id`: optional metadata filter.
+- `conversation_type`: `auto`, `direct`, `group`, or `system`; default `auto`.
+- `query`: optional metadata filter. Body columns are searched only when `include_body=true`.
+- `since`: optional timestamp lower bound using the DB timestamp column representation.
+- `cursor`: unsupported initially; any non-empty value is rejected.
+- `limit`: bounded to `1..50`; default `20`.
+- `include_body`: default `false`.
+- `include_attachment_metadata`: default `false`; returns filename/size metadata only, never path values or download refs.
+
+Returned data adds:
+
+- `read_only=true`
+- `message_body_values_returned`: equals `include_body`.
+- `raw_rows_returned=false`
+- `file_paths_returned=false`
+- `source_tables[]`
+- `messages[]` entries with normalized fields:
+  - `message_id` / `id`
+  - `conversation_id`
+  - `conversation_type`
+  - `sender_id`
+  - `sender_name`
+  - `receiver_id`
+  - `text` / `content_text` (empty when `include_body=false`)
+  - `content_type`
+  - `timestamp` / `created_at`
+  - `subject`
+  - `read`
+  - `source=local_readonly`
+  - `raw_ref=msglib:<table>`
+  - `attachments[]` with `file_id`, `file_name`, `size_bytes`, and empty `download_ref`
+- `next_cursor=null`
+
+Initial allowlisted message sources:
+
+- Direct: `tblPersonMsg`.
+- Group: `tblMsgGroupPersonMsg`.
+- System: `TD_SystemMessageRecord`.
+- Attachment metadata: `TD_ReceiveFileRecord`, with `FilePath`/path-like values intentionally excluded.
+
 ## Security and scope rules

 - The sidecar must open DB files with read-only connection settings.
@@ -190,7 +242,8 @@ Initial message source candidates:
 - The schema/display-source operations return schema, columns, counts, and display-source availability only.
 - `display_entities` may return bounded contact/group `jid` and `display_name` metadata only.
 - `message_sources` may return message-source table names, required/present column names, roles, availability flags, and row counts only.
- It does not return message bodies, message text, file path values, attachment contents, or raw row sets.
+- `list_messages` may return bounded normalized message metadata. It returns `text`/`content_text` only when `include_body=true`; the standard verification path uses `include_body=false`.
+- It does not return file path values, attachment contents, raw row sets, sends, downloads, hooks, injection, or DB writes.
 - It does not implement send, file download, mark-read, login, hooks, injection, or DB writes.
 - The Go MCP layer remains the coordinator; this process is a bounded local reader.

@@ -232,6 +285,14 @@ Committed C36 implementation:
 - Verification script calls `message_sources` only when copied DB env paths are supplied, and prints only sanitized `message_source_count` plus message-source summaries containing source/table/role/availability/row-count metadata.
 - Evidence-backed C36 verification against the copied DB returned six message-source summaries without printing message body values, file path values, raw rows, or copied DB contents.

+Committed C38 implementation:
+
+- Sidecar op: `list_messages`.
+- Go wrapper: `ListMessages(ctx, ListMessagesOptions)`.
+- The op reads only allowlisted direct/group/system message tables and optional received-file metadata from a copied `MsgLib.db`, clamps `limit` to `1..50`, rejects non-empty cursor, and never returns raw rows or file path values.
+- `scripts/verify-msglib-sidecar.ps1` calls `list_messages` only when copied DB env paths are supplied, with `include_body=false`, and prints only `message_list_count`, `message_list_sources`, safety booleans, and omission booleans.
+- Evidence-backed C38 smoke returned normalized message count/source-table evidence without printing message text, body values, file paths, download refs, raw rows, sends, downloads, hooks, injection, or DB writes.
+
 ## Verification

 Default verification builds and checks `self_check` only:
@@ -257,3 +318,5 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-en
 ```

 Expected printed evidence is sanitized: contact/group counts, `msglib:<source_table>` refs, `message_source_count`, message-source table/role/row-count summaries, `receive_display_smoke=true`, populated-field booleans, and no entity values, message bodies, file path values, or raw rows.
+
+C38 provider verification additionally prints `message_list_count`, `message_list_sources`, `message_list_body_values_returned=false`, `message_list_raw_rows_returned=false`, `message_list_file_paths_returned=false`, `message_list_content_values_omitted=true`, and `message_list_download_refs_omitted=true`.
--- a/docs/source-discovery/capability-source-matrix.md
+++ b/docs/source-discovery/capability-source-matrix.md
@@ -17,7 +17,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`

 | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
 | --- | --- | --- | --- | --- | --- |
-| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design and keeps PacketReader default until reconciliation | C38 sidecar bounded DB message listing |
+| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through x86 read-only sidecar `list_messages`; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md`; `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design; C38 implements sidecar/Go-wrapper `list_messages` with sanitized copied-DB verification, while PacketReader remains MCP default until reconciliation | C39 DB-backed receive source adapter design/test slice |
 | `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
 | `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
 | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac

 ## Deferred source work

-`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, and C37 defines the bounded DB-backed receive-source design. The next step is sidecar-level `list_messages` with sanitized verification, not MCP integration yet.
+`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, C37 defines the bounded DB-backed receive-source design, and C38 implements sidecar/Go-wrapper `list_messages` with sanitized verification. The next step is a DB-backed receive source adapter test slice; do not silently merge MsgLib rows into the MCP default until PacketReader-vs-DB reconciliation and operator source-selection behavior are explicit.
--- a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md
+++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md
@@ -2679,20 +2679,60 @@ Bounded DB-backed receive-source design result:
 - First evidence-backed copied-DB smoke uses `include_body=false` and prints only counts/source refs/booleans.
 - Do not return copied-DB `MsgText`, `MessageBody`, `FilePath`, raw rows, sends, downloads, hooks, injection, or DB writes in verification output.

- [ ] **Step 1: Write failing Go wrapper test**
+- [x] **Step 1: Write failing Go wrapper test**

-Use fake sidecar to assert request args and decoded normalized message shape.
+Added `TestListMessagesPassesBoundedArgsAndDecodesNormalizedMessages`. RED failed because `Client.ListMessages` and `ListMessagesOptions` did not exist.

- [ ] **Step 2: Implement sidecar `list_messages`**
+- [x] **Step 2: Implement sidecar `list_messages`**

-Read allowlisted columns only, clamp limit, reject cursor, hide file paths, and support `include_body=false`.
+Implemented Go wrapper `ListMessages(ctx, ListMessagesOptions)` and x86 sidecar op `list_messages`. The wrapper clamps `limit` to `1..50` and rejects non-empty cursor. The sidecar reads allowlisted direct/group/system message tables, uses copied DB read-only mode, returns normalized messages, hides raw rows, and excludes file path/download refs from attachment metadata.

- [ ] **Step 3: Update verification and docs**
+- [x] **Step 3: Update verification and docs**

-Extend `verify-msglib-sidecar.ps1` with sanitized `list_messages` smoke and run full gate.
+Extended `verify-msglib-sidecar.ps1` with sanitized copied-DB `list_messages` smoke using `include_body=false`, then updated status, matrix, contract, and this plan. Full verification gate is required before commit/push.

 ---

+## Loop C38 Result
+
+Sidecar bounded DB message listing result:
+
+- Added `list_messages` to `native/MsgLibReadSidecar` and bumped sidecar version to `0.4.0`.
+- Added Go wrapper types and `Client.ListMessages(ctx, options)` in `internal/msglib`.
+- Fake-sidecar unit coverage verifies bounded args, decoded normalized messages, safety flags, and no body values when `include_body=false`.
+- Evidence-backed copied-DB smoke returned only sanitized `message_list_count`, `message_list_sources`, body/raw/file-path booleans, and omission booleans.
+- C38 does not wire MsgLib DB rows into the MCP receive default; PacketReader remains the default receive source.
+
+---
+
+## Loop C39: DB-backed receive source adapter test slice
+
+**Goal:** Add the smallest Go adapter/test layer that can translate `msglib.ListMessages` results into the existing receive-message domain model, while keeping MCP source selection explicit and PacketReader default unchanged.
+
+**Planned files:**
+- Modify or create: `internal/isphere` DB-backed source adapter files/tests, or a narrow `internal/tools` adapter if the existing receive model is tool-local.
+- Modify: `internal/msglib` only if C38 return shape exposes a decoding gap.
+- Modify docs/status/matrix/plan after implementation.
+
+**Planned behavior:**
+- Use an injected/fake MsgLib list provider first; do not require real DB in unit tests.
+- Map normalized MsgLib fields to the current `isphere_receive_messages` envelope without returning raw rows or file paths.
+- Keep source selection explicit. Do not silently merge MsgLib rows into default MCP receive output.
+- Verification should continue to clear MsgLib env in the standard `verify-go-mcp.ps1`; optional copied-DB verification remains separate.
+
+- [ ] **Step 1: Locate current receive-message model boundary**
+
+Inspect `internal/isphere` and `internal/tools` receive-message code and decide whether the adapter belongs below the tool layer or inside the tool registration layer.
+
+- [ ] **Step 2: Write failing adapter test**
+
+Use a fake MsgLib list provider returning redacted normalized messages and assert mapped receive-message fields, source refs, attachment metadata behavior, and explicit source-selection handling.
+
+- [ ] **Step 3: Implement minimal adapter and update docs**
+
+Implement only enough code to pass the adapter test; defer MCP default wiring until reconciliation/source-selection behavior is proven.
+
+---
 ## Self-Review

 - Spec coverage: the plan follows the user's cyclic requirement: plan first, implement one loop at a time, update the next loop after each result, and ask only on blockers.