docs: design bounded db receive source

docs/current-status-card.md

@@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme
 
 ## Current loop
 
-Current loop: `Stage C / Loop C37 - bounded DB-backed receive-source design`.
+Current loop: `Stage C / Loop C38 - sidecar bounded DB message listing`.
 
 Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.
 
@@ -105,10 +105,11 @@ Current loop output so far:
 34. C34: `docs/go-mcp-runbook.md` now documents the operator setup path for optional MsgLib display-name enrichment, including the required env vars, partial-config failure rule, standard no-MsgLib verification, optional provider smoke, optional MCP enrichment smoke, and sanitized output boundaries.
 35. C35: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` maps `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, recent, and file-record schema columns to the future `isphere_receive_messages` contract; decision is metadata-only `message_sources` sidecar first, not body-returning DB listing yet.
 36. C36: added metadata-only MsgLib `message_sources` across the x86 sidecar, Go wrapper, and sidecar verification; evidence-backed smoke returned six message-source summaries with source/table/role/availability/row-count metadata only.
+37. C37: `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md` defines the bounded DB-backed receive-message design: sidecar op `list_messages`, hard limit, no cursor initially, explicit `include_body`, no raw rows, no file path values, PacketReader remains default until reconciliation.
 
 ## Next business mainline
 
-1. Run Stage C Loop C37: design the bounded DB-backed receive-message source contract now that `message_sources` metadata is available; do not expose body rows until strict limits and output rules are specified.
+1. Run Stage C Loop C38: implement sidecar-level bounded `list_messages` plus Go wrapper tests and sanitized verification; do not wire DB rows into MCP yet.
 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated.
 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated.
 4. Keep UIA selector/report work as fallback support, not as the primary roadmap.

docs/source-discovery/2026-07-10-db-backed-receive-source-design.md

@@ -0,0 +1,148 @@
+# Bounded DB-backed receive-message source design
+
+Date: 2026-07-10
+
+## Goal
+
+Design the smallest safe implementation slice for using copied/operator-local `MsgLib.db` as a future source for `isphere_receive_messages`.
+
+This is a design node only. It does not add row-reading code and does not change the MCP tool surface.
+
+## Inputs
+
+Evidence now available:
+
+- C35 mapped `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, file-record, and recent-index tables to the receive-message contract.
+- C36 added metadata-only `message_sources`, proving candidate table availability, required/present columns, roles, and row counts without exposing message bodies, file path values, or raw rows.
+- Current MCP `isphere_receive_messages` is PacketReader-log backed and already supports `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, safe `source_preference`, and empty `cursor`.
+
+## Non-goals
+
+C37 and the next implementation slice must not:
+
+- Add send, download, mark-read, login, hook, injection, or DB mutation behavior.
+- Return raw DB rows.
+- Return local `FilePath` values from file-record tables.
+- Replace PacketReader as the only receive source.
+- Claim production ingestion completeness.
+- Print copied-DB message bodies in evidence-backed smoke output.
+
+## Proposed sidecar operation: `list_messages`
+
+Add a future sidecar operation named `list_messages` under protocol `isphere.msglib.v1`.
+
+### Required args
+
+- `sqlite_dll_path`
+- `db_path`
+
+### Optional args
+
+- `password`: default `123`.
+- `conversation_id`: exact normalized conversation id or DB conversation key.
+- `conversation_type`: `direct`, `group`, `system`, or empty/`auto`.
+- `query`: bounded keyword filter over allowlisted columns only.
+- `since`: ISO-8601/RFC3339 timestamp.
+- `limit`: default `20`, clamp to `1..50` for the first DB row-listing slice.
+- `cursor`: must be empty in the first slice.
+- `include_body`: default `false` in sidecar verification; MCP integration may explicitly set this later after redacted/synthetic tests pass.
+- `include_attachment_metadata`: default `true`; returns only safe attachment fields.
+
+### Returned top-level data
+
+- `read_only=true`
+- `raw_rows_returned=false`
+- `file_paths_returned=false`
+- `message_body_values_returned=<include_body>`
+- `source_tables`: unique allowlisted tables used.
+- `messages`: normalized bounded message objects.
+- `next_cursor=null` in the first slice.
+
+### Normalized message fields
+
+| Field | Required behavior |
+| --- | --- |
+| `message_id` / `id` | From table primary key: `tblPersonMsg.Id`, `tblMsgGroupPersonMsg.Id`, or `TD_SystemMessageRecord.MsgGuid`. |
+| `conversation_id` | Direct: prefer `ObjPersonId`, fallback `SndPersonId|RecvPersonId`; Group: `MsgGroupId`; System: stable `system` or sender-derived key. |
+| `conversation_type` | `direct`, `group`, or `system`. |
+| `sender_id` | Direct/group: `SndPersonId`; System: `SendPersonJid`. |
+| `sender_name` | Direct/group: `SndPerson`; System: `SendPersonName`; later can be overwritten/enriched by display-entity maps. |
+| `receiver_id` | Direct: `RecvPersonId`; Group: `MsgGroupId`; System: null or stable system receiver. |
+| `text` / `content_text` | Only when `include_body=true`; select `MsgText` first, fallback `MessageBody`; system messages use `MsgText`. |
+| `content_type` | Derived from `MsgType` and safe attachment metadata; unknown when mapping is unclear. |
+| `subject` | System: `MsgTitle`; direct/group: null until a DB column equivalent is proven. |
+| `timestamp` / `created_at` | Direct: `ActionTime`; Group: `MsgTime`; System: `MsgTime`; parse into RFC3339 where possible. |
+| `read` | Direct/group/system read-state columns mapped to boolean-like value when possible. |
+| `receipt_id` | Null in first slice unless a safe receipt id mapping is proven; DB receipt state may remain in internal metadata only. |
+| `attachments` | Join `TD_ReceiveFileRecord` by message id; include `file_id`, `file_name`, `size_bytes`, and null `download_ref`; never return `FilePath`. |
+| `source` | `local_readonly`. |
+| `raw_ref` | A stable non-row-dump ref such as `msglib:tblPersonMsg`, `msglib:tblMsgGroupPersonMsg`, or `msglib:TD_SystemMessageRecord`. |
+
+## Query and ordering policy
+
+First DB-backed row-listing slice should:
+
+1. Read only allowlisted columns.
+2. Filter source tables by `conversation_type` before querying rows when supplied.
+3. Filter `conversation_id` against the normalized candidate conversation key.
+4. Filter `since` on `ActionTime`, `MsgTime`, or system `MsgTime`.
+5. Filter `query` only over allowlisted identity/title/body columns; if `include_body=false`, query may still use body columns internally but must not print the values.
+6. Sort descending by timestamp, then descending by primary key/id for deterministic output.
+7. Clamp `limit` to at most 50.
+8. Reject non-empty `cursor` until pagination is explicitly implemented.
+
+## Body-output policy
+
+The first sidecar implementation may support `include_body`, but verification must separate two paths:
+
+1. **Default/evidence-backed real copied DB smoke:** `include_body=false`; output summaries only, no message body values.
+2. **Synthetic/redacted fixture test path:** `include_body=true` may be tested with redacted fixture values only.
+
+Before MCP integration sets `include_body=true` against operator-local data, acceptance must prove:
+
+- strict limit is enforced;
+- output contains only normalized message objects, not raw rows;
+- file path values are always omitted;
+- audit/source fields make the DB source explicit;
+- user/operator local data remains uncommitted.
+
+## Attachment policy
+
+Attachment metadata can use `TD_ReceiveFileRecord` only as a safe list source:
+
+- Join key: `ReceiveMsgGuid` equals normalized message id.
+- Return allowed: `ReceiveMsgGuid` as file/message ref, `FileName`, `FileSize`, `SourceID`, `SourceName`, `FileType`, `SendPersonJid`, `SendPersonName`, `SendTime` only if needed for normalized metadata.
+- Never return: `FilePath`.
+- `download_ref` remains null until cache/download mapping is separately validated.
+
+## Relationship with PacketReader source
+
+PacketReader remains the default receive source until DB-vs-log reconciliation is tested. DB-backed receive should enter in stages:
+
+1. C38: sidecar-level bounded `list_messages` operation, with real copied DB smoke using `include_body=false` only.
+2. C39: Go wrapper/fake-sidecar tests and possibly synthetic/redacted fixture flow for `include_body=true`.
+3. C40: optional Go `ReceiveMessagesSource` implementation behind explicit env/source configuration.
+4. Later: decide `auto` merge/reconciliation policy between PacketReader and MsgLib.
+
+The first MCP integration should not silently merge PacketReader and MsgLib rows. It should be explicit through env config or a source-mode decision recorded in docs.
+
+## C38 recommended implementation route
+
+Implement sidecar `list_messages` first, not MCP integration.
+
+C38 acceptance gate:
+
+- Add Go fake-sidecar tests for `ListMessages(ctx, options)` request shape and decoded normalized fields.
+- Add sidecar op `list_messages` with allowlisted table/column queries.
+- Default sidecar smoke still returns no DB provider checks unless env is configured.
+- Evidence-backed copied DB smoke calls `list_messages` with `include_body=false` and prints only:
+  - `message_list_count`
+  - `message_list_sources`
+  - `message_body_values_returned=false`
+  - `raw_rows_returned=false`
+  - `file_paths_returned=false`
+- No real copied-DB `MsgText`, `MessageBody`, `FilePath`, raw rows, entity values, sends, downloads, hooks, injection, or DB writes.
+
+## Decision
+
+Proceed to C38 with sidecar-level `list_messages` plus Go wrapper tests and sanitized verification. Do not wire DB-backed rows into `isphere_receive_messages` MCP until C38 proves the bounded sidecar behavior.

docs/source-discovery/capability-source-matrix.md

@@ -17,7 +17,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
 
 | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
 | --- | --- | --- | --- | --- | --- |
-| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources` readiness with table/column/count summaries only | C37 bounded DB-backed receive-source design |
+| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design and keeps PacketReader default until reconciliation | C38 sidecar bounded DB message listing |
 | `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
 | `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
 | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
 
 ## Deferred source work
 
-`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, and C36 adds metadata-only `message_sources` readiness. The next step is bounded DB-backed receive-source design.
+`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, and C37 defines the bounded DB-backed receive-source design. The next step is sidecar-level `list_messages` with sanitized verification, not MCP integration yet.

docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md

@@ -2638,17 +2638,58 @@ MsgLib message-source metadata sidecar result:
 - Define direct/group source selection, ordering, limit, body-column policy, attachment metadata policy, and fallback behavior versus PacketReader.
 - Decide whether C38 should add sidecar-level bounded row listing or first add a redacted synthetic DB fixture.
 
-- [ ] **Step 1: Write bounded DB receive-source contract**
+- [x] **Step 1: Write bounded DB receive-source contract**
 
-Define request args, returned fields, limits, and no-output boundaries for DB-backed receive listing.
+Created `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md`. It defines sidecar op `list_messages`, args, return fields, hard limit `1..50`, no cursor initially, `include_body` policy, attachment policy, no raw rows, no file path values, and PacketReader remains default until reconciliation.
 
-- [ ] **Step 2: Choose implementation route**
+- [x] **Step 2: Choose implementation route**
 
-Select between sidecar row listing, Go source abstraction first, or synthetic fixture-first tests.
+Decision: C38 should implement sidecar-level `list_messages` plus Go wrapper fake-sidecar tests and sanitized evidence-backed verification. Do not wire DB rows into `isphere_receive_messages` MCP in C38.
 
-- [ ] **Step 3: Update next implementation loop**
+- [x] **Step 3: Update next implementation loop**
 
-Record C37 decision and prepare C38.
+Updated matrix/status/plan. Verification for this design-only loop will use `git diff --check` and `go test ./...` before commit.
+
+---
+
+## Loop C37 Result
+
+Bounded DB-backed receive-source design result:
+
+- Created `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md`.
+- Selected future sidecar op `list_messages` with strict limit, empty cursor initially, explicit `include_body`, safe attachment metadata, no raw rows, and no file path values.
+- PacketReader remains the default receive source; MsgLib DB rows should not silently merge into MCP until reconciliation is tested.
+- C38 should implement sidecar-level bounded listing and sanitized verification only; no MCP integration yet.
+
+---
+
+## Loop C38: Sidecar bounded DB message listing
+
+**Goal:** Implement sidecar-level `list_messages` for copied/operator-local MsgLib DBs with strict limits and sanitized verification, without wiring DB rows into MCP yet.
+
+**Planned files:**
+- Modify: `internal/msglib/sidecar_client.go` and tests.
+- Modify: `native/MsgLibReadSidecar/Program.cs`.
+- Modify: `scripts/verify-msglib-sidecar.ps1`.
+- Modify docs/status/matrix/plan.
+
+**Planned behavior:**
+- Add Go wrapper such as `ListMessages(ctx, options)`.
+- Add sidecar op `list_messages`.
+- First evidence-backed copied-DB smoke uses `include_body=false` and prints only counts/source refs/booleans.
+- Do not return copied-DB `MsgText`, `MessageBody`, `FilePath`, raw rows, sends, downloads, hooks, injection, or DB writes in verification output.
+
+- [ ] **Step 1: Write failing Go wrapper test**
+
+Use fake sidecar to assert request args and decoded normalized message shape.
+
+- [ ] **Step 2: Implement sidecar `list_messages`**
+
+Read allowlisted columns only, clamp limit, reject cursor, hide file paths, and support `include_body=false`.
+
+- [ ] **Step 3: Update verification and docs**
+
+Extend `verify-msglib-sidecar.ps1` with sanitized `list_messages` smoke and run full gate.
 
 ---