diff --git a/docs/current-status-card.md b/docs/current-status-card.md index a04e461..dbb3386 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Stage C / Loop C37 - bounded DB-backed receive-source design`. +Current loop: `Stage C / Loop C38 - sidecar bounded DB message listing`. Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. @@ -105,10 +105,11 @@ Current loop output so far: 34. C34: `docs/go-mcp-runbook.md` now documents the operator setup path for optional MsgLib display-name enrichment, including the required env vars, partial-config failure rule, standard no-MsgLib verification, optional provider smoke, optional MCP enrichment smoke, and sanitized output boundaries. 35. C35: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` maps `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, recent, and file-record schema columns to the future `isphere_receive_messages` contract; decision is metadata-only `message_sources` sidecar first, not body-returning DB listing yet. 36. C36: added metadata-only MsgLib `message_sources` across the x86 sidecar, Go wrapper, and sidecar verification; evidence-backed smoke returned six message-source summaries with source/table/role/availability/row-count metadata only. +37. C37: `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md` defines the bounded DB-backed receive-message design: sidecar op `list_messages`, hard limit, no cursor initially, explicit `include_body`, no raw rows, no file path values, PacketReader remains default until reconciliation. ## Next business mainline -1. Run Stage C Loop C37: design the bounded DB-backed receive-message source contract now that `message_sources` metadata is available; do not expose body rows until strict limits and output rules are specified. +1. Run Stage C Loop C38: implement sidecar-level bounded `list_messages` plus Go wrapper tests and sanitized verification; do not wire DB rows into MCP yet. 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated. 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated. 4. Keep UIA selector/report work as fallback support, not as the primary roadmap. diff --git a/docs/source-discovery/2026-07-10-db-backed-receive-source-design.md b/docs/source-discovery/2026-07-10-db-backed-receive-source-design.md new file mode 100644 index 0000000..5a7fa15 --- /dev/null +++ b/docs/source-discovery/2026-07-10-db-backed-receive-source-design.md @@ -0,0 +1,148 @@ +# Bounded DB-backed receive-message source design + +Date: 2026-07-10 + +## Goal + +Design the smallest safe implementation slice for using copied/operator-local `MsgLib.db` as a future source for `isphere_receive_messages`. + +This is a design node only. It does not add row-reading code and does not change the MCP tool surface. + +## Inputs + +Evidence now available: + +- C35 mapped `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, file-record, and recent-index tables to the receive-message contract. +- C36 added metadata-only `message_sources`, proving candidate table availability, required/present columns, roles, and row counts without exposing message bodies, file path values, or raw rows. +- Current MCP `isphere_receive_messages` is PacketReader-log backed and already supports `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, safe `source_preference`, and empty `cursor`. + +## Non-goals + +C37 and the next implementation slice must not: + +- Add send, download, mark-read, login, hook, injection, or DB mutation behavior. +- Return raw DB rows. +- Return local `FilePath` values from file-record tables. +- Replace PacketReader as the only receive source. +- Claim production ingestion completeness. +- Print copied-DB message bodies in evidence-backed smoke output. + +## Proposed sidecar operation: `list_messages` + +Add a future sidecar operation named `list_messages` under protocol `isphere.msglib.v1`. + +### Required args + +- `sqlite_dll_path` +- `db_path` + +### Optional args + +- `password`: default `123`. +- `conversation_id`: exact normalized conversation id or DB conversation key. +- `conversation_type`: `direct`, `group`, `system`, or empty/`auto`. +- `query`: bounded keyword filter over allowlisted columns only. +- `since`: ISO-8601/RFC3339 timestamp. +- `limit`: default `20`, clamp to `1..50` for the first DB row-listing slice. +- `cursor`: must be empty in the first slice. +- `include_body`: default `false` in sidecar verification; MCP integration may explicitly set this later after redacted/synthetic tests pass. +- `include_attachment_metadata`: default `true`; returns only safe attachment fields. + +### Returned top-level data + +- `read_only=true` +- `raw_rows_returned=false` +- `file_paths_returned=false` +- `message_body_values_returned=` +- `source_tables`: unique allowlisted tables used. +- `messages`: normalized bounded message objects. +- `next_cursor=null` in the first slice. + +### Normalized message fields + +| Field | Required behavior | +| --- | --- | +| `message_id` / `id` | From table primary key: `tblPersonMsg.Id`, `tblMsgGroupPersonMsg.Id`, or `TD_SystemMessageRecord.MsgGuid`. | +| `conversation_id` | Direct: prefer `ObjPersonId`, fallback `SndPersonId|RecvPersonId`; Group: `MsgGroupId`; System: stable `system` or sender-derived key. | +| `conversation_type` | `direct`, `group`, or `system`. | +| `sender_id` | Direct/group: `SndPersonId`; System: `SendPersonJid`. | +| `sender_name` | Direct/group: `SndPerson`; System: `SendPersonName`; later can be overwritten/enriched by display-entity maps. | +| `receiver_id` | Direct: `RecvPersonId`; Group: `MsgGroupId`; System: null or stable system receiver. | +| `text` / `content_text` | Only when `include_body=true`; select `MsgText` first, fallback `MessageBody`; system messages use `MsgText`. | +| `content_type` | Derived from `MsgType` and safe attachment metadata; unknown when mapping is unclear. | +| `subject` | System: `MsgTitle`; direct/group: null until a DB column equivalent is proven. | +| `timestamp` / `created_at` | Direct: `ActionTime`; Group: `MsgTime`; System: `MsgTime`; parse into RFC3339 where possible. | +| `read` | Direct/group/system read-state columns mapped to boolean-like value when possible. | +| `receipt_id` | Null in first slice unless a safe receipt id mapping is proven; DB receipt state may remain in internal metadata only. | +| `attachments` | Join `TD_ReceiveFileRecord` by message id; include `file_id`, `file_name`, `size_bytes`, and null `download_ref`; never return `FilePath`. | +| `source` | `local_readonly`. | +| `raw_ref` | A stable non-row-dump ref such as `msglib:tblPersonMsg`, `msglib:tblMsgGroupPersonMsg`, or `msglib:TD_SystemMessageRecord`. | + +## Query and ordering policy + +First DB-backed row-listing slice should: + +1. Read only allowlisted columns. +2. Filter source tables by `conversation_type` before querying rows when supplied. +3. Filter `conversation_id` against the normalized candidate conversation key. +4. Filter `since` on `ActionTime`, `MsgTime`, or system `MsgTime`. +5. Filter `query` only over allowlisted identity/title/body columns; if `include_body=false`, query may still use body columns internally but must not print the values. +6. Sort descending by timestamp, then descending by primary key/id for deterministic output. +7. Clamp `limit` to at most 50. +8. Reject non-empty `cursor` until pagination is explicitly implemented. + +## Body-output policy + +The first sidecar implementation may support `include_body`, but verification must separate two paths: + +1. **Default/evidence-backed real copied DB smoke:** `include_body=false`; output summaries only, no message body values. +2. **Synthetic/redacted fixture test path:** `include_body=true` may be tested with redacted fixture values only. + +Before MCP integration sets `include_body=true` against operator-local data, acceptance must prove: + +- strict limit is enforced; +- output contains only normalized message objects, not raw rows; +- file path values are always omitted; +- audit/source fields make the DB source explicit; +- user/operator local data remains uncommitted. + +## Attachment policy + +Attachment metadata can use `TD_ReceiveFileRecord` only as a safe list source: + +- Join key: `ReceiveMsgGuid` equals normalized message id. +- Return allowed: `ReceiveMsgGuid` as file/message ref, `FileName`, `FileSize`, `SourceID`, `SourceName`, `FileType`, `SendPersonJid`, `SendPersonName`, `SendTime` only if needed for normalized metadata. +- Never return: `FilePath`. +- `download_ref` remains null until cache/download mapping is separately validated. + +## Relationship with PacketReader source + +PacketReader remains the default receive source until DB-vs-log reconciliation is tested. DB-backed receive should enter in stages: + +1. C38: sidecar-level bounded `list_messages` operation, with real copied DB smoke using `include_body=false` only. +2. C39: Go wrapper/fake-sidecar tests and possibly synthetic/redacted fixture flow for `include_body=true`. +3. C40: optional Go `ReceiveMessagesSource` implementation behind explicit env/source configuration. +4. Later: decide `auto` merge/reconciliation policy between PacketReader and MsgLib. + +The first MCP integration should not silently merge PacketReader and MsgLib rows. It should be explicit through env config or a source-mode decision recorded in docs. + +## C38 recommended implementation route + +Implement sidecar `list_messages` first, not MCP integration. + +C38 acceptance gate: + +- Add Go fake-sidecar tests for `ListMessages(ctx, options)` request shape and decoded normalized fields. +- Add sidecar op `list_messages` with allowlisted table/column queries. +- Default sidecar smoke still returns no DB provider checks unless env is configured. +- Evidence-backed copied DB smoke calls `list_messages` with `include_body=false` and prints only: + - `message_list_count` + - `message_list_sources` + - `message_body_values_returned=false` + - `raw_rows_returned=false` + - `file_paths_returned=false` +- No real copied-DB `MsgText`, `MessageBody`, `FilePath`, raw rows, entity values, sends, downloads, hooks, injection, or DB writes. + +## Decision + +Proceed to C38 with sidecar-level `list_messages` plus Go wrapper tests and sanitized verification. Do not wire DB-backed rows into `isphere_receive_messages` MCP until C38 proves the bounded sidecar behavior. diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index c9b8219..cb319be 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -17,7 +17,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | --- | --- | --- | --- | --- | --- | -| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources` readiness with table/column/count summaries only | C37 bounded DB-backed receive-source design | +| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design and keeps PacketReader default until reconciliation | C38 sidecar bounded DB message listing | | `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design | | `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design | | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence | @@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac ## Deferred source work -`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, and C36 adds metadata-only `message_sources` readiness. The next step is bounded DB-backed receive-source design. +`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, and C37 defines the bounded DB-backed receive-source design. The next step is sidecar-level `list_messages` with sanitized verification, not MCP integration yet. diff --git a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md index c0a2d81..8a3117e 100644 --- a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md +++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md @@ -2638,17 +2638,58 @@ MsgLib message-source metadata sidecar result: - Define direct/group source selection, ordering, limit, body-column policy, attachment metadata policy, and fallback behavior versus PacketReader. - Decide whether C38 should add sidecar-level bounded row listing or first add a redacted synthetic DB fixture. -- [ ] **Step 1: Write bounded DB receive-source contract** +- [x] **Step 1: Write bounded DB receive-source contract** -Define request args, returned fields, limits, and no-output boundaries for DB-backed receive listing. +Created `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md`. It defines sidecar op `list_messages`, args, return fields, hard limit `1..50`, no cursor initially, `include_body` policy, attachment policy, no raw rows, no file path values, and PacketReader remains default until reconciliation. -- [ ] **Step 2: Choose implementation route** +- [x] **Step 2: Choose implementation route** -Select between sidecar row listing, Go source abstraction first, or synthetic fixture-first tests. +Decision: C38 should implement sidecar-level `list_messages` plus Go wrapper fake-sidecar tests and sanitized evidence-backed verification. Do not wire DB rows into `isphere_receive_messages` MCP in C38. -- [ ] **Step 3: Update next implementation loop** +- [x] **Step 3: Update next implementation loop** -Record C37 decision and prepare C38. +Updated matrix/status/plan. Verification for this design-only loop will use `git diff --check` and `go test ./...` before commit. + +--- + +## Loop C37 Result + +Bounded DB-backed receive-source design result: + +- Created `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md`. +- Selected future sidecar op `list_messages` with strict limit, empty cursor initially, explicit `include_body`, safe attachment metadata, no raw rows, and no file path values. +- PacketReader remains the default receive source; MsgLib DB rows should not silently merge into MCP until reconciliation is tested. +- C38 should implement sidecar-level bounded listing and sanitized verification only; no MCP integration yet. + +--- + +## Loop C38: Sidecar bounded DB message listing + +**Goal:** Implement sidecar-level `list_messages` for copied/operator-local MsgLib DBs with strict limits and sanitized verification, without wiring DB rows into MCP yet. + +**Planned files:** +- Modify: `internal/msglib/sidecar_client.go` and tests. +- Modify: `native/MsgLibReadSidecar/Program.cs`. +- Modify: `scripts/verify-msglib-sidecar.ps1`. +- Modify docs/status/matrix/plan. + +**Planned behavior:** +- Add Go wrapper such as `ListMessages(ctx, options)`. +- Add sidecar op `list_messages`. +- First evidence-backed copied-DB smoke uses `include_body=false` and prints only counts/source refs/booleans. +- Do not return copied-DB `MsgText`, `MessageBody`, `FilePath`, raw rows, sends, downloads, hooks, injection, or DB writes in verification output. + +- [ ] **Step 1: Write failing Go wrapper test** + +Use fake sidecar to assert request args and decoded normalized message shape. + +- [ ] **Step 2: Implement sidecar `list_messages`** + +Read allowlisted columns only, clamp limit, reject cursor, hide file paths, and support `include_body=false`. + +- [ ] **Step 3: Update verification and docs** + +Extend `verify-msglib-sidecar.ps1` with sanitized `list_messages` smoke and run full gate. ---