docs: design bounded db receive source
This commit is contained in:
zhaoyilun
@@ -0,0 +1,148 @@
|
||||
# Bounded DB-backed receive-message source design
|
||||
|
||||
Date: 2026-07-10
|
||||
|
||||
## Goal
|
||||
|
||||
Design the smallest safe implementation slice for using copied/operator-local `MsgLib.db` as a future source for `isphere_receive_messages`.
|
||||
|
||||
This is a design node only. It does not add row-reading code and does not change the MCP tool surface.
|
||||
|
||||
## Inputs
|
||||
|
||||
Evidence now available:
|
||||
|
||||
- C35 mapped `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, file-record, and recent-index tables to the receive-message contract.
|
||||
- C36 added metadata-only `message_sources`, proving candidate table availability, required/present columns, roles, and row counts without exposing message bodies, file path values, or raw rows.
|
||||
- Current MCP `isphere_receive_messages` is PacketReader-log backed and already supports `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, safe `source_preference`, and empty `cursor`.
|
||||
|
||||
## Non-goals
|
||||
|
||||
C37 and the next implementation slice must not:
|
||||
|
||||
- Add send, download, mark-read, login, hook, injection, or DB mutation behavior.
|
||||
- Return raw DB rows.
|
||||
- Return local `FilePath` values from file-record tables.
|
||||
- Replace PacketReader as the only receive source.
|
||||
- Claim production ingestion completeness.
|
||||
- Print copied-DB message bodies in evidence-backed smoke output.
|
||||
|
||||
## Proposed sidecar operation: `list_messages`
|
||||
|
||||
Add a future sidecar operation named `list_messages` under protocol `isphere.msglib.v1`.
|
||||
|
||||
### Required args
|
||||
|
||||
- `sqlite_dll_path`
|
||||
- `db_path`
|
||||
|
||||
### Optional args
|
||||
|
||||
- `password`: default `123`.
|
||||
- `conversation_id`: exact normalized conversation id or DB conversation key.
|
||||
- `conversation_type`: `direct`, `group`, `system`, or empty/`auto`.
|
||||
- `query`: bounded keyword filter over allowlisted columns only.
|
||||
- `since`: ISO-8601/RFC3339 timestamp.
|
||||
- `limit`: default `20`, clamp to `1..50` for the first DB row-listing slice.
|
||||
- `cursor`: must be empty in the first slice.
|
||||
- `include_body`: default `false` in sidecar verification; MCP integration may explicitly set this later after redacted/synthetic tests pass.
|
||||
- `include_attachment_metadata`: default `true`; returns only safe attachment fields.
|
||||
|
||||
### Returned top-level data
|
||||
|
||||
- `read_only=true`
|
||||
- `raw_rows_returned=false`
|
||||
- `file_paths_returned=false`
|
||||
- `message_body_values_returned=<include_body>`
|
||||
- `source_tables`: unique allowlisted tables used.
|
||||
- `messages`: normalized bounded message objects.
|
||||
- `next_cursor=null` in the first slice.
|
||||
|
||||
### Normalized message fields
|
||||
|
||||
| Field | Required behavior |
|
||||
| --- | --- |
|
||||
| `message_id` / `id` | From table primary key: `tblPersonMsg.Id`, `tblMsgGroupPersonMsg.Id`, or `TD_SystemMessageRecord.MsgGuid`. |
|
||||
| `conversation_id` | Direct: prefer `ObjPersonId`, fallback `SndPersonId|RecvPersonId`; Group: `MsgGroupId`; System: stable `system` or sender-derived key. |
|
||||
| `conversation_type` | `direct`, `group`, or `system`. |
|
||||
| `sender_id` | Direct/group: `SndPersonId`; System: `SendPersonJid`. |
|
||||
| `sender_name` | Direct/group: `SndPerson`; System: `SendPersonName`; later can be overwritten/enriched by display-entity maps. |
|
||||
| `receiver_id` | Direct: `RecvPersonId`; Group: `MsgGroupId`; System: null or stable system receiver. |
|
||||
| `text` / `content_text` | Only when `include_body=true`; select `MsgText` first, fallback `MessageBody`; system messages use `MsgText`. |
|
||||
| `content_type` | Derived from `MsgType` and safe attachment metadata; unknown when mapping is unclear. |
|
||||
| `subject` | System: `MsgTitle`; direct/group: null until a DB column equivalent is proven. |
|
||||
| `timestamp` / `created_at` | Direct: `ActionTime`; Group: `MsgTime`; System: `MsgTime`; parse into RFC3339 where possible. |
|
||||
| `read` | Direct/group/system read-state columns mapped to boolean-like value when possible. |
|
||||
| `receipt_id` | Null in first slice unless a safe receipt id mapping is proven; DB receipt state may remain in internal metadata only. |
|
||||
| `attachments` | Join `TD_ReceiveFileRecord` by message id; include `file_id`, `file_name`, `size_bytes`, and null `download_ref`; never return `FilePath`. |
|
||||
| `source` | `local_readonly`. |
|
||||
| `raw_ref` | A stable non-row-dump ref such as `msglib:tblPersonMsg`, `msglib:tblMsgGroupPersonMsg`, or `msglib:TD_SystemMessageRecord`. |
|
||||
|
||||
## Query and ordering policy
|
||||
|
||||
First DB-backed row-listing slice should:
|
||||
|
||||
1. Read only allowlisted columns.
|
||||
2. Filter source tables by `conversation_type` before querying rows when supplied.
|
||||
3. Filter `conversation_id` against the normalized candidate conversation key.
|
||||
4. Filter `since` on `ActionTime`, `MsgTime`, or system `MsgTime`.
|
||||
5. Filter `query` only over allowlisted identity/title/body columns; if `include_body=false`, query may still use body columns internally but must not print the values.
|
||||
6. Sort descending by timestamp, then descending by primary key/id for deterministic output.
|
||||
7. Clamp `limit` to at most 50.
|
||||
8. Reject non-empty `cursor` until pagination is explicitly implemented.
|
||||
|
||||
## Body-output policy
|
||||
|
||||
The first sidecar implementation may support `include_body`, but verification must separate two paths:
|
||||
|
||||
1. **Default/evidence-backed real copied DB smoke:** `include_body=false`; output summaries only, no message body values.
|
||||
2. **Synthetic/redacted fixture test path:** `include_body=true` may be tested with redacted fixture values only.
|
||||
|
||||
Before MCP integration sets `include_body=true` against operator-local data, acceptance must prove:
|
||||
|
||||
- strict limit is enforced;
|
||||
- output contains only normalized message objects, not raw rows;
|
||||
- file path values are always omitted;
|
||||
- audit/source fields make the DB source explicit;
|
||||
- user/operator local data remains uncommitted.
|
||||
|
||||
## Attachment policy
|
||||
|
||||
Attachment metadata can use `TD_ReceiveFileRecord` only as a safe list source:
|
||||
|
||||
- Join key: `ReceiveMsgGuid` equals normalized message id.
|
||||
- Return allowed: `ReceiveMsgGuid` as file/message ref, `FileName`, `FileSize`, `SourceID`, `SourceName`, `FileType`, `SendPersonJid`, `SendPersonName`, `SendTime` only if needed for normalized metadata.
|
||||
- Never return: `FilePath`.
|
||||
- `download_ref` remains null until cache/download mapping is separately validated.
|
||||
|
||||
## Relationship with PacketReader source
|
||||
|
||||
PacketReader remains the default receive source until DB-vs-log reconciliation is tested. DB-backed receive should enter in stages:
|
||||
|
||||
1. C38: sidecar-level bounded `list_messages` operation, with real copied DB smoke using `include_body=false` only.
|
||||
2. C39: Go wrapper/fake-sidecar tests and possibly synthetic/redacted fixture flow for `include_body=true`.
|
||||
3. C40: optional Go `ReceiveMessagesSource` implementation behind explicit env/source configuration.
|
||||
4. Later: decide `auto` merge/reconciliation policy between PacketReader and MsgLib.
|
||||
|
||||
The first MCP integration should not silently merge PacketReader and MsgLib rows. It should be explicit through env config or a source-mode decision recorded in docs.
|
||||
|
||||
## C38 recommended implementation route
|
||||
|
||||
Implement sidecar `list_messages` first, not MCP integration.
|
||||
|
||||
C38 acceptance gate:
|
||||
|
||||
- Add Go fake-sidecar tests for `ListMessages(ctx, options)` request shape and decoded normalized fields.
|
||||
- Add sidecar op `list_messages` with allowlisted table/column queries.
|
||||
- Default sidecar smoke still returns no DB provider checks unless env is configured.
|
||||
- Evidence-backed copied DB smoke calls `list_messages` with `include_body=false` and prints only:
|
||||
- `message_list_count`
|
||||
- `message_list_sources`
|
||||
- `message_body_values_returned=false`
|
||||
- `raw_rows_returned=false`
|
||||
- `file_paths_returned=false`
|
||||
- No real copied-DB `MsgText`, `MessageBody`, `FilePath`, raw rows, entity values, sends, downloads, hooks, injection, or DB writes.
|
||||
|
||||
## Decision
|
||||
|
||||
Proceed to C38 with sidecar-level `list_messages` plus Go wrapper tests and sanitized verification. Do not wire DB-backed rows into `isphere_receive_messages` MCP until C38 proves the bounded sidecar behavior.
|
||||
@@ -17,7 +17,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
|
||||
|
||||
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources` readiness with table/column/count summaries only | C37 bounded DB-backed receive-source design |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design and keeps PacketReader default until reconciliation | C38 sidecar bounded DB message listing |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
|
||||
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
|
||||
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
|
||||
|
||||
## Deferred source work
|
||||
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, and C36 adds metadata-only `message_sources` readiness. The next step is bounded DB-backed receive-source design.
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, and C37 defines the bounded DB-backed receive-source design. The next step is sidecar-level `list_messages` with sanitized verification, not MCP integration yet.
|
||||
|
||||
Reference in New Issue
Block a user