149 lines
7.7 KiB
Markdown
149 lines
7.7 KiB
Markdown
# Bounded DB-backed receive-message source design
|
|
|
|
Date: 2026-07-10
|
|
|
|
## Goal
|
|
|
|
Design the smallest safe implementation slice for using copied/operator-local `MsgLib.db` as a future source for `isphere_receive_messages`.
|
|
|
|
This is a design node only. It does not add row-reading code and does not change the MCP tool surface.
|
|
|
|
## Inputs
|
|
|
|
Evidence now available:
|
|
|
|
- C35 mapped `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, file-record, and recent-index tables to the receive-message contract.
|
|
- C36 added metadata-only `message_sources`, proving candidate table availability, required/present columns, roles, and row counts without exposing message bodies, file path values, or raw rows.
|
|
- Current MCP `isphere_receive_messages` is PacketReader-log backed and already supports `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, safe `source_preference`, and empty `cursor`.
|
|
|
|
## Non-goals
|
|
|
|
C37 and the next implementation slice must not:
|
|
|
|
- Add send, download, mark-read, login, hook, injection, or DB mutation behavior.
|
|
- Return raw DB rows.
|
|
- Return local `FilePath` values from file-record tables.
|
|
- Replace PacketReader as the only receive source.
|
|
- Claim production ingestion completeness.
|
|
- Print copied-DB message bodies in evidence-backed smoke output.
|
|
|
|
## Proposed sidecar operation: `list_messages`
|
|
|
|
Add a future sidecar operation named `list_messages` under protocol `isphere.msglib.v1`.
|
|
|
|
### Required args
|
|
|
|
- `sqlite_dll_path`
|
|
- `db_path`
|
|
|
|
### Optional args
|
|
|
|
- `password`: default `123`.
|
|
- `conversation_id`: exact normalized conversation id or DB conversation key.
|
|
- `conversation_type`: `direct`, `group`, `system`, or empty/`auto`.
|
|
- `query`: bounded keyword filter over allowlisted columns only.
|
|
- `since`: ISO-8601/RFC3339 timestamp.
|
|
- `limit`: default `20`, clamp to `1..50` for the first DB row-listing slice.
|
|
- `cursor`: must be empty in the first slice.
|
|
- `include_body`: default `false` in sidecar verification; MCP integration may explicitly set this later after redacted/synthetic tests pass.
|
|
- `include_attachment_metadata`: default `true`; returns only safe attachment fields.
|
|
|
|
### Returned top-level data
|
|
|
|
- `read_only=true`
|
|
- `raw_rows_returned=false`
|
|
- `file_paths_returned=false`
|
|
- `message_body_values_returned=<include_body>`
|
|
- `source_tables`: unique allowlisted tables used.
|
|
- `messages`: normalized bounded message objects.
|
|
- `next_cursor=null` in the first slice.
|
|
|
|
### Normalized message fields
|
|
|
|
| Field | Required behavior |
|
|
| --- | --- |
|
|
| `message_id` / `id` | From table primary key: `tblPersonMsg.Id`, `tblMsgGroupPersonMsg.Id`, or `TD_SystemMessageRecord.MsgGuid`. |
|
|
| `conversation_id` | Direct: prefer `ObjPersonId`, fallback `SndPersonId|RecvPersonId`; Group: `MsgGroupId`; System: stable `system` or sender-derived key. |
|
|
| `conversation_type` | `direct`, `group`, or `system`. |
|
|
| `sender_id` | Direct/group: `SndPersonId`; System: `SendPersonJid`. |
|
|
| `sender_name` | Direct/group: `SndPerson`; System: `SendPersonName`; later can be overwritten/enriched by display-entity maps. |
|
|
| `receiver_id` | Direct: `RecvPersonId`; Group: `MsgGroupId`; System: null or stable system receiver. |
|
|
| `text` / `content_text` | Only when `include_body=true`; select `MsgText` first, fallback `MessageBody`; system messages use `MsgText`. |
|
|
| `content_type` | Derived from `MsgType` and safe attachment metadata; unknown when mapping is unclear. |
|
|
| `subject` | System: `MsgTitle`; direct/group: null until a DB column equivalent is proven. |
|
|
| `timestamp` / `created_at` | Direct: `ActionTime`; Group: `MsgTime`; System: `MsgTime`; parse into RFC3339 where possible. |
|
|
| `read` | Direct/group/system read-state columns mapped to boolean-like value when possible. |
|
|
| `receipt_id` | Null in first slice unless a safe receipt id mapping is proven; DB receipt state may remain in internal metadata only. |
|
|
| `attachments` | Join `TD_ReceiveFileRecord` by message id; include `file_id`, `file_name`, `size_bytes`, and null `download_ref`; never return `FilePath`. |
|
|
| `source` | `local_readonly`. |
|
|
| `raw_ref` | A stable non-row-dump ref such as `msglib:tblPersonMsg`, `msglib:tblMsgGroupPersonMsg`, or `msglib:TD_SystemMessageRecord`. |
|
|
|
|
## Query and ordering policy
|
|
|
|
First DB-backed row-listing slice should:
|
|
|
|
1. Read only allowlisted columns.
|
|
2. Filter source tables by `conversation_type` before querying rows when supplied.
|
|
3. Filter `conversation_id` against the normalized candidate conversation key.
|
|
4. Filter `since` on `ActionTime`, `MsgTime`, or system `MsgTime`.
|
|
5. Filter `query` only over allowlisted identity/title/body columns; if `include_body=false`, query may still use body columns internally but must not print the values.
|
|
6. Sort descending by timestamp, then descending by primary key/id for deterministic output.
|
|
7. Clamp `limit` to at most 50.
|
|
8. Reject non-empty `cursor` until pagination is explicitly implemented.
|
|
|
|
## Body-output policy
|
|
|
|
The first sidecar implementation may support `include_body`, but verification must separate two paths:
|
|
|
|
1. **Default/evidence-backed real copied DB smoke:** `include_body=false`; output summaries only, no message body values.
|
|
2. **Synthetic/redacted fixture test path:** `include_body=true` may be tested with redacted fixture values only.
|
|
|
|
Before MCP integration sets `include_body=true` against operator-local data, acceptance must prove:
|
|
|
|
- strict limit is enforced;
|
|
- output contains only normalized message objects, not raw rows;
|
|
- file path values are always omitted;
|
|
- audit/source fields make the DB source explicit;
|
|
- user/operator local data remains uncommitted.
|
|
|
|
## Attachment policy
|
|
|
|
Attachment metadata can use `TD_ReceiveFileRecord` only as a safe list source:
|
|
|
|
- Join key: `ReceiveMsgGuid` equals normalized message id.
|
|
- Return allowed: `ReceiveMsgGuid` as file/message ref, `FileName`, `FileSize`, `SourceID`, `SourceName`, `FileType`, `SendPersonJid`, `SendPersonName`, `SendTime` only if needed for normalized metadata.
|
|
- Never return: `FilePath`.
|
|
- `download_ref` remains null until cache/download mapping is separately validated.
|
|
|
|
## Relationship with PacketReader source
|
|
|
|
PacketReader remains the default receive source until DB-vs-log reconciliation is tested. DB-backed receive should enter in stages:
|
|
|
|
1. C38: sidecar-level bounded `list_messages` operation, with real copied DB smoke using `include_body=false` only.
|
|
2. C39: Go wrapper/fake-sidecar tests and possibly synthetic/redacted fixture flow for `include_body=true`.
|
|
3. C40: optional Go `ReceiveMessagesSource` implementation behind explicit env/source configuration.
|
|
4. Later: decide `auto` merge/reconciliation policy between PacketReader and MsgLib.
|
|
|
|
The first MCP integration should not silently merge PacketReader and MsgLib rows. It should be explicit through env config or a source-mode decision recorded in docs.
|
|
|
|
## C38 recommended implementation route
|
|
|
|
Implement sidecar `list_messages` first, not MCP integration.
|
|
|
|
C38 acceptance gate:
|
|
|
|
- Add Go fake-sidecar tests for `ListMessages(ctx, options)` request shape and decoded normalized fields.
|
|
- Add sidecar op `list_messages` with allowlisted table/column queries.
|
|
- Default sidecar smoke still returns no DB provider checks unless env is configured.
|
|
- Evidence-backed copied DB smoke calls `list_messages` with `include_body=false` and prints only:
|
|
- `message_list_count`
|
|
- `message_list_sources`
|
|
- `message_body_values_returned=false`
|
|
- `raw_rows_returned=false`
|
|
- `file_paths_returned=false`
|
|
- No real copied-DB `MsgText`, `MessageBody`, `FilePath`, raw rows, entity values, sends, downloads, hooks, injection, or DB writes.
|
|
|
|
## Decision
|
|
|
|
Proceed to C38 with sidecar-level `list_messages` plus Go wrapper tests and sanitized verification. Do not wire DB-backed rows into `isphere_receive_messages` MCP until C38 proves the bounded sidecar behavior.
|