# Bounded DB-backed receive-message source design

Date: 2026-07-10

## Goal

Design the smallest safe implementation slice for using copied/operator-local `MsgLib.db` as a future source for `isphere_receive_messages`.

This is a design node only. It does not add row-reading code and does not change the MCP tool surface.

## Inputs

Evidence now available:

- C35 mapped `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, file-record, and recent-index tables to the receive-message contract.
- C36 added metadata-only `message_sources`, proving candidate table availability, required/present columns, roles, and row counts without exposing message bodies, file path values, or raw rows.
- Current MCP `isphere_receive_messages` is PacketReader-log backed and already supports `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, safe `source_preference`, and empty `cursor`.

## Non-goals

C37 and the next implementation slice must not:

- Add send, download, mark-read, login, hook, injection, or DB mutation behavior.
- Return raw DB rows.
- Return local `FilePath` values from file-record tables.
- Replace PacketReader as the only receive source.
- Claim production ingestion completeness.
- Print copied-DB message bodies in evidence-backed smoke output.

## Proposed sidecar operation: `list_messages`

Add a future sidecar operation named `list_messages` under protocol `isphere.msglib.v1`.

### Required args

- `sqlite_dll_path`
- `db_path`

### Optional args

- `password`: default `123`.
- `conversation_id`: exact normalized conversation id or DB conversation key.
- `conversation_type`: `direct`, `group`, `system`, or empty/`auto`.
- `query`: bounded keyword filter over allowlisted columns only.
- `since`: ISO-8601/RFC3339 timestamp.
- `limit`: default `20`, clamp to `1..50` for the first DB row-listing slice.
- `cursor`: must be empty in the first slice.
- `include_body`: default `false` in sidecar verification; MCP integration may explicitly set this later after redacted/synthetic tests pass.
- `include_attachment_metadata`: default `true`; returns only safe attachment fields.

### Returned top-level data

- `read_only=true`
- `raw_rows_returned=false`
- `file_paths_returned=false`
- `message_body_values_returned=<include_body>`
- `source_tables`: unique allowlisted tables used.
- `messages`: normalized bounded message objects.
- `next_cursor=null` in the first slice.

### Normalized message fields

| Field | Required behavior |
| --- | --- |
| `message_id` / `id` | From table primary key: `tblPersonMsg.Id`, `tblMsgGroupPersonMsg.Id`, or `TD_SystemMessageRecord.MsgGuid`. |
| `conversation_id` | Direct: prefer `ObjPersonId`, fallback `SndPersonId|RecvPersonId`; Group: `MsgGroupId`; System: stable `system` or sender-derived key. |
| `conversation_type` | `direct`, `group`, or `system`. |
| `sender_id` | Direct/group: `SndPersonId`; System: `SendPersonJid`. |
| `sender_name` | Direct/group: `SndPerson`; System: `SendPersonName`; later can be overwritten/enriched by display-entity maps. |
| `receiver_id` | Direct: `RecvPersonId`; Group: `MsgGroupId`; System: null or stable system receiver. |
| `text` / `content_text` | Only when `include_body=true`; select `MsgText` first, fallback `MessageBody`; system messages use `MsgText`. |
| `content_type` | Derived from `MsgType` and safe attachment metadata; unknown when mapping is unclear. |
| `subject` | System: `MsgTitle`; direct/group: null until a DB column equivalent is proven. |
| `timestamp` / `created_at` | Direct: `ActionTime`; Group: `MsgTime`; System: `MsgTime`; parse into RFC3339 where possible. |
| `read` | Direct/group/system read-state columns mapped to boolean-like value when possible. |
| `receipt_id` | Null in first slice unless a safe receipt id mapping is proven; DB receipt state may remain in internal metadata only. |
| `attachments` | Join `TD_ReceiveFileRecord` by message id; include `file_id`, `file_name`, `size_bytes`, and null `download_ref`; never return `FilePath`. |
| `source` | `local_readonly`. |
| `raw_ref` | A stable non-row-dump ref such as `msglib:tblPersonMsg`, `msglib:tblMsgGroupPersonMsg`, or `msglib:TD_SystemMessageRecord`. |

## Query and ordering policy

First DB-backed row-listing slice should:

1. Read only allowlisted columns.
2. Filter source tables by `conversation_type` before querying rows when supplied.
3. Filter `conversation_id` against the normalized candidate conversation key.
4. Filter `since` on `ActionTime`, `MsgTime`, or system `MsgTime`.
5. Filter `query` only over allowlisted identity/title/body columns; if `include_body=false`, query may still use body columns internally but must not print the values.
6. Sort descending by timestamp, then descending by primary key/id for deterministic output.
7. Clamp `limit` to at most 50.
8. Reject non-empty `cursor` until pagination is explicitly implemented.

## Body-output policy

The first sidecar implementation may support `include_body`, but verification must separate two paths:

1. **Default/evidence-backed real copied DB smoke:** `include_body=false`; output summaries only, no message body values.
2. **Synthetic/redacted fixture test path:** `include_body=true` may be tested with redacted fixture values only.

Before MCP integration sets `include_body=true` against operator-local data, acceptance must prove:

- strict limit is enforced;
- output contains only normalized message objects, not raw rows;
- file path values are always omitted;
- audit/source fields make the DB source explicit;
- user/operator local data remains uncommitted.

## Attachment policy

Attachment metadata can use `TD_ReceiveFileRecord` only as a safe list source:

- Join key: `ReceiveMsgGuid` equals normalized message id.
- Return allowed: `ReceiveMsgGuid` as file/message ref, `FileName`, `FileSize`, `SourceID`, `SourceName`, `FileType`, `SendPersonJid`, `SendPersonName`, `SendTime` only if needed for normalized metadata.
- Never return: `FilePath`.
- `download_ref` remains null until cache/download mapping is separately validated.

## Relationship with PacketReader source

PacketReader remains the default receive source until DB-vs-log reconciliation is tested. DB-backed receive should enter in stages:

1. C38: sidecar-level bounded `list_messages` operation, with real copied DB smoke using `include_body=false` only.
2. C39: Go wrapper/fake-sidecar tests and possibly synthetic/redacted fixture flow for `include_body=true`.
3. C40: optional Go `ReceiveMessagesSource` implementation behind explicit env/source configuration.
4. Later: decide `auto` merge/reconciliation policy between PacketReader and MsgLib.

The first MCP integration should not silently merge PacketReader and MsgLib rows. It should be explicit through env config or a source-mode decision recorded in docs.

## C38 recommended implementation route

Implement sidecar `list_messages` first, not MCP integration.

C38 acceptance gate:

- Add Go fake-sidecar tests for `ListMessages(ctx, options)` request shape and decoded normalized fields.
- Add sidecar op `list_messages` with allowlisted table/column queries.
- Default sidecar smoke still returns no DB provider checks unless env is configured.
- Evidence-backed copied DB smoke calls `list_messages` with `include_body=false` and prints only:
  - `message_list_count`
  - `message_list_sources`
  - `message_body_values_returned=false`
  - `raw_rows_returned=false`
  - `file_paths_returned=false`
- No real copied-DB `MsgText`, `MessageBody`, `FilePath`, raw rows, entity values, sends, downloads, hooks, injection, or DB writes.

## Decision

Proceed to C38 with sidecar-level `list_messages` plus Go wrapper tests and sanitized verification. Do not wire DB-backed rows into `isphere_receive_messages` MCP until C38 proves the bounded sidecar behavior.