isphere-ai-bridge/docs/source-discovery/2026-07-10-db-backed-receive-source-design.md

Bounded DB-backed receive-message source design

Date: 2026-07-10

Goal

Design the smallest safe implementation slice for using copied/operator-local MsgLib.db as a future source for isphere_receive_messages.

This is a design node only. It does not add row-reading code and does not change the MCP tool surface.

Inputs

Evidence now available:

• C35 mapped tblPersonMsg, tblMsgGroupPersonMsg, TD_SystemMessageRecord, receipt, file-record, and recent-index tables to the receive-message contract.
• C36 added metadata-only message_sources, proving candidate table availability, required/present columns, roles, and row counts without exposing message bodies, file path values, or raw rows.
• Current MCP isphere_receive_messages is PacketReader-log backed and already supports conversation_id, query, since, limit, include_attachment_metadata, safe source_preference, and empty cursor.

Non-goals

C37 and the next implementation slice must not:

• Add send, download, mark-read, login, hook, injection, or DB mutation behavior.
• Return raw DB rows.
• Return local FilePath values from file-record tables.
• Replace PacketReader as the only receive source.
• Claim production ingestion completeness.
• Print copied-DB message bodies in evidence-backed smoke output.

Proposed sidecar operation: list_messages

Add a future sidecar operation named list_messages under protocol isphere.msglib.v1.

Required args

• sqlite_dll_path
• db_path

Optional args

• password: default 123.
• conversation_id: exact normalized conversation id or DB conversation key.
• conversation_type: direct, group, system, or empty/auto.
• query: bounded keyword filter over allowlisted columns only.
• since: ISO-8601/RFC3339 timestamp.
• limit: default 20, clamp to 1..50 for the first DB row-listing slice.
• cursor: must be empty in the first slice.
• include_body: default false in sidecar verification; MCP integration may explicitly set this later after redacted/synthetic tests pass.
• include_attachment_metadata: default true; returns only safe attachment fields.

Returned top-level data

• read_only=true
• raw_rows_returned=false
• file_paths_returned=false
• message_body_values_returned=<include_body>
• source_tables: unique allowlisted tables used.
• messages: normalized bounded message objects.
• next_cursor=null in the first slice.

Normalized message fields

Field	Required behavior
message_id / id	From table primary key: tblPersonMsg.Id, tblMsgGroupPersonMsg.Id, or TD_SystemMessageRecord.MsgGuid.
conversation_id	Direct: prefer ObjPersonId, fallback `SndPersonId
conversation_type	direct, group, or system.
sender_id	Direct/group: SndPersonId; System: SendPersonJid.
sender_name	Direct/group: SndPerson; System: SendPersonName; later can be overwritten/enriched by display-entity maps.
receiver_id	Direct: RecvPersonId; Group: MsgGroupId; System: null or stable system receiver.
text / content_text	Only when include_body=true; select MsgText first, fallback MessageBody; system messages use MsgText.
content_type	Derived from MsgType and safe attachment metadata; unknown when mapping is unclear.
subject	System: MsgTitle; direct/group: null until a DB column equivalent is proven.
timestamp / created_at	Direct: ActionTime; Group: MsgTime; System: MsgTime; parse into RFC3339 where possible.
read	Direct/group/system read-state columns mapped to boolean-like value when possible.
receipt_id	Null in first slice unless a safe receipt id mapping is proven; DB receipt state may remain in internal metadata only.
attachments	Join TD_ReceiveFileRecord by message id; include file_id, file_name, size_bytes, and null download_ref; never return FilePath.
source	local_readonly.
raw_ref	A stable non-row-dump ref such as msglib:tblPersonMsg, msglib:tblMsgGroupPersonMsg, or msglib:TD_SystemMessageRecord.

Query and ordering policy

First DB-backed row-listing slice should:

1. Read only allowlisted columns.
2. Filter source tables by conversation_type before querying rows when supplied.
3. Filter conversation_id against the normalized candidate conversation key.
4. Filter since on ActionTime, MsgTime, or system MsgTime.
5. Filter query only over allowlisted identity/title/body columns; if include_body=false, query may still use body columns internally but must not print the values.
6. Sort descending by timestamp, then descending by primary key/id for deterministic output.
7. Clamp limit to at most 50.
8. Reject non-empty cursor until pagination is explicitly implemented.

Body-output policy

The first sidecar implementation may support include_body, but verification must separate two paths:

1. Default/evidence-backed real copied DB smoke: include_body=false; output summaries only, no message body values.
2. Synthetic/redacted fixture test path: include_body=true may be tested with redacted fixture values only.

Before MCP integration sets include_body=true against operator-local data, acceptance must prove:

• strict limit is enforced;
• output contains only normalized message objects, not raw rows;
• file path values are always omitted;
• audit/source fields make the DB source explicit;
• user/operator local data remains uncommitted.

Attachment policy

Attachment metadata can use TD_ReceiveFileRecord only as a safe list source:

• Join key: ReceiveMsgGuid equals normalized message id.
• Return allowed: ReceiveMsgGuid as file/message ref, FileName, FileSize, SourceID, SourceName, FileType, SendPersonJid, SendPersonName, SendTime only if needed for normalized metadata.
• Never return: FilePath.
• download_ref remains null until cache/download mapping is separately validated.

Relationship with PacketReader source

PacketReader remains the default receive source until DB-vs-log reconciliation is tested. DB-backed receive should enter in stages:

1. C38: sidecar-level bounded list_messages operation, with real copied DB smoke using include_body=false only.
2. C39: Go wrapper/fake-sidecar tests and possibly synthetic/redacted fixture flow for include_body=true.
3. C40: optional Go ReceiveMessagesSource implementation behind explicit env/source configuration.
4. Later: decide auto merge/reconciliation policy between PacketReader and MsgLib.

The first MCP integration should not silently merge PacketReader and MsgLib rows. It should be explicit through env config or a source-mode decision recorded in docs.

C38 recommended implementation route

Implement sidecar list_messages first, not MCP integration.

C38 acceptance gate:

• Add Go fake-sidecar tests for ListMessages(ctx, options) request shape and decoded normalized fields.
• Add sidecar op list_messages with allowlisted table/column queries.
• Default sidecar smoke still returns no DB provider checks unless env is configured.
• Evidence-backed copied DB smoke calls list_messages with include_body=false and prints only:
  • message_list_count
  • message_list_sources
  • message_body_values_returned=false
  • raw_rows_returned=false
  • file_paths_returned=false
• No real copied-DB MsgText, MessageBody, FilePath, raw rows, entity values, sends, downloads, hooks, injection, or DB writes.

Decision

Proceed to C38 with sidecar-level list_messages plus Go wrapper tests and sanitized verification. Do not wire DB-backed rows into isphere_receive_messages MCP until C38 proves the bounded sidecar behavior.