docs: precheck msglib message source
This commit is contained in:
zhaoyilun
@@ -0,0 +1,116 @@
|
||||
# MsgLib.db message-table source precheck
|
||||
|
||||
Date: 2026-07-10
|
||||
|
||||
## Question
|
||||
|
||||
Can copied `MsgLib.db` message tables become a future DB-backed source for `isphere_receive_messages`, and what is the safest next implementation slice?
|
||||
|
||||
## Evidence used
|
||||
|
||||
Committed evidence:
|
||||
|
||||
- `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`
|
||||
- `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
|
||||
- `docs/msglib-read-sidecar-contract.md`
|
||||
|
||||
Ignored local evidence consulted in this loop:
|
||||
|
||||
- C26 schema-only x86 JSON under `runs/offline-evidence-intake/.../metadata/`.
|
||||
|
||||
Scope of this loop:
|
||||
|
||||
- Read table names, column names, schema SQL, and row counts only.
|
||||
- Do not read or print message row values.
|
||||
- Do not read or print message bodies, file paths, contact/group values, or raw rows.
|
||||
- Do not add code that exposes DB message rows yet.
|
||||
|
||||
## Candidate DB tables
|
||||
|
||||
| Table | Safe evidence | Candidate role | C35 decision |
|
||||
| --- | --- | --- | --- |
|
||||
| `tblPersonMsg` | Present in all opened copied DBs; count-only evidence shows thousands of rows in inspected copies | direct/person-to-person messages | strong candidate for future DB-backed direct messages |
|
||||
| `tblMsgGroupPersonMsg` | Present in all opened copied DBs; count-only evidence shows hundreds/thousands of rows in inspected copies | group/discussion messages | strong candidate for future DB-backed group messages |
|
||||
| `TD_SystemMessageRecord` | Present with count-only evidence | system/auth/notification messages | secondary candidate after direct/group message flow |
|
||||
| `TD_GroupReceiptMessage` | Present in richer schema | per-person group receipt state by `MsgGuid`/`PersonID` | useful join candidate for receipt enrichment |
|
||||
| `TD_ReceiveFileRecord` | Present with message-guid and file metadata columns | received-file metadata linked by message guid | useful for attachment metadata, but `FilePath` must not be exposed by default |
|
||||
| `TD_SendFileRecord` | Present with message-guid and file metadata columns | sent-file metadata linked by message guid | later outbound/file-source evidence only |
|
||||
| `tblRecent` | Maintained by triggers from person/group message inserts | recent conversation index | useful for conversation ordering/display, not primary message body source |
|
||||
|
||||
## Field mapping hypothesis
|
||||
|
||||
### Direct messages: `tblPersonMsg`
|
||||
|
||||
| MCP receive field | Candidate DB column | Confidence | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `message_id` / `id` | `Id` | high | primary key in schema |
|
||||
| `conversation_type` | constant `direct` | high | table is direct/person message source |
|
||||
| `conversation_id` | `ObjPersonId`, fallback normalized `SndPersonId|RecvPersonId` | medium | trigger writes `ObjPersonId` into `tblRecent.PersonId`; need row-level validation before finalizing |
|
||||
| `sender_id` | `SndPersonId` | high | schema names sender identity explicitly |
|
||||
| `sender_name` | `SndPerson` or optional display-entity map | high | value exposure should follow existing display-name policy |
|
||||
| `receiver_id` | `RecvPersonId` | high | schema names receiver identity explicitly |
|
||||
| `text` / `content_text` | `MsgText` or `MessageBody` | medium | body columns exist, but C35 does not read values; future body read needs strict limit/audit contract |
|
||||
| `content_type` | `MsgType` | medium | values not inspected in C35 |
|
||||
| `timestamp` / `created_at` | `ActionTime` | high | direct message timestamp column |
|
||||
| `read` | `MsgReadState` | medium | boolean-like column; semantics need row-level validation |
|
||||
| `receipt_id` / receipt state | `IsReceipt`, `MsgReceiptState`, `ReceiptTime`, `MsgRecaptionState` | low/medium | schema supports state but not the same as PacketReader receipt id |
|
||||
| `subject` | none | low | direct DB table does not expose PacketReader `<subject>` equivalent in schema |
|
||||
| `attachments` | join `TD_ReceiveFileRecord.ReceiveMsgGuid = tblPersonMsg.Id` | medium | `FilePath` is sensitive and should remain hidden unless a separate download/cache contract exists |
|
||||
|
||||
### Group messages: `tblMsgGroupPersonMsg`
|
||||
|
||||
| MCP receive field | Candidate DB column | Confidence | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `message_id` / `id` | `Id` | high | primary key in schema |
|
||||
| `conversation_type` | constant `group` | high | table is group/discussion message source |
|
||||
| `conversation_id` | `MsgGroupId` | high | group id column and recent trigger key |
|
||||
| `conversation.display_name` | `MsgGroupName` or `SessionTitle`, plus optional display-entity map | high | use existing MsgLib display source priority where possible |
|
||||
| `sender_id` | `SndPersonId` | high | schema names sender identity explicitly |
|
||||
| `sender_name` | `SndPerson` or optional display-entity map | high | value exposure should follow existing display-name policy |
|
||||
| `receiver_id` | `MsgGroupId` | medium | group id is the receive target in MCP contract terms |
|
||||
| `text` / `content_text` | `MsgText` or `MessageBody` | medium | body columns exist, but C35 does not read values; future body read needs strict limit/audit contract |
|
||||
| `content_type` | `MsgType`, possibly `GroupType` for conversation class | medium | values not inspected in C35 |
|
||||
| `timestamp` / `created_at` | `MsgTime` | high | group message timestamp column |
|
||||
| `read` | `MsgReadState` | medium | boolean-like column; semantics need row-level validation |
|
||||
| `receipt_id` / receipt state | `IsReceipt`, `MsgReceiptState`, `MsgRecaptionState`; optional join `TD_GroupReceiptMessage.MsgGuid` | low/medium | DB state exists but PacketReader receipt id equivalence is unproven |
|
||||
| `subject` | none | low | group DB table does not expose PacketReader `<subject>` equivalent in schema |
|
||||
| `attachments` | join `TD_ReceiveFileRecord.ReceiveMsgGuid = tblMsgGroupPersonMsg.Id` | medium | hide `FilePath`; list only safe file metadata until cache/download mapping exists |
|
||||
|
||||
### System messages: `TD_SystemMessageRecord`
|
||||
|
||||
| MCP receive field | Candidate DB column | Confidence | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `message_id` / `id` | `MsgGuid` | high | primary key in schema |
|
||||
| `conversation_type` | constant `system` | medium | source is system/auth record table |
|
||||
| `sender_id` / `sender_name` | `SendPersonJid` / `SendPersonName` | medium | values not inspected in C35 |
|
||||
| `text` / `content_text` | `MsgText` | medium | body-like column exists; future body read requires a strict contract |
|
||||
| `subject` | `MsgTitle` | medium | title-like field exists |
|
||||
| `timestamp` | `MsgTime` | high | system message timestamp column |
|
||||
| `read` | `MsgReadState` | medium | bit-like field exists |
|
||||
|
||||
## Important gaps
|
||||
|
||||
1. `MsgText` and `MessageBody` are body columns. C35 confirms only that they exist, not whether their values are safe, decoded, complete, or aligned with PacketReader bodies.
|
||||
2. `MsgLib.db` may not contain every received message. Earlier SaveToDB evidence noted some messages may not be stored to the small DB, so DB-backed receive should not replace PacketReader as the only source until reconciliation is tested.
|
||||
3. `subject` is strong in PacketReader XML but not clearly represented in `tblPersonMsg`/`tblMsgGroupPersonMsg` schema.
|
||||
4. Receipt semantics are not identical yet. DB columns show receipt/read state, but C35 does not prove the PacketReader receipt id mapping.
|
||||
5. Attachment metadata can likely join through `TD_ReceiveFileRecord`, but `FilePath` is sensitive and should remain hidden by default.
|
||||
6. Row ordering and incremental pagination need an explicit timestamp/id strategy before DB-backed listing is exposed.
|
||||
|
||||
## Decision
|
||||
|
||||
`MsgLib.db` is a strong future source for DB-backed `isphere_receive_messages`, but it should not jump directly to body-returning message listing.
|
||||
|
||||
Recommended next implementation slice:
|
||||
|
||||
1. Add a metadata-only sidecar operation for message-source readiness, for example `message_sources`.
|
||||
2. Return table availability, required/present columns, count-only summaries, and the committed field-map shape.
|
||||
3. Wire a Go wrapper and verification script for this metadata-only operation.
|
||||
4. Keep message body values, raw rows, local file paths, and DB row dumps out of scope until the metadata operation is verified.
|
||||
|
||||
## Proposed C36 acceptance gate
|
||||
|
||||
- Default sidecar smoke still passes with `self_check` only.
|
||||
- Evidence-backed sidecar smoke prints only `message_source_count`, table names, required/present columns, and row counts.
|
||||
- No `MsgText`, `MessageBody`, `FilePath`, entity values, raw rows, sends, downloads, hooks, injection, or DB writes.
|
||||
- Capability matrix records whether C37 may safely design a bounded DB-backed receive source.
|
||||
@@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
|
||||
|
||||
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; DB-backed message listing is still not implemented | C35 MsgLib message-table source precheck |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C35 MsgLib message-table source precheck |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C35 MsgLib message-table source precheck |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables to the receive contract but keeps body/row reads out of scope | C36 metadata-only MsgLib message-source sidecar |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar |
|
||||
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
|
||||
| `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration |
|
||||
| `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated |
|
||||
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
|
||||
|
||||
## Deferred source work
|
||||
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, and C34 documents the operator setup/verification path. The next step is a MsgLib message-table source precheck for future DB-backed receive-message ingestion.
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, and C35 maps MsgLib message tables to receive-message contract fields without reading row values. The next step is a metadata-only sidecar operation for message-source readiness.
|
||||
|
||||
Reference in New Issue
Block a user