diff --git a/docs/current-status-card.md b/docs/current-status-card.md index b38b5d7..de3e9a7 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Stage C / Loop C35 - MsgLib message-table source precheck`. +Current loop: `Stage C / Loop C36 - MsgLib message-source metadata sidecar`. Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. @@ -103,10 +103,11 @@ Current loop output so far: 32. C32: `isphere_receive_messages` can now use the optional MsgLib display-entity source to fill safe `sender_name` and `conversation.display_name` fields while preserving message text/content/attachment behavior. 33. C33: optional real copied-DB MCP smoke now also constructs a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verifies receive-message display enrichment through MCP; output stays sanitized to counts, booleans, and `msglib:` refs only. 34. C34: `docs/go-mcp-runbook.md` now documents the operator setup path for optional MsgLib display-name enrichment, including the required env vars, partial-config failure rule, standard no-MsgLib verification, optional provider smoke, optional MCP enrichment smoke, and sanitized output boundaries. +35. C35: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` maps `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, recent, and file-record schema columns to the future `isphere_receive_messages` contract; decision is metadata-only `message_sources` sidecar first, not body-returning DB listing yet. ## Next business mainline -1. Run Stage C Loop C35: precheck whether copied `MsgLib.db` message tables can support a future DB-backed `isphere_receive_messages` source, starting with schema/column mapping only and no message-body or raw-row output. +1. Run Stage C Loop C36: add a metadata-only MsgLib sidecar operation for message-source readiness, returning table availability, required/present columns, and row counts only. 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated. 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated. 4. Keep UIA selector/report work as fallback support, not as the primary roadmap. diff --git a/docs/source-discovery/2026-07-10-msglib-message-source-precheck.md b/docs/source-discovery/2026-07-10-msglib-message-source-precheck.md new file mode 100644 index 0000000..6213d47 --- /dev/null +++ b/docs/source-discovery/2026-07-10-msglib-message-source-precheck.md @@ -0,0 +1,116 @@ +# MsgLib.db message-table source precheck + +Date: 2026-07-10 + +## Question + +Can copied `MsgLib.db` message tables become a future DB-backed source for `isphere_receive_messages`, and what is the safest next implementation slice? + +## Evidence used + +Committed evidence: + +- `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` +- `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` +- `docs/msglib-read-sidecar-contract.md` + +Ignored local evidence consulted in this loop: + +- C26 schema-only x86 JSON under `runs/offline-evidence-intake/.../metadata/`. + +Scope of this loop: + +- Read table names, column names, schema SQL, and row counts only. +- Do not read or print message row values. +- Do not read or print message bodies, file paths, contact/group values, or raw rows. +- Do not add code that exposes DB message rows yet. + +## Candidate DB tables + +| Table | Safe evidence | Candidate role | C35 decision | +| --- | --- | --- | --- | +| `tblPersonMsg` | Present in all opened copied DBs; count-only evidence shows thousands of rows in inspected copies | direct/person-to-person messages | strong candidate for future DB-backed direct messages | +| `tblMsgGroupPersonMsg` | Present in all opened copied DBs; count-only evidence shows hundreds/thousands of rows in inspected copies | group/discussion messages | strong candidate for future DB-backed group messages | +| `TD_SystemMessageRecord` | Present with count-only evidence | system/auth/notification messages | secondary candidate after direct/group message flow | +| `TD_GroupReceiptMessage` | Present in richer schema | per-person group receipt state by `MsgGuid`/`PersonID` | useful join candidate for receipt enrichment | +| `TD_ReceiveFileRecord` | Present with message-guid and file metadata columns | received-file metadata linked by message guid | useful for attachment metadata, but `FilePath` must not be exposed by default | +| `TD_SendFileRecord` | Present with message-guid and file metadata columns | sent-file metadata linked by message guid | later outbound/file-source evidence only | +| `tblRecent` | Maintained by triggers from person/group message inserts | recent conversation index | useful for conversation ordering/display, not primary message body source | + +## Field mapping hypothesis + +### Direct messages: `tblPersonMsg` + +| MCP receive field | Candidate DB column | Confidence | Notes | +| --- | --- | --- | --- | +| `message_id` / `id` | `Id` | high | primary key in schema | +| `conversation_type` | constant `direct` | high | table is direct/person message source | +| `conversation_id` | `ObjPersonId`, fallback normalized `SndPersonId|RecvPersonId` | medium | trigger writes `ObjPersonId` into `tblRecent.PersonId`; need row-level validation before finalizing | +| `sender_id` | `SndPersonId` | high | schema names sender identity explicitly | +| `sender_name` | `SndPerson` or optional display-entity map | high | value exposure should follow existing display-name policy | +| `receiver_id` | `RecvPersonId` | high | schema names receiver identity explicitly | +| `text` / `content_text` | `MsgText` or `MessageBody` | medium | body columns exist, but C35 does not read values; future body read needs strict limit/audit contract | +| `content_type` | `MsgType` | medium | values not inspected in C35 | +| `timestamp` / `created_at` | `ActionTime` | high | direct message timestamp column | +| `read` | `MsgReadState` | medium | boolean-like column; semantics need row-level validation | +| `receipt_id` / receipt state | `IsReceipt`, `MsgReceiptState`, `ReceiptTime`, `MsgRecaptionState` | low/medium | schema supports state but not the same as PacketReader receipt id | +| `subject` | none | low | direct DB table does not expose PacketReader `` equivalent in schema | +| `attachments` | join `TD_ReceiveFileRecord.ReceiveMsgGuid = tblPersonMsg.Id` | medium | `FilePath` is sensitive and should remain hidden unless a separate download/cache contract exists | + +### Group messages: `tblMsgGroupPersonMsg` + +| MCP receive field | Candidate DB column | Confidence | Notes | +| --- | --- | --- | --- | +| `message_id` / `id` | `Id` | high | primary key in schema | +| `conversation_type` | constant `group` | high | table is group/discussion message source | +| `conversation_id` | `MsgGroupId` | high | group id column and recent trigger key | +| `conversation.display_name` | `MsgGroupName` or `SessionTitle`, plus optional display-entity map | high | use existing MsgLib display source priority where possible | +| `sender_id` | `SndPersonId` | high | schema names sender identity explicitly | +| `sender_name` | `SndPerson` or optional display-entity map | high | value exposure should follow existing display-name policy | +| `receiver_id` | `MsgGroupId` | medium | group id is the receive target in MCP contract terms | +| `text` / `content_text` | `MsgText` or `MessageBody` | medium | body columns exist, but C35 does not read values; future body read needs strict limit/audit contract | +| `content_type` | `MsgType`, possibly `GroupType` for conversation class | medium | values not inspected in C35 | +| `timestamp` / `created_at` | `MsgTime` | high | group message timestamp column | +| `read` | `MsgReadState` | medium | boolean-like column; semantics need row-level validation | +| `receipt_id` / receipt state | `IsReceipt`, `MsgReceiptState`, `MsgRecaptionState`; optional join `TD_GroupReceiptMessage.MsgGuid` | low/medium | DB state exists but PacketReader receipt id equivalence is unproven | +| `subject` | none | low | group DB table does not expose PacketReader `` equivalent in schema | +| `attachments` | join `TD_ReceiveFileRecord.ReceiveMsgGuid = tblMsgGroupPersonMsg.Id` | medium | hide `FilePath`; list only safe file metadata until cache/download mapping exists | + +### System messages: `TD_SystemMessageRecord` + +| MCP receive field | Candidate DB column | Confidence | Notes | +| --- | --- | --- | --- | +| `message_id` / `id` | `MsgGuid` | high | primary key in schema | +| `conversation_type` | constant `system` | medium | source is system/auth record table | +| `sender_id` / `sender_name` | `SendPersonJid` / `SendPersonName` | medium | values not inspected in C35 | +| `text` / `content_text` | `MsgText` | medium | body-like column exists; future body read requires a strict contract | +| `subject` | `MsgTitle` | medium | title-like field exists | +| `timestamp` | `MsgTime` | high | system message timestamp column | +| `read` | `MsgReadState` | medium | bit-like field exists | + +## Important gaps + +1. `MsgText` and `MessageBody` are body columns. C35 confirms only that they exist, not whether their values are safe, decoded, complete, or aligned with PacketReader bodies. +2. `MsgLib.db` may not contain every received message. Earlier SaveToDB evidence noted some messages may not be stored to the small DB, so DB-backed receive should not replace PacketReader as the only source until reconciliation is tested. +3. `subject` is strong in PacketReader XML but not clearly represented in `tblPersonMsg`/`tblMsgGroupPersonMsg` schema. +4. Receipt semantics are not identical yet. DB columns show receipt/read state, but C35 does not prove the PacketReader receipt id mapping. +5. Attachment metadata can likely join through `TD_ReceiveFileRecord`, but `FilePath` is sensitive and should remain hidden by default. +6. Row ordering and incremental pagination need an explicit timestamp/id strategy before DB-backed listing is exposed. + +## Decision + +`MsgLib.db` is a strong future source for DB-backed `isphere_receive_messages`, but it should not jump directly to body-returning message listing. + +Recommended next implementation slice: + +1. Add a metadata-only sidecar operation for message-source readiness, for example `message_sources`. +2. Return table availability, required/present columns, count-only summaries, and the committed field-map shape. +3. Wire a Go wrapper and verification script for this metadata-only operation. +4. Keep message body values, raw rows, local file paths, and DB row dumps out of scope until the metadata operation is verified. + +## Proposed C36 acceptance gate + +- Default sidecar smoke still passes with `self_check` only. +- Evidence-backed sidecar smoke prints only `message_source_count`, table names, required/present columns, and row counts. +- No `MsgText`, `MessageBody`, `FilePath`, entity values, raw rows, sends, downloads, hooks, injection, or DB writes. +- Capability matrix records whether C37 may safely design a bounded DB-backed receive source. diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index 68d13cb..57ce89f 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | --- | --- | --- | --- | --- | --- | -| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; DB-backed message listing is still not implemented | C35 MsgLib message-table source precheck | -| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C35 MsgLib message-table source precheck | -| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C35 MsgLib message-table source precheck | +| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables to the receive contract but keeps body/row reads out of scope | C36 metadata-only MsgLib message-source sidecar | +| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar | +| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar | | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence | | `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration | | `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated | @@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac ## Deferred source work -`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, and C34 documents the operator setup/verification path. The next step is a MsgLib message-table source precheck for future DB-backed receive-message ingestion. +`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, and C35 maps MsgLib message tables to receive-message contract fields without reading row values. The next step is a metadata-only sidecar operation for message-source readiness. diff --git a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md index cf915f5..a21cce7 100644 --- a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md +++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md @@ -2560,17 +2560,57 @@ MsgLib enrichment operator runbook result: - Map candidate message tables/columns to the `isphere_receive_messages` contract fields. - Do not print or commit message bodies, file paths, entity values, raw rows, or copied DB contents. -- [ ] **Step 1: Collect message-table schema evidence** +- [x] **Step 1: Collect message-table schema evidence** -Review existing schema extraction docs and optional sidecar metadata for `tblPersonMsg`, `tblMsgGroupPersonMsg`, `tblRecent`, and related file/message tables. +Reviewed committed C26 schema extraction docs and the ignored C26 schema-only x86 JSON for table/column names and row counts only. Confirmed `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, `TD_GroupReceiptMessage`, `TD_ReceiveFileRecord`, `TD_SendFileRecord`, and `tblRecent` are the relevant message-source candidates. -- [ ] **Step 2: Map candidate fields to receive-message contract** +- [x] **Step 2: Map candidate fields to receive-message contract** -Document which fields may map to message id, conversation id/type, sender/receiver, timestamp, body/text, subject, receipt/read state, and attachment metadata; mark gaps explicitly. +Created `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` with direct/group/system field mapping hypotheses. It marks `MsgText`/`MessageBody`, `FilePath`, receipt semantics, subject gaps, pagination/order, and PacketReader-vs-DB completeness as explicit unresolved gaps. -- [ ] **Step 3: Decide next implementation slice** +- [x] **Step 3: Decide next implementation slice** -Choose between a metadata-only sidecar extension, a bounded DB-backed receive source, or keeping PacketReader as primary until more evidence exists. +Decision: do not jump directly to body-returning DB message listing. C36 should add a metadata-only MsgLib sidecar operation such as `message_sources`, returning table availability, required/present columns, and row counts only. Verification passed: `git diff --check`, `go test ./...`, and evidence-backed `scripts\verify-msglib-sidecar.ps1`. + +--- + +## Loop C35 Result + +MsgLib message-table source precheck result: + +- Created `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md`. +- `tblPersonMsg` is a strong direct-message candidate; `tblMsgGroupPersonMsg` is a strong group-message candidate; `TD_SystemMessageRecord`, `TD_GroupReceiptMessage`, and file-record tables are secondary/join candidates. +- C35 mapped candidate columns to the `isphere_receive_messages` contract without reading row values. +- Decision: next implementation should be metadata-only `message_sources`, not DB message body listing yet. + +--- + +## Loop C36: MsgLib message-source metadata sidecar + +**Goal:** Add a bounded metadata-only sidecar operation that reports MsgLib message-source readiness for future DB-backed receive-message work. + +**Planned files:** +- Modify: `native/MsgLibReadSidecar/Program.cs` and/or protocol files. +- Modify: `internal/msglib/sidecar_client.go` and tests. +- Modify: `scripts/verify-msglib-sidecar.ps1`. +- Modify: `docs/msglib-read-sidecar-contract.md`, matrix/status/plan. + +**Planned behavior:** +- Add op such as `message_sources`. +- Return table availability, required/present columns, safe row counts, and source role metadata only. +- Do not return `MsgText`, `MessageBody`, `FilePath`, entity values, raw rows, sends, downloads, hooks, injection, or DB writes. + +- [ ] **Step 1: Write failing Go client/sidecar tests for message-source metadata** + +Test request/response decoding and bounded metadata shape with a fake sidecar before implementation. + +- [ ] **Step 2: Implement sidecar metadata operation** + +Expose message-source readiness using existing read-only schema/count helpers and allowlisted table/column maps. + +- [ ] **Step 3: Verify default and evidence-backed smokes** + +Update `scripts\verify-msglib-sidecar.ps1` to assert sanitized message-source metadata when copied DB env is configured, then run the full gate. ---