From 1d6dd2f50422bb657d55fb4bd48d2ad06f45f693 Mon Sep 17 00:00:00 2001 From: zhaoyilun Date: Fri, 10 Jul 2026 09:29:53 +0800 Subject: [PATCH] docs: add receive source reconciliation precheck --- docs/current-status-card.md | 8 +- ...receive-message-reconciliation-precheck.md | 151 ++++++++++++++++++ .../capability-source-matrix.md | 4 +- ...7-10-core-business-capabilities-roadmap.md | 15 +- 4 files changed, 170 insertions(+), 8 deletions(-) create mode 100644 docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md diff --git a/docs/current-status-card.md b/docs/current-status-card.md index d40eac5..9c52a86 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Business Roadmap / R1 - Receive-message source reconciliation precheck`. +Current loop: `Business Roadmap / R2 - Contact and group search quality hardening`. Plan file: `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`. @@ -120,4 +120,8 @@ The active plan is now organized around the four business goals rather than the 3. Receive/send files: map received file metadata to cache/download behavior first; send-file remains behind the send connector plus upload evidence. 4. Keep UIA selector/report work as fallback support, not as the primary roadmap. -Next node: R1 receive-message source reconciliation precheck. +R1 receive-message source reconciliation precheck is complete: + +- Created `docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md`. +- Decision: do not change default receive routing yet; empty/`auto` remains PacketReader/log-backed and `msglib_readonly` remains explicit. +- Next node: R2 contact/group search quality hardening. diff --git a/docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md b/docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md new file mode 100644 index 0000000..ca00fe5 --- /dev/null +++ b/docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md @@ -0,0 +1,151 @@ +# Receive Message Reconciliation Precheck + +Date: 2026-07-10 + +## Goal + +Compare PacketReader and MsgLib receive-message sources using sanitized identifiers, counts, source refs, and booleans only. + +This is a precheck node. It does not change MCP default routing, does not merge sources, and does not add pagination, send, download, hook, injection, or DB-write behavior. + +## Sources + +### PacketReader source + +Current role: default `isphere_receive_messages` source when `source_preference` is empty, `auto`, or `local_readonly`. + +Evidence: + +- Decrypted `PacketReader.ProcessPacket` lines expose XMPP `` stanzas. +- Current Go source: `internal/isphere.EncryptedPacketLogSource`. +- Current configuration: + - `ISPHERE_PACKET_LOG_FILE` + - `ISPHERE_PACKET_LOG_DIR` +- Current normalized fields include: + - `message_id` + - `conversation_id` + - `conversation_type` + - `sender_id` + - `receiver_id` + - `content_text` + - `subject` + - `timestamp` + - `receipt_id` + - read flag + +### MsgLib DB source + +Current role: explicit copied-DB receive source only when `source_preference="msglib_readonly"` and full MsgLib env is configured. + +Evidence: + +- C26 proved copied `MsgLib.db` opens through 32-bit `System.Data.SQLite` with password `123`. +- C38 added sidecar `list_messages`. +- C39 added `internal/isphere.MsgLibMessageSource`. +- C40 added explicit `msglib_readonly` tool selection. +- C41 wired env-configured MsgLib receive and optional copied-DB MCP smoke. + +Current optional smoke prints sanitized evidence only: + +```json +{ + "db_receive_smoke": true, + "db_receive_message_count": 5, + "db_receive_sources": ["msglib:tblPersonMsg"], + "message_body_values_printed": false, + "file_paths_returned": false, + "raw_rows_returned": false +} +``` + +## Candidate match keys + +Use these match keys for a future reconciliation helper: + +```text +strong: message_id exact match when both sources expose the same id +medium: same conversation id, same sender id, timestamp within a small window +medium: same group id, same sender id, same subject/body-present flag +weak: filename plus nearby timestamp for file-transfer messages +not allowed in committed evidence: message body values, local file paths, raw rows +``` + +Recommended future scoring: + +| Score | Meaning | Required sanitized evidence | +| --- | --- | --- | +| strong | Same message id and compatible source refs. | `message_id`, `raw_ref`, source name. | +| medium | Same conversation/sender and close timestamp. | `conversation_id`, `sender_id`, timestamp bucket. | +| medium | Same group/sender and same subject/body-present state. | `conversation_type=group`, group id, sender id, body-present boolean. | +| weak | Same file name/size near the same time. | file name, size, timestamp bucket; no path values. | +| no match | Different ids and no close timestamp/participant overlap. | sanitized ids/booleans only. | + +## Known mismatches + +1. **Completeness may differ.** PacketReader sees decrypted transport/client message stanzas; MsgLib DB may not store every message. +2. **Subject coverage differs.** PacketReader exposes XML `` strongly; direct/group MsgLib tables do not have a clearly equivalent subject column. +3. **Receipt semantics differ.** PacketReader receipt ids and MsgLib receipt/read-state columns are not proven equivalent. +4. **Timestamp format differs.** PacketReader uses XMPP/log epoch-like values; MsgLib uses table-specific timestamp columns such as `ActionTime`, `MsgTime`, and system `MsgTime`. +5. **Conversation id shape differs.** PacketReader may use normalized `sender|receiver` for direct messages; MsgLib direct messages may prefer `ObjPersonId`. +6. **Attachment mapping is incomplete.** MsgLib can expose safe attachment metadata through `TD_ReceiveFileRecord`, but `download_ref` and local cache path remain unresolved. +7. **Body output policy differs.** PacketReader configured-source verification uses redacted/synthetic bodies. Real copied-DB optional smoke must not print body values even if the internal explicit DB receive call can detect body presence. + +## Safe evidence allowed in logs + +Allowed in committed docs or verification output: + +- Counts: message count, source count, table count. +- Source refs: `packetlog_message`, `msglib:tblPersonMsg`, `msglib:tblMsgGroupPersonMsg`, `msglib:TD_SystemMessageRecord`. +- Booleans: + - body values present internally + - body values printed externally + - file paths returned + - raw rows returned + - sender/conversation display populated +- Sanitized field names and table names. +- Timestamp bucket or normalized time window when values are not personally revealing. + +Not allowed in committed docs or verification output: + +- Real message body values. +- Real local file paths. +- Raw DB rows. +- Raw decrypted logs. +- User credentials or login metadata values. +- Attachment contents. +- Production send/download side effects. + +## Decision for default routing + +Do not change default routing yet. + +Current rule remains: + +| `source_preference` | Receive source | +| --- | --- | +| empty | PacketReader/log-backed source | +| `auto` | PacketReader/log-backed source | +| `local_readonly` | PacketReader/log-backed source | +| `msglib_readonly` | Explicit MsgLib DB receive source, only when env is configured | + +Reason: + +- PacketReader is still the safest default because it is the current parsed receive source and was implemented first with redacted fixtures. +- MsgLib DB is now a valid explicit receive source, but it needs reconciliation before it can become part of `auto`. +- The current safe evidence is enough to design a reconciliation helper, but not enough to automatically merge or replace sources. + +## Next implementation node + +Recommended next business node: **R2 contact/group search quality hardening**. + +Reason: + +- Send-message and send-file both depend on reliable target resolution. +- Search results need deterministic ranking and de-duplication before write-side connector work. +- Receive-source reconciliation can continue later as a helper node before any `auto` merge, but it should not block target search hardening. + +Future receive-specific node after R2/R3: + +- Add a local-only reconciliation helper that compares PacketReader and MsgLib messages by sanitized keys and outputs only match counts, source refs, and mismatch categories. +- Do not implement pagination until reconciliation proves which source should drive ordering. +- Do not implement file download until R3 proves message-to-cache/download mapping. diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index a5e2a7b..c9e05cd 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -17,7 +17,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | --- | --- | --- | --- | --- | --- | -| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through x86 read-only sidecar `list_messages` selected explicitly by `source_preference="msglib_readonly"`; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md`; `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design; C38 implements sidecar/Go-wrapper `list_messages` with sanitized copied-DB verification; C39 adds a Go adapter from `ListMessages` to the receive-message domain model; C40 adds tool-level explicit `msglib_readonly` source selection; C41 wires env-configured MsgLib receive source and optional copied-DB smoke while `auto` remains PacketReader/log-backed | R1 receive-message source reconciliation precheck | +| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through x86 read-only sidecar `list_messages` selected explicitly by `source_preference="msglib_readonly"`; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md`; `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md`; `docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design; C38 implements sidecar/Go-wrapper `list_messages` with sanitized copied-DB verification; C39 adds a Go adapter from `ListMessages` to the receive-message domain model; C40 adds tool-level explicit `msglib_readonly` source selection; C41 wires env-configured MsgLib receive source and optional copied-DB smoke; R1 precheck keeps `auto` PacketReader/log-backed until a future reconciliation helper is implemented | future receive reconciliation helper after R2/R3 | | `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | R2 contact search quality hardening | | `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | R2 group search quality hardening | | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | R3 receive-file download mapping precheck | @@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac ## Deferred source work -`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, C37 defines the bounded DB-backed receive-source design, C38 implements sidecar/Go-wrapper `list_messages` with sanitized verification, C39 adds a Go adapter from MsgLib list results to the receive-message domain model, C40 adds explicit tool-level `msglib_readonly` source selection, and C41 wires env-configured explicit MsgLib receive with optional sanitized smoke. The active roadmap is now `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`; next step is R1 receive-source reconciliation. +`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, C37 defines the bounded DB-backed receive-source design, C38 implements sidecar/Go-wrapper `list_messages` with sanitized verification, C39 adds a Go adapter from MsgLib list results to the receive-message domain model, C40 adds explicit tool-level `msglib_readonly` source selection, C41 wires env-configured explicit MsgLib receive with optional sanitized smoke, and R1 documents source reconciliation keys. The active roadmap is `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`; next step is R2 contact/group search quality hardening. diff --git a/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md b/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md index b6a676a..0186230 100644 --- a/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md +++ b/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md @@ -27,7 +27,7 @@ | --- | --- | --- | --- | | Search contacts | Registered MCP tool; log/JID candidates; optional copied-DB MsgLib display enrichment. | Result quality hardening: de-duplication, ranking, stable IDs, optional richer fields. | R2 | | Search groups | Registered MCP tool; groupchat/conference candidates; optional copied-DB MsgLib display enrichment. | Group member count, owner refs, nested/group hierarchy quality. | R2 | -| Receive messages | PacketReader source works; copied `MsgLib.db` explicit receive works with `source_preference="msglib_readonly"`. | Reconciliation between PacketReader and MsgLib, source confidence, pagination policy. | R1 | +| Receive messages | PacketReader source works; copied `MsgLib.db` explicit receive works with `source_preference="msglib_readonly"`; R1 precheck says default should remain PacketReader until a future reconciliation helper is implemented. | Future reconciliation helper, source confidence scoring, pagination policy. | future receive helper after R2/R3 | | Send messages | Contract exists; no validated write connector. | Need connector discovery, preview-only path, idempotency and audit before production send. | R5 then R6 | | Receive files | List mode exists from message-derived file metadata. | Need message-to-cache/download mapping and output-dir download behavior. | R3 then R4 | | Send files | Contract exists; no validated upload/send connector. | Depends on send-message connector plus upload/file-transfer evidence. | R7 then R8 | @@ -73,7 +73,7 @@ go test ./... **Steps:** -- [ ] **Step 1: Write the precheck note skeleton** +- [x] **Step 1: Write the precheck note skeleton** Create `docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md` with these headings: @@ -99,7 +99,7 @@ Compare PacketReader and MsgLib receive-message sources using sanitized identifi ## Next implementation node ``` -- [ ] **Step 2: Fill candidate match keys** +- [x] **Step 2: Fill candidate match keys** Record these exact candidate keys: @@ -111,7 +111,7 @@ weak: filename plus nearby timestamp for file-transfer messages not allowed in committed evidence: message body values, local file paths, raw rows ``` -- [ ] **Step 3: Run verification** +- [x] **Step 3: Run verification** ```powershell git diff --check @@ -129,6 +129,13 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-en **Stop condition:** - If safe sanitized keys are not enough to compare sources, stop and request either more evidence or permission for a focused local-only diagnostic script. +**R1 Result:** + +- Created `docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md`. +- Candidate match keys are defined as strong/medium/weak using message id, conversation/sender ids, timestamp windows, body-present booleans, and safe file metadata. +- Default routing remains unchanged: empty/`auto`/`local_readonly` use PacketReader; `msglib_readonly` is explicit only. +- Decision: next business node is R2 contact/group search quality hardening. A receive-specific reconciliation helper can be added later before any `auto` merge. + --- ### R2: Contact and group search quality hardening