docs: add receive source reconciliation precheck
This commit is contained in:
zhaoyilun
@@ -0,0 +1,151 @@
|
||||
# Receive Message Reconciliation Precheck
|
||||
|
||||
Date: 2026-07-10
|
||||
|
||||
## Goal
|
||||
|
||||
Compare PacketReader and MsgLib receive-message sources using sanitized identifiers, counts, source refs, and booleans only.
|
||||
|
||||
This is a precheck node. It does not change MCP default routing, does not merge sources, and does not add pagination, send, download, hook, injection, or DB-write behavior.
|
||||
|
||||
## Sources
|
||||
|
||||
### PacketReader source
|
||||
|
||||
Current role: default `isphere_receive_messages` source when `source_preference` is empty, `auto`, or `local_readonly`.
|
||||
|
||||
Evidence:
|
||||
|
||||
- Decrypted `PacketReader.ProcessPacket` lines expose XMPP `<message>` stanzas.
|
||||
- Current Go source: `internal/isphere.EncryptedPacketLogSource`.
|
||||
- Current configuration:
|
||||
- `ISPHERE_PACKET_LOG_FILE`
|
||||
- `ISPHERE_PACKET_LOG_DIR`
|
||||
- Current normalized fields include:
|
||||
- `message_id`
|
||||
- `conversation_id`
|
||||
- `conversation_type`
|
||||
- `sender_id`
|
||||
- `receiver_id`
|
||||
- `content_text`
|
||||
- `subject`
|
||||
- `timestamp`
|
||||
- `receipt_id`
|
||||
- read flag
|
||||
|
||||
### MsgLib DB source
|
||||
|
||||
Current role: explicit copied-DB receive source only when `source_preference="msglib_readonly"` and full MsgLib env is configured.
|
||||
|
||||
Evidence:
|
||||
|
||||
- C26 proved copied `MsgLib.db` opens through 32-bit `System.Data.SQLite` with password `123`.
|
||||
- C38 added sidecar `list_messages`.
|
||||
- C39 added `internal/isphere.MsgLibMessageSource`.
|
||||
- C40 added explicit `msglib_readonly` tool selection.
|
||||
- C41 wired env-configured MsgLib receive and optional copied-DB MCP smoke.
|
||||
|
||||
Current optional smoke prints sanitized evidence only:
|
||||
|
||||
```json
|
||||
{
|
||||
"db_receive_smoke": true,
|
||||
"db_receive_message_count": 5,
|
||||
"db_receive_sources": ["msglib:tblPersonMsg"],
|
||||
"message_body_values_printed": false,
|
||||
"file_paths_returned": false,
|
||||
"raw_rows_returned": false
|
||||
}
|
||||
```
|
||||
|
||||
## Candidate match keys
|
||||
|
||||
Use these match keys for a future reconciliation helper:
|
||||
|
||||
```text
|
||||
strong: message_id exact match when both sources expose the same id
|
||||
medium: same conversation id, same sender id, timestamp within a small window
|
||||
medium: same group id, same sender id, same subject/body-present flag
|
||||
weak: filename plus nearby timestamp for file-transfer messages
|
||||
not allowed in committed evidence: message body values, local file paths, raw rows
|
||||
```
|
||||
|
||||
Recommended future scoring:
|
||||
|
||||
| Score | Meaning | Required sanitized evidence |
|
||||
| --- | --- | --- |
|
||||
| strong | Same message id and compatible source refs. | `message_id`, `raw_ref`, source name. |
|
||||
| medium | Same conversation/sender and close timestamp. | `conversation_id`, `sender_id`, timestamp bucket. |
|
||||
| medium | Same group/sender and same subject/body-present state. | `conversation_type=group`, group id, sender id, body-present boolean. |
|
||||
| weak | Same file name/size near the same time. | file name, size, timestamp bucket; no path values. |
|
||||
| no match | Different ids and no close timestamp/participant overlap. | sanitized ids/booleans only. |
|
||||
|
||||
## Known mismatches
|
||||
|
||||
1. **Completeness may differ.** PacketReader sees decrypted transport/client message stanzas; MsgLib DB may not store every message.
|
||||
2. **Subject coverage differs.** PacketReader exposes XML `<subject>` strongly; direct/group MsgLib tables do not have a clearly equivalent subject column.
|
||||
3. **Receipt semantics differ.** PacketReader receipt ids and MsgLib receipt/read-state columns are not proven equivalent.
|
||||
4. **Timestamp format differs.** PacketReader uses XMPP/log epoch-like values; MsgLib uses table-specific timestamp columns such as `ActionTime`, `MsgTime`, and system `MsgTime`.
|
||||
5. **Conversation id shape differs.** PacketReader may use normalized `sender|receiver` for direct messages; MsgLib direct messages may prefer `ObjPersonId`.
|
||||
6. **Attachment mapping is incomplete.** MsgLib can expose safe attachment metadata through `TD_ReceiveFileRecord`, but `download_ref` and local cache path remain unresolved.
|
||||
7. **Body output policy differs.** PacketReader configured-source verification uses redacted/synthetic bodies. Real copied-DB optional smoke must not print body values even if the internal explicit DB receive call can detect body presence.
|
||||
|
||||
## Safe evidence allowed in logs
|
||||
|
||||
Allowed in committed docs or verification output:
|
||||
|
||||
- Counts: message count, source count, table count.
|
||||
- Source refs: `packetlog_message`, `msglib:tblPersonMsg`, `msglib:tblMsgGroupPersonMsg`, `msglib:TD_SystemMessageRecord`.
|
||||
- Booleans:
|
||||
- body values present internally
|
||||
- body values printed externally
|
||||
- file paths returned
|
||||
- raw rows returned
|
||||
- sender/conversation display populated
|
||||
- Sanitized field names and table names.
|
||||
- Timestamp bucket or normalized time window when values are not personally revealing.
|
||||
|
||||
Not allowed in committed docs or verification output:
|
||||
|
||||
- Real message body values.
|
||||
- Real local file paths.
|
||||
- Raw DB rows.
|
||||
- Raw decrypted logs.
|
||||
- User credentials or login metadata values.
|
||||
- Attachment contents.
|
||||
- Production send/download side effects.
|
||||
|
||||
## Decision for default routing
|
||||
|
||||
Do not change default routing yet.
|
||||
|
||||
Current rule remains:
|
||||
|
||||
| `source_preference` | Receive source |
|
||||
| --- | --- |
|
||||
| empty | PacketReader/log-backed source |
|
||||
| `auto` | PacketReader/log-backed source |
|
||||
| `local_readonly` | PacketReader/log-backed source |
|
||||
| `msglib_readonly` | Explicit MsgLib DB receive source, only when env is configured |
|
||||
|
||||
Reason:
|
||||
|
||||
- PacketReader is still the safest default because it is the current parsed receive source and was implemented first with redacted fixtures.
|
||||
- MsgLib DB is now a valid explicit receive source, but it needs reconciliation before it can become part of `auto`.
|
||||
- The current safe evidence is enough to design a reconciliation helper, but not enough to automatically merge or replace sources.
|
||||
|
||||
## Next implementation node
|
||||
|
||||
Recommended next business node: **R2 contact/group search quality hardening**.
|
||||
|
||||
Reason:
|
||||
|
||||
- Send-message and send-file both depend on reliable target resolution.
|
||||
- Search results need deterministic ranking and de-duplication before write-side connector work.
|
||||
- Receive-source reconciliation can continue later as a helper node before any `auto` merge, but it should not block target search hardening.
|
||||
|
||||
Future receive-specific node after R2/R3:
|
||||
|
||||
- Add a local-only reconciliation helper that compares PacketReader and MsgLib messages by sanitized keys and outputs only match counts, source refs, and mismatch categories.
|
||||
- Do not implement pagination until reconciliation proves which source should drive ordering.
|
||||
- Do not implement file download until R3 proves message-to-cache/download mapping.
|
||||
@@ -17,7 +17,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
|
||||
|
||||
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through x86 read-only sidecar `list_messages` selected explicitly by `source_preference="msglib_readonly"`; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md`; `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design; C38 implements sidecar/Go-wrapper `list_messages` with sanitized copied-DB verification; C39 adds a Go adapter from `ListMessages` to the receive-message domain model; C40 adds tool-level explicit `msglib_readonly` source selection; C41 wires env-configured MsgLib receive source and optional copied-DB smoke while `auto` remains PacketReader/log-backed | R1 receive-message source reconciliation precheck |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through x86 read-only sidecar `list_messages` selected explicitly by `source_preference="msglib_readonly"`; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md`; `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md`; `docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design; C38 implements sidecar/Go-wrapper `list_messages` with sanitized copied-DB verification; C39 adds a Go adapter from `ListMessages` to the receive-message domain model; C40 adds tool-level explicit `msglib_readonly` source selection; C41 wires env-configured MsgLib receive source and optional copied-DB smoke; R1 precheck keeps `auto` PacketReader/log-backed until a future reconciliation helper is implemented | future receive reconciliation helper after R2/R3 |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | R2 contact search quality hardening |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | R2 group search quality hardening |
|
||||
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | R3 receive-file download mapping precheck |
|
||||
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
|
||||
|
||||
## Deferred source work
|
||||
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, C37 defines the bounded DB-backed receive-source design, C38 implements sidecar/Go-wrapper `list_messages` with sanitized verification, C39 adds a Go adapter from MsgLib list results to the receive-message domain model, C40 adds explicit tool-level `msglib_readonly` source selection, and C41 wires env-configured explicit MsgLib receive with optional sanitized smoke. The active roadmap is now `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`; next step is R1 receive-source reconciliation.
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, C37 defines the bounded DB-backed receive-source design, C38 implements sidecar/Go-wrapper `list_messages` with sanitized verification, C39 adds a Go adapter from MsgLib list results to the receive-message domain model, C40 adds explicit tool-level `msglib_readonly` source selection, C41 wires env-configured explicit MsgLib receive with optional sanitized smoke, and R1 documents source reconciliation keys. The active roadmap is `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`; next step is R2 contact/group search quality hardening.
|
||||
|
||||
Reference in New Issue
Block a user