docs: add receive source reconciliation precheck
This commit is contained in:
zhaoyilun
@@ -0,0 +1,151 @@
|
||||
# Receive Message Reconciliation Precheck
|
||||
|
||||
Date: 2026-07-10
|
||||
|
||||
## Goal
|
||||
|
||||
Compare PacketReader and MsgLib receive-message sources using sanitized identifiers, counts, source refs, and booleans only.
|
||||
|
||||
This is a precheck node. It does not change MCP default routing, does not merge sources, and does not add pagination, send, download, hook, injection, or DB-write behavior.
|
||||
|
||||
## Sources
|
||||
|
||||
### PacketReader source
|
||||
|
||||
Current role: default `isphere_receive_messages` source when `source_preference` is empty, `auto`, or `local_readonly`.
|
||||
|
||||
Evidence:
|
||||
|
||||
- Decrypted `PacketReader.ProcessPacket` lines expose XMPP `<message>` stanzas.
|
||||
- Current Go source: `internal/isphere.EncryptedPacketLogSource`.
|
||||
- Current configuration:
|
||||
- `ISPHERE_PACKET_LOG_FILE`
|
||||
- `ISPHERE_PACKET_LOG_DIR`
|
||||
- Current normalized fields include:
|
||||
- `message_id`
|
||||
- `conversation_id`
|
||||
- `conversation_type`
|
||||
- `sender_id`
|
||||
- `receiver_id`
|
||||
- `content_text`
|
||||
- `subject`
|
||||
- `timestamp`
|
||||
- `receipt_id`
|
||||
- read flag
|
||||
|
||||
### MsgLib DB source
|
||||
|
||||
Current role: explicit copied-DB receive source only when `source_preference="msglib_readonly"` and full MsgLib env is configured.
|
||||
|
||||
Evidence:
|
||||
|
||||
- C26 proved copied `MsgLib.db` opens through 32-bit `System.Data.SQLite` with password `123`.
|
||||
- C38 added sidecar `list_messages`.
|
||||
- C39 added `internal/isphere.MsgLibMessageSource`.
|
||||
- C40 added explicit `msglib_readonly` tool selection.
|
||||
- C41 wired env-configured MsgLib receive and optional copied-DB MCP smoke.
|
||||
|
||||
Current optional smoke prints sanitized evidence only:
|
||||
|
||||
```json
|
||||
{
|
||||
"db_receive_smoke": true,
|
||||
"db_receive_message_count": 5,
|
||||
"db_receive_sources": ["msglib:tblPersonMsg"],
|
||||
"message_body_values_printed": false,
|
||||
"file_paths_returned": false,
|
||||
"raw_rows_returned": false
|
||||
}
|
||||
```
|
||||
|
||||
## Candidate match keys
|
||||
|
||||
Use these match keys for a future reconciliation helper:
|
||||
|
||||
```text
|
||||
strong: message_id exact match when both sources expose the same id
|
||||
medium: same conversation id, same sender id, timestamp within a small window
|
||||
medium: same group id, same sender id, same subject/body-present flag
|
||||
weak: filename plus nearby timestamp for file-transfer messages
|
||||
not allowed in committed evidence: message body values, local file paths, raw rows
|
||||
```
|
||||
|
||||
Recommended future scoring:
|
||||
|
||||
| Score | Meaning | Required sanitized evidence |
|
||||
| --- | --- | --- |
|
||||
| strong | Same message id and compatible source refs. | `message_id`, `raw_ref`, source name. |
|
||||
| medium | Same conversation/sender and close timestamp. | `conversation_id`, `sender_id`, timestamp bucket. |
|
||||
| medium | Same group/sender and same subject/body-present state. | `conversation_type=group`, group id, sender id, body-present boolean. |
|
||||
| weak | Same file name/size near the same time. | file name, size, timestamp bucket; no path values. |
|
||||
| no match | Different ids and no close timestamp/participant overlap. | sanitized ids/booleans only. |
|
||||
|
||||
## Known mismatches
|
||||
|
||||
1. **Completeness may differ.** PacketReader sees decrypted transport/client message stanzas; MsgLib DB may not store every message.
|
||||
2. **Subject coverage differs.** PacketReader exposes XML `<subject>` strongly; direct/group MsgLib tables do not have a clearly equivalent subject column.
|
||||
3. **Receipt semantics differ.** PacketReader receipt ids and MsgLib receipt/read-state columns are not proven equivalent.
|
||||
4. **Timestamp format differs.** PacketReader uses XMPP/log epoch-like values; MsgLib uses table-specific timestamp columns such as `ActionTime`, `MsgTime`, and system `MsgTime`.
|
||||
5. **Conversation id shape differs.** PacketReader may use normalized `sender|receiver` for direct messages; MsgLib direct messages may prefer `ObjPersonId`.
|
||||
6. **Attachment mapping is incomplete.** MsgLib can expose safe attachment metadata through `TD_ReceiveFileRecord`, but `download_ref` and local cache path remain unresolved.
|
||||
7. **Body output policy differs.** PacketReader configured-source verification uses redacted/synthetic bodies. Real copied-DB optional smoke must not print body values even if the internal explicit DB receive call can detect body presence.
|
||||
|
||||
## Safe evidence allowed in logs
|
||||
|
||||
Allowed in committed docs or verification output:
|
||||
|
||||
- Counts: message count, source count, table count.
|
||||
- Source refs: `packetlog_message`, `msglib:tblPersonMsg`, `msglib:tblMsgGroupPersonMsg`, `msglib:TD_SystemMessageRecord`.
|
||||
- Booleans:
|
||||
- body values present internally
|
||||
- body values printed externally
|
||||
- file paths returned
|
||||
- raw rows returned
|
||||
- sender/conversation display populated
|
||||
- Sanitized field names and table names.
|
||||
- Timestamp bucket or normalized time window when values are not personally revealing.
|
||||
|
||||
Not allowed in committed docs or verification output:
|
||||
|
||||
- Real message body values.
|
||||
- Real local file paths.
|
||||
- Raw DB rows.
|
||||
- Raw decrypted logs.
|
||||
- User credentials or login metadata values.
|
||||
- Attachment contents.
|
||||
- Production send/download side effects.
|
||||
|
||||
## Decision for default routing
|
||||
|
||||
Do not change default routing yet.
|
||||
|
||||
Current rule remains:
|
||||
|
||||
| `source_preference` | Receive source |
|
||||
| --- | --- |
|
||||
| empty | PacketReader/log-backed source |
|
||||
| `auto` | PacketReader/log-backed source |
|
||||
| `local_readonly` | PacketReader/log-backed source |
|
||||
| `msglib_readonly` | Explicit MsgLib DB receive source, only when env is configured |
|
||||
|
||||
Reason:
|
||||
|
||||
- PacketReader is still the safest default because it is the current parsed receive source and was implemented first with redacted fixtures.
|
||||
- MsgLib DB is now a valid explicit receive source, but it needs reconciliation before it can become part of `auto`.
|
||||
- The current safe evidence is enough to design a reconciliation helper, but not enough to automatically merge or replace sources.
|
||||
|
||||
## Next implementation node
|
||||
|
||||
Recommended next business node: **R2 contact/group search quality hardening**.
|
||||
|
||||
Reason:
|
||||
|
||||
- Send-message and send-file both depend on reliable target resolution.
|
||||
- Search results need deterministic ranking and de-duplication before write-side connector work.
|
||||
- Receive-source reconciliation can continue later as a helper node before any `auto` merge, but it should not block target search hardening.
|
||||
|
||||
Future receive-specific node after R2/R3:
|
||||
|
||||
- Add a local-only reconciliation helper that compares PacketReader and MsgLib messages by sanitized keys and outputs only match counts, source refs, and mismatch categories.
|
||||
- Do not implement pagination until reconciliation proves which source should drive ordering.
|
||||
- Do not implement file download until R3 proves message-to-cache/download mapping.
|
||||
Reference in New Issue
Block a user