From ea38346ce371e6895855d80c1668456849d8df68 Mon Sep 17 00:00:00 2001 From: zhaoyilun Date: Fri, 10 Jul 2026 09:44:08 +0800 Subject: [PATCH] docs: add file download mapping precheck --- docs/current-status-card.md | 8 +- ...26-07-10-file-download-mapping-precheck.md | 124 ++++++++++++++++++ .../capability-source-matrix.md | 2 +- ...7-10-core-business-capabilities-roadmap.md | 32 ++++- 4 files changed, 163 insertions(+), 3 deletions(-) create mode 100644 docs/source-discovery/2026-07-10-file-download-mapping-precheck.md diff --git a/docs/current-status-card.md b/docs/current-status-card.md index 7b61cda..9873aaf 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Business Roadmap / R3 - Receive-file download mapping precheck`. +Current loop: `Business Roadmap / R3b - File cache mapping diagnostic`. Plan file: `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`. @@ -133,3 +133,9 @@ R2 contact/group search quality hardening is complete: - Search groups now has deterministic exact-match-first ranking and case-insensitive de-duplication for groupchat/conference candidates and optional MsgLib display enrichment. - MCP output still preserves `source` and `raw_ref`; no send or download behavior was introduced. - Next node: R3 receive-file download mapping precheck. + +R3 receive-file download mapping precheck is complete: + +- Created `docs/source-discovery/2026-07-10-file-download-mapping-precheck.md`. +- Decision: do not implement download mode yet; current evidence supports list mode and DB attachment metadata, but not a validated cache/download mapping. +- Next node: R3b file cache mapping diagnostic with sanitized counts only. diff --git a/docs/source-discovery/2026-07-10-file-download-mapping-precheck.md b/docs/source-discovery/2026-07-10-file-download-mapping-precheck.md new file mode 100644 index 0000000..5821400 --- /dev/null +++ b/docs/source-discovery/2026-07-10-file-download-mapping-precheck.md @@ -0,0 +1,124 @@ +# File Download Mapping Precheck + +Date: 2026-07-10 + +## Goal + +Determine whether `isphere_receive_files` can safely move from list mode to download mode. + +This precheck uses only sanitized counts, table/column names, source refs, booleans, extension counts, and path-shape descriptions. It does not commit local file paths, file contents, decrypted message bodies, raw DB rows, DB credentials, or attachment contents. + +## Current product behavior + +`isphere_receive_files` currently supports list mode only. + +Current list source: + +- `internal/isphere.ListFilesFromMessages` extracts file candidates from normalized PacketReader messages. +- It returns metadata such as `file_id`, `message_id`, `conversation_id`, `file_name`, `mime_type`, `created_at`, `source`, and `raw_ref`. +- `download_ref`, `saved_path`, and `sha256` remain empty. +- `internal/tools/isphere_files.go` rejects non-list modes and rejects `output_dir` until download mode is implemented. + +## Evidence inventory + +### PacketReader file-transfer evidence + +Safe count evidence from C12: + +| Source | Parsed message evidence | Cache/download signal | +| --- | --- | --- | +| `PacketReader.ProcessPacket` | 107 file-candidate messages; 123 `FILE_TRANSFER` hits; extension counts include `.doc`, `.docx`, `.pdf`, `.png`, `.xlsx`, `.zip`. | 0 importal-hash path hits; 0 download-keyword hits. | +| `Smark.SendReceive` | 5 file-candidate messages; extension counts include `.doc`, `.docx`, `.pdf`, `.zip`. | 0 importal-hash path hits; 0 download-keyword hits. | +| `SaveToDB` | 9 generic file-extension hits, but no parseable XMPP message candidates. | 0 importal-hash path hits; 0 download-keyword hits. | + +Conclusion: PacketReader is sufficient for a file list, but not sufficient for local file resolution. + +### MsgLib file-record evidence + +Validated copied-DB table metadata: + +- `TD_ReceiveFileRecord`: `ReceiveMsgGuid`, `FileName`, `FileSize`, `FilePath`, `SourceID`, `SourceName`, `FileType`, `SendPersonJid`, `SendPersonName`, `SendTime`. +- `TD_SendFileRecord`: `SendMsgGuid`, `FileName`, `FileSize`, `FilePath`, `SourceID`, `SourceName`, `FileType`, `ReceivePersonJid`, `ReceivePersonName`, `ReceiveTime`. + +Current sidecar behavior in `native/MsgLibReadSidecar/Program.cs`: + +- Joins `TD_ReceiveFileRecord` by `ReceiveMsgGuid`. +- Returns `file_id`, `file_name`, `size_bytes`, and an empty `download_ref`. +- Uses `SafeFileName(...)` for file names. +- Does not return `FilePath`; smoke output keeps `file_paths_returned=false`. + +Conclusion: MsgLib can provide stronger received-file metadata than PacketReader, but it still does not prove a cache-file path or client download operation. + +### Cache path-shape evidence + +Ignored archive metadata shows: + +```json +{ + "importal_hash_shape_files": 11, + "hash_dir_length_set": [32], + "extension_counts": { + ".doc": 2, + ".docx": 3, + ".jpg": 2, + ".pdf": 2, + ".xlsx": 2 + }, + "raw_paths_printed": false +} +``` + +Conclusion: local cache files exist under a stable hash-directory shape, but no committed evidence maps a message id, `ReceiveMsgGuid`, `SourceID`, filename, or size to a specific cache file. + +## Candidate mappings + +| Candidate | Evidence strength | Status | +| --- | --- | --- | +| `PacketReader message_id + body filename` -> file list row | Medium for list mode. | Already implemented for metadata listing. | +| `TD_ReceiveFileRecord.ReceiveMsgGuid` -> MsgLib message id | Strong for DB attachment-to-message relation. | Usable for metadata enrichment after source reconciliation. | +| `TD_ReceiveFileRecord.FileName + FileSize` -> cache file filename/size | Medium if a local diagnostic proves unique matches. | Not validated yet. | +| `TD_ReceiveFileRecord.SourceID` -> cache hash directory or download token | Unknown. | Needs focused diagnostic; do not use as `download_ref` yet. | +| `TD_ReceiveFileRecord.FilePath` -> local file path | Potentially direct but sensitive. | Do not return or commit; may only be used in a local diagnostic that prints counts/booleans, not values. | +| UI/client operation -> trigger download | Unknown. | Later fallback if cache mapping fails. | + +## Decision + +Do not implement R4 download mode yet. + +R3 chooses option 3 from the acceptance gate: **additional evidence request**, concretely a new R3b local-only cache mapping diagnostic. + +Reason: + +- There is enough evidence to list received files. +- There is enough evidence to identify DB attachment records. +- There is not enough evidence to safely copy or download the file because no validated mapping exists between message/file metadata and a local cache object. + +## R3b diagnostic requirements + +The next node should add or run a local-only diagnostic that outputs only sanitized evidence: + +- Inputs: + - copied `MsgLib.db` and bundled SQLite DLL env already used by MsgLib verification; + - operator-local cache root or ignored archive-list source. +- Allowed output: + - count of `TD_ReceiveFileRecord` rows inspected; + - count of rows with filename, size, source id, and receive message id present; + - cache candidate counts by extension and hash-directory shape; + - match counts by safe strategy: filename-only, filename+size, source-id-shape, message-id-linked; + - booleans: `raw_paths_printed=false`, `file_contents_read=false`, `file_paths_returned=false`. +- Not allowed output: + - local paths; + - raw DB rows; + - real message bodies; + - attachment contents; + - copied file names if they are not explicitly redacted. + +## Next implementation node + +Next node: **R3b file cache mapping diagnostic**. + +Possible R3b outcomes: + +1. If filename+size or source-id mapping is unique and repeatable, proceed to R4 cache-file resolver/download implementation. +2. If cache mapping is ambiguous but the client exposes a safe UI/API download action, proceed to a UI/client download connector node. +3. If neither mapping is proven, request additional operator evidence and keep `isphere_receive_files` in list-only mode. diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index 23b14b9..0b244b0 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -20,7 +20,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through x86 read-only sidecar `list_messages` selected explicitly by `source_preference="msglib_readonly"`; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md`; `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md`; `docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design; C38 implements sidecar/Go-wrapper `list_messages` with sanitized copied-DB verification; C39 adds a Go adapter from `ListMessages` to the receive-message domain model; C40 adds tool-level explicit `msglib_readonly` source selection; C41 wires env-configured MsgLib receive source and optional copied-DB smoke; R1 precheck keeps `auto` PacketReader/log-backed until a future reconciliation helper is implemented | future receive reconciliation helper after R2/R3 | | `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; `internal/isphere/contacts_test.go`; `internal/tools/isphere_contacts_test.go` | C34 documents optional MsgLib contact display enrichment; R2 adds deterministic exact-match-first ranking, case-insensitive de-duplication across log/MsgLib candidates, and preserves `source`/`raw_ref` | core contact search complete; optional richer fields later | | `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; `internal/isphere/groups_test.go`; `internal/tools/isphere_groups_test.go`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents optional MsgLib group display enrichment; R2 adds deterministic exact-match-first ranking, case-insensitive de-duplication across log/MsgLib candidates, and preserves `source`/`raw_ref` | core group search complete; optional member/owner hierarchy later | -| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | R3 receive-file download mapping precheck | +| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | MsgLib `TD_ReceiveFileRecord` safe metadata through explicit copied-DB path; decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck`; `docs/source-discovery/2026-07-10-file-download-mapping-precheck.md` | C22 verifies safe file-list contract args; R3 confirms list metadata is available but no validated message/file-record-to-cache-file mapping exists; R4 remains blocked | R3b file cache mapping diagnostic | | `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | R5 send-message connector discovery | | `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | R7 send-file connector discovery | diff --git a/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md b/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md index b22f62c..5c1dd87 100644 --- a/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md +++ b/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md @@ -29,7 +29,7 @@ | Search groups | Registered MCP tool; deterministic exact-match ranking and case-insensitive de-duplication across groupchat/conference candidates and optional copied-DB MsgLib display enrichment. | Optional member count, owner refs, and nested/group hierarchy remain future enrichment, not a core search blocker. | complete for core search; optional enrichment later | | Receive messages | PacketReader source works; copied `MsgLib.db` explicit receive works with `source_preference="msglib_readonly"`; R1 precheck says default should remain PacketReader until a future reconciliation helper is implemented. | Future reconciliation helper, source confidence scoring, pagination policy. | future receive helper after R2/R3 | | Send messages | Contract exists; no validated write connector. | Need connector discovery, preview-only path, idempotency and audit before production send. | R5 then R6 | -| Receive files | List mode exists from message-derived file metadata. | Need message-to-cache/download mapping and output-dir download behavior. | R3 then R4 | +| Receive files | List mode exists from message-derived file metadata; R3 precheck confirms cache/download mapping is still unvalidated. | Need sanitized cache mapping diagnostic before output-dir download behavior. | R3b then R4 if validated | | Send files | Contract exists; no validated upload/send connector. | Depends on send-message connector plus upload/file-transfer evidence. | R7 then R8 | ## Node Sequence @@ -229,6 +229,36 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1 3. additional evidence request. - No file content or local file path values are committed. +**R3 Result:** + +- Created `docs/source-discovery/2026-07-10-file-download-mapping-precheck.md`. +- PacketReader remains sufficient for file metadata list mode, but has 0 cache path and 0 download keyword hits in the C12 safe count evidence. +- MsgLib `TD_ReceiveFileRecord` provides safe attachment metadata (`ReceiveMsgGuid`, filename, size, source id), but current sidecar intentionally returns empty `download_ref` and `file_paths_returned=false`. +- Ignored cache evidence shows 11 importal hash-shape files under 32-character hash directories, but no validated message/file-record-to-cache-file mapping. +- Decision: do not start R4 yet; insert R3b file cache mapping diagnostic as the next node. + +--- + +### R3b: File cache mapping diagnostic + +**Purpose:** Prove or reject a safe mapping from MsgLib received-file metadata to local cache files before implementing download mode. + +**Files:** +- Create or modify: `scripts/verify-file-cache-mapping.ps1` or an equivalent local-only diagnostic. +- Modify docs: this roadmap, `docs/current-status-card.md`, and `docs/source-discovery/capability-source-matrix.md`. + +**Diagnostic policy:** +- Read copied DB/cache metadata only; do not read attachment contents. +- Do not print local paths, raw rows, real message bodies, or unredacted file names. +- Output counts, extension groups, hash-directory shape, match strategy counts, and safety booleans only. + +**Acceptance gate:** +- Diagnostic output includes `raw_paths_printed=false`, `file_contents_read=false`, and `file_paths_returned=false`. +- Diagnostic chooses one route: R4 cache-file resolver, UI/client download connector, or additional evidence request. + +**Stop condition:** +- If no unique/repeatable cache mapping exists, do not implement R4 download mode. + --- ### R4: Receive-file download implementation