From 3d68a18097c63e090cd8060e4a9ab0d504e2688a Mon Sep 17 00:00:00 2001 From: zhaoyilun Date: Fri, 10 Jul 2026 00:40:02 +0800 Subject: [PATCH] docs: confirm file receive metadata evidence --- docs/current-status-card.md | 13 ++-- .../2026-07-09-n12-pre-zyl-schema-notes.md | 15 +++++ .../capability-source-matrix.md | 2 +- ...tage-c-log-backed-receive-messages-loop.md | 67 ++++++++++++++++++- 4 files changed, 87 insertions(+), 10 deletions(-) diff --git a/docs/current-status-card.md b/docs/current-status-card.md index 8b55d1c..21d3bb6 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -1,6 +1,6 @@ # Current Status Card -Date: 2026-07-09 +Date: 2026-07-10 Branch: `main` Loop base: `e993ce1 docs: trim safety constraints from mcp roadmap` Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-report-hardening'` @@ -29,7 +29,7 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor ## What cannot be claimed externally -- The project has not yet implemented business MCP tools for message sending, file receiving, or file sending; `isphere_receive_messages`, `isphere_search_contacts`, and `isphere_search_groups` have operator-local encrypted log-file support but not a production ingestion path. +- The project has not yet implemented business MCP tools for message sending, file receiving/download, or file sending; `isphere_receive_messages`, `isphere_search_contacts`, and `isphere_search_groups` have operator-local encrypted log-file support but not a production ingestion path. - The project has not completed production iSphere integration. - The project has not proven a stable real message source for the full business workflow beyond decrypted PacketReader log evidence and synthetic/redacted tests. - The project has not implemented real message sending or real file upload. @@ -41,7 +41,7 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor - The repository has a Go MCP + C# WinHelper foundation for controlled Windows-side read-only observation. - The helper protocol boundary is explicit and keeps Windows/UIA work separate from the Go MCP service layer. - The current code can support future MCP business tools by routing low-level Windows/UIA fallback work through a bounded helper layer. -- The project has repeatable verification scripts for the helper/MCP foundation and the current five-tool surface. +- The project has repeatable verification scripts for the helper/MCP foundation and the current seven-tool surface. - The project now has a corrected business contract and plan for core MCP communication tools, plus registered receive/contact/group read tools that can use `ISPHERE_PACKET_LOG_FILE` for operator-local encrypted PacketReader log lines. ## N13/N14/N15 positioning @@ -50,7 +50,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Stage C / Loop C12 - file receive source evidence precheck`. +Current loop: `Stage C / Loop C13 - log-backed file metadata extraction`. Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. @@ -74,10 +74,11 @@ Current loop output so far: 9. C9: group source evidence precheck confirmed groupchat/conference/MUC hits in ignored decrypted logs. 10. C10: log-backed group candidate extraction from groupchat/conference JIDs. 11. C11: `isphere_search_groups` MCP registration and seven-tool verification. +12. C12: file evidence precheck confirmed enough PacketReader/Smark evidence for file metadata listing, but not enough for real cache/download mapping. ## Next business mainline -1. Run Stage C Loop C12: verify current evidence for `isphere_receive_files` before implementation. -2. Implement file metadata extraction only after file-transfer/cache evidence is recorded. +1. Run Stage C Loop C13: implement log-backed file metadata extraction from normalized messages. +2. Register `isphere_receive_files` list mode after C13 source behavior passes; leave actual download/cache mapping as a later connector node. 3. Add `isphere_send_message` and `isphere_send_file` after source discovery and connector selection. 4. Keep UIA selector/report work as fallback support, not as the primary roadmap. diff --git a/docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md b/docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md index fc2c506..0152223 100644 --- a/docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md +++ b/docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md @@ -93,6 +93,21 @@ Candidate mapping to `isphere_receive_messages`: `LoginInfo` can map the local logged-in account identity, but it contains sensitive user metadata. Use only field names and identifiers needed for normalization; do not include raw personal values in committed docs or test fixtures. +## C12 file-transfer evidence precheck + +A temporary ignored scanner decrypted only the selected N12-pre log files and wrote count-only metadata to `runs/offline-evidence-intake/zyl-qqfile-20260709/metadata/c12-file-evidence-counts.json`. No raw decrypted message content is committed. + +Count summary: + +| Source | Decrypted | File-transfer / file evidence | Cache/download evidence | +| --- | ---: | --- | --- | +| `PacketReader.ProcessPacket` | 245/245 | 57 `` hits, 123 `FILE_TRANSFER` hits, 57 `FILE_TRANSFER_CANCEL` hits, 91 body file-extension hits, 107 parsed file-candidate messages | 0 `zyl\importal\` hits, 0 download-keyword hits | +| `Smark.SendReceive` | 1,078/1,078 | 5 `FILE_TRANSFER` hits, 4 body file-extension hits, 5 parsed file-candidate messages | 0 `zyl\importal\` hits, 0 download-keyword hits | +| `SaveToDB` | 14/14 | 9 generic file-extension hits and 2 attachment-keyword hits; no parseable XMPP message candidates | 0 `zyl\importal\` hits, 0 download-keyword hits | +| `archive-list.txt` | n/a | 11 cached file entries under 11 `zyl\importal\` directories; extensions observed: `.doc`, `.docx`, `.jpg`, `.pdf`, `.xlsx` | cache entries exist, but no confirmed link from message log to cache hash yet | + +Decision: file metadata listing can proceed from log-backed normalized messages. Real `download_ref`, `saved_path`, and cache hash resolution remain deferred until the message-to-cache mapping or a client-side download connector is validated. + ## Updated source decision The previous matrix selected `MsgLib.db` as the first source subject to schema inspection. Stage B2 changes that decision: diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index 384b8e5..e31b992 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -20,7 +20,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; wrapped `MsgLib.db` after DB wrapper is solved | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | C5 env loader wired through ISPHERE_PACKET_LOG_FILE; default source remains empty when unset | fixture-backed configured-source operator smoke | | `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages | decrypted roster/contact stanzas from `Smark.SendReceive`; `MsgLib.db` contact/conversation tables after DB wrapper is solved; UIA helper source if display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#log-decryption-result` | C8 MCP tool registered and verify-go-mcp covers default/configured contact smoke; JID-only display/account fallback | enrich display names from roster/db/uia evidence | | `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs | `MsgLib.db` group/conversation tables after DB wrapper is solved; UIA helper source if group display names are missing | C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C11 MCP tool registered and verify-go-mcp covers default/configured group smoke; JID-only display fallback | enrich group display/member data from roster/db/uia evidence | -| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas plus `zyl\importal\` cache mapping | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | candidate pending cache-reference mapping | fourth | +| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas for metadata listing; `zyl\importal\` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C12 evidence confirmed metadata-list source: PacketReader has 107 parsed file candidates; no log-to-cache/download mapping yet | extract file metadata candidates from normalized messages | ## Stage C entry rule diff --git a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md index 526f2cd..b5465b3 100644 --- a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md +++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md @@ -1223,15 +1223,15 @@ Implemented and verified in this loop: - If evidence exists, write C13 around file metadata extraction from normalized messages/cache references. - If evidence is insufficient, mark `isphere_receive_files` blocked on additional file-transfer/cache mapping evidence and move to source hardening/docs instead. -- [ ] **Step 1: Inspect current source evidence for file-transfer identifiers** +- [x] **Step 1: Inspect current source evidence for file-transfer identifiers** Use committed schema notes and ignored summaries/log scans. Do not commit raw logs or decrypted content. -- [ ] **Step 2: Update matrix and next loop** +- [x] **Step 2: Update matrix and next loop** Record whether file receive implementation can proceed from current evidence. Rewrite C13 accordingly. -- [ ] **Step 3: Verify docs-only change and commit** +- [x] **Step 3: Verify docs-only change and commit** Run: @@ -1245,6 +1245,67 @@ Expected: all exit 0. Commit exact C12 paths only. --- +## Loop C12 Result + +Evidence precheck result: + +- A temporary ignored scanner decrypted selected N12-pre PacketReader/Smark/SaveToDB logs and wrote only count metadata to `runs/offline-evidence-intake/zyl-qqfile-20260709/metadata/c12-file-evidence-counts.json`. +- `PacketReader.ProcessPacket`: 245/245 decrypted, 57 `` hits, 123 `FILE_TRANSFER` hits, 57 `FILE_TRANSFER_CANCEL` hits, 91 body file-extension hits, and 107 parsed file-candidate messages. +- `Smark.SendReceive`: 1,078/1,078 decrypted, 5 `FILE_TRANSFER` hits, 4 body file-extension hits, and 5 parsed file-candidate messages. +- `SaveToDB`: 14/14 decrypted, 9 generic file-extension hits and 2 attachment-keyword hits, but no parseable XMPP message candidates. +- `archive-list.txt`: 11 `zyl\importal\` cached file entries across 11 hash directories, with `.doc`, `.docx`, `.jpg`, `.pdf`, and `.xlsx` extensions. +- Decision: `isphere_receive_files` can proceed for read-only file metadata listing from normalized messages, but actual `download_ref`/cache-path resolution remains blocked until a log-to-cache hash mapping or client download connector is validated. + +--- + +## Loop C13: Log-backed file metadata extraction + +**Goal:** Extract read-only file metadata candidates from normalized log-backed messages, using message id plus filename as a stable local file id while cache/download resolution remains deferred. + +**Planned files:** +- Create: `internal/isphere/files.go` +- Create: `internal/isphere/files_test.go` +- Modify: `docs/source-discovery/capability-source-matrix.md` +- Modify: `docs/current-status-card.md` +- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md` + +**Planned behavior:** +- Consume `[]isphere.Message` records returned by the existing log-backed receive source. +- Treat a message as a file candidate when `Subject` contains `FILE_TRANSFER` or `Text` contains a supported file extension. +- Extract file name from message text without logging or committing real raw filenames; tests must use synthetic/redacted names. +- File fields: `FileID`, `MessageID`, `ConversationID`, `FileName`, `SizeBytes` nil/unknown, `MimeType` inferred from extension where possible, `CreatedAt` from message timestamp, `DownloadRef` empty, `SavedPath` empty, `SHA256` empty, `Source` `local_readonly`, `RawRef` `packetlog_file_transfer`. +- Query supports `ConversationID`, `MessageID`, `FileID`, `NameContains`, and `Limit`; default/zero limit returns all candidates. +- This loop does not register the MCP tool yet. Registration/list-mode smoke becomes C14 after source behavior passes tests. + +- [ ] **Step 1: Write failing file extraction test** + +Create `internal/isphere/files_test.go` and run: + +```powershell +go test ./internal/isphere -run TestListFilesFromMessagesExtractsFileTransferCandidates -v +``` + +Expected initial failure: file query/types/functions do not exist. + +- [ ] **Step 2: Implement file metadata extraction/search** + +Create `internal/isphere/files.go` with minimal pure Go logic over normalized messages and synthetic/redacted fixtures. + +- [ ] **Step 3: Run full verification and rewrite Loop C14** + +Run: + +```powershell +git diff --check +go test ./... +go build ./cmd/isphere-mcp +powershell -NoProfile -ExecutionPolicy Bypass -File scripts/verify-go-mcp.ps1 +``` + +Expected: all exit 0. Remove any generated root `isphere-mcp.exe` after build. Rewrite Loop C14 around registering `isphere_receive_files` list mode from the C13 source behavior. + +--- + ## Self-Review - Spec coverage: the plan follows the user's cyclic requirement: plan first, implement one loop at a time, update the next loop after each result, and ask only on blockers.