docs: confirm file receive metadata evidence
This commit is contained in:
zhaoyilun
@@ -1,6 +1,6 @@
|
||||
# Current Status Card
|
||||
|
||||
Date: 2026-07-09
|
||||
Date: 2026-07-10
|
||||
Branch: `main`
|
||||
Loop base: `e993ce1 docs: trim safety constraints from mcp roadmap`
|
||||
Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-report-hardening'`
|
||||
@@ -29,7 +29,7 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor
|
||||
|
||||
## What cannot be claimed externally
|
||||
|
||||
- The project has not yet implemented business MCP tools for message sending, file receiving, or file sending; `isphere_receive_messages`, `isphere_search_contacts`, and `isphere_search_groups` have operator-local encrypted log-file support but not a production ingestion path.
|
||||
- The project has not yet implemented business MCP tools for message sending, file receiving/download, or file sending; `isphere_receive_messages`, `isphere_search_contacts`, and `isphere_search_groups` have operator-local encrypted log-file support but not a production ingestion path.
|
||||
- The project has not completed production iSphere integration.
|
||||
- The project has not proven a stable real message source for the full business workflow beyond decrypted PacketReader log evidence and synthetic/redacted tests.
|
||||
- The project has not implemented real message sending or real file upload.
|
||||
@@ -41,7 +41,7 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor
|
||||
- The repository has a Go MCP + C# WinHelper foundation for controlled Windows-side read-only observation.
|
||||
- The helper protocol boundary is explicit and keeps Windows/UIA work separate from the Go MCP service layer.
|
||||
- The current code can support future MCP business tools by routing low-level Windows/UIA fallback work through a bounded helper layer.
|
||||
- The project has repeatable verification scripts for the helper/MCP foundation and the current five-tool surface.
|
||||
- The project has repeatable verification scripts for the helper/MCP foundation and the current seven-tool surface.
|
||||
- The project now has a corrected business contract and plan for core MCP communication tools, plus registered receive/contact/group read tools that can use `ISPHERE_PACKET_LOG_FILE` for operator-local encrypted PacketReader log lines.
|
||||
|
||||
## N13/N14/N15 positioning
|
||||
@@ -50,7 +50,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme
|
||||
|
||||
## Current loop
|
||||
|
||||
Current loop: `Stage C / Loop C12 - file receive source evidence precheck`.
|
||||
Current loop: `Stage C / Loop C13 - log-backed file metadata extraction`.
|
||||
|
||||
Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.
|
||||
|
||||
@@ -74,10 +74,11 @@ Current loop output so far:
|
||||
9. C9: group source evidence precheck confirmed groupchat/conference/MUC hits in ignored decrypted logs.
|
||||
10. C10: log-backed group candidate extraction from groupchat/conference JIDs.
|
||||
11. C11: `isphere_search_groups` MCP registration and seven-tool verification.
|
||||
12. C12: file evidence precheck confirmed enough PacketReader/Smark evidence for file metadata listing, but not enough for real cache/download mapping.
|
||||
|
||||
## Next business mainline
|
||||
|
||||
1. Run Stage C Loop C12: verify current evidence for `isphere_receive_files` before implementation.
|
||||
2. Implement file metadata extraction only after file-transfer/cache evidence is recorded.
|
||||
1. Run Stage C Loop C13: implement log-backed file metadata extraction from normalized messages.
|
||||
2. Register `isphere_receive_files` list mode after C13 source behavior passes; leave actual download/cache mapping as a later connector node.
|
||||
3. Add `isphere_send_message` and `isphere_send_file` after source discovery and connector selection.
|
||||
4. Keep UIA selector/report work as fallback support, not as the primary roadmap.
|
||||
|
||||
@@ -93,6 +93,21 @@ Candidate mapping to `isphere_receive_messages`:
|
||||
|
||||
`LoginInfo` can map the local logged-in account identity, but it contains sensitive user metadata. Use only field names and identifiers needed for normalization; do not include raw personal values in committed docs or test fixtures.
|
||||
|
||||
## C12 file-transfer evidence precheck
|
||||
|
||||
A temporary ignored scanner decrypted only the selected N12-pre log files and wrote count-only metadata to `runs/offline-evidence-intake/zyl-qqfile-20260709/metadata/c12-file-evidence-counts.json`. No raw decrypted message content is committed.
|
||||
|
||||
Count summary:
|
||||
|
||||
| Source | Decrypted | File-transfer / file evidence | Cache/download evidence |
|
||||
| --- | ---: | --- | --- |
|
||||
| `PacketReader.ProcessPacket` | 245/245 | 57 `<subject>` hits, 123 `FILE_TRANSFER` hits, 57 `FILE_TRANSFER_CANCEL` hits, 91 body file-extension hits, 107 parsed file-candidate messages | 0 `zyl\importal\<hash>` hits, 0 download-keyword hits |
|
||||
| `Smark.SendReceive` | 1,078/1,078 | 5 `FILE_TRANSFER` hits, 4 body file-extension hits, 5 parsed file-candidate messages | 0 `zyl\importal\<hash>` hits, 0 download-keyword hits |
|
||||
| `SaveToDB` | 14/14 | 9 generic file-extension hits and 2 attachment-keyword hits; no parseable XMPP message candidates | 0 `zyl\importal\<hash>` hits, 0 download-keyword hits |
|
||||
| `archive-list.txt` | n/a | 11 cached file entries under 11 `zyl\importal\<hash>` directories; extensions observed: `.doc`, `.docx`, `.jpg`, `.pdf`, `.xlsx` | cache entries exist, but no confirmed link from message log to cache hash yet |
|
||||
|
||||
Decision: file metadata listing can proceed from log-backed normalized messages. Real `download_ref`, `saved_path`, and cache hash resolution remain deferred until the message-to-cache mapping or a client-side download connector is validated.
|
||||
|
||||
## Updated source decision
|
||||
|
||||
The previous matrix selected `MsgLib.db` as the first source subject to schema inspection. Stage B2 changes that decision:
|
||||
|
||||
@@ -20,7 +20,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; wrapped `MsgLib.db` after DB wrapper is solved | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | C5 env loader wired through ISPHERE_PACKET_LOG_FILE; default source remains empty when unset | fixture-backed configured-source operator smoke |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages | decrypted roster/contact stanzas from `Smark.SendReceive`; `MsgLib.db` contact/conversation tables after DB wrapper is solved; UIA helper source if display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#log-decryption-result` | C8 MCP tool registered and verify-go-mcp covers default/configured contact smoke; JID-only display/account fallback | enrich display names from roster/db/uia evidence |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs | `MsgLib.db` group/conversation tables after DB wrapper is solved; UIA helper source if group display names are missing | C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C11 MCP tool registered and verify-go-mcp covers default/configured group smoke; JID-only display fallback | enrich group display/member data from roster/db/uia evidence |
|
||||
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas plus `zyl\importal\<hash>` cache mapping | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | candidate pending cache-reference mapping | fourth |
|
||||
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas for metadata listing; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C12 evidence confirmed metadata-list source: PacketReader has 107 parsed file candidates; no log-to-cache/download mapping yet | extract file metadata candidates from normalized messages |
|
||||
|
||||
## Stage C entry rule
|
||||
|
||||
|
||||
@@ -1223,15 +1223,15 @@ Implemented and verified in this loop:
|
||||
- If evidence exists, write C13 around file metadata extraction from normalized messages/cache references.
|
||||
- If evidence is insufficient, mark `isphere_receive_files` blocked on additional file-transfer/cache mapping evidence and move to source hardening/docs instead.
|
||||
|
||||
- [ ] **Step 1: Inspect current source evidence for file-transfer identifiers**
|
||||
- [x] **Step 1: Inspect current source evidence for file-transfer identifiers**
|
||||
|
||||
Use committed schema notes and ignored summaries/log scans. Do not commit raw logs or decrypted content.
|
||||
|
||||
- [ ] **Step 2: Update matrix and next loop**
|
||||
- [x] **Step 2: Update matrix and next loop**
|
||||
|
||||
Record whether file receive implementation can proceed from current evidence. Rewrite C13 accordingly.
|
||||
|
||||
- [ ] **Step 3: Verify docs-only change and commit**
|
||||
- [x] **Step 3: Verify docs-only change and commit**
|
||||
|
||||
Run:
|
||||
|
||||
@@ -1245,6 +1245,67 @@ Expected: all exit 0. Commit exact C12 paths only.
|
||||
|
||||
---
|
||||
|
||||
## Loop C12 Result
|
||||
|
||||
Evidence precheck result:
|
||||
|
||||
- A temporary ignored scanner decrypted selected N12-pre PacketReader/Smark/SaveToDB logs and wrote only count metadata to `runs/offline-evidence-intake/zyl-qqfile-20260709/metadata/c12-file-evidence-counts.json`.
|
||||
- `PacketReader.ProcessPacket`: 245/245 decrypted, 57 `<subject>` hits, 123 `FILE_TRANSFER` hits, 57 `FILE_TRANSFER_CANCEL` hits, 91 body file-extension hits, and 107 parsed file-candidate messages.
|
||||
- `Smark.SendReceive`: 1,078/1,078 decrypted, 5 `FILE_TRANSFER` hits, 4 body file-extension hits, and 5 parsed file-candidate messages.
|
||||
- `SaveToDB`: 14/14 decrypted, 9 generic file-extension hits and 2 attachment-keyword hits, but no parseable XMPP message candidates.
|
||||
- `archive-list.txt`: 11 `zyl\importal\<hash>` cached file entries across 11 hash directories, with `.doc`, `.docx`, `.jpg`, `.pdf`, and `.xlsx` extensions.
|
||||
- Decision: `isphere_receive_files` can proceed for read-only file metadata listing from normalized messages, but actual `download_ref`/cache-path resolution remains blocked until a log-to-cache hash mapping or client download connector is validated.
|
||||
|
||||
---
|
||||
|
||||
## Loop C13: Log-backed file metadata extraction
|
||||
|
||||
**Goal:** Extract read-only file metadata candidates from normalized log-backed messages, using message id plus filename as a stable local file id while cache/download resolution remains deferred.
|
||||
|
||||
**Planned files:**
|
||||
- Create: `internal/isphere/files.go`
|
||||
- Create: `internal/isphere/files_test.go`
|
||||
- Modify: `docs/source-discovery/capability-source-matrix.md`
|
||||
- Modify: `docs/current-status-card.md`
|
||||
- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`
|
||||
|
||||
**Planned behavior:**
|
||||
- Consume `[]isphere.Message` records returned by the existing log-backed receive source.
|
||||
- Treat a message as a file candidate when `Subject` contains `FILE_TRANSFER` or `Text` contains a supported file extension.
|
||||
- Extract file name from message text without logging or committing real raw filenames; tests must use synthetic/redacted names.
|
||||
- File fields: `FileID`, `MessageID`, `ConversationID`, `FileName`, `SizeBytes` nil/unknown, `MimeType` inferred from extension where possible, `CreatedAt` from message timestamp, `DownloadRef` empty, `SavedPath` empty, `SHA256` empty, `Source` `local_readonly`, `RawRef` `packetlog_file_transfer`.
|
||||
- Query supports `ConversationID`, `MessageID`, `FileID`, `NameContains`, and `Limit`; default/zero limit returns all candidates.
|
||||
- This loop does not register the MCP tool yet. Registration/list-mode smoke becomes C14 after source behavior passes tests.
|
||||
|
||||
- [ ] **Step 1: Write failing file extraction test**
|
||||
|
||||
Create `internal/isphere/files_test.go` and run:
|
||||
|
||||
```powershell
|
||||
go test ./internal/isphere -run TestListFilesFromMessagesExtractsFileTransferCandidates -v
|
||||
```
|
||||
|
||||
Expected initial failure: file query/types/functions do not exist.
|
||||
|
||||
- [ ] **Step 2: Implement file metadata extraction/search**
|
||||
|
||||
Create `internal/isphere/files.go` with minimal pure Go logic over normalized messages and synthetic/redacted fixtures.
|
||||
|
||||
- [ ] **Step 3: Run full verification and rewrite Loop C14**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
git diff --check
|
||||
go test ./...
|
||||
go build ./cmd/isphere-mcp
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/verify-go-mcp.ps1
|
||||
```
|
||||
|
||||
Expected: all exit 0. Remove any generated root `isphere-mcp.exe` after build. Rewrite Loop C14 around registering `isphere_receive_files` list mode from the C13 source behavior.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
- Spec coverage: the plan follows the user's cyclic requirement: plan first, implement one loop at a time, update the next loop after each result, and ask only on blockers.
|
||||
|
||||
Reference in New Issue
Block a user