diff --git a/docs/current-status-card.md b/docs/current-status-card.md index 85750a2..2bf1672 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -47,24 +47,29 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Stage B1 - source discovery from N12-pre zyl package`. +Current loop: `Stage C / Loop C1 - log decryptor and PacketReader parser`. -Plan file: `docs/superpowers/plans/2026-07-09-mcp-core-source-discovery-loop.md`. +Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. -Loop output: +Loop rule: -1. Evidence index for downloaded `zyl.rar` package. -2. Capability-source matrix for the four read tools: - - `isphere_search_contacts` - - `isphere_search_groups` - - `isphere_receive_messages` - - `isphere_receive_files` -3. Selected first Stage C implementation slice. +1. Write or update the plan before coding. +2. Implement one loop only. +3. Verify, commit exact paths, then update the next loop based on actual results. +4. Continue automatically while the next step is clear; ask the user only when blocked or when business direction changes. + +Current loop output: + +1. `internal/isphere/logcodec.DecryptLine` for DES/CBC/PKCS7 Base64 log lines. +2. `internal/isphere/packetlog.ParseMessageLog` for redacted `PacketReader.ProcessPacket` XMPP `` stanzas. +3. Passing unit tests with synthetic/redacted fixtures only. +4. Updated Loop C2 plan before source abstraction starts. ## Next business mainline -1. Complete Stage B1 evidence index and capability-source matrix. -2. Open Stage C with `isphere_receive_messages` as the first likely slice if evidence supports it. -3. Add contact/group/file read tools after their source decisions are recorded. -4. Add `isphere_send_message` and `isphere_send_file` after source discovery and connector selection. -5. Keep UIA selector/report work as fallback support, not as the primary roadmap. +1. Complete Stage C Loop C1: log decryptor + PacketReader parser. +2. Rewrite and execute Loop C2: log-backed `ReceiveMessages` source abstraction. +3. Register `isphere_receive_messages` only after parser/source tests pass. +4. Add contact/group/file read tools after their source decisions are recorded. +5. Add `isphere_send_message` and `isphere_send_file` after source discovery and connector selection. +6. Keep UIA selector/report work as fallback support, not as the primary roadmap. diff --git a/docs/source-discovery/README.md b/docs/source-discovery/README.md index 63cc979..55e2ac2 100644 --- a/docs/source-discovery/README.md +++ b/docs/source-discovery/README.md @@ -1,39 +1,43 @@ # Source Discovery Ledger Date: 2026-07-09 -Current loop: Stage B1 - N12-pre zyl evidence source discovery +Current loop: Stage C / Loop C1 - log decryptor and PacketReader parser ## Objective -Find the best real capability source for the MCP core read tools before implementing Stage C. +Build the first business read capability path for `isphere_receive_messages` from decrypted `PacketReader.ProcessPacket` XMPP message logs. ## Inputs -- `runs/offline-evidence-intake/zyl-qqfile-20260709/archives/zyl.rar` -- `runs/offline-evidence-intake/zyl-qqfile-20260709/metadata/archive-list.txt` -- `runs/offline-evidence-intake/zyl-qqfile-20260709/metadata/sha256.txt` +- `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md` +- `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` +- `docs/source-discovery/capability-source-matrix.md` - `docs/mcp-core-tools-contract.md` - `docs/mcp-core-business-plan.md` -## Loop cadence +## Loop protocol -1. Inventory evidence. -2. Identify source candidates. -3. Map candidates to MCP tools. -4. Select primary/fallback source per tool. -5. Open one Stage C implementation slice. -6. Verify with `go test ./...`, `go build ./cmd/isphere-mcp`, and MCP verification script. -7. Commit exact paths. +1. Write or update the plan before coding. +2. Implement one loop only. +3. Run loop-specific tests. +4. Run full verification with `go test ./...` and `go build ./cmd/isphere-mcp`. +5. Commit exact paths. +6. Update the next loop based on actual results. +7. Continue automatically while the next step is clear; ask the user only when blocked or when the business direction changes. -## Current target order +## Current loop output -1. `isphere_receive_messages` -2. `isphere_search_contacts` -3. `isphere_search_groups` -4. `isphere_receive_files` +Loop C1 must produce: -Reason: message source discovery is the current blocker; contacts/groups can be derived or selected after the message source shape is known. +1. `internal/isphere/logcodec.DecryptLine` for DES/CBC/PKCS7 Base64 log lines. +2. `internal/isphere/packetlog.ParseMessageLog` for redacted `PacketReader.ProcessPacket` XMPP `` stanzas. +3. Passing unit tests with synthetic/redacted fixtures only. +4. A plan update for Loop C2 before any source abstraction code starts. ## Current plan -See `docs/superpowers/plans/2026-07-09-mcp-core-source-discovery-loop.md`. +See `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. + +## Last completed discovery result + +Stage B2 showed that selected `MsgLib.db` files are not direct SQLite. Decrypted `PacketReader.ProcessPacket` XML is now the first implementable source for `isphere_receive_messages`. diff --git a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md new file mode 100644 index 0000000..408e51c --- /dev/null +++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md @@ -0,0 +1,455 @@ +# Stage C Log-backed Receive Messages Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the first business read capability for `isphere_receive_messages` using decrypted `PacketReader.ProcessPacket` XMPP message logs from the N12-pre zyl evidence package. + +**Architecture:** Work in short closed loops. Each loop has one deliverable, a verification gate, a commit, and a plan-rewrite step before the next loop starts. Stage C starts with pure Go parsing code and redacted fixtures, then adds a source abstraction, then registers the MCP tool only after parser/source behavior is verified. + +**Tech Stack:** Go 1.23.4, standard library `crypto/des`, `crypto/cipher`, `encoding/base64`, `encoding/xml`, existing Go MCP SDK, PowerShell verification scripts. + +## Global Constraints + +- Repository root: `E:\coding\codex\isphere-ai-bridge`. +- Remote name: `gitea`; active branch: `main`. +- Code search rule from `AGENTS.md`: use `git ls-files`, `ag`, or `grep -R`; do not use `rg`. +- Current evidence decision source: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`. +- Current capability matrix: `docs/source-discovery/capability-source-matrix.md`. +- Raw evidence and extracted logs stay under `runs/`; never commit raw DB/log files or decrypted user message content. +- Tests use synthetic or redacted fixtures only. +- Every implementation loop must run `go test ./...` and `go build ./cmd/isphere-mcp` before commit. +- Every loop must update this plan or a loop notes document before the next loop starts. +- If a loop output contradicts the current plan, update the next loop rather than pushing ahead. + +--- + +## Loop Protocol + +Each loop follows the same 7-step protocol: + +1. State the active loop id and one-sentence goal. +2. Read the current plan section for that loop. +3. Implement only that loop's files. +4. Run the loop-specific test first. +5. Run full verification: `go test ./...`, `go build ./cmd/isphere-mcp`. +6. Commit exact paths. +7. Rewrite the next loop section based on the actual result. + +Stop and ask the user only when: + +- The current loop cannot proceed because required evidence is missing. +- The actual output contradicts the product goal and no local fallback is clear. +- A decision affects business scope, such as switching from log-backed read to UIA or DB-wrapper work as the mainline. + +--- + +## File Structure + +### Loop C1 files + +- Create: `E:\coding\codex\isphere-ai-bridge\internal\isphere\logcodec\codec.go` +- Create: `E:\coding\codex\isphere-ai-bridge\internal\isphere\logcodec\codec_test.go` +- Create: `E:\coding\codex\isphere-ai-bridge\internal\isphere\packetlog\message.go` +- Create: `E:\coding\codex\isphere-ai-bridge\internal\isphere\packetlog\message_test.go` +- Modify: `E:\coding\codex\isphere-ai-bridge\docs\superpowers\plans\2026-07-09-stage-c-log-backed-receive-messages-loop.md` + +### Later loop files + +- Create: `E:\coding\codex\isphere-ai-bridge\internal\isphere\source.go` +- Create: `E:\coding\codex\isphere-ai-bridge\internal\isphere\source_test.go` +- Create: `E:\coding\codex\isphere-ai-bridge\internal\tools\isphere_read.go` +- Create: `E:\coding\codex\isphere-ai-bridge\internal\tools\isphere_read_test.go` +- Modify: `E:\coding\codex\isphere-ai-bridge\internal\mcpserver\server.go` +- Modify: `E:\coding\codex\isphere-ai-bridge\internal\tools\winhelper_test.go` +- Modify: `E:\coding\codex\isphere-ai-bridge\docs\source-discovery\capability-source-matrix.md` + +--- + +## Loop C1: Log decryptor and PacketReader parser + +**Goal:** Produce pure Go code that decrypts one Base64 DES/CBC/PKCS7 log line and parses a redacted `PacketReader.ProcessPacket` XMPP `` stanza into a normalized internal message. + +**Files:** +- Create: `internal/isphere/logcodec/codec.go` +- Create: `internal/isphere/logcodec/codec_test.go` +- Create: `internal/isphere/packetlog/message.go` +- Create: `internal/isphere/packetlog/message_test.go` +- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md` + +**Interfaces:** +- Produces: `logcodec.DecryptLine(ciphertextBase64 string) (string, error)`. +- Produces: `packetlog.ParseMessageLog(plaintext string) (packetlog.Message, error)`. +- Produces: `packetlog.Message` with fields `ID`, `From`, `To`, `Type`, `Body`, `Subject`, `SendTime`, `ReceiptID`, `ReceiptType`, `Consumed`, and `Read`. + +- [ ] **Step 1: Write failing decrypt test** + +Create `internal/isphere/logcodec/codec_test.go`: + +```go +package logcodec + +import "testing" + +func TestDecryptLineWithKnownPolicyRoundTrip(t *testing.T) { + plaintext := "--------------------------------------------------------------------------------------------------------------------------------------------\r\n2026/7/7 15:30:07\r\nredacted" + ciphertext := encryptLineForTest(t, plaintext) + + got, err := DecryptLine(ciphertext) + if err != nil { + t.Fatalf("DecryptLine returned error: %v", err) + } + if got != plaintext { + t.Fatalf("DecryptLine() = %q, want %q", got, plaintext) + } +} +``` + +Run: + +```powershell +go test ./internal/isphere/logcodec -run TestDecryptLineWithKnownPolicyRoundTrip -v +``` + +Expected: FAIL because the package/function does not exist. + +- [ ] **Step 2: Implement decryptor and test helper** + +Create `internal/isphere/logcodec/codec.go`: + +```go +package logcodec + +import ( + "crypto/cipher" + "crypto/des" + "encoding/base64" + "errors" + "fmt" +) + +var ( + policyKey = []byte("hyhccdtm") + policyIV = []byte{0x12, 0x34, 0x56, 0x78, 0x90, 0xAB, 0xCD, 0xEF} +) + +func DecryptLine(ciphertextBase64 string) (string, error) { + ciphertext, err := base64.StdEncoding.DecodeString(ciphertextBase64) + if err != nil { + return "", fmt.Errorf("decode base64 log line: %w", err) + } + if len(ciphertext) == 0 || len(ciphertext)%des.BlockSize != 0 { + return "", fmt.Errorf("ciphertext length %d is not a positive DES block multiple", len(ciphertext)) + } + + block, err := des.NewCipher(policyKey) + if err != nil { + return "", fmt.Errorf("create DES cipher: %w", err) + } + plain := make([]byte, len(ciphertext)) + cipher.NewCBCDecrypter(block, policyIV).CryptBlocks(plain, ciphertext) + plain, err = unpadPKCS7(plain, des.BlockSize) + if err != nil { + return "", err + } + return string(plain), nil +} + +func unpadPKCS7(data []byte, blockSize int) ([]byte, error) { + if len(data) == 0 || len(data)%blockSize != 0 { + return nil, errors.New("pkcs7 data is empty or not block aligned") + } + pad := int(data[len(data)-1]) + if pad == 0 || pad > blockSize || pad > len(data) { + return nil, fmt.Errorf("invalid pkcs7 padding length %d", pad) + } + for i := len(data) - pad; i < len(data); i++ { + if int(data[i]) != pad { + return nil, errors.New("invalid pkcs7 padding bytes") + } + } + return data[:len(data)-pad], nil +} +``` + +Append this helper to `internal/isphere/logcodec/codec_test.go`: + +```go +func encryptLineForTest(t *testing.T, plaintext string) string { + t.Helper() + block, err := des.NewCipher(policyKey) + if err != nil { + t.Fatalf("NewCipher: %v", err) + } + plain := padPKCS7ForTest([]byte(plaintext), des.BlockSize) + ciphertext := make([]byte, len(plain)) + cipher.NewCBCEncrypter(block, policyIV).CryptBlocks(ciphertext, plain) + return base64.StdEncoding.EncodeToString(ciphertext) +} + +func padPKCS7ForTest(data []byte, blockSize int) []byte { + pad := blockSize - len(data)%blockSize + out := append([]byte(nil), data...) + for i := 0; i < pad; i++ { + out = append(out, byte(pad)) + } + return out +} +``` + +Add imports in `codec_test.go`: + +```go +import ( + "crypto/cipher" + "crypto/des" + "encoding/base64" + "testing" +) +``` + +- [ ] **Step 3: Verify decryptor test passes** + +Run: + +```powershell +go test ./internal/isphere/logcodec -run TestDecryptLineWithKnownPolicyRoundTrip -v +``` + +Expected: PASS. + +- [ ] **Step 4: Write failing PacketReader parser test** + +Create `internal/isphere/packetlog/message_test.go`: + +```go +package packetlog + +import "testing" + +func TestParseMessageLogExtractsXMPPMessage(t *testing.T) { + plaintext := "--------------------------------------------------------------------------------------------------------------------------------------------\r\n2026/7/7 15:30:07\r\n\r\n redacted.docx\r\n \r\n FILE_TRANSFER_CANCEL\r\n \r\n \r\n \r\n \r\n \r\n" + + got, err := ParseMessageLog(plaintext) + if err != nil { + t.Fatalf("ParseMessageLog returned error: %v", err) + } + if got.ID != "2de080bf-f5cc-45e2-b229-2df6bcd4afe3" { + t.Fatalf("ID = %q", got.ID) + } + if got.From != "sender@imopenfire1-lanzhou/imp_pc_4.1.2.6842" || got.To != "receiver@imopenfire1-lanzhou" { + t.Fatalf("from/to = %q/%q", got.From, got.To) + } + if got.Body != "redacted.docx" || got.Subject != "FILE_TRANSFER_CANCEL" { + t.Fatalf("body/subject = %q/%q", got.Body, got.Subject) + } + if got.SendTime != "1783423807000" || got.ReceiptID != "167c08c3d710489f812c13dc23d4e404" || got.ReceiptType != "1" { + t.Fatalf("receipt fields = %+v", got) + } + if !got.Consumed || !got.Read { + t.Fatalf("consumed/read = %v/%v", got.Consumed, got.Read) + } +} +``` + +Run: + +```powershell +go test ./internal/isphere/packetlog -run TestParseMessageLogExtractsXMPPMessage -v +``` + +Expected: FAIL because the package/function does not exist. + +- [ ] **Step 5: Implement PacketReader parser** + +Create `internal/isphere/packetlog/message.go`: + +```go +package packetlog + +import ( + "encoding/xml" + "errors" + "fmt" + "strings" +) + +type Message struct { + ID string + From string + To string + Type string + Body string + Subject string + SendTime string + ReceiptID string + ReceiptType string + Consumed bool + Read bool +} + +type xmppMessage struct { + XMLName xml.Name `xml:"message"` + ID string `xml:"id,attr"` + From string `xml:"from,attr"` + To string `xml:"to,attr"` + Type string `xml:"type,attr"` + Body string `xml:"body"` + Subject string `xml:"subject"` + Receipt xmppReceipt `xml:"received"` + ISphere xmppISphere `xml:"isphere"` + Consumed *struct{} `xml:"consumed"` + Readed *struct{} `xml:"readed"` +} + +type xmppReceipt struct { + ID string `xml:"id,attr"` + Type string `xml:"type,attr"` +} + +type xmppISphere struct { + SendTime string `xml:"sendtime,attr"` +} + +func ParseMessageLog(plaintext string) (Message, error) { + start := strings.Index(plaintext, "