diff --git a/docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md b/docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md new file mode 100644 index 0000000..fc2c506 --- /dev/null +++ b/docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md @@ -0,0 +1,112 @@ +# N12-pre zyl Schema and Log Inspection Notes + +Date: 2026-07-09 +Evidence id: `zyl-qqfile-20260709` +Stage: B2 selected DB/log inspection + +## Files inspected under `runs/` + +| Label | Extracted path | Result | +| --- | --- | --- | +| iSphere latest DB | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/msgdb-isphere-90005981/MsgLib.db` | high-entropy binary; not direct SQLite | +| IMPP same-account DB | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/msgdb-impp-90005981/MsgLib.db` | high-entropy binary; not direct SQLite | +| IMPP large DB | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/msgdb-impp-27091282/MsgLib.db` | high-entropy binary; not direct SQLite | +| `Smark.SendReceive` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-smark-90005981-20260707/2026-07-07_1.log` | decryptable line-based log | +| `SaveToDB` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-savetodb-90005981-20260707/2026-07-07_1.log` | decryptable line-based log | +| `PacketReader.ProcessPacket` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-packetreader-90005981-20260707/2026-07-07_1.log` | decryptable line-based XMPP message log | +| `LoginInfo` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-logininfo-90005981-20260707/2026-07-07_1.log` | decryptable login metadata XML | + +## `MsgLib.db` inspection result + +The three selected `MsgLib.db` files are not directly readable with Python `sqlite3` read-only mode. + +Observed result: + +```text +DatabaseError('file is not a database') +``` + +Observed binary traits: + +| File | Size bytes | SHA256 | Header / format signal | +| --- | ---: | --- | --- | +| `msgdb-isphere-90005981/MsgLib.db` | 2,839,552 | `4f308a26e4d528175853dac8889b69f371b4cad779f75ce2e31d2f83043f86a2` | high entropy, first bytes start `45 1b bd 2b`; not `SQLite format 3` | +| `msgdb-impp-90005981/MsgLib.db` | 2,776,064 | `6707d06126e3cd9a535f6d3879123e31377f0cb7d350b7a6e0cdfe1b34656943` | high entropy, first bytes start `45 1b bd 2b`; not `SQLite format 3` | +| `msgdb-impp-27091282/MsgLib.db` | 12,198,912 | `fe60ddcfbaa70523b21ec080f6d7e633298c9a9d710b7e9c36d91996bf4dc0f0` | high entropy, first bytes start `45 1b bd 2b`; not `SQLite format 3` | + +Tried the known `IMPP.Util.Log.LogSecurityPolicy` DES/CBC key against the first DB blocks. It did not reveal a SQLite header. Current conclusion: `MsgLib.db` remains an important source candidate, but it is not the first implementable Stage C source until the DB wrapper/encryption/loading path is identified. + +## Log decryption result + +The selected logs are Base64 line-based ciphertext. The existing local iSphere log policy finding applies to these logs: + +- policy owner: `IMPP.Util.Log.LogSecurityPolicy` +- cipher: DES +- mode: CBC +- padding: PKCS7 +- key string: `hyhccdtm` +- IV bytes: `12 34 56 78 90 AB CD EF` +- plaintext encoding: UTF-8 + +Decryption succeeded for all selected log lines: + +| Log | Lines | Decrypted | Useful structure | +| --- | ---: | ---: | --- | +| `SaveToDB` | 14 | 14 | NLog-style records from `IMPP.Client.control.MessageCenter.Chat_OnMessageArrived`; message type `com.vision.smack.packet.ReceiptMessage`; includes `from:` and `body:` fields | +| `Smark.SendReceive` | 1,078 | 1,078 | XMPP send/receive stanzas; includes IQ and message traffic | +| `PacketReader.ProcessPacket` | 245 | 245 | XMPP `` stanzas; includes `id`, `from`, `to`, `type`, ``, ``, `` | +| `LoginInfo` | 1 | 1 | XML login metadata; contains account/user/dept/contact fields and should be treated as sensitive metadata | + +## Field mapping evidence + +`PacketReader.ProcessPacket` is now the strongest immediate source for `isphere_receive_messages` because decrypted lines expose message-level XML shape. + +Observed redacted shape: + +```xml + + ... + + ... + + + + +``` + +Candidate mapping to `isphere_receive_messages`: + +| MCP field | Candidate source | +| --- | --- | +| `message_id` | `` | +| `conversation_id` | normalized `from` / `to` JID pair or group JID | +| `conversation_type` | `` and JID shape | +| `sender` | `from` JID, stripped of resource such as `/imp_pc_4.1.2.6842` | +| `timestamp` | log timestamp line plus `` when present | +| `content_text` | `...` | +| attachment/file hint | `` plus body filename when file-transfer subject is present | +| receipt/status | ``, ``, `` elements when present | + +`SaveToDB` is useful as a behavioral trace but not the first parser source. It records `Chat_OnMessageArrived` and `ReceiptMessage` entries with `from:` and `body:` fields, and indicates some messages meet a “not stored to small DB” condition. This explains why `MsgLib.db` may not contain every received message even after its wrapper is solved. + +`Smark.SendReceive` is useful as a lower-level transport trace. It includes XMPP IQ/message traffic and can become a fallback source or reconciliation source after the `PacketReader` parser is stable. + +`LoginInfo` can map the local logged-in account identity, but it contains sensitive user metadata. Use only field names and identifiers needed for normalization; do not include raw personal values in committed docs or test fixtures. + +## Updated source decision + +The previous matrix selected `MsgLib.db` as the first source subject to schema inspection. Stage B2 changes that decision: + +1. `MsgLib.db` remains a high-value candidate but is blocked by a non-SQLite wrapper/encryption layer. +2. `PacketReader.ProcessPacket` decrypted XMPP XML is the first implementable source for `isphere_receive_messages`. +3. `Smark.SendReceive` and `SaveToDB` are fallback/reconciliation sources. +4. File receiving likely needs `PacketReader` file-transfer XML plus a later map to `zyl\importal\` document cache directories. + +## Next node + +Open Stage C with a log-backed source abstraction first: + +1. Add a small decrypt/parser package for `IMPP.Util.Log.LogSecurityPolicy` style lines. +2. Parse decrypted `PacketReader.ProcessPacket` XML into normalized message structs. +3. Use redacted fixture strings in tests, not raw user message text. +4. Keep `MsgLib.db` work as a separate follow-up node for DB wrapper/encryption analysis. diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index dbb71ec..5eab1de 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -3,6 +3,7 @@ Date: 2026-07-09 Stage: B source discovery Evidence index: `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md` +Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` ## Source priority @@ -16,63 +17,24 @@ Evidence index: `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md` | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | --- | --- | --- | --- | --- | --- | -| `isphere_receive_messages` | `zyl\iSphere\8c5e81a7f6dd1169.10083\27000000\Account\90005981\MsgLib.db` | `zyl\Impp\Account\90005981\MsgLib.db` plus `Smark.SendReceive`, `SaveToDB`, and `PacketReader.ProcessPacket` logs | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#first-extraction-target-list` | selected for schema inspection; implementation starts after message table mapping is confirmed | first | -| `isphere_search_contacts` | contact/conversation/person tables inside `MsgLib.db`, if schema exposes display names or account ids | infer from message rows; then UIA helper source through current WinHelper if DB lacks contact display names | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#msglibdb-candidates` | candidate pending DB schema inspection | second | -| `isphere_search_groups` | group/conversation tables inside `MsgLib.db`, if schema exposes group ids or group display names | infer from group-message rows; then UIA helper source through current WinHelper if DB lacks group display names | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#msglibdb-candidates` | candidate pending DB schema inspection | third | -| `isphere_receive_files` | importal hashed document cache mapped through `MsgLib.db` attachment/file-reference rows | `Smark.SendReceive` and `SaveToDB` logs for file-reference traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#receiveddocument-cache-candidates` | candidate pending DB schema and cache-reference inspection | fourth | +| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; wrapped `MsgLib.db` after DB wrapper is solved | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | selected for Stage C log-backed parser | first | +| `isphere_search_contacts` | decrypted packet/send-receive JIDs and later roster stanzas from `Smark.SendReceive` | `MsgLib.db` contact/conversation tables after DB wrapper is solved; UIA helper source if display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#log-decryption-result` | candidate pending parser coverage | second | +| `isphere_search_groups` | decrypted XMPP group/message stanzas if groupchat/group JIDs appear | `MsgLib.db` group/conversation tables after DB wrapper is solved; UIA helper source if group display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | candidate pending group stanza evidence | third | +| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas plus `zyl\importal\` cache mapping | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | candidate pending cache-reference mapping | fourth | ## Stage C entry rule -Open implementation only after the selected row links to a concrete file or directory in the evidence index and the schema/log inspection notes state which fields map to the MCP output contract. - -## Stage B2 extraction and inspection order - -| Order | Target | Why | Expected output | -| --- | --- | --- | --- | -| 1 | `zyl\iSphere\8c5e81a7f6dd1169.10083\27000000\Account\90005981\MsgLib.db` | latest visible iSphere message DB; best candidate for `isphere_receive_messages` | schema summary, table list, sample redacted row shapes | -| 2 | `zyl\Impp\Account\90005981\MsgLib.db` | same account id in older/parallel IMPP tree | schema comparison with iSphere DB | -| 3 | `zyl\Impp\Account\27091282\MsgLib.db` | largest DB, likely richer schema/data variety | schema comparison and attachment/contact table clues | -| 4 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\SaveToDB\2026-07-07_1.log` | may reveal which DB table receives message data | log field summary and DB/table clues | -| 5 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\Smark.SendReceive\2026-07-07_1.log` | send/receive trace near latest DB timestamp | message-id or payload-shape clues | -| 6 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\PacketReader.ProcessPacket\2026-07-07_1.log` | packet processing trace near latest DB timestamp | packet-to-message mapping clues | -| 7 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\LoginInfo\2026-07-07_1.log` | account/session context for source mapping | account id and environment context | - -## Tool-by-tool decision notes - -### `isphere_receive_messages` - -Selected first because the latest `90005981` iSphere `MsgLib.db` and same-day logs align around `2026-07-07 15:30-15:43`. Stage B2 must inspect the DB schema and produce a field mapping for: - -- message id -- conversation id or peer id -- sender id/name -- timestamp -- message body or content reference -- attachment/file reference if present - -After schema mapping, Stage C can start with a local DB-backed source abstraction before registering the business MCP tool. - -### `isphere_search_contacts` - -Candidate path is DB-first. If the DB contains contact/person tables, use those. If not, derive a minimal contact search result from message conversation rows and sender/peer identifiers, then use UIA as fallback for display-name enrichment. - -### `isphere_search_groups` - -Candidate path is DB-first. If the DB contains group/conversation tables, use those. If not, infer group-like conversations from message rows and use UIA as fallback for display-name enrichment. - -### `isphere_receive_files` - -Candidate path is DB/cache mapping. The hashed `zyl\importal\` directories appear to hold received document cache files. This tool needs DB/log references to map those hashed directories back to conversations/messages before implementation. +Open implementation only for the selected row whose evidence maps concrete fields to the MCP output contract. After Stage B2, `isphere_receive_messages` satisfies the entry rule through decrypted `PacketReader.ProcessPacket` XML shape. ## Stage C first slice recommendation -Start Stage C with a narrow `isphere_receive_messages` source abstraction: +Start Stage C with a narrow log-backed `isphere_receive_messages` source abstraction: -1. Extract the selected DB/log files into `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/`. -2. Inspect schema and record a schema note under `docs/source-discovery/`. -3. Implement a local DB source interface only after the table/field mapping is confirmed. -4. Keep MCP registration as the next commit after the source abstraction test passes. +1. Implement DES/CBC/PKCS7 line decryption for `IMPP.Util.Log.LogSecurityPolicy` format. +2. Implement a parser for decrypted `PacketReader.ProcessPacket` XMPP `` stanzas. +3. Normalize only redacted test fixtures into message structs first. +4. Add the MCP tool registration after the parser/source abstraction passes tests. -## Stop condition for this matrix +## Deferred source work -If DB schema inspection shows no message table or no usable message fields, do not start Stage C implementation. Return to Stage B and promote the log source or UIA helper source to the first candidate. +`MsgLib.db` is deferred because the selected DB files are not direct SQLite and did not decrypt to a SQLite header with the known log policy key. A separate DB-wrapper analysis node is required before using DB tables as a production read source.