admin/isphere-ai-bridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: inspect n12-pre message source formats

Browse Source
This commit is contained in:
zhaoyilun
2026-07-09 22:38:14 +08:00
parent 57b38d8244
commit 1eed0f62f2
2 changed files with 125 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

112
docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md Normal file
View File

@@ -0,0 +1,112 @@
# N12-pre zyl Schema and Log Inspection Notes
Date: 2026-07-09
Evidence id: `zyl-qqfile-20260709`
Stage: B2 selected DB/log inspection
## Files inspected under `runs/`
| Label | Extracted path | Result |
| --- | --- | --- |
| iSphere latest DB | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/msgdb-isphere-90005981/MsgLib.db` | high-entropy binary; not direct SQLite |
| IMPP same-account DB | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/msgdb-impp-90005981/MsgLib.db` | high-entropy binary; not direct SQLite |
| IMPP large DB | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/msgdb-impp-27091282/MsgLib.db` | high-entropy binary; not direct SQLite |
| `Smark.SendReceive` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-smark-90005981-20260707/2026-07-07_1.log` | decryptable line-based log |
| `SaveToDB` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-savetodb-90005981-20260707/2026-07-07_1.log` | decryptable line-based log |
| `PacketReader.ProcessPacket` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-packetreader-90005981-20260707/2026-07-07_1.log` | decryptable line-based XMPP message log |
| `LoginInfo` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-logininfo-90005981-20260707/2026-07-07_1.log` | decryptable login metadata XML |
## `MsgLib.db` inspection result
The three selected `MsgLib.db` files are not directly readable with Python `sqlite3` read-only mode.
Observed result:
```text
DatabaseError('file is not a database')
```
Observed binary traits:
| File | Size bytes | SHA256 | Header / format signal |
| --- | ---: | --- | --- |
| `msgdb-isphere-90005981/MsgLib.db` | 2,839,552 | `4f308a26e4d528175853dac8889b69f371b4cad779f75ce2e31d2f83043f86a2` | high entropy, first bytes start `45 1b bd 2b`; not `SQLite format 3` |
| `msgdb-impp-90005981/MsgLib.db` | 2,776,064 | `6707d06126e3cd9a535f6d3879123e31377f0cb7d350b7a6e0cdfe1b34656943` | high entropy, first bytes start `45 1b bd 2b`; not `SQLite format 3` |
| `msgdb-impp-27091282/MsgLib.db` | 12,198,912 | `fe60ddcfbaa70523b21ec080f6d7e633298c9a9d710b7e9c36d91996bf4dc0f0` | high entropy, first bytes start `45 1b bd 2b`; not `SQLite format 3` |
Tried the known `IMPP.Util.Log.LogSecurityPolicy` DES/CBC key against the first DB blocks. It did not reveal a SQLite header. Current conclusion: `MsgLib.db` remains an important source candidate, but it is not the first implementable Stage C source until the DB wrapper/encryption/loading path is identified.
## Log decryption result
The selected logs are Base64 line-based ciphertext. The existing local iSphere log policy finding applies to these logs:
- policy owner: `IMPP.Util.Log.LogSecurityPolicy`
- cipher: DES
- mode: CBC
- padding: PKCS7
- key string: `hyhccdtm`
- IV bytes: `12 34 56 78 90 AB CD EF`
- plaintext encoding: UTF-8
Decryption succeeded for all selected log lines:
| Log | Lines | Decrypted | Useful structure |
| --- | ---: | ---: | --- |
| `SaveToDB` | 14 | 14 | NLog-style records from `IMPP.Client.control.MessageCenter.Chat_OnMessageArrived`; message type `com.vision.smack.packet.ReceiptMessage`; includes `from:` and `body:` fields |
| `Smark.SendReceive` | 1,078 | 1,078 | XMPP send/receive stanzas; includes IQ and message traffic |
| `PacketReader.ProcessPacket` | 245 | 245 | XMPP `<message>` stanzas; includes `id`, `from`, `to`, `type`, `<body>`, `<subject>`, `<isphere ... sendtime=...>` |
| `LoginInfo` | 1 | 1 | XML login metadata; contains account/user/dept/contact fields and should be treated as sensitive metadata |
## Field mapping evidence
`PacketReader.ProcessPacket` is now the strongest immediate source for `isphere_receive_messages` because decrypted lines expose message-level XML shape.
Observed redacted shape:
```xml
<message id="..." from="<jid>/imp_pc_4.1.2.6842" to="<jid>" type="chat">
<body>...</body>
<received xmlns="urn:xmpp:receipts" id="..." type="1" stamp="" />
<subject>...</subject>
<isphere xmlns="isphere.im" type="1001" sendtime="..." version="...">
<alert type="0" />
</isphere>
</message>
```
Candidate mapping to `isphere_receive_messages`:
| MCP field | Candidate source |
| --- | --- |
| `message_id` | `<message id="...">` |
| `conversation_id` | normalized `from` / `to` JID pair or group JID |
| `conversation_type` | `<message type="chat|groupchat|...">` and JID shape |
| `sender` | `from` JID, stripped of resource such as `/imp_pc_4.1.2.6842` |
| `timestamp` | log timestamp line plus `<isphere sendtime="...">` when present |
| `content_text` | `<body>...</body>` |
| attachment/file hint | `<subject>` plus body filename when file-transfer subject is present |
| receipt/status | `<received>`, `<consumed>`, `<readed>` elements when present |
`SaveToDB` is useful as a behavioral trace but not the first parser source. It records `Chat_OnMessageArrived` and `ReceiptMessage` entries with `from:` and `body:` fields, and indicates some messages meet a “not stored to small DB” condition. This explains why `MsgLib.db` may not contain every received message even after its wrapper is solved.
`Smark.SendReceive` is useful as a lower-level transport trace. It includes XMPP IQ/message traffic and can become a fallback source or reconciliation source after the `PacketReader` parser is stable.
`LoginInfo` can map the local logged-in account identity, but it contains sensitive user metadata. Use only field names and identifiers needed for normalization; do not include raw personal values in committed docs or test fixtures.
## Updated source decision
The previous matrix selected `MsgLib.db` as the first source subject to schema inspection. Stage B2 changes that decision:
1. `MsgLib.db` remains a high-value candidate but is blocked by a non-SQLite wrapper/encryption layer.
2. `PacketReader.ProcessPacket` decrypted XMPP XML is the first implementable source for `isphere_receive_messages`.
3. `Smark.SendReceive` and `SaveToDB` are fallback/reconciliation sources.
4. File receiving likely needs `PacketReader` file-transfer XML plus a later map to `zyl\importal\<hash>` document cache directories.
## Next node
Open Stage C with a log-backed source abstraction first:
1. Add a small decrypt/parser package for `IMPP.Util.Log.LogSecurityPolicy` style lines.
2. Parse decrypted `PacketReader.ProcessPacket` XML into normalized message structs.
3. Use redacted fixture strings in tests, not raw user message text.
4. Keep `MsgLib.db` work as a separate follow-up node for DB wrapper/encryption analysis.

64
docs/source-discovery/capability-source-matrix.md
View File

@@ -3,6 +3,7 @@
Date: 2026-07-09 Date: 2026-07-09
Stage: B source discovery Stage: B source discovery
Evidence index: `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md` Evidence index: `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md`
Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
## Source priority ## Source priority
@@ -16,63 +17,24 @@ Evidence index: `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md`
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
| --- | --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- | --- |
| `isphere_receive_messages` | `zyl\iSphere\8c5e81a7f6dd1169.10083\27000000\Account\90005981\MsgLib.db` | `zyl\Impp\Account\90005981\MsgLib.db` plus `Smark.SendReceive`, `SaveToDB`, and `PacketReader.ProcessPacket` logs | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#first-extraction-target-list` | selected for schema inspection; implementation starts after message table mapping is confirmed | first | | `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; wrapped `MsgLib.db` after DB wrapper is solved | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | selected for Stage C log-backed parser | first |
| `isphere_search_contacts` | contact/conversation/person tables inside `MsgLib.db`, if schema exposes display names or account ids | infer from message rows; then UIA helper source through current WinHelper if DB lacks contact display names | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#msglibdb-candidates` | candidate pending DB schema inspection | second | | `isphere_search_contacts` | decrypted packet/send-receive JIDs and later roster stanzas from `Smark.SendReceive` | `MsgLib.db` contact/conversation tables after DB wrapper is solved; UIA helper source if display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#log-decryption-result` | candidate pending parser coverage | second |
| `isphere_search_groups` | group/conversation tables inside `MsgLib.db`, if schema exposes group ids or group display names | infer from group-message rows; then UIA helper source through current WinHelper if DB lacks group display names | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#msglibdb-candidates` | candidate pending DB schema inspection | third | | `isphere_search_groups` | decrypted XMPP group/message stanzas if groupchat/group JIDs appear | `MsgLib.db` group/conversation tables after DB wrapper is solved; UIA helper source if group display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | candidate pending group stanza evidence | third |
| `isphere_receive_files` | importal hashed document cache mapped through `MsgLib.db` attachment/file-reference rows | `Smark.SendReceive` and `SaveToDB` logs for file-reference traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#receiveddocument-cache-candidates` | candidate pending DB schema and cache-reference inspection | fourth | | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas plus `zyl\importal\<hash>` cache mapping | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | candidate pending cache-reference mapping | fourth |
## Stage C entry rule ## Stage C entry rule
Open implementation only after the selected row links to a concrete file or directory in the evidence index and the schema/log inspection notes state which fields map to the MCP output contract. Open implementation only for the selected row whose evidence maps concrete fields to the MCP output contract. After Stage B2, `isphere_receive_messages` satisfies the entry rule through decrypted `PacketReader.ProcessPacket` XML shape.
## Stage B2 extraction and inspection order
| Order | Target | Why | Expected output |
| --- | --- | --- | --- |
| 1 | `zyl\iSphere\8c5e81a7f6dd1169.10083\27000000\Account\90005981\MsgLib.db` | latest visible iSphere message DB; best candidate for `isphere_receive_messages` | schema summary, table list, sample redacted row shapes |
| 2 | `zyl\Impp\Account\90005981\MsgLib.db` | same account id in older/parallel IMPP tree | schema comparison with iSphere DB |
| 3 | `zyl\Impp\Account\27091282\MsgLib.db` | largest DB, likely richer schema/data variety | schema comparison and attachment/contact table clues |
| 4 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\SaveToDB\2026-07-07_1.log` | may reveal which DB table receives message data | log field summary and DB/table clues |
| 5 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\Smark.SendReceive\2026-07-07_1.log` | send/receive trace near latest DB timestamp | message-id or payload-shape clues |
| 6 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\PacketReader.ProcessPacket\2026-07-07_1.log` | packet processing trace near latest DB timestamp | packet-to-message mapping clues |
| 7 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\LoginInfo\2026-07-07_1.log` | account/session context for source mapping | account id and environment context |
## Tool-by-tool decision notes
### `isphere_receive_messages`
Selected first because the latest `90005981` iSphere `MsgLib.db` and same-day logs align around `2026-07-07 15:30-15:43`. Stage B2 must inspect the DB schema and produce a field mapping for:
- message id
- conversation id or peer id
- sender id/name
- timestamp
- message body or content reference
- attachment/file reference if present
After schema mapping, Stage C can start with a local DB-backed source abstraction before registering the business MCP tool.
### `isphere_search_contacts`
Candidate path is DB-first. If the DB contains contact/person tables, use those. If not, derive a minimal contact search result from message conversation rows and sender/peer identifiers, then use UIA as fallback for display-name enrichment.
### `isphere_search_groups`
Candidate path is DB-first. If the DB contains group/conversation tables, use those. If not, infer group-like conversations from message rows and use UIA as fallback for display-name enrichment.
### `isphere_receive_files`
Candidate path is DB/cache mapping. The hashed `zyl\importal\<hash>` directories appear to hold received document cache files. This tool needs DB/log references to map those hashed directories back to conversations/messages before implementation.
## Stage C first slice recommendation ## Stage C first slice recommendation
Start Stage C with a narrow `isphere_receive_messages` source abstraction: Start Stage C with a narrow log-backed `isphere_receive_messages` source abstraction:
1. Extract the selected DB/log files into `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/`. 1. Implement DES/CBC/PKCS7 line decryption for `IMPP.Util.Log.LogSecurityPolicy` format.
2. Inspect schema and record a schema note under `docs/source-discovery/`. 2. Implement a parser for decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas.
3. Implement a local DB source interface only after the table/field mapping is confirmed. 3. Normalize only redacted test fixtures into message structs first.
4. Keep MCP registration as the next commit after the source abstraction test passes. 4. Add the MCP tool registration after the parser/source abstraction passes tests.
## Stop condition for this matrix ## Deferred source work
If DB schema inspection shows no message table or no usable message fields, do not start Stage C implementation. Return to Stage B and promote the log source or UIA helper source to the first candidate. `MsgLib.db` is deferred because the selected DB files are not direct SQLite and did not decrypt to a SQLite header with the known log policy key. A separate DB-wrapper analysis node is required before using DB tables as a production read source.