admin/isphere-ai-bridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: inspect n12-pre message source formats

Browse Source
This commit is contained in:
zhaoyilun
2026-07-09 22:38:14 +08:00
parent 57b38d8244
commit 1eed0f62f2
2 changed files with 125 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

112
docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md Normal file
View File

@@ -0,0 +1,112 @@
# N12-pre zyl Schema and Log Inspection Notes
Date: 2026-07-09
Evidence id: `zyl-qqfile-20260709`
Stage: B2 selected DB/log inspection
## Files inspected under `runs/`
| Label | Extracted path | Result |
| --- | --- | --- |
| iSphere latest DB | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/msgdb-isphere-90005981/MsgLib.db` | high-entropy binary; not direct SQLite |
| IMPP same-account DB | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/msgdb-impp-90005981/MsgLib.db` | high-entropy binary; not direct SQLite |
| IMPP large DB | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/msgdb-impp-27091282/MsgLib.db` | high-entropy binary; not direct SQLite |
| `Smark.SendReceive` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-smark-90005981-20260707/2026-07-07_1.log` | decryptable line-based log |
| `SaveToDB` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-savetodb-90005981-20260707/2026-07-07_1.log` | decryptable line-based log |
| `PacketReader.ProcessPacket` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-packetreader-90005981-20260707/2026-07-07_1.log` | decryptable line-based XMPP message log |
| `LoginInfo` log | `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/log-logininfo-90005981-20260707/2026-07-07_1.log` | decryptable login metadata XML |
## `MsgLib.db` inspection result
The three selected `MsgLib.db` files are not directly readable with Python `sqlite3` read-only mode.
Observed result:
```text
DatabaseError('file is not a database')
```
Observed binary traits:
| File | Size bytes | SHA256 | Header / format signal |
| --- | ---: | --- | --- |
| `msgdb-isphere-90005981/MsgLib.db` | 2,839,552 | `4f308a26e4d528175853dac8889b69f371b4cad779f75ce2e31d2f83043f86a2` | high entropy, first bytes start `45 1b bd 2b`; not `SQLite format 3` |
| `msgdb-impp-90005981/MsgLib.db` | 2,776,064 | `6707d06126e3cd9a535f6d3879123e31377f0cb7d350b7a6e0cdfe1b34656943` | high entropy, first bytes start `45 1b bd 2b`; not `SQLite format 3` |
| `msgdb-impp-27091282/MsgLib.db` | 12,198,912 | `fe60ddcfbaa70523b21ec080f6d7e633298c9a9d710b7e9c36d91996bf4dc0f0` | high entropy, first bytes start `45 1b bd 2b`; not `SQLite format 3` |
Tried the known `IMPP.Util.Log.LogSecurityPolicy` DES/CBC key against the first DB blocks. It did not reveal a SQLite header. Current conclusion: `MsgLib.db` remains an important source candidate, but it is not the first implementable Stage C source until the DB wrapper/encryption/loading path is identified.
## Log decryption result
The selected logs are Base64 line-based ciphertext. The existing local iSphere log policy finding applies to these logs:
- policy owner: `IMPP.Util.Log.LogSecurityPolicy`
- cipher: DES
- mode: CBC
- padding: PKCS7
- key string: `hyhccdtm`
- IV bytes: `12 34 56 78 90 AB CD EF`
- plaintext encoding: UTF-8
Decryption succeeded for all selected log lines:
| Log | Lines | Decrypted | Useful structure |
| --- | ---: | ---: | --- |
| `SaveToDB` | 14 | 14 | NLog-style records from `IMPP.Client.control.MessageCenter.Chat_OnMessageArrived`; message type `com.vision.smack.packet.ReceiptMessage`; includes `from:` and `body:` fields |
| `Smark.SendReceive` | 1,078 | 1,078 | XMPP send/receive stanzas; includes IQ and message traffic |
| `PacketReader.ProcessPacket` | 245 | 245 | XMPP `<message>` stanzas; includes `id`, `from`, `to`, `type`, `<body>`, `<subject>`, `<isphere ... sendtime=...>` |
| `LoginInfo` | 1 | 1 | XML login metadata; contains account/user/dept/contact fields and should be treated as sensitive metadata |
## Field mapping evidence
`PacketReader.ProcessPacket` is now the strongest immediate source for `isphere_receive_messages` because decrypted lines expose message-level XML shape.
Observed redacted shape:
```xml
<message id="..." from="<jid>/imp_pc_4.1.2.6842" to="<jid>" type="chat">
<body>...</body>
<received xmlns="urn:xmpp:receipts" id="..." type="1" stamp="" />
<subject>...</subject>
<isphere xmlns="isphere.im" type="1001" sendtime="..." version="...">
<alert type="0" />
</isphere>
</message>
```
Candidate mapping to `isphere_receive_messages`:
| MCP field | Candidate source |
| --- | --- |
| `message_id` | `<message id="...">` |
| `conversation_id` | normalized `from` / `to` JID pair or group JID |
| `conversation_type` | `<message type="chat|groupchat|...">` and JID shape |
| `sender` | `from` JID, stripped of resource such as `/imp_pc_4.1.2.6842` |
| `timestamp` | log timestamp line plus `<isphere sendtime="...">` when present |
| `content_text` | `<body>...</body>` |
| attachment/file hint | `<subject>` plus body filename when file-transfer subject is present |
| receipt/status | `<received>`, `<consumed>`, `<readed>` elements when present |
`SaveToDB` is useful as a behavioral trace but not the first parser source. It records `Chat_OnMessageArrived` and `ReceiptMessage` entries with `from:` and `body:` fields, and indicates some messages meet a “not stored to small DB” condition. This explains why `MsgLib.db` may not contain every received message even after its wrapper is solved.
`Smark.SendReceive` is useful as a lower-level transport trace. It includes XMPP IQ/message traffic and can become a fallback source or reconciliation source after the `PacketReader` parser is stable.
`LoginInfo` can map the local logged-in account identity, but it contains sensitive user metadata. Use only field names and identifiers needed for normalization; do not include raw personal values in committed docs or test fixtures.
## Updated source decision
The previous matrix selected `MsgLib.db` as the first source subject to schema inspection. Stage B2 changes that decision:
1. `MsgLib.db` remains a high-value candidate but is blocked by a non-SQLite wrapper/encryption layer.
2. `PacketReader.ProcessPacket` decrypted XMPP XML is the first implementable source for `isphere_receive_messages`.
3. `Smark.SendReceive` and `SaveToDB` are fallback/reconciliation sources.
4. File receiving likely needs `PacketReader` file-transfer XML plus a later map to `zyl\importal\<hash>` document cache directories.
## Next node
Open Stage C with a log-backed source abstraction first:
1. Add a small decrypt/parser package for `IMPP.Util.Log.LogSecurityPolicy` style lines.
2. Parse decrypted `PacketReader.ProcessPacket` XML into normalized message structs.
3. Use redacted fixture strings in tests, not raw user message text.
4. Keep `MsgLib.db` work as a separate follow-up node for DB wrapper/encryption analysis.

64
docs/source-discovery/capability-source-matrix.md
View File

@@ -3,6 +3,7 @@
Date: 2026-07-09
Stage: B source discovery
Evidence index: `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md`
Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
## Source priority
@@ -16,63 +17,24 @@ Evidence index: `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md`
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
| --- | --- | --- | --- | --- | --- |
| `isphere_receive_messages` | `zyl\iSphere\8c5e81a7f6dd1169.10083\27000000\Account\90005981\MsgLib.db` | `zyl\Impp\Account\90005981\MsgLib.db` plus `Smark.SendReceive`, `SaveToDB`, and `PacketReader.ProcessPacket` logs | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#first-extraction-target-list` | selected for schema inspection; implementation starts after message table mapping is confirmed | first |
| `isphere_search_contacts` | contact/conversation/person tables inside `MsgLib.db`, if schema exposes display names or account ids | infer from message rows; then UIA helper source through current WinHelper if DB lacks contact display names | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#msglibdb-candidates` | candidate pending DB schema inspection | second |
| `isphere_search_groups` | group/conversation tables inside `MsgLib.db`, if schema exposes group ids or group display names | infer from group-message rows; then UIA helper source through current WinHelper if DB lacks group display names | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#msglibdb-candidates` | candidate pending DB schema inspection | third |
| `isphere_receive_files` | importal hashed document cache mapped through `MsgLib.db` attachment/file-reference rows | `Smark.SendReceive` and `SaveToDB` logs for file-reference traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-index.md#receiveddocument-cache-candidates` | candidate pending DB schema and cache-reference inspection | fourth |
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; wrapped `MsgLib.db` after DB wrapper is solved | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | selected for Stage C log-backed parser | first |
| `isphere_search_contacts` | decrypted packet/send-receive JIDs and later roster stanzas from `Smark.SendReceive` | `MsgLib.db` contact/conversation tables after DB wrapper is solved; UIA helper source if display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#log-decryption-result` | candidate pending parser coverage | second |
| `isphere_search_groups` | decrypted XMPP group/message stanzas if groupchat/group JIDs appear | `MsgLib.db` group/conversation tables after DB wrapper is solved; UIA helper source if group display names are missing | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | candidate pending group stanza evidence | third |
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas plus `zyl\importal\<hash>` cache mapping | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence` | candidate pending cache-reference mapping | fourth |
## Stage C entry rule
Open implementation only after the selected row links to a concrete file or directory in the evidence index and the schema/log inspection notes state which fields map to the MCP output contract.
## Stage B2 extraction and inspection order
| Order | Target | Why | Expected output |
| --- | --- | --- | --- |
| 1 | `zyl\iSphere\8c5e81a7f6dd1169.10083\27000000\Account\90005981\MsgLib.db` | latest visible iSphere message DB; best candidate for `isphere_receive_messages` | schema summary, table list, sample redacted row shapes |
| 2 | `zyl\Impp\Account\90005981\MsgLib.db` | same account id in older/parallel IMPP tree | schema comparison with iSphere DB |
| 3 | `zyl\Impp\Account\27091282\MsgLib.db` | largest DB, likely richer schema/data variety | schema comparison and attachment/contact table clues |
| 4 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\SaveToDB\2026-07-07_1.log` | may reveal which DB table receives message data | log field summary and DB/table clues |
| 5 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\Smark.SendReceive\2026-07-07_1.log` | send/receive trace near latest DB timestamp | message-id or payload-shape clues |
| 6 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\PacketReader.ProcessPacket\2026-07-07_1.log` | packet processing trace near latest DB timestamp | packet-to-message mapping clues |
| 7 | `zyl\importal\localcache\8c5e81a7f6dd1169.10083\27000000\Log\90005981\LoginInfo\2026-07-07_1.log` | account/session context for source mapping | account id and environment context |
## Tool-by-tool decision notes
### `isphere_receive_messages`
Selected first because the latest `90005981` iSphere `MsgLib.db` and same-day logs align around `2026-07-07 15:30-15:43`. Stage B2 must inspect the DB schema and produce a field mapping for:
- message id
- conversation id or peer id
- sender id/name
- timestamp
- message body or content reference
- attachment/file reference if present
After schema mapping, Stage C can start with a local DB-backed source abstraction before registering the business MCP tool.
### `isphere_search_contacts`
Candidate path is DB-first. If the DB contains contact/person tables, use those. If not, derive a minimal contact search result from message conversation rows and sender/peer identifiers, then use UIA as fallback for display-name enrichment.
### `isphere_search_groups`
Candidate path is DB-first. If the DB contains group/conversation tables, use those. If not, infer group-like conversations from message rows and use UIA as fallback for display-name enrichment.
### `isphere_receive_files`
Candidate path is DB/cache mapping. The hashed `zyl\importal\<hash>` directories appear to hold received document cache files. This tool needs DB/log references to map those hashed directories back to conversations/messages before implementation.
Open implementation only for the selected row whose evidence maps concrete fields to the MCP output contract. After Stage B2, `isphere_receive_messages` satisfies the entry rule through decrypted `PacketReader.ProcessPacket` XML shape.
## Stage C first slice recommendation
Start Stage C with a narrow `isphere_receive_messages` source abstraction:
Start Stage C with a narrow log-backed `isphere_receive_messages` source abstraction:
1. Extract the selected DB/log files into `runs/offline-evidence-intake/zyl-qqfile-20260709/extracted/`.
2. Inspect schema and record a schema note under `docs/source-discovery/`.
3. Implement a local DB source interface only after the table/field mapping is confirmed.
4. Keep MCP registration as the next commit after the source abstraction test passes.
1. Implement DES/CBC/PKCS7 line decryption for `IMPP.Util.Log.LogSecurityPolicy` format.
2. Implement a parser for decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas.
3. Normalize only redacted test fixtures into message structs first.
4. Add the MCP tool registration after the parser/source abstraction passes tests.
## Stop condition for this matrix
## Deferred source work
If DB schema inspection shows no message table or no usable message fields, do not start Stage C implementation. Return to Stage B and promote the log source or UIA helper source to the first candidate.
`MsgLib.db` is deferred because the selected DB files are not direct SQLite and did not decrypt to a SQLite header with the known log policy key. A separate DB-wrapper analysis node is required before using DB tables as a production read source.