admin/isphere-ai-bridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add msglib readonly sidecar

Browse Source
This commit is contained in:
zhaoyilun
2026-07-10 02:45:55 +08:00
parent b442138e5c
commit 5b05228ba8
8 changed files with 849 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/current-status-card.md
View File

@@ -27,8 +27,9 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor
- `scripts/verify-win-helper.ps1` and `scripts/verify-go-mcp.ps1` provide repeatable local checks for the helper, eight-tool MCP surface, default empty-source receive/contact/group/file calls, synthetic configured-file receive/contact/group/file smoke, and synthetic configured-directory receive/contact/group/file smoke.
- `isphere_receive_messages` now returns the contract-facing `ok`, `conversation`, `messages`, `next_cursor`, and `audit` envelope while retaining legacy parser-native message fields for compatibility.
- `isphere_receive_messages` accepts the read-contract argument set for the current local-readonly path: `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, `source_preference`, `preview`, and empty `cursor`.
- `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path, but display names/member data remain JID-derived/unknown until a bounded x86 `MsgLib.db` sidecar/helper is implemented.
- `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path, but display names/member data remain JID-derived/unknown until the new x86 `MsgLibReadSidecar` is wrapped from Go and integrated.
- `isphere_receive_files` list mode accepts and validates the safe file-list contract args for the current local-readonly path while keeping download blocked.
- `native/MsgLibReadSidecar` now provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks; it is not yet wired into Go MCP tools.
- N12-pre and N12R documents define safe offline evidence intake and internal-sandbox live UIA capture procedures.
- N13/N14/N15 produced selector catalog/report validation infrastructure, but those are supporting validation assets only.
@@ -55,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme
## Current loop
Current loop: `Stage C / Loop C27 - bounded MsgLib sidecar/helper contract`.
Current loop: `Stage C / Loop C28 - Go MsgLib sidecar client wrapper`.
Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.
@@ -94,10 +95,11 @@ Current loop output so far:
24. C24: `MsgLib.db` readability precheck found high-entropy non-SQLite files with a consistent wrapper/encryption signature; wrapper candidates include `HYHC.IMPP.DAL.dll`, `Utilities.Lib.SQLiteInteraction*.dll`, `Utilities.Lib.DBModel.dll`, and `RepairDatabase.exe`.
25. C25: static .NET wrapper analysis recovered the DAL open path: `HYHC.IMPP.DAL.IMPPDAL` configures `System.Data.SQLite` through `Utilities.Lib.DBSQLite`, sets `DatabaseName` to the DB path and `Password` to `123`, and exposes candidate display/message tables such as `TD_Roster`, `tblRecent`, `tblPersonMsg`, `tblMsgGroupPersonMsg`, and `TD_WorkGroupAuth`.
26. C26: copied `MsgLib.db` schema extraction succeeded through a 32-bit .NET `System.Data.SQLite` read-only probe with password `123`; all three copied DB candidates opened, with confirmed contact/group/message/file schema tables.
27. C27: added `native/MsgLibReadSidecar` with protocol `isphere.msglib.v1`, x86 build/verify scripts, and safe `self_check`, `schema_summary`, and `display_sources` operations; evidence-backed smoke returned 8 display-source candidates without row/message values.
## Next business mainline
1. Run Stage C Loop C27: define and implement the first bounded x86 `MsgLib.db` sidecar/helper contract for schema/display-name reads; keep message bodies, file downloads, sends, writes, and DB mutation out of scope.
1. Run Stage C Loop C28: add a Go client wrapper for `MsgLibReadSidecar` using env-configured sidecar/SQLite/DB paths, with fake-sidecar tests first; do not expose a new MCP tool or read message bodies yet.
2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated.
3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated.
4. Keep UIA selector/report work as fallback support, not as the primary roadmap.

146
docs/msglib-read-sidecar-contract.md Normal file
View File

@@ -0,0 +1,146 @@
# MsgLib read-only sidecar contract
Date: 2026-07-10
## Purpose
`MsgLibReadSidecar` is the bounded 32-bit .NET process used by the Go MCP layer to access copied or operator-local `MsgLib.db` files through the client-compatible `System.Data.SQLite` provider.
It exists because C26 proved the database opens with the bundled 32-bit SQLite provider and password `123`, while a 64-bit probe fails at provider loading. Go should coordinate the process; the sidecar owns only the narrow Windows/.NET DB read boundary.
## Protocol
- Protocol id: `isphere.msglib.v1`
- Transport: one JSON request on stdin, one JSON response on stdout.
- Process architecture: x86.
- Database mode: read-only only.
- Raw rows: not returned by the first contract.
- Message bodies: not returned by the first contract.
- DB mutation: never allowed.
Request envelope:
```json
{
"protocol": "isphere.msglib.v1",
"request_id": "example-1",
"op": "self_check",
"args": {}
}
```
Response envelope:
```json
{
"protocol": "isphere.msglib.v1",
"request_id": "example-1",
"op": "self_check",
"ok": true,
"data": {},
"error": null,
"warnings": []
}
```
## Operations
### `self_check`
Returns sidecar identity, protocol, runtime, process bitness, and capability flags.
Key response fields:
- `sidecar_name`
- `sidecar_version`
- `protocol`
- `runtime`
- `process_bits`
- `requires_process_bits`
- `db_mutation_allowed=false`
- `message_body_reads_allowed=false`
### `schema_summary`
Validates provider load and read-only DB open, then returns schema metadata only.
Required args:
- `sqlite_dll_path`: path to bundled `System.Data.SQLite.dll`.
- `db_path`: path to a copied/operator-local `MsgLib.db`.
Optional args:
- `password`: default `123`.
- `include_counts`: default `true`; returns row counts only for target tables.
- `max_tables`: default `128`, clamped to `1..256`.
- `target_tables`: optional list; defaults to known display/message/file metadata tables.
Returned data:
- `metadata_only=true`
- `read_only=true`
- `message_body_values_returned=false`
- `sqlite_version`
- `tables`: table/view names only.
- `columns`: table column names/types only.
- `target_row_counts`: safe row counts for target tables only.
### `display_sources`
Same safe provider/DB validation as `schema_summary`, but additionally maps schema availability to product-facing display source candidates.
Returned data adds:
- `display_sources[]` entries with:
- `source`
- `table`
- `available`
- `required_columns`
- `present_columns`
Initial display sources:
- `contacts_roster` from `TD_Roster`.
- `contacts_effigy` from `TD_CustomEffigy`.
- `recent_display` from `tblRecent`.
- `person_message_names` from `tblPersonMsg`.
- `group_message_names` from `tblMsgGroupPersonMsg`.
- `work_group_auth` from `TD_WorkGroupAuth`.
- `received_file_metadata` from `TD_ReceiveFileRecord`.
- `sent_file_metadata` from `TD_SendFileRecord`.
## Security and scope rules
- The sidecar must open DB files with read-only connection settings.
- `db_path` must point to a file named `MsgLib.db`.
- The first contract returns schema, columns, counts, and display-source availability only.
- It does not return message bodies, contact row values, file paths, or raw row sets.
- It does not implement send, file download, mark-read, login, hooks, injection, or DB writes.
- The Go MCP layer remains the coordinator; this process is a bounded local reader.
## Current implementation
Committed C27 implementation:
- Source: `native/MsgLibReadSidecar/`.
- Build script: `scripts/build-msglib-sidecar.ps1`.
- Verification script: `scripts/verify-msglib-sidecar.ps1`.
- Default verification builds an x86 executable and checks `self_check`.
- Evidence-backed verification with C26 paths checked provider load/read-only DB open and returned 8 display-source candidates.
## Verification
Default verification builds and checks `self_check` only:
```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1
```
Evidence-backed provider verification is opt-in through environment variables and should use copied DB files only:
```powershell
$env:ISPHERE_MSGLIB_SQLITE_DLL = "<path-to-System.Data.SQLite.dll>"
$env:ISPHERE_MSGLIB_DB = "<path-to-copied-MsgLib.db>"
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1
```

8
docs/source-discovery/capability-source-matrix.md
View File

@@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
| --- | --- | --- | --- | --- | --- |
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C26 validates copied `MsgLib.db` read-only open/schema, but DB-backed message listing is not implemented yet | C27 bounded x86 MsgLib sidecar/helper contract for safe display/schema reads |
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C26 validates read-only DB open/schema with the recovered wrapper password; product code still needs a bounded x86 sidecar/helper before DB-backed display names can be exposed | C27 bounded x86 MsgLib sidecar/helper contract |
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C26 validates read-only DB open/schema with the recovered wrapper password; product code still needs a bounded x86 sidecar/helper before DB-backed group/member names can be exposed | C27 bounded x86 MsgLib sidecar/helper contract |
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C27 adds a bounded x86 `MsgLibReadSidecar` with schema/display-source operations; DB-backed message listing is still not implemented | C28 Go sidecar client wrapper before MCP/tool integration |
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C27 adds and verifies the bounded x86 sidecar/helper contract; Go MCP still needs a sidecar client wrapper before DB-backed display names can be exposed | C28 Go sidecar client wrapper |
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C27 adds and verifies the bounded x86 sidecar/helper contract; Go MCP still needs a sidecar client wrapper before DB-backed group/member names can be exposed | C28 Go sidecar client wrapper |
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
| `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration |
| `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated |
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
## Deferred source work
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. Because the working provider path is x86 .NET, DB-backed production reads should enter through a bounded sidecar/helper contract rather than direct Go SQLite access.
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary; the next step is a Go sidecar client wrapper and then contact/group display-name integration.

105
docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md
View File

@@ -2106,40 +2106,113 @@ Copied `MsgLib.db` read-only schema extraction result:
**Goal:** Turn the validated C26 read-only DB path into the first production-shaped helper boundary for DB-backed display-name/source metadata, without exposing message bodies or adding write behavior.
**Planned files:**
- Modify or create docs that define the sidecar/helper JSON contract.
- Optionally create a minimal committed C# sidecar/helper skeleton if the contract can stay narrow and testable.
- Modify verification docs/scripts only if an executable smoke is added.
**Files:**
- Create: `native/MsgLibReadSidecar/Program.cs`
- Create: `native/MsgLibReadSidecar/SidecarProtocol.cs`
- Create: `scripts/build-msglib-sidecar.ps1`
- Create: `scripts/verify-msglib-sidecar.ps1`
- Create: `docs/msglib-read-sidecar-contract.md`
- Modify: `docs/source-discovery/capability-source-matrix.md`
- Modify: `docs/current-status-card.md`
- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`
- Do not commit `System.Data.SQLite.dll`, copied DBs, raw rows, message bodies, personal values, file paths, or generated probe binaries.
**Planned behavior:**
- Treat the DB reader as a dedicated x86 .NET boundary because C26 proved 32-bit SQLite loading is required.
- Start with safe operations such as:
- `msglib_self_check`: validates provider load and read-only open against a copied DB path.
- `msglib_list_display_sources`: returns table/column availability and limited display-name source counts, not raw message content.
- Implement safe operations:
- `self_check`: sidecar identity, runtime, x86 bitness, and no-mutation flags.
- `schema_summary`: provider load/read-only DB open plus schema/columns/counts only.
- `display_sources`: schema availability mapped to display-source candidates only.
- Keep Go MCP as the coordinator and use a bounded process/helper contract for Windows/.NET DB access.
- Keep message-body reads, file downloads, sends, writes, hooks, injection, and DB mutation out of scope.
- [ ] **Step 1: Define the sidecar/helper contract**
- [x] **Step 1: Define the sidecar/helper contract**
Write the request/response envelope, allowed operations, path rules, limit rules, and redaction rules.
Created `docs/msglib-read-sidecar-contract.md` with protocol `isphere.msglib.v1`, request/response envelopes, safe operation contracts, and scope rules.
- [ ] **Step 2: Decide skeleton vs docs-only for this loop**
- [x] **Step 2: Implement minimal C# x86 sidecar skeleton**
If the contract is stable enough, create a minimal C# x86 sidecar skeleton with self-check only. Otherwise keep C27 as a contract/design gate and make C28 the skeleton implementation.
Created `native/MsgLibReadSidecar` and build/verify scripts. The sidecar compiles x86 and uses reflection to load a caller-supplied `System.Data.SQLite.dll`, so the SQLite binary is not committed.
- [ ] **Step 3: Update status/matrix/plan**
- [x] **Step 3: Verify default and evidence-backed smokes**
Record the implementation boundary and next code slice based on actual C27 output.
Verified default and evidence-backed sidecar smokes with `scripts\verify-msglib-sidecar.ps1`; evidence-backed smoke used copied DB paths through environment variables and returned `display_source_count=8`.
- [ ] **Step 4: Verify and commit**
- [x] **Step 4: Verify and commit**
Run the appropriate docs/code verification before commit:
Run final verification before commit:
```powershell
git diff --check
go test ./...
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/verify-go-mcp.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build-msglib-sidecar.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1 # with copied DB env paths
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1
```
Final verification passed: `git diff --check`, `go test ./...`, explicit `go build ./cmd/isphere-mcp`, default sidecar smoke, evidence-backed sidecar smoke, and `scripts\verify-go-mcp.ps1`.
---
## Loop C27 Result
Bounded MsgLib sidecar result:
- Added `native/MsgLibReadSidecar` as a dedicated x86 .NET process with protocol `isphere.msglib.v1`.
- Added `self_check`, `schema_summary`, and `display_sources` operations.
- `schema_summary` and `display_sources` require caller-supplied `sqlite_dll_path` and `db_path`, open only files named `MsgLib.db`, set read-only SQLite connection options, and return schema/column/count/display-source metadata only.
- Added `scripts/build-msglib-sidecar.ps1`, compiling with 32-bit .NET Framework `csc.exe` and `/platform:x86`.
- Added `scripts/verify-msglib-sidecar.ps1`, with default self-check verification plus opt-in evidence-backed provider verification through `ISPHERE_MSGLIB_SQLITE_DLL` and `ISPHERE_MSGLIB_DB`.
- Default sidecar verification passed with `process_bits=32`.
- Evidence-backed sidecar verification passed against the copied DB and returned `display_source_count=8`.
- The sidecar is not yet wrapped by Go and is not yet used by MCP tools.
---
## Loop C28: Go `MsgLibReadSidecar` client wrapper
**Goal:** Add a Go-side client wrapper that can call the C27 x86 sidecar through a bounded JSON process contract, without exposing a new MCP tool or reading message bodies yet.
**Planned files:**
- Create: `internal/msglib/sidecar_client.go`
- Create: `internal/msglib/sidecar_client_test.go`
- Optionally update `docs/msglib-read-sidecar-contract.md`
- Modify: `docs/current-status-card.md`
- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`
**Planned behavior:**
- Implement a Go client for `isphere.msglib.v1` with request ids, stdin/stdout JSON, timeout, and structured errors.
- Test with a fake sidecar executable/script first; do not require real `System.Data.SQLite.dll` or real DB in normal unit tests.
- Add env config shape but do not wire MCP tools yet:
- `ISPHERE_MSGLIB_SIDECAR_EXE`
- `ISPHERE_MSGLIB_SQLITE_DLL`
- `ISPHERE_MSGLIB_DB`
- Expose only Go functions for `SelfCheck` and `DisplaySources` metadata.
- Do not add message-body reads, file downloads, sends, writes, hooks, injection, DB mutation, or a public MCP tool in this loop.
- [ ] **Step 1: Write failing fake-sidecar client tests**
Test request envelope, timeout/error handling, and display-source metadata decoding against a fake process.
- [ ] **Step 2: Implement sidecar client**
Use `context.Context`, `exec.CommandContext`, stdin JSON, stdout JSON, stderr capture, and small typed structs.
- [ ] **Step 3: Verify and update docs**
Run Go tests and update status/plan. Keep real evidence-backed sidecar smoke optional through `scripts/verify-msglib-sidecar.ps1`.
- [ ] **Step 4: Full verification and commit**
Run:
```powershell
git diff --check
go test ./...
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build-msglib-sidecar.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1
```
---