feat: add msglib sidecar go client
This commit is contained in:
zhaoyilun
@@ -27,9 +27,9 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor
|
||||
- `scripts/verify-win-helper.ps1` and `scripts/verify-go-mcp.ps1` provide repeatable local checks for the helper, eight-tool MCP surface, default empty-source receive/contact/group/file calls, synthetic configured-file receive/contact/group/file smoke, and synthetic configured-directory receive/contact/group/file smoke.
|
||||
- `isphere_receive_messages` now returns the contract-facing `ok`, `conversation`, `messages`, `next_cursor`, and `audit` envelope while retaining legacy parser-native message fields for compatibility.
|
||||
- `isphere_receive_messages` accepts the read-contract argument set for the current local-readonly path: `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, `source_preference`, `preview`, and empty `cursor`.
|
||||
- `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path, but display names/member data remain JID-derived/unknown until the new x86 `MsgLibReadSidecar` is wrapped from Go and integrated.
|
||||
- `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path, but display names/member data remain JID-derived/unknown until the x86 `MsgLibReadSidecar` is extended for bounded display-entity reads and integrated.
|
||||
- `isphere_receive_files` list mode accepts and validates the safe file-list contract args for the current local-readonly path while keeping download blocked.
|
||||
- `native/MsgLibReadSidecar` now provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks; it is not yet wired into Go MCP tools.
|
||||
- `native/MsgLibReadSidecar` provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks, and `internal/msglib` now wraps it from Go; it is not yet wired into MCP tools and does not return display row values yet.
|
||||
- N12-pre and N12R documents define safe offline evidence intake and internal-sandbox live UIA capture procedures.
|
||||
- N13/N14/N15 produced selector catalog/report validation infrastructure, but those are supporting validation assets only.
|
||||
|
||||
@@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme
|
||||
|
||||
## Current loop
|
||||
|
||||
Current loop: `Stage C / Loop C28 - Go MsgLib sidecar client wrapper`.
|
||||
Current loop: `Stage C / Loop C29 - bounded MsgLib display entity extraction`.
|
||||
|
||||
Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.
|
||||
|
||||
@@ -96,10 +96,11 @@ Current loop output so far:
|
||||
25. C25: static .NET wrapper analysis recovered the DAL open path: `HYHC.IMPP.DAL.IMPPDAL` configures `System.Data.SQLite` through `Utilities.Lib.DBSQLite`, sets `DatabaseName` to the DB path and `Password` to `123`, and exposes candidate display/message tables such as `TD_Roster`, `tblRecent`, `tblPersonMsg`, `tblMsgGroupPersonMsg`, and `TD_WorkGroupAuth`.
|
||||
26. C26: copied `MsgLib.db` schema extraction succeeded through a 32-bit .NET `System.Data.SQLite` read-only probe with password `123`; all three copied DB candidates opened, with confirmed contact/group/message/file schema tables.
|
||||
27. C27: added `native/MsgLibReadSidecar` with protocol `isphere.msglib.v1`, x86 build/verify scripts, and safe `self_check`, `schema_summary`, and `display_sources` operations; evidence-backed smoke returned 8 display-source candidates without row/message values.
|
||||
28. C28: added `internal/msglib` Go client wrapper for the sidecar protocol with env config, fake-sidecar unit tests, timeout handling, structured sidecar errors, `SelfCheck`, and `DisplaySources`; no MCP tool wiring or message/display row reads yet.
|
||||
|
||||
## Next business mainline
|
||||
|
||||
1. Run Stage C Loop C28: add a Go client wrapper for `MsgLibReadSidecar` using env-configured sidecar/SQLite/DB paths, with fake-sidecar tests first; do not expose a new MCP tool or read message bodies yet.
|
||||
1. Run Stage C Loop C29: extend the sidecar/client contract for bounded contact/group display-entity extraction only, with tests first; do not expose message bodies, file paths, downloads, sends, writes, hooks, injection, or raw row dumps.
|
||||
2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated.
|
||||
3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated.
|
||||
4. Keep UIA selector/report work as fallback support, not as the primary roadmap.
|
||||
|
||||
@@ -129,6 +129,19 @@ Committed C27 implementation:
|
||||
- Default verification builds an x86 executable and checks `self_check`.
|
||||
- Evidence-backed verification with C26 paths checked provider load/read-only DB open and returned 8 display-source candidates.
|
||||
|
||||
## Go client wrapper
|
||||
|
||||
Committed C28 implementation:
|
||||
|
||||
- Source: `internal/msglib/`.
|
||||
- `ConfigFromEnv` reads `ISPHERE_MSGLIB_SIDECAR_EXE`, `ISPHERE_MSGLIB_SQLITE_DLL`, `ISPHERE_MSGLIB_DB`, and optional `ISPHERE_MSGLIB_PASSWORD`.
|
||||
- `NewClient` creates a process-backed client for protocol `isphere.msglib.v1`.
|
||||
- `SelfCheck` calls `self_check` and decodes sidecar identity, protocol, bitness, and read-only/no-mutation flags.
|
||||
- `DisplaySources` calls `display_sources` and decodes display-source availability metadata.
|
||||
- The client uses stdin/stdout JSON, request ids, `context.Context`, per-call timeout, stderr capture, and typed `SidecarError` values.
|
||||
- Unit tests use a fake sidecar process; normal tests do not require real `System.Data.SQLite.dll` or copied `MsgLib.db`.
|
||||
- C28 does not register a new MCP tool and does not return message bodies, file paths, raw rows, contact row values, or group row values.
|
||||
|
||||
## Verification
|
||||
|
||||
Default verification builds and checks `self_check` only:
|
||||
|
||||
@@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
|
||||
|
||||
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C27 adds a bounded x86 `MsgLibReadSidecar` with schema/display-source operations; DB-backed message listing is still not implemented | C28 Go sidecar client wrapper before MCP/tool integration |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C27 adds and verifies the bounded x86 sidecar/helper contract; Go MCP still needs a sidecar client wrapper before DB-backed display names can be exposed | C28 Go sidecar client wrapper |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C27 adds and verifies the bounded x86 sidecar/helper contract; Go MCP still needs a sidecar client wrapper before DB-backed group/member names can be exposed | C28 Go sidecar client wrapper |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C28 adds the Go `internal/msglib` sidecar wrapper for schema/display-source operations; DB-backed message listing is still not implemented | Defer DB-backed message listing; first run C29 bounded contact/group display-entity extraction |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C28 adds the Go sidecar client wrapper; the sidecar still needs a bounded display-entity operation before DB-backed display names can be exposed | C29 bounded contact display-entity extraction |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C28 adds the Go sidecar client wrapper; the sidecar still needs a bounded display-entity operation before DB-backed group/member names can be exposed | C29 bounded group display-entity extraction |
|
||||
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
|
||||
| `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration |
|
||||
| `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated |
|
||||
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
|
||||
|
||||
## Deferred source work
|
||||
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary; the next step is a Go sidecar client wrapper and then contact/group display-name integration.
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, and C28 adds the Go `internal/msglib` process client. The next step is a bounded display-entity operation before contact/group display-name integration.
|
||||
|
||||
@@ -2174,10 +2174,11 @@ Bounded MsgLib sidecar result:
|
||||
|
||||
**Goal:** Add a Go-side client wrapper that can call the C27 x86 sidecar through a bounded JSON process contract, without exposing a new MCP tool or reading message bodies yet.
|
||||
|
||||
**Planned files:**
|
||||
**Files:**
|
||||
- Create: `internal/msglib/sidecar_client.go`
|
||||
- Create: `internal/msglib/sidecar_client_test.go`
|
||||
- Optionally update `docs/msglib-read-sidecar-contract.md`
|
||||
- Modify: `docs/msglib-read-sidecar-contract.md`
|
||||
- Modify: `docs/source-discovery/capability-source-matrix.md`
|
||||
- Modify: `docs/current-status-card.md`
|
||||
- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`
|
||||
|
||||
@@ -2191,25 +2192,89 @@ Bounded MsgLib sidecar result:
|
||||
- Expose only Go functions for `SelfCheck` and `DisplaySources` metadata.
|
||||
- Do not add message-body reads, file downloads, sends, writes, hooks, injection, DB mutation, or a public MCP tool in this loop.
|
||||
|
||||
- [ ] **Step 1: Write failing fake-sidecar client tests**
|
||||
- [x] **Step 1: Write failing fake-sidecar client tests**
|
||||
|
||||
Test request envelope, timeout/error handling, and display-source metadata decoding against a fake process.
|
||||
Created `internal/msglib/sidecar_client_test.go` with fake process tests for request envelope, `self_check`, `display_sources`, structured sidecar errors, timeout handling, and env config. Verified RED with undefined `Config`, `Client`, `Protocol`, and `SidecarError` symbols.
|
||||
|
||||
- [ ] **Step 2: Implement sidecar client**
|
||||
- [x] **Step 2: Implement sidecar client**
|
||||
|
||||
Use `context.Context`, `exec.CommandContext`, stdin JSON, stdout JSON, stderr capture, and small typed structs.
|
||||
Created `internal/msglib/sidecar_client.go` with protocol constants, `ConfigFromEnv`, `NewClient`, `SelfCheck`, `DisplaySources`, request ids, `exec.CommandContext`, stdin/stdout JSON, timeout, stderr capture, response validation, and typed `SidecarError` handling.
|
||||
|
||||
- [ ] **Step 3: Verify and update docs**
|
||||
- [x] **Step 3: Verify and update docs**
|
||||
|
||||
Run Go tests and update status/plan. Keep real evidence-backed sidecar smoke optional through `scripts/verify-msglib-sidecar.ps1`.
|
||||
Ran package tests for `internal/msglib` and updated the sidecar contract, capability matrix, current status card, and this plan. Real evidence-backed sidecar smoke remains covered by `scripts\verify-msglib-sidecar.ps1` rather than normal Go unit tests.
|
||||
|
||||
- [ ] **Step 4: Full verification and commit**
|
||||
- [x] **Step 4: Full verification and commit**
|
||||
|
||||
Run before commit:
|
||||
|
||||
```powershell
|
||||
git diff --check
|
||||
go test ./...
|
||||
go build ./cmd/isphere-mcp
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build-msglib-sidecar.ps1
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1 # with copied DB env paths
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1
|
||||
```
|
||||
|
||||
Final verification passed: `git diff --check`, `go test ./...`, explicit `go build ./cmd/isphere-mcp`, default sidecar smoke, evidence-backed sidecar smoke, and `scripts\verify-go-mcp.ps1`.
|
||||
|
||||
---
|
||||
|
||||
## Loop C28 Result
|
||||
|
||||
Go sidecar client result:
|
||||
|
||||
- Added `internal/msglib` as the internal Go process client for `isphere.msglib.v1`.
|
||||
- Added fake-sidecar unit coverage for envelope validation, `SelfCheck`, `DisplaySources`, structured `SidecarError`, timeout behavior, and env config.
|
||||
- `ConfigFromEnv` reads `ISPHERE_MSGLIB_SIDECAR_EXE`, `ISPHERE_MSGLIB_SQLITE_DLL`, `ISPHERE_MSGLIB_DB`, and optional `ISPHERE_MSGLIB_PASSWORD`.
|
||||
- C28 still does not register a new MCP tool, does not read message bodies, does not return contact/group row values, and does not implement file download/send/write behavior.
|
||||
- Next gap: C29 must add a bounded display-entity operation before DB-backed contact/group display names can be exposed.
|
||||
|
||||
---
|
||||
|
||||
## Loop C29: Bounded `MsgLib.db` display entity extraction
|
||||
|
||||
**Goal:** Extend the sidecar/client boundary from schema/display-source availability to bounded contact/group display-entity metadata, so a later loop can enrich `isphere_search_contacts` and `isphere_search_groups` without returning raw rows, message bodies, file paths, downloads, sends, writes, hooks, injection, or DB mutations.
|
||||
|
||||
**Planned files:**
|
||||
- Modify: `native/MsgLibReadSidecar/Program.cs`
|
||||
- Modify: `docs/msglib-read-sidecar-contract.md`
|
||||
- Modify: `internal/msglib/sidecar_client.go`
|
||||
- Modify: `internal/msglib/sidecar_client_test.go`
|
||||
- Modify: `scripts/verify-msglib-sidecar.ps1`
|
||||
- Modify: `docs/current-status-card.md`
|
||||
- Modify: `docs/source-discovery/capability-source-matrix.md`
|
||||
- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`
|
||||
|
||||
**Planned behavior:**
|
||||
- Add a sidecar operation tentatively named `display_entities`.
|
||||
- Request args: `sqlite_dll_path`, `db_path`, optional `password`, `entity_type` (`contacts` or `groups`), optional `query`, and bounded `limit`.
|
||||
- Response data: normalized metadata only, such as `entity_type`, `source_table`, `jid`, `display_name`, `confidence`, and `matched_columns`.
|
||||
- Never return message bodies, file paths, raw SQL row maps, attachment contents, send/write capabilities, or unbounded result sets.
|
||||
- Use tests first. Normal Go tests should still use fake sidecar data; real copied DB verification stays opt-in through script/env paths.
|
||||
|
||||
- [ ] **Step 1: Write failing display-entity contract tests**
|
||||
|
||||
Add sidecar/client tests for bounded contact/group entity decoding, query/limit validation, and rejection of unsupported entity types.
|
||||
|
||||
- [ ] **Step 2: Implement sidecar `display_entities` operation**
|
||||
|
||||
Implement allowlisted table/column reads only for contact/group identity metadata, with parameterized query/limit and no raw row dumps.
|
||||
|
||||
- [ ] **Step 3: Extend Go client and verification script**
|
||||
|
||||
Add Go wrapper types/methods and optional evidence-backed verification that reports only counts/source-table names, not personal values.
|
||||
|
||||
- [ ] **Step 4: Update docs, run full verification, and commit**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
git diff --check
|
||||
go test ./...
|
||||
go build ./cmd/isphere-mcp
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build-msglib-sidecar.ps1
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1
|
||||
|
||||
Reference in New Issue
Block a user