diff --git a/docs/current-status-card.md b/docs/current-status-card.md index 7adafb8..3bcf671 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -27,8 +27,9 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor - `scripts/verify-win-helper.ps1` and `scripts/verify-go-mcp.ps1` provide repeatable local checks for the helper, eight-tool MCP surface, default empty-source receive/contact/group/file calls, synthetic configured-file receive/contact/group/file smoke, and synthetic configured-directory receive/contact/group/file smoke. - `isphere_receive_messages` now returns the contract-facing `ok`, `conversation`, `messages`, `next_cursor`, and `audit` envelope while retaining legacy parser-native message fields for compatibility. - `isphere_receive_messages` accepts the read-contract argument set for the current local-readonly path: `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, `source_preference`, `preview`, and empty `cursor`. -- `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path, but display names/member data remain JID-derived/unknown until a bounded x86 `MsgLib.db` sidecar/helper is implemented. +- `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path, but display names/member data remain JID-derived/unknown until the new x86 `MsgLibReadSidecar` is wrapped from Go and integrated. - `isphere_receive_files` list mode accepts and validates the safe file-list contract args for the current local-readonly path while keeping download blocked. +- `native/MsgLibReadSidecar` now provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks; it is not yet wired into Go MCP tools. - N12-pre and N12R documents define safe offline evidence intake and internal-sandbox live UIA capture procedures. - N13/N14/N15 produced selector catalog/report validation infrastructure, but those are supporting validation assets only. @@ -55,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Stage C / Loop C27 - bounded MsgLib sidecar/helper contract`. +Current loop: `Stage C / Loop C28 - Go MsgLib sidecar client wrapper`. Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. @@ -94,10 +95,11 @@ Current loop output so far: 24. C24: `MsgLib.db` readability precheck found high-entropy non-SQLite files with a consistent wrapper/encryption signature; wrapper candidates include `HYHC.IMPP.DAL.dll`, `Utilities.Lib.SQLiteInteraction*.dll`, `Utilities.Lib.DBModel.dll`, and `RepairDatabase.exe`. 25. C25: static .NET wrapper analysis recovered the DAL open path: `HYHC.IMPP.DAL.IMPPDAL` configures `System.Data.SQLite` through `Utilities.Lib.DBSQLite`, sets `DatabaseName` to the DB path and `Password` to `123`, and exposes candidate display/message tables such as `TD_Roster`, `tblRecent`, `tblPersonMsg`, `tblMsgGroupPersonMsg`, and `TD_WorkGroupAuth`. 26. C26: copied `MsgLib.db` schema extraction succeeded through a 32-bit .NET `System.Data.SQLite` read-only probe with password `123`; all three copied DB candidates opened, with confirmed contact/group/message/file schema tables. +27. C27: added `native/MsgLibReadSidecar` with protocol `isphere.msglib.v1`, x86 build/verify scripts, and safe `self_check`, `schema_summary`, and `display_sources` operations; evidence-backed smoke returned 8 display-source candidates without row/message values. ## Next business mainline -1. Run Stage C Loop C27: define and implement the first bounded x86 `MsgLib.db` sidecar/helper contract for schema/display-name reads; keep message bodies, file downloads, sends, writes, and DB mutation out of scope. +1. Run Stage C Loop C28: add a Go client wrapper for `MsgLibReadSidecar` using env-configured sidecar/SQLite/DB paths, with fake-sidecar tests first; do not expose a new MCP tool or read message bodies yet. 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated. 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated. 4. Keep UIA selector/report work as fallback support, not as the primary roadmap. diff --git a/docs/msglib-read-sidecar-contract.md b/docs/msglib-read-sidecar-contract.md new file mode 100644 index 0000000..6abe77f --- /dev/null +++ b/docs/msglib-read-sidecar-contract.md @@ -0,0 +1,146 @@ +# MsgLib read-only sidecar contract + +Date: 2026-07-10 + +## Purpose + +`MsgLibReadSidecar` is the bounded 32-bit .NET process used by the Go MCP layer to access copied or operator-local `MsgLib.db` files through the client-compatible `System.Data.SQLite` provider. + +It exists because C26 proved the database opens with the bundled 32-bit SQLite provider and password `123`, while a 64-bit probe fails at provider loading. Go should coordinate the process; the sidecar owns only the narrow Windows/.NET DB read boundary. + +## Protocol + +- Protocol id: `isphere.msglib.v1` +- Transport: one JSON request on stdin, one JSON response on stdout. +- Process architecture: x86. +- Database mode: read-only only. +- Raw rows: not returned by the first contract. +- Message bodies: not returned by the first contract. +- DB mutation: never allowed. + +Request envelope: + +```json +{ + "protocol": "isphere.msglib.v1", + "request_id": "example-1", + "op": "self_check", + "args": {} +} +``` + +Response envelope: + +```json +{ + "protocol": "isphere.msglib.v1", + "request_id": "example-1", + "op": "self_check", + "ok": true, + "data": {}, + "error": null, + "warnings": [] +} +``` + +## Operations + +### `self_check` + +Returns sidecar identity, protocol, runtime, process bitness, and capability flags. + +Key response fields: + +- `sidecar_name` +- `sidecar_version` +- `protocol` +- `runtime` +- `process_bits` +- `requires_process_bits` +- `db_mutation_allowed=false` +- `message_body_reads_allowed=false` + +### `schema_summary` + +Validates provider load and read-only DB open, then returns schema metadata only. + +Required args: + +- `sqlite_dll_path`: path to bundled `System.Data.SQLite.dll`. +- `db_path`: path to a copied/operator-local `MsgLib.db`. + +Optional args: + +- `password`: default `123`. +- `include_counts`: default `true`; returns row counts only for target tables. +- `max_tables`: default `128`, clamped to `1..256`. +- `target_tables`: optional list; defaults to known display/message/file metadata tables. + +Returned data: + +- `metadata_only=true` +- `read_only=true` +- `message_body_values_returned=false` +- `sqlite_version` +- `tables`: table/view names only. +- `columns`: table column names/types only. +- `target_row_counts`: safe row counts for target tables only. + +### `display_sources` + +Same safe provider/DB validation as `schema_summary`, but additionally maps schema availability to product-facing display source candidates. + +Returned data adds: + +- `display_sources[]` entries with: + - `source` + - `table` + - `available` + - `required_columns` + - `present_columns` + +Initial display sources: + +- `contacts_roster` from `TD_Roster`. +- `contacts_effigy` from `TD_CustomEffigy`. +- `recent_display` from `tblRecent`. +- `person_message_names` from `tblPersonMsg`. +- `group_message_names` from `tblMsgGroupPersonMsg`. +- `work_group_auth` from `TD_WorkGroupAuth`. +- `received_file_metadata` from `TD_ReceiveFileRecord`. +- `sent_file_metadata` from `TD_SendFileRecord`. + +## Security and scope rules + +- The sidecar must open DB files with read-only connection settings. +- `db_path` must point to a file named `MsgLib.db`. +- The first contract returns schema, columns, counts, and display-source availability only. +- It does not return message bodies, contact row values, file paths, or raw row sets. +- It does not implement send, file download, mark-read, login, hooks, injection, or DB writes. +- The Go MCP layer remains the coordinator; this process is a bounded local reader. + +## Current implementation + +Committed C27 implementation: + +- Source: `native/MsgLibReadSidecar/`. +- Build script: `scripts/build-msglib-sidecar.ps1`. +- Verification script: `scripts/verify-msglib-sidecar.ps1`. +- Default verification builds an x86 executable and checks `self_check`. +- Evidence-backed verification with C26 paths checked provider load/read-only DB open and returned 8 display-source candidates. + +## Verification + +Default verification builds and checks `self_check` only: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1 +``` + +Evidence-backed provider verification is opt-in through environment variables and should use copied DB files only: + +```powershell +$env:ISPHERE_MSGLIB_SQLITE_DLL = "" +$env:ISPHERE_MSGLIB_DB = "" +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1 +``` diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index 6394821..68f545f 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | --- | --- | --- | --- | --- | --- | -| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C26 validates copied `MsgLib.db` read-only open/schema, but DB-backed message listing is not implemented yet | C27 bounded x86 MsgLib sidecar/helper contract for safe display/schema reads | -| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C26 validates read-only DB open/schema with the recovered wrapper password; product code still needs a bounded x86 sidecar/helper before DB-backed display names can be exposed | C27 bounded x86 MsgLib sidecar/helper contract | -| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C26 validates read-only DB open/schema with the recovered wrapper password; product code still needs a bounded x86 sidecar/helper before DB-backed group/member names can be exposed | C27 bounded x86 MsgLib sidecar/helper contract | +| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C27 adds a bounded x86 `MsgLibReadSidecar` with schema/display-source operations; DB-backed message listing is still not implemented | C28 Go sidecar client wrapper before MCP/tool integration | +| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C27 adds and verifies the bounded x86 sidecar/helper contract; Go MCP still needs a sidecar client wrapper before DB-backed display names can be exposed | C28 Go sidecar client wrapper | +| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C27 adds and verifies the bounded x86 sidecar/helper contract; Go MCP still needs a sidecar client wrapper before DB-backed group/member names can be exposed | C28 Go sidecar client wrapper | | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence | | `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration | | `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated | @@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac ## Deferred source work -`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. Because the working provider path is x86 .NET, DB-backed production reads should enter through a bounded sidecar/helper contract rather than direct Go SQLite access. +`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary; the next step is a Go sidecar client wrapper and then contact/group display-name integration. diff --git a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md index 52a3c18..f7b99b4 100644 --- a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md +++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md @@ -2106,40 +2106,113 @@ Copied `MsgLib.db` read-only schema extraction result: **Goal:** Turn the validated C26 read-only DB path into the first production-shaped helper boundary for DB-backed display-name/source metadata, without exposing message bodies or adding write behavior. -**Planned files:** -- Modify or create docs that define the sidecar/helper JSON contract. -- Optionally create a minimal committed C# sidecar/helper skeleton if the contract can stay narrow and testable. -- Modify verification docs/scripts only if an executable smoke is added. +**Files:** +- Create: `native/MsgLibReadSidecar/Program.cs` +- Create: `native/MsgLibReadSidecar/SidecarProtocol.cs` +- Create: `scripts/build-msglib-sidecar.ps1` +- Create: `scripts/verify-msglib-sidecar.ps1` +- Create: `docs/msglib-read-sidecar-contract.md` +- Modify: `docs/source-discovery/capability-source-matrix.md` +- Modify: `docs/current-status-card.md` +- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md` - Do not commit `System.Data.SQLite.dll`, copied DBs, raw rows, message bodies, personal values, file paths, or generated probe binaries. **Planned behavior:** - Treat the DB reader as a dedicated x86 .NET boundary because C26 proved 32-bit SQLite loading is required. -- Start with safe operations such as: - - `msglib_self_check`: validates provider load and read-only open against a copied DB path. - - `msglib_list_display_sources`: returns table/column availability and limited display-name source counts, not raw message content. +- Implement safe operations: + - `self_check`: sidecar identity, runtime, x86 bitness, and no-mutation flags. + - `schema_summary`: provider load/read-only DB open plus schema/columns/counts only. + - `display_sources`: schema availability mapped to display-source candidates only. - Keep Go MCP as the coordinator and use a bounded process/helper contract for Windows/.NET DB access. - Keep message-body reads, file downloads, sends, writes, hooks, injection, and DB mutation out of scope. -- [ ] **Step 1: Define the sidecar/helper contract** +- [x] **Step 1: Define the sidecar/helper contract** -Write the request/response envelope, allowed operations, path rules, limit rules, and redaction rules. +Created `docs/msglib-read-sidecar-contract.md` with protocol `isphere.msglib.v1`, request/response envelopes, safe operation contracts, and scope rules. -- [ ] **Step 2: Decide skeleton vs docs-only for this loop** +- [x] **Step 2: Implement minimal C# x86 sidecar skeleton** -If the contract is stable enough, create a minimal C# x86 sidecar skeleton with self-check only. Otherwise keep C27 as a contract/design gate and make C28 the skeleton implementation. +Created `native/MsgLibReadSidecar` and build/verify scripts. The sidecar compiles x86 and uses reflection to load a caller-supplied `System.Data.SQLite.dll`, so the SQLite binary is not committed. -- [ ] **Step 3: Update status/matrix/plan** +- [x] **Step 3: Verify default and evidence-backed smokes** -Record the implementation boundary and next code slice based on actual C27 output. +Verified default and evidence-backed sidecar smokes with `scripts\verify-msglib-sidecar.ps1`; evidence-backed smoke used copied DB paths through environment variables and returned `display_source_count=8`. -- [ ] **Step 4: Verify and commit** +- [x] **Step 4: Verify and commit** -Run the appropriate docs/code verification before commit: +Run final verification before commit: ```powershell git diff --check go test ./... -powershell -NoProfile -ExecutionPolicy Bypass -File scripts/verify-go-mcp.ps1 +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build-msglib-sidecar.ps1 +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1 +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1 # with copied DB env paths +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1 +``` + +Final verification passed: `git diff --check`, `go test ./...`, explicit `go build ./cmd/isphere-mcp`, default sidecar smoke, evidence-backed sidecar smoke, and `scripts\verify-go-mcp.ps1`. + +--- + +## Loop C27 Result + +Bounded MsgLib sidecar result: + +- Added `native/MsgLibReadSidecar` as a dedicated x86 .NET process with protocol `isphere.msglib.v1`. +- Added `self_check`, `schema_summary`, and `display_sources` operations. +- `schema_summary` and `display_sources` require caller-supplied `sqlite_dll_path` and `db_path`, open only files named `MsgLib.db`, set read-only SQLite connection options, and return schema/column/count/display-source metadata only. +- Added `scripts/build-msglib-sidecar.ps1`, compiling with 32-bit .NET Framework `csc.exe` and `/platform:x86`. +- Added `scripts/verify-msglib-sidecar.ps1`, with default self-check verification plus opt-in evidence-backed provider verification through `ISPHERE_MSGLIB_SQLITE_DLL` and `ISPHERE_MSGLIB_DB`. +- Default sidecar verification passed with `process_bits=32`. +- Evidence-backed sidecar verification passed against the copied DB and returned `display_source_count=8`. +- The sidecar is not yet wrapped by Go and is not yet used by MCP tools. + +--- + +## Loop C28: Go `MsgLibReadSidecar` client wrapper + +**Goal:** Add a Go-side client wrapper that can call the C27 x86 sidecar through a bounded JSON process contract, without exposing a new MCP tool or reading message bodies yet. + +**Planned files:** +- Create: `internal/msglib/sidecar_client.go` +- Create: `internal/msglib/sidecar_client_test.go` +- Optionally update `docs/msglib-read-sidecar-contract.md` +- Modify: `docs/current-status-card.md` +- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md` + +**Planned behavior:** +- Implement a Go client for `isphere.msglib.v1` with request ids, stdin/stdout JSON, timeout, and structured errors. +- Test with a fake sidecar executable/script first; do not require real `System.Data.SQLite.dll` or real DB in normal unit tests. +- Add env config shape but do not wire MCP tools yet: + - `ISPHERE_MSGLIB_SIDECAR_EXE` + - `ISPHERE_MSGLIB_SQLITE_DLL` + - `ISPHERE_MSGLIB_DB` +- Expose only Go functions for `SelfCheck` and `DisplaySources` metadata. +- Do not add message-body reads, file downloads, sends, writes, hooks, injection, DB mutation, or a public MCP tool in this loop. + +- [ ] **Step 1: Write failing fake-sidecar client tests** + +Test request envelope, timeout/error handling, and display-source metadata decoding against a fake process. + +- [ ] **Step 2: Implement sidecar client** + +Use `context.Context`, `exec.CommandContext`, stdin JSON, stdout JSON, stderr capture, and small typed structs. + +- [ ] **Step 3: Verify and update docs** + +Run Go tests and update status/plan. Keep real evidence-backed sidecar smoke optional through `scripts/verify-msglib-sidecar.ps1`. + +- [ ] **Step 4: Full verification and commit** + +Run: + +```powershell +git diff --check +go test ./... +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build-msglib-sidecar.ps1 +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sidecar.ps1 +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1 ``` --- diff --git a/native/MsgLibReadSidecar/Program.cs b/native/MsgLibReadSidecar/Program.cs new file mode 100644 index 0000000..0867a41 --- /dev/null +++ b/native/MsgLibReadSidecar/Program.cs @@ -0,0 +1,329 @@ +using System; +using System.Collections.Generic; +using System.Data.Common; +using System.IO; +using System.Reflection; + +namespace MsgLibReadSidecar +{ + internal static class Program + { + private const string Version = "0.1.0"; + private const string DefaultPassword = "123"; + + private static readonly HashSet DefaultTargetTables = new HashSet(StringComparer.OrdinalIgnoreCase) + { + "TD_Roster", "TD_CustomEffigy", "tblRecent", "tblChatLevel", "tblPersonMsg", "tblMsgGroupPersonMsg", + "TD_WorkGroupAuth", "TD_ReceiveFileRecord", "TD_SendFileRecord", "TD_SystemMessageRecord", "TD_MsgReadBurn" + }; + + [STAThread] + private static int Main(string[] args) + { + string requestText = Console.In.ReadToEnd(); + Dictionary request; + string requestId = ""; + string op = ""; + + if (!SidecarProtocol.TryParseRequest(requestText, out request, out requestId, out op)) + { + Console.WriteLine(SidecarProtocol.ToJson(SidecarProtocol.Failure(requestId, op, "INVALID_REQUEST", "request must be a valid isphere.msglib.v1 JSON object"))); + return 0; + } + + try + { + Dictionary opArgs = SidecarProtocol.GetObject(request, "args"); + Dictionary response; + switch (op) + { + case "self_check": + response = SidecarProtocol.Success(requestId, op, SelfCheckData()); + break; + case "schema_summary": + response = SidecarProtocol.Success(requestId, op, SchemaSummary(opArgs, false)); + break; + case "display_sources": + response = SidecarProtocol.Success(requestId, op, SchemaSummary(opArgs, true)); + break; + default: + response = SidecarProtocol.Failure(requestId, op, "UNSUPPORTED_OP", "unsupported op: " + op); + break; + } + Console.WriteLine(SidecarProtocol.ToJson(response)); + return 0; + } + catch (Exception ex) + { + Console.Error.WriteLine(ex.ToString()); + Console.WriteLine(SidecarProtocol.ToJson(SidecarProtocol.Failure(requestId, op, "SIDECAR_FAILED", ex.Message))); + return 0; + } + } + + private static Dictionary SelfCheckData() + { + return new Dictionary + { + { "sidecar_name", "MsgLibReadSidecar" }, + { "sidecar_version", Version }, + { "protocol", SidecarProtocol.Protocol }, + { "runtime", ".NET Framework " + Environment.Version }, + { "process_bits", IntPtr.Size * 8 }, + { "requires_process_bits", 32 }, + { "db_mutation_allowed", false }, + { "message_body_reads_allowed", false } + }; + } + + private static Dictionary SchemaSummary(Dictionary args, bool displayOnly) + { + string sqliteDllPath = SidecarProtocol.GetString(args, "sqlite_dll_path", ""); + string dbPath = SidecarProtocol.GetString(args, "db_path", ""); + string password = SidecarProtocol.GetString(args, "password", DefaultPassword); + bool includeCounts = SidecarProtocol.GetBool(args, "include_counts", true); + int maxTables = Clamp(SidecarProtocol.GetInt(args, "max_tables", 128), 1, 256); + List requestedTargets = SidecarProtocol.GetStringList(args, "target_tables"); + HashSet targetTables = requestedTargets.Count == 0 ? DefaultTargetTables : new HashSet(requestedTargets, StringComparer.OrdinalIgnoreCase); + + ValidateInputs(sqliteDllPath, dbPath); + + var result = new Dictionary + { + { "metadata_only", true }, + { "read_only", true }, + { "message_body_values_returned", false }, + { "sqlite_dll_path", sqliteDllPath }, + { "db_path", dbPath }, + { "provider", "System.Data.SQLite" }, + { "process_bits", IntPtr.Size * 8 } + }; + + using (DbConnection connection = OpenSqliteConnection(sqliteDllPath, dbPath, password)) + { + connection.Open(); + TryExecuteNonQuery(connection, "PRAGMA query_only=ON"); + result["sqlite_version"] = ExecuteScalar(connection, "select sqlite_version()"); + List> tables = ReadMasterTables(connection, maxTables); + List> columns = ReadColumns(connection, tables, displayOnly ? targetTables : null); + result["tables"] = tables; + result["columns"] = columns; + result["target_row_counts"] = includeCounts ? ReadTargetCounts(connection, tables, targetTables) : new List>(); + if (displayOnly) + { + result["display_sources"] = BuildDisplaySources(tables, columns); + } + } + + return result; + } + + private static void ValidateInputs(string sqliteDllPath, string dbPath) + { + if (string.IsNullOrWhiteSpace(sqliteDllPath) || !File.Exists(sqliteDllPath)) + { + throw new ArgumentException("sqlite_dll_path must point to an existing System.Data.SQLite.dll"); + } + if (string.IsNullOrWhiteSpace(dbPath) || !File.Exists(dbPath)) + { + throw new ArgumentException("db_path must point to an existing copied MsgLib.db"); + } + if (!string.Equals(Path.GetFileName(dbPath), "MsgLib.db", StringComparison.OrdinalIgnoreCase)) + { + throw new ArgumentException("db_path must point to a copied MsgLib.db file"); + } + } + + private static DbConnection OpenSqliteConnection(string sqliteDllPath, string dbPath, string password) + { + Assembly assembly = Assembly.LoadFrom(sqliteDllPath); + Type builderType = assembly.GetType("System.Data.SQLite.SQLiteConnectionStringBuilder", true); + object builder = Activator.CreateInstance(builderType); + SetPropertyIfExists(builderType, builder, "DataSource", dbPath); + SetPropertyIfExists(builderType, builder, "Password", password); + SetPropertyIfExists(builderType, builder, "ReadOnly", true); + SetPropertyIfExists(builderType, builder, "Pooling", false); + SetPropertyIfExists(builderType, builder, "FailIfMissing", true); + string connectionString = Convert.ToString(builderType.GetProperty("ConnectionString").GetValue(builder, null)); + Type connectionType = assembly.GetType("System.Data.SQLite.SQLiteConnection", true); + object connection = Activator.CreateInstance(connectionType, new object[] { connectionString }); + DbConnection dbConnection = connection as DbConnection; + if (dbConnection == null) + { + throw new InvalidOperationException("SQLiteConnection is not a DbConnection"); + } + return dbConnection; + } + + private static void SetPropertyIfExists(Type type, object target, string name, object value) + { + PropertyInfo property = type.GetProperty(name); + if (property != null && property.CanWrite) + { + property.SetValue(target, value, null); + } + } + + private static List> ReadMasterTables(DbConnection connection, int maxTables) + { + var rows = new List>(); + using (DbCommand command = connection.CreateCommand()) + { + command.CommandText = "select type, name, tbl_name from sqlite_master where type in ('table','view') order by type, name"; + using (DbDataReader reader = command.ExecuteReader()) + { + while (reader.Read() && rows.Count < maxTables) + { + rows.Add(new Dictionary + { + { "type", Convert.ToString(reader["type"]) }, + { "name", Convert.ToString(reader["name"]) }, + { "tbl_name", Convert.ToString(reader["tbl_name"]) } + }); + } + } + } + return rows; + } + + private static List> ReadColumns(DbConnection connection, List> tables, HashSet onlyTables) + { + var rows = new List>(); + foreach (Dictionary table in tables) + { + string tableName = Convert.ToString(table["name"]); + string tableType = Convert.ToString(table["type"]); + if (!string.Equals(tableType, "table", StringComparison.OrdinalIgnoreCase)) continue; + if (onlyTables != null && !onlyTables.Contains(tableName)) continue; + + using (DbCommand command = connection.CreateCommand()) + { + command.CommandText = "PRAGMA table_info(" + QuoteIdent(tableName) + ")"; + using (DbDataReader reader = command.ExecuteReader()) + { + while (reader.Read()) + { + rows.Add(new Dictionary + { + { "table", tableName }, + { "cid", Convert.ToString(reader["cid"]) }, + { "name", Convert.ToString(reader["name"]) }, + { "type", Convert.ToString(reader["type"]) }, + { "notnull", Convert.ToString(reader["notnull"]) }, + { "pk", Convert.ToString(reader["pk"]) } + }); + } + } + } + } + return rows; + } + + private static List> ReadTargetCounts(DbConnection connection, List> tables, HashSet targetTables) + { + var rows = new List>(); + foreach (Dictionary table in tables) + { + string tableName = Convert.ToString(table["name"]); + string tableType = Convert.ToString(table["type"]); + if (!string.Equals(tableType, "table", StringComparison.OrdinalIgnoreCase)) continue; + if (!targetTables.Contains(tableName)) continue; + rows.Add(new Dictionary + { + { "table", tableName }, + { "row_count", ExecuteScalar(connection, "select count(*) from " + QuoteIdent(tableName)) } + }); + } + return rows; + } + + private static List> BuildDisplaySources(List> tables, List> columns) + { + var tableSet = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (Dictionary table in tables) + { + if (string.Equals(Convert.ToString(table["type"]), "table", StringComparison.OrdinalIgnoreCase)) + { + tableSet.Add(Convert.ToString(table["name"])); + } + } + + var columnMap = new Dictionary>(StringComparer.OrdinalIgnoreCase); + foreach (Dictionary column in columns) + { + string table = Convert.ToString(column["table"]); + string name = Convert.ToString(column["name"]); + if (!columnMap.ContainsKey(table)) columnMap[table] = new HashSet(StringComparer.OrdinalIgnoreCase); + columnMap[table].Add(name); + } + + var sources = new List>(); + AddSource(sources, tableSet, columnMap, "contacts_roster", "TD_Roster", new string[] { "JId", "Nick", "GroupName" }); + AddSource(sources, tableSet, columnMap, "contacts_effigy", "TD_CustomEffigy", new string[] { "PersonJid", "PersonName" }); + AddSource(sources, tableSet, columnMap, "recent_display", "tblRecent", new string[] { "PersonId", "PersonName", "FType" }); + AddSource(sources, tableSet, columnMap, "person_message_names", "tblPersonMsg", new string[] { "SndPerson", "SndPersonId", "RecvPerson", "RecvPersonId", "ObjPerson", "ObjPersonId" }); + AddSource(sources, tableSet, columnMap, "group_message_names", "tblMsgGroupPersonMsg", new string[] { "MsgGroupId", "MsgGroupName", "SessionPersonId", "SessionPersonName", "GroupType" }); + AddSource(sources, tableSet, columnMap, "work_group_auth", "TD_WorkGroupAuth", new string[] { "GroupJID", "GroupName", "ChildGroupJid" }); + AddSource(sources, tableSet, columnMap, "received_file_metadata", "TD_ReceiveFileRecord", new string[] { "FileName", "SendPersonJid", "SendPersonName" }); + AddSource(sources, tableSet, columnMap, "sent_file_metadata", "TD_SendFileRecord", new string[] { "FileName", "ReceivePersonJid", "ReceivePersonName" }); + return sources; + } + + private static void AddSource(List> sources, HashSet tableSet, Dictionary> columnMap, string sourceName, string table, string[] columns) + { + var presentColumns = new List(); + if (columnMap.ContainsKey(table)) + { + foreach (string column in columns) + { + if (columnMap[table].Contains(column)) presentColumns.Add(column); + } + } + sources.Add(new Dictionary + { + { "source", sourceName }, + { "table", table }, + { "available", tableSet.Contains(table) && presentColumns.Count > 0 }, + { "required_columns", columns }, + { "present_columns", presentColumns.ToArray() } + }); + } + + private static string ExecuteScalar(DbConnection connection, string sql) + { + using (DbCommand command = connection.CreateCommand()) + { + command.CommandText = sql; + object value = command.ExecuteScalar(); + return Convert.ToString(value); + } + } + + private static void TryExecuteNonQuery(DbConnection connection, string sql) + { + try + { + using (DbCommand command = connection.CreateCommand()) + { + command.CommandText = sql; + command.ExecuteNonQuery(); + } + } + catch + { + } + } + + private static string QuoteIdent(string value) + { + return "\"" + value.Replace("\"", "\"\"") + "\""; + } + + private static int Clamp(int value, int min, int max) + { + if (value < min) return min; + if (value > max) return max; + return value; + } + } +} diff --git a/native/MsgLibReadSidecar/SidecarProtocol.cs b/native/MsgLibReadSidecar/SidecarProtocol.cs new file mode 100644 index 0000000..a6ed132 --- /dev/null +++ b/native/MsgLibReadSidecar/SidecarProtocol.cs @@ -0,0 +1,142 @@ +using System; +using System.Collections.Generic; +using System.Web.Script.Serialization; + +namespace MsgLibReadSidecar +{ + internal static class SidecarProtocol + { + public const string Protocol = "isphere.msglib.v1"; + private static readonly JavaScriptSerializer Serializer = new JavaScriptSerializer { MaxJsonLength = 16 * 1024 * 1024, RecursionLimit = 256 }; + + public static bool TryParseRequest(string text, out Dictionary request, out string requestId, out string op) + { + request = null; + requestId = ""; + op = ""; + + if (string.IsNullOrWhiteSpace(text)) + { + return false; + } + + try + { + object decoded = Serializer.DeserializeObject(text); + request = decoded as Dictionary; + if (request == null) + { + return false; + } + + string protocol = GetString(request, "protocol", ""); + requestId = GetString(request, "request_id", ""); + op = GetString(request, "op", ""); + + return protocol == Protocol && !string.IsNullOrWhiteSpace(op); + } + catch + { + return false; + } + } + + public static Dictionary Success(string requestId, string op, object data) + { + return Envelope(requestId, op, true, data, null); + } + + public static Dictionary Failure(string requestId, string op, string code, string message) + { + return Envelope(requestId, op, false, null, new Dictionary { { "code", code }, { "message", message } }); + } + + private static Dictionary Envelope(string requestId, string op, bool ok, object data, object error) + { + return new Dictionary + { + { "protocol", Protocol }, + { "request_id", requestId ?? "" }, + { "op", op ?? "" }, + { "ok", ok }, + { "data", data }, + { "error", error }, + { "warnings", new object[0] } + }; + } + + public static string ToJson(object value) + { + return Serializer.Serialize(value); + } + + public static Dictionary GetObject(Dictionary source, string key) + { + if (source == null || !source.ContainsKey(key) || source[key] == null) + { + return new Dictionary(); + } + return source[key] as Dictionary ?? new Dictionary(); + } + + public static string GetString(Dictionary source, string key, string defaultValue) + { + if (source == null || !source.ContainsKey(key) || source[key] == null) + { + return defaultValue; + } + return Convert.ToString(source[key]); + } + + public static bool GetBool(Dictionary source, string key, bool defaultValue) + { + if (source == null || !source.ContainsKey(key) || source[key] == null) + { + return defaultValue; + } + try { return Convert.ToBoolean(source[key]); } + catch { return defaultValue; } + } + + public static int GetInt(Dictionary source, string key, int defaultValue) + { + if (source == null || !source.ContainsKey(key) || source[key] == null) + { + return defaultValue; + } + try { return Convert.ToInt32(source[key]); } + catch { return defaultValue; } + } + + public static List GetStringList(Dictionary source, string key) + { + var values = new List(); + if (source == null || !source.ContainsKey(key) || source[key] == null) + { + return values; + } + object raw = source[key]; + object[] array = raw as object[]; + if (array != null) + { + foreach (object item in array) + { + if (item != null) values.Add(Convert.ToString(item)); + } + return values; + } + System.Collections.ArrayList list = raw as System.Collections.ArrayList; + if (list != null) + { + foreach (object item in list) + { + if (item != null) values.Add(Convert.ToString(item)); + } + return values; + } + string single = Convert.ToString(raw); + if (!string.IsNullOrWhiteSpace(single)) values.Add(single); + return values; + } + } +} diff --git a/scripts/build-msglib-sidecar.ps1 b/scripts/build-msglib-sidecar.ps1 new file mode 100644 index 0000000..41b62bd --- /dev/null +++ b/scripts/build-msglib-sidecar.ps1 @@ -0,0 +1,53 @@ +param( + [string]$OutputDir = "runs/msglib-sidecar", + [string]$Configuration = "Release" +) + +$ErrorActionPreference = "Stop" + +$repo = Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..") +$sourceDir = Join-Path $repo "native/MsgLibReadSidecar" +$outDirPath = Join-Path $repo $OutputDir +$outExe = Join-Path $outDirPath "MsgLibReadSidecar.exe" + +if (-not (Test-Path -LiteralPath $sourceDir)) { + throw "source directory not found: $sourceDir" +} + +$csc = "$env:WINDIR\Microsoft.NET\Framework\v4.0.30319\csc.exe" +if (-not (Test-Path -LiteralPath $csc)) { + throw "32-bit .NET Framework csc.exe v4.0.30319 not found" +} + +New-Item -ItemType Directory -Force -Path $outDirPath | Out-Null + +$sources = Get-ChildItem -LiteralPath $sourceDir -Filter '*.cs' | Sort-Object Name | ForEach-Object { $_.FullName } +if (-not $sources) { + throw "no C# sources found in $sourceDir" +} + +& $csc ` + /nologo ` + /target:exe ` + /platform:x86 ` + /optimize+ ` + /warn:0 ` + /out:$outExe ` + /reference:System.dll ` + /reference:System.Core.dll ` + /reference:System.Data.dll ` + /reference:System.Web.Extensions.dll ` + $sources + +if ($LASTEXITCODE -ne 0) { + throw "MsgLib sidecar compilation failed with exit code $LASTEXITCODE" +} + +[ordered]@{ + ok = $true + configuration = $Configuration + platform = "x86" + csc = $csc + source_dir = $sourceDir + output = $outExe +} | ConvertTo-Json -Depth 4 -Compress diff --git a/scripts/verify-msglib-sidecar.ps1 b/scripts/verify-msglib-sidecar.ps1 new file mode 100644 index 0000000..26457be --- /dev/null +++ b/scripts/verify-msglib-sidecar.ps1 @@ -0,0 +1,81 @@ +param( + [string]$SidecarExe = "runs/msglib-sidecar/MsgLibReadSidecar.exe", + [string]$SQLiteDllPath = $env:ISPHERE_MSGLIB_SQLITE_DLL, + [string]$DbPath = $env:ISPHERE_MSGLIB_DB, + [switch]$SkipBuild +) + +$ErrorActionPreference = "Stop" +$repo = Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..") +$sidecarPath = Join-Path $repo $SidecarExe + +function Invoke-SidecarJson([hashtable]$Request) { + $json = $Request | ConvertTo-Json -Depth 10 -Compress + $output = $json | & $sidecarPath + if ($LASTEXITCODE -ne 0) { + throw "sidecar exited with code $LASTEXITCODE" + } + try { + return $output | ConvertFrom-Json + } + catch { + throw "sidecar output was not JSON: $output" + } +} + +if (-not $SkipBuild) { + & powershell -NoProfile -ExecutionPolicy Bypass -File (Join-Path $repo "scripts\build-msglib-sidecar.ps1") | Out-Host + if ($LASTEXITCODE -ne 0) { + throw "build-msglib-sidecar.ps1 failed with exit code $LASTEXITCODE" + } +} + +if (-not (Test-Path -LiteralPath $sidecarPath)) { + throw "sidecar not found: $sidecarPath" +} + +$self = Invoke-SidecarJson @{ + protocol = "isphere.msglib.v1" + request_id = "verify-self-check" + op = "self_check" + args = @{} +} +if (-not $self.ok -or $self.data.sidecar_name -ne "MsgLibReadSidecar" -or $self.data.process_bits -ne 32) { + throw "self_check failed: $($self | ConvertTo-Json -Depth 8 -Compress)" +} + +$providerChecked = $false +$displaySources = $null +if ($SQLiteDllPath -and $DbPath) { + if (-not (Test-Path -LiteralPath $SQLiteDllPath)) { + throw "SQLite DLL not found: $SQLiteDllPath" + } + if (-not (Test-Path -LiteralPath $DbPath)) { + throw "MsgLib DB not found: $DbPath" + } + $displaySources = Invoke-SidecarJson @{ + protocol = "isphere.msglib.v1" + request_id = "verify-display-sources" + op = "display_sources" + args = @{ + sqlite_dll_path = $SQLiteDllPath + db_path = $DbPath + password = "123" + include_counts = $true + max_tables = 64 + } + } + if (-not $displaySources.ok -or -not $displaySources.data.metadata_only -or $displaySources.data.message_body_values_returned) { + throw "display_sources failed: $($displaySources | ConvertTo-Json -Depth 12 -Compress)" + } + $providerChecked = $true +} + +[ordered]@{ + ok = $true + sidecar = (Resolve-Path -LiteralPath $sidecarPath).Path + version = $self.data.sidecar_version + process_bits = $self.data.process_bits + provider_checked = $providerChecked + display_source_count = if ($displaySources) { $displaySources.data.display_sources.Count } else { 0 } +} | ConvertTo-Json -Depth 6 -Compress