diff --git a/docs/current-status-card.md b/docs/current-status-card.md index de3e9a7..a04e461 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Stage C / Loop C36 - MsgLib message-source metadata sidecar`. +Current loop: `Stage C / Loop C37 - bounded DB-backed receive-source design`. Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. @@ -104,10 +104,11 @@ Current loop output so far: 33. C33: optional real copied-DB MCP smoke now also constructs a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verifies receive-message display enrichment through MCP; output stays sanitized to counts, booleans, and `msglib:` refs only. 34. C34: `docs/go-mcp-runbook.md` now documents the operator setup path for optional MsgLib display-name enrichment, including the required env vars, partial-config failure rule, standard no-MsgLib verification, optional provider smoke, optional MCP enrichment smoke, and sanitized output boundaries. 35. C35: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` maps `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, recent, and file-record schema columns to the future `isphere_receive_messages` contract; decision is metadata-only `message_sources` sidecar first, not body-returning DB listing yet. +36. C36: added metadata-only MsgLib `message_sources` across the x86 sidecar, Go wrapper, and sidecar verification; evidence-backed smoke returned six message-source summaries with source/table/role/availability/row-count metadata only. ## Next business mainline -1. Run Stage C Loop C36: add a metadata-only MsgLib sidecar operation for message-source readiness, returning table availability, required/present columns, and row counts only. +1. Run Stage C Loop C37: design the bounded DB-backed receive-message source contract now that `message_sources` metadata is available; do not expose body rows until strict limits and output rules are specified. 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated. 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated. 4. Keep UIA selector/report work as fallback support, not as the primary roadmap. diff --git a/docs/msglib-read-sidecar-contract.md b/docs/msglib-read-sidecar-contract.md index 1d074ef..f9c5b7c 100644 --- a/docs/msglib-read-sidecar-contract.md +++ b/docs/msglib-read-sidecar-contract.md @@ -145,13 +145,52 @@ Initial allowlisted identity sources: - Contacts: `TD_Roster(JId,Nick)`, `TD_CustomEffigy(PersonJid,PersonName)`, `tblRecent(PersonId,PersonName)`, `tblChatLevel(PersonId,PersonName)`, selected `tblPersonMsg` id/name pairs only. - Groups: `TD_WorkGroupAuth(GroupJID,GroupName)`, `TD_WorkGroupAuth(ChildGroupJid,GroupName)`, selected `tblMsgGroupPersonMsg` group/session id/name pairs only. +### `message_sources` + +Same safe provider/DB validation as `display_sources`, but reports whether candidate message tables are ready for a future DB-backed `isphere_receive_messages` source. + +Required args: + +- `sqlite_dll_path` +- `db_path` + +Optional args: + +- `password`: default `123`. + +Returned data adds: + +- `metadata_only=true` +- `read_only=true` +- `message_body_values_returned=false` +- `raw_rows_returned=false` +- `file_paths_returned=false` +- `message_sources[]` entries with: + - `source` + - `table` + - `role` + - `available` + - `required_columns` + - `present_columns` + - `row_count` + +Initial message source candidates: + +- `direct_messages` from `tblPersonMsg`. +- `group_messages` from `tblMsgGroupPersonMsg`. +- `system_messages` from `TD_SystemMessageRecord`. +- `group_receipts` from `TD_GroupReceiptMessage`. +- `received_file_metadata` from `TD_ReceiveFileRecord`. +- `recent_conversation_index` from `tblRecent`. + ## Security and scope rules - The sidecar must open DB files with read-only connection settings. - `db_path` must point to a file named `MsgLib.db`. - The schema/display-source operations return schema, columns, counts, and display-source availability only. - `display_entities` may return bounded contact/group `jid` and `display_name` metadata only. -- It does not return message bodies, message text, file paths, attachment contents, or raw row sets. +- `message_sources` may return message-source table names, required/present column names, roles, availability flags, and row counts only. +- It does not return message bodies, message text, file path values, attachment contents, or raw row sets. - It does not implement send, file download, mark-read, login, hooks, injection, or DB writes. - The Go MCP layer remains the coordinator; this process is a bounded local reader. @@ -186,6 +225,13 @@ Committed C29 implementation: - Evidence-backed C29 verification against the copied DB returned contact/group entity counts and source-table names without printing entity values. - C31 adds optional `scripts/verify-msglib-mcp-enrichment.ps1` to prove MCP-level contact/group enrichment through the real sidecar while printing only counts and `msglib:` refs. C33 extends the same optional script to create a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verify receive-message `sender_name` plus `conversation.display_name` enrichment; the printed result remains limited to counts, booleans, and source refs. +Committed C36 implementation: + +- Sidecar op: `message_sources`. +- Go wrapper: `MessageSources(ctx)`. +- Verification script calls `message_sources` only when copied DB env paths are supplied, and prints only sanitized `message_source_count` plus message-source summaries containing source/table/role/availability/row-count metadata. +- Evidence-backed C36 verification against the copied DB returned six message-source summaries without printing message body values, file path values, raw rows, or copied DB contents. + ## Verification Default verification builds and checks `self_check` only: @@ -210,4 +256,4 @@ $env:ISPHERE_MSGLIB_DB = "" powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-enrichment.ps1 ``` -Expected printed evidence is sanitized: contact/group counts, `msglib:` refs, `receive_display_smoke=true`, populated-field booleans, and no entity values, message bodies, file paths, or raw rows. +Expected printed evidence is sanitized: contact/group counts, `msglib:` refs, `message_source_count`, message-source table/role/row-count summaries, `receive_display_smoke=true`, populated-field booleans, and no entity values, message bodies, file path values, or raw rows. diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index 57ce89f..c9b8219 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | --- | --- | --- | --- | --- | --- | -| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables to the receive contract but keeps body/row reads out of scope | C36 metadata-only MsgLib message-source sidecar | -| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar | -| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar | +| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources` readiness with table/column/count summaries only | C37 bounded DB-backed receive-source design | +| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design | +| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design | | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence | | `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration | | `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated | @@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac ## Deferred source work -`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, and C35 maps MsgLib message tables to receive-message contract fields without reading row values. The next step is a metadata-only sidecar operation for message-source readiness. +`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, and C36 adds metadata-only `message_sources` readiness. The next step is bounded DB-backed receive-source design. diff --git a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md index a21cce7..c0a2d81 100644 --- a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md +++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md @@ -2600,17 +2600,55 @@ MsgLib message-table source precheck result: - Return table availability, required/present columns, safe row counts, and source role metadata only. - Do not return `MsgText`, `MessageBody`, `FilePath`, entity values, raw rows, sends, downloads, hooks, injection, or DB writes. -- [ ] **Step 1: Write failing Go client/sidecar tests for message-source metadata** +- [x] **Step 1: Write failing Go client/sidecar tests for message-source metadata** -Test request/response decoding and bounded metadata shape with a fake sidecar before implementation. +Added `TestMessageSourcesPassesConfiguredPathsAndDecodesMetadata`; RED failed with `client.MessageSources undefined`. -- [ ] **Step 2: Implement sidecar metadata operation** +- [x] **Step 2: Implement sidecar metadata operation** -Expose message-source readiness using existing read-only schema/count helpers and allowlisted table/column maps. +Implemented Go `MessageSources(ctx)` plus x86 sidecar op `message_sources`. The op returns metadata-only source/table/role/availability/required/present columns/row counts for allowlisted message-source tables. -- [ ] **Step 3: Verify default and evidence-backed smokes** +- [x] **Step 3: Verify default and evidence-backed smokes** -Update `scripts\verify-msglib-sidecar.ps1` to assert sanitized message-source metadata when copied DB env is configured, then run the full gate. +Updated `scripts\verify-msglib-sidecar.ps1` to call `message_sources` when copied DB env is configured. Verification passed: `git diff --check`, `go test ./...`, default sidecar smoke, evidence-backed sidecar smoke, optional `scripts\verify-msglib-mcp-enrichment.ps1`, and deterministic `scripts\verify-go-mcp.ps1`. Default sidecar smoke returned `message_source_count=0`; evidence-backed smoke returned `message_source_count=6`, `message_body_values_returned=false`, `raw_rows_returned=false`, and `file_paths_returned=false`. + +--- + +## Loop C36 Result + +MsgLib message-source metadata sidecar result: + +- Added sidecar op `message_sources` and Go wrapper `MessageSources(ctx)`. +- Added fake-sidecar unit coverage for request args and decoded metadata shape. +- Evidence-backed sidecar smoke returned six message-source summaries: direct, group, system, group receipt, received-file metadata, and recent conversation index. +- Output remains metadata-only: table/column/count summaries, no `MsgText`/`MessageBody` values, no file path values, no raw rows. + +--- + +## Loop C37: Bounded DB-backed receive-source design + +**Goal:** Design the smallest safe implementation slice for DB-backed `isphere_receive_messages` using C35 field mapping and C36 `message_sources` readiness metadata. + +**Planned files:** +- Modify or create a design note under `docs/source-discovery/`. +- Modify matrix/status/plan. +- Do not implement body-returning DB listing until the contract and acceptance gate are explicit. + +**Planned behavior:** +- Define direct/group source selection, ordering, limit, body-column policy, attachment metadata policy, and fallback behavior versus PacketReader. +- Decide whether C38 should add sidecar-level bounded row listing or first add a redacted synthetic DB fixture. + +- [ ] **Step 1: Write bounded DB receive-source contract** + +Define request args, returned fields, limits, and no-output boundaries for DB-backed receive listing. + +- [ ] **Step 2: Choose implementation route** + +Select between sidecar row listing, Go source abstraction first, or synthetic fixture-first tests. + +- [ ] **Step 3: Update next implementation loop** + +Record C37 decision and prepare C38. --- diff --git a/internal/msglib/sidecar_client.go b/internal/msglib/sidecar_client.go index 6337e0c..e20a9f6 100644 --- a/internal/msglib/sidecar_client.go +++ b/internal/msglib/sidecar_client.go @@ -87,6 +87,16 @@ type DisplaySource struct { PresentColumns []string `json:"present_columns"` } +type MessageSource struct { + Source string `json:"source"` + Table string `json:"table"` + Role string `json:"role"` + Available bool `json:"available"` + RequiredColumns []string `json:"required_columns"` + PresentColumns []string `json:"present_columns"` + RowCount string `json:"row_count"` +} + type DisplayEntitiesOptions struct { EntityType string Query string @@ -139,6 +149,19 @@ func (c *Client) DisplaySources(ctx context.Context) ([]DisplaySource, error) { return out.DisplaySources, nil } +func (c *Client) MessageSources(ctx context.Context) ([]MessageSource, error) { + args := map[string]any{ + "sqlite_dll_path": c.config.SQLiteDLLPath, + "db_path": c.config.DBPath, + "password": c.config.Password, + } + var out messageSourcesData + if err := c.call(ctx, "message_sources", args, &out); err != nil { + return nil, err + } + return out.MessageSources, nil +} + func (c *Client) DisplayEntities(ctx context.Context, opts DisplayEntitiesOptions) ([]DisplayEntity, error) { entityType := strings.TrimSpace(opts.EntityType) if entityType != EntityTypeContacts && entityType != EntityTypeGroups { @@ -174,6 +197,10 @@ type displaySourcesData struct { DisplaySources []DisplaySource `json:"display_sources"` } +type messageSourcesData struct { + MessageSources []MessageSource `json:"message_sources"` +} + type displayEntitiesData struct { Entities []DisplayEntity `json:"entities"` } diff --git a/internal/msglib/sidecar_client_test.go b/internal/msglib/sidecar_client_test.go index c87e32a..dad7972 100644 --- a/internal/msglib/sidecar_client_test.go +++ b/internal/msglib/sidecar_client_test.go @@ -55,6 +55,32 @@ func TestDisplaySourcesPassesConfiguredPathsAndDecodesMetadata(t *testing.T) { } } +func TestMessageSourcesPassesConfiguredPathsAndDecodesMetadata(t *testing.T) { + cfg := Config{ + SQLiteDLLPath: `C:\fixture\System.Data.SQLite.dll`, + DBPath: `C:\fixture\MsgLib.db`, + Password: "123", + } + client := newFakeClient(t, "message_sources_ok", cfg) + + got, err := client.MessageSources(context.Background()) + if err != nil { + t.Fatalf("MessageSources returned error: %v", err) + } + if len(got) != 2 { + t.Fatalf("len(MessageSources) = %d, want 2", len(got)) + } + if got[0].Source != "direct_messages" || got[0].Table != "tblPersonMsg" || got[0].Role != "direct" || !got[0].Available { + t.Fatalf("unexpected first message source: %+v", got[0]) + } + if got[0].RowCount != "2132" || strings.Join(got[0].RequiredColumns, ",") != "Id,SndPersonId,RecvPersonId,ObjPersonId,ActionTime,MsgText,MessageBody" { + t.Fatalf("unexpected first message source metadata: %+v", got[0]) + } + if got[1].Source != "group_messages" || got[1].Table != "tblMsgGroupPersonMsg" || got[1].Role != "group" || got[1].RowCount != "2884" { + t.Fatalf("unexpected second message source: %+v", got[1]) + } +} + func TestDisplayEntitiesPassesBoundedArgsAndDecodesMetadata(t *testing.T) { cfg := Config{ SQLiteDLLPath: `C:\fixture\System.Data.SQLite.dll`, @@ -240,6 +266,47 @@ func fakeSidecarMain() int { "error": nil, "warnings": []any{}, }) + case "message_sources": + args, _ := req["args"].(map[string]any) + if args["sqlite_dll_path"] != `C:\fixture\System.Data.SQLite.dll` || args["db_path"] != `C:\fixture\MsgLib.db` || args["password"] != "123" { + fmt.Fprintf(os.Stderr, "bad message_sources args: %#v\n", args) + return 2 + } + return writeFakeResponse(map[string]any{ + "protocol": Protocol, + "request_id": requestID, + "op": op, + "ok": true, + "data": map[string]any{ + "metadata_only": true, + "read_only": true, + "message_body_values_returned": false, + "raw_rows_returned": false, + "file_paths_returned": false, + "message_sources": []any{ + map[string]any{ + "source": "direct_messages", + "table": "tblPersonMsg", + "role": "direct", + "available": true, + "required_columns": []any{"Id", "SndPersonId", "RecvPersonId", "ObjPersonId", "ActionTime", "MsgText", "MessageBody"}, + "present_columns": []any{"Id", "SndPersonId", "RecvPersonId", "ObjPersonId", "ActionTime", "MsgText", "MessageBody"}, + "row_count": "2132", + }, + map[string]any{ + "source": "group_messages", + "table": "tblMsgGroupPersonMsg", + "role": "group", + "available": true, + "required_columns": []any{"Id", "SndPersonId", "MsgGroupId", "MsgGroupName", "MsgTime", "MsgText", "MessageBody"}, + "present_columns": []any{"Id", "SndPersonId", "MsgGroupId", "MsgGroupName", "MsgTime", "MsgText", "MessageBody"}, + "row_count": "2884", + }, + }, + }, + "error": nil, + "warnings": []any{}, + }) case "display_entities": args, _ := req["args"].(map[string]any) if args["sqlite_dll_path"] != `C:\fixture\System.Data.SQLite.dll` || args["db_path"] != `C:\fixture\MsgLib.db` || args["password"] != "123" || args["entity_type"] != "contacts" || args["query"] != "alice" || args["limit"] != float64(2) { diff --git a/native/MsgLibReadSidecar/Program.cs b/native/MsgLibReadSidecar/Program.cs index 06d6b84..f50e98b 100644 --- a/native/MsgLibReadSidecar/Program.cs +++ b/native/MsgLibReadSidecar/Program.cs @@ -8,7 +8,7 @@ namespace MsgLibReadSidecar { internal static class Program { - private const string Version = "0.2.0"; + private const string Version = "0.3.0"; private const string DefaultPassword = "123"; private static readonly HashSet DefaultTargetTables = new HashSet(StringComparer.OrdinalIgnoreCase) @@ -46,6 +46,9 @@ namespace MsgLibReadSidecar case "display_sources": response = SidecarProtocol.Success(requestId, op, SchemaSummary(opArgs, true)); break; + case "message_sources": + response = SidecarProtocol.Success(requestId, op, MessageSources(opArgs)); + break; case "display_entities": response = SidecarProtocol.Success(requestId, op, DisplayEntities(opArgs)); break; @@ -123,6 +126,40 @@ namespace MsgLibReadSidecar return result; } + private static Dictionary MessageSources(Dictionary args) + { + string sqliteDllPath = SidecarProtocol.GetString(args, "sqlite_dll_path", ""); + string dbPath = SidecarProtocol.GetString(args, "db_path", ""); + string password = SidecarProtocol.GetString(args, "password", DefaultPassword); + + ValidateInputs(sqliteDllPath, dbPath); + + var result = new Dictionary + { + { "metadata_only", true }, + { "read_only", true }, + { "message_body_values_returned", false }, + { "raw_rows_returned", false }, + { "file_paths_returned", false }, + { "provider", "System.Data.SQLite" }, + { "process_bits", IntPtr.Size * 8 } + }; + + using (DbConnection connection = OpenSqliteConnection(sqliteDllPath, dbPath, password)) + { + connection.Open(); + TryExecuteNonQuery(connection, "PRAGMA query_only=ON"); + List> tables = ReadMasterTables(connection, 256); + HashSet messageTables = BuildMessageSourceTableSet(); + List> columns = ReadColumns(connection, tables, messageTables); + Dictionary> columnMap = BuildColumnMap(columns); + Dictionary rowCounts = ReadTargetCountMap(connection, tables, messageTables); + result["message_sources"] = BuildMessageSources(tables, columnMap, rowCounts); + } + + return result; + } + private static Dictionary DisplayEntities(Dictionary args) { @@ -288,6 +325,16 @@ namespace MsgLibReadSidecar return rows; } + private static Dictionary ReadTargetCountMap(DbConnection connection, List> tables, HashSet targetTables) + { + var rowCounts = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (Dictionary item in ReadTargetCounts(connection, tables, targetTables)) + { + rowCounts[Convert.ToString(item["table"])] = Convert.ToString(item["row_count"]); + } + return rowCounts; + } + private static List> BuildDisplaySources(List> tables, List> columns) { var tableSet = new HashSet(StringComparer.OrdinalIgnoreCase); @@ -320,6 +367,115 @@ namespace MsgLibReadSidecar return sources; } + private sealed class MessageSourceSpec + { + public string Source; + public string Table; + public string Role; + public string[] RequiredColumns; + } + + private static HashSet BuildMessageSourceTableSet() + { + var tables = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (MessageSourceSpec spec in BuildMessageSourceSpecs()) + { + tables.Add(spec.Table); + } + return tables; + } + + private static List BuildMessageSourceSpecs() + { + return new List + { + new MessageSourceSpec + { + Source = "direct_messages", + Table = "tblPersonMsg", + Role = "direct", + RequiredColumns = new string[] { "Id", "SndPersonId", "RecvPersonId", "ObjPersonId", "ActionTime", "MsgText", "MessageBody" } + }, + new MessageSourceSpec + { + Source = "group_messages", + Table = "tblMsgGroupPersonMsg", + Role = "group", + RequiredColumns = new string[] { "Id", "SndPersonId", "MsgGroupId", "MsgGroupName", "MsgTime", "MsgText", "MessageBody" } + }, + new MessageSourceSpec + { + Source = "system_messages", + Table = "TD_SystemMessageRecord", + Role = "system", + RequiredColumns = new string[] { "MsgGuid", "MsgType", "MsgTitle", "MsgText", "SendPersonJid", "MsgTime", "MsgReadState" } + }, + new MessageSourceSpec + { + Source = "group_receipts", + Table = "TD_GroupReceiptMessage", + Role = "receipt_metadata", + RequiredColumns = new string[] { "MsgGuid", "PersonID", "ReceiptTime" } + }, + new MessageSourceSpec + { + Source = "received_file_metadata", + Table = "TD_ReceiveFileRecord", + Role = "attachment_metadata", + RequiredColumns = new string[] { "ReceiveMsgGuid", "FileName", "FileSize", "SourceID", "SourceName", "FileType", "SendPersonJid", "SendPersonName", "SendTime" } + }, + new MessageSourceSpec + { + Source = "recent_conversation_index", + Table = "tblRecent", + Role = "conversation_index", + RequiredColumns = new string[] { "Id", "PersonId", "PersonName", "ActionTime", "FType" } + } + }; + } + + private static List> BuildMessageSources(List> tables, Dictionary> columnMap, Dictionary rowCounts) + { + var tableSet = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (Dictionary table in tables) + { + if (string.Equals(Convert.ToString(table["type"]), "table", StringComparison.OrdinalIgnoreCase)) + { + tableSet.Add(Convert.ToString(table["name"])); + } + } + + var sources = new List>(); + foreach (MessageSourceSpec spec in BuildMessageSourceSpecs()) + { + AddMessageSource(sources, tableSet, columnMap, rowCounts, spec); + } + return sources; + } + + private static void AddMessageSource(List> sources, HashSet tableSet, Dictionary> columnMap, Dictionary rowCounts, MessageSourceSpec spec) + { + var presentColumns = new List(); + if (columnMap.ContainsKey(spec.Table)) + { + foreach (string column in spec.RequiredColumns) + { + if (columnMap[spec.Table].Contains(column)) presentColumns.Add(column); + } + } + string rowCount = rowCounts.ContainsKey(spec.Table) ? rowCounts[spec.Table] : "0"; + sources.Add(new Dictionary + { + { "source", spec.Source }, + { "table", spec.Table }, + { "role", spec.Role }, + { "available", tableSet.Contains(spec.Table) && presentColumns.Count == spec.RequiredColumns.Length }, + { "required_columns", spec.RequiredColumns }, + { "present_columns", presentColumns.ToArray() }, + { "row_count", rowCount } + }); + } + private sealed class DisplayEntitySpec { diff --git a/scripts/verify-msglib-sidecar.ps1 b/scripts/verify-msglib-sidecar.ps1 index 85f7397..19fd88c 100644 --- a/scripts/verify-msglib-sidecar.ps1 +++ b/scripts/verify-msglib-sidecar.ps1 @@ -47,6 +47,8 @@ if (-not $self.ok -or $self.data.sidecar_name -ne "MsgLibReadSidecar" -or $self. $providerChecked = $false $displaySources = $null $displayEntitySummaries = @() +$messageSources = $null +$messageSourceSummaries = @() if ($SQLiteDllPath -and $DbPath) { if (-not (Test-Path -LiteralPath $SQLiteDllPath)) { throw "SQLite DLL not found: $SQLiteDllPath" @@ -70,6 +72,29 @@ if ($SQLiteDllPath -and $DbPath) { throw "display_sources failed" } + $messageSources = Invoke-SidecarJson @{ + protocol = "isphere.msglib.v1" + request_id = "verify-message-sources" + op = "message_sources" + args = @{ + sqlite_dll_path = $SQLiteDllPath + db_path = $DbPath + password = "123" + } + } + if (-not $messageSources.ok -or -not $messageSources.data.metadata_only -or -not $messageSources.data.read_only -or $messageSources.data.message_body_values_returned -or $messageSources.data.raw_rows_returned -or $messageSources.data.file_paths_returned) { + throw "message_sources failed" + } + foreach ($source in @($messageSources.data.message_sources)) { + $messageSourceSummaries += [ordered]@{ + source = $source.source + table = $source.table + role = $source.role + available = $source.available + row_count = $source.row_count + } + } + foreach ($entityType in @("contacts", "groups")) { $displayEntities = Invoke-SidecarJson @{ protocol = "isphere.msglib.v1" @@ -104,4 +129,9 @@ if ($SQLiteDllPath -and $DbPath) { provider_checked = $providerChecked display_source_count = if ($displaySources) { $displaySources.data.display_sources.Count } else { 0 } display_entity_summaries = $displayEntitySummaries + message_source_count = if ($messageSources) { $messageSources.data.message_sources.Count } else { 0 } + message_source_summaries = $messageSourceSummaries + message_body_values_returned = $false + raw_rows_returned = $false + file_paths_returned = $false } | ConvertTo-Json -Depth 8 -Compress