diff --git a/docs/current-status-card.md b/docs/current-status-card.md index dbb3386..6044f95 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -29,7 +29,7 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor - `isphere_receive_messages` accepts the read-contract argument set for the current local-readonly path: `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, `source_preference`, `preview`, and empty `cursor`. - `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path and can optionally enrich results from the env-configured C29 `MsgLib.db` display-entity reader; default behavior remains log/JID-backed when MsgLib env vars are absent. - `isphere_receive_files` list mode accepts and validates the safe file-list contract args for the current local-readonly path while keeping download blocked. -- `native/MsgLibReadSidecar` provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks and bounded contact/group display-entity reads; `internal/msglib` wraps it from Go, and C30 wires it as an optional MCP contact/group enrichment source. +- `native/MsgLibReadSidecar` provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks, bounded contact/group display-entity reads, message-source metadata, and sidecar-level bounded message listing; `internal/msglib` wraps it from Go, and C30 wires display metadata as an optional MCP contact/group enrichment source. - N12-pre and N12R documents define safe offline evidence intake and internal-sandbox live UIA capture procedures. - N13/N14/N15 produced selector catalog/report validation infrastructure, but those are supporting validation assets only. @@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Stage C / Loop C38 - sidecar bounded DB message listing`. +Current loop: `Stage C / Loop C39 - DB-backed receive source adapter test slice`. Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. @@ -106,10 +106,11 @@ Current loop output so far: 35. C35: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` maps `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, recent, and file-record schema columns to the future `isphere_receive_messages` contract; decision is metadata-only `message_sources` sidecar first, not body-returning DB listing yet. 36. C36: added metadata-only MsgLib `message_sources` across the x86 sidecar, Go wrapper, and sidecar verification; evidence-backed smoke returned six message-source summaries with source/table/role/availability/row-count metadata only. 37. C37: `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md` defines the bounded DB-backed receive-message design: sidecar op `list_messages`, hard limit, no cursor initially, explicit `include_body`, no raw rows, no file path values, PacketReader remains default until reconciliation. +38. C38: added sidecar/Go-wrapper `list_messages`, clamped to `1..50`, cursor rejected, optional attachment metadata without path/download refs, and copied-DB verification with `include_body=false`; printed evidence is only count/source-table/safety booleans. ## Next business mainline -1. Run Stage C Loop C38: implement sidecar-level bounded `list_messages` plus Go wrapper tests and sanitized verification; do not wire DB rows into MCP yet. +1. Run Stage C Loop C39: design and test a DB-backed receive source adapter around `msglib.ListMessages`, with source selection explicit and PacketReader remaining the default until reconciliation is verified. 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated. 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated. 4. Keep UIA selector/report work as fallback support, not as the primary roadmap. diff --git a/docs/msglib-read-sidecar-contract.md b/docs/msglib-read-sidecar-contract.md index f9c5b7c..bd30d5c 100644 --- a/docs/msglib-read-sidecar-contract.md +++ b/docs/msglib-read-sidecar-contract.md @@ -14,8 +14,8 @@ It exists because C26 proved the database opens with the bundled 32-bit SQLite p - Transport: one JSON request on stdin, one JSON response on stdout. - Process architecture: x86. - Database mode: read-only only. -- Raw rows: not returned by the first contract. -- Message bodies: not returned by the first contract. +- Raw rows: never returned. +- Message bodies: returned only when a future caller explicitly sets `list_messages.include_body=true`; current verification and MCP defaults keep it `false`. - DB mutation: never allowed. Request envelope: @@ -58,7 +58,7 @@ Key response fields: - `process_bits` - `requires_process_bits` - `db_mutation_allowed=false` -- `message_body_reads_allowed=false` +- `message_body_reads_allowed=true` for explicit `list_messages.include_body` opt-in; verification keeps it disabled. ### `schema_summary` @@ -183,6 +183,58 @@ Initial message source candidates: - `received_file_metadata` from `TD_ReceiveFileRecord`. - `recent_conversation_index` from `tblRecent`. +### `list_messages` + +Same safe provider/DB validation as `message_sources`, but returns a bounded normalized message list for future DB-backed receive-message work. C38 keeps this at sidecar/Go-wrapper level only; it is not wired into MCP receive by default. + +Required args: + +- `sqlite_dll_path` +- `db_path` + +Optional args: + +- `password`: default `123`. +- `conversation_id`: optional metadata filter. +- `conversation_type`: `auto`, `direct`, `group`, or `system`; default `auto`. +- `query`: optional metadata filter. Body columns are searched only when `include_body=true`. +- `since`: optional timestamp lower bound using the DB timestamp column representation. +- `cursor`: unsupported initially; any non-empty value is rejected. +- `limit`: bounded to `1..50`; default `20`. +- `include_body`: default `false`. +- `include_attachment_metadata`: default `false`; returns filename/size metadata only, never path values or download refs. + +Returned data adds: + +- `read_only=true` +- `message_body_values_returned`: equals `include_body`. +- `raw_rows_returned=false` +- `file_paths_returned=false` +- `source_tables[]` +- `messages[]` entries with normalized fields: + - `message_id` / `id` + - `conversation_id` + - `conversation_type` + - `sender_id` + - `sender_name` + - `receiver_id` + - `text` / `content_text` (empty when `include_body=false`) + - `content_type` + - `timestamp` / `created_at` + - `subject` + - `read` + - `source=local_readonly` + - `raw_ref=msglib:` + - `attachments[]` with `file_id`, `file_name`, `size_bytes`, and empty `download_ref` +- `next_cursor=null` + +Initial allowlisted message sources: + +- Direct: `tblPersonMsg`. +- Group: `tblMsgGroupPersonMsg`. +- System: `TD_SystemMessageRecord`. +- Attachment metadata: `TD_ReceiveFileRecord`, with `FilePath`/path-like values intentionally excluded. + ## Security and scope rules - The sidecar must open DB files with read-only connection settings. @@ -190,7 +242,8 @@ Initial message source candidates: - The schema/display-source operations return schema, columns, counts, and display-source availability only. - `display_entities` may return bounded contact/group `jid` and `display_name` metadata only. - `message_sources` may return message-source table names, required/present column names, roles, availability flags, and row counts only. -- It does not return message bodies, message text, file path values, attachment contents, or raw row sets. +- `list_messages` may return bounded normalized message metadata. It returns `text`/`content_text` only when `include_body=true`; the standard verification path uses `include_body=false`. +- It does not return file path values, attachment contents, raw row sets, sends, downloads, hooks, injection, or DB writes. - It does not implement send, file download, mark-read, login, hooks, injection, or DB writes. - The Go MCP layer remains the coordinator; this process is a bounded local reader. @@ -232,6 +285,14 @@ Committed C36 implementation: - Verification script calls `message_sources` only when copied DB env paths are supplied, and prints only sanitized `message_source_count` plus message-source summaries containing source/table/role/availability/row-count metadata. - Evidence-backed C36 verification against the copied DB returned six message-source summaries without printing message body values, file path values, raw rows, or copied DB contents. +Committed C38 implementation: + +- Sidecar op: `list_messages`. +- Go wrapper: `ListMessages(ctx, ListMessagesOptions)`. +- The op reads only allowlisted direct/group/system message tables and optional received-file metadata from a copied `MsgLib.db`, clamps `limit` to `1..50`, rejects non-empty cursor, and never returns raw rows or file path values. +- `scripts/verify-msglib-sidecar.ps1` calls `list_messages` only when copied DB env paths are supplied, with `include_body=false`, and prints only `message_list_count`, `message_list_sources`, safety booleans, and omission booleans. +- Evidence-backed C38 smoke returned normalized message count/source-table evidence without printing message text, body values, file paths, download refs, raw rows, sends, downloads, hooks, injection, or DB writes. + ## Verification Default verification builds and checks `self_check` only: @@ -257,3 +318,5 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-en ``` Expected printed evidence is sanitized: contact/group counts, `msglib:` refs, `message_source_count`, message-source table/role/row-count summaries, `receive_display_smoke=true`, populated-field booleans, and no entity values, message bodies, file path values, or raw rows. + +C38 provider verification additionally prints `message_list_count`, `message_list_sources`, `message_list_body_values_returned=false`, `message_list_raw_rows_returned=false`, `message_list_file_paths_returned=false`, `message_list_content_values_omitted=true`, and `message_list_download_refs_omitted=true`. diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index cb319be..1491c4a 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -17,7 +17,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | --- | --- | --- | --- | --- | --- | -| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design and keeps PacketReader default until reconciliation | C38 sidecar bounded DB message listing | +| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through x86 read-only sidecar `list_messages`; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md`; `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design; C38 implements sidecar/Go-wrapper `list_messages` with sanitized copied-DB verification, while PacketReader remains MCP default until reconciliation | C39 DB-backed receive source adapter design/test slice | | `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design | | `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design | | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence | @@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac ## Deferred source work -`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, and C37 defines the bounded DB-backed receive-source design. The next step is sidecar-level `list_messages` with sanitized verification, not MCP integration yet. +`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, C37 defines the bounded DB-backed receive-source design, and C38 implements sidecar/Go-wrapper `list_messages` with sanitized verification. The next step is a DB-backed receive source adapter test slice; do not silently merge MsgLib rows into the MCP default until PacketReader-vs-DB reconciliation and operator source-selection behavior are explicit. diff --git a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md index 8a3117e..38ff6bf 100644 --- a/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md +++ b/docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md @@ -2679,20 +2679,60 @@ Bounded DB-backed receive-source design result: - First evidence-backed copied-DB smoke uses `include_body=false` and prints only counts/source refs/booleans. - Do not return copied-DB `MsgText`, `MessageBody`, `FilePath`, raw rows, sends, downloads, hooks, injection, or DB writes in verification output. -- [ ] **Step 1: Write failing Go wrapper test** +- [x] **Step 1: Write failing Go wrapper test** -Use fake sidecar to assert request args and decoded normalized message shape. +Added `TestListMessagesPassesBoundedArgsAndDecodesNormalizedMessages`. RED failed because `Client.ListMessages` and `ListMessagesOptions` did not exist. -- [ ] **Step 2: Implement sidecar `list_messages`** +- [x] **Step 2: Implement sidecar `list_messages`** -Read allowlisted columns only, clamp limit, reject cursor, hide file paths, and support `include_body=false`. +Implemented Go wrapper `ListMessages(ctx, ListMessagesOptions)` and x86 sidecar op `list_messages`. The wrapper clamps `limit` to `1..50` and rejects non-empty cursor. The sidecar reads allowlisted direct/group/system message tables, uses copied DB read-only mode, returns normalized messages, hides raw rows, and excludes file path/download refs from attachment metadata. -- [ ] **Step 3: Update verification and docs** +- [x] **Step 3: Update verification and docs** -Extend `verify-msglib-sidecar.ps1` with sanitized `list_messages` smoke and run full gate. +Extended `verify-msglib-sidecar.ps1` with sanitized copied-DB `list_messages` smoke using `include_body=false`, then updated status, matrix, contract, and this plan. Full verification gate is required before commit/push. --- +## Loop C38 Result + +Sidecar bounded DB message listing result: + +- Added `list_messages` to `native/MsgLibReadSidecar` and bumped sidecar version to `0.4.0`. +- Added Go wrapper types and `Client.ListMessages(ctx, options)` in `internal/msglib`. +- Fake-sidecar unit coverage verifies bounded args, decoded normalized messages, safety flags, and no body values when `include_body=false`. +- Evidence-backed copied-DB smoke returned only sanitized `message_list_count`, `message_list_sources`, body/raw/file-path booleans, and omission booleans. +- C38 does not wire MsgLib DB rows into the MCP receive default; PacketReader remains the default receive source. + +--- + +## Loop C39: DB-backed receive source adapter test slice + +**Goal:** Add the smallest Go adapter/test layer that can translate `msglib.ListMessages` results into the existing receive-message domain model, while keeping MCP source selection explicit and PacketReader default unchanged. + +**Planned files:** +- Modify or create: `internal/isphere` DB-backed source adapter files/tests, or a narrow `internal/tools` adapter if the existing receive model is tool-local. +- Modify: `internal/msglib` only if C38 return shape exposes a decoding gap. +- Modify docs/status/matrix/plan after implementation. + +**Planned behavior:** +- Use an injected/fake MsgLib list provider first; do not require real DB in unit tests. +- Map normalized MsgLib fields to the current `isphere_receive_messages` envelope without returning raw rows or file paths. +- Keep source selection explicit. Do not silently merge MsgLib rows into default MCP receive output. +- Verification should continue to clear MsgLib env in the standard `verify-go-mcp.ps1`; optional copied-DB verification remains separate. + +- [ ] **Step 1: Locate current receive-message model boundary** + +Inspect `internal/isphere` and `internal/tools` receive-message code and decide whether the adapter belongs below the tool layer or inside the tool registration layer. + +- [ ] **Step 2: Write failing adapter test** + +Use a fake MsgLib list provider returning redacted normalized messages and assert mapped receive-message fields, source refs, attachment metadata behavior, and explicit source-selection handling. + +- [ ] **Step 3: Implement minimal adapter and update docs** + +Implement only enough code to pass the adapter test; defer MCP default wiring until reconciliation/source-selection behavior is proven. + +--- ## Self-Review - Spec coverage: the plan follows the user's cyclic requirement: plan first, implement one loop at a time, update the next loop after each result, and ask only on blockers. diff --git a/internal/msglib/sidecar_client.go b/internal/msglib/sidecar_client.go index e20a9f6..eb5bab0 100644 --- a/internal/msglib/sidecar_client.go +++ b/internal/msglib/sidecar_client.go @@ -22,6 +22,9 @@ const ( DefaultDisplayEntityLimit = 25 MaxDisplayEntityLimit = 100 + + DefaultListMessagesLimit = 20 + MaxListMessagesLimit = 50 ) var requestCounter uint64 @@ -97,6 +100,54 @@ type MessageSource struct { RowCount string `json:"row_count"` } +type ListMessagesOptions struct { + ConversationID string + ConversationType string + Query string + Since string + Cursor string + Limit int + IncludeBody bool + IncludeAttachmentMetadata bool +} + +type ListMessagesResult struct { + ReadOnly bool `json:"read_only"` + MessageBodyValuesReturned bool `json:"message_body_values_returned"` + RawRowsReturned bool `json:"raw_rows_returned"` + FilePathsReturned bool `json:"file_paths_returned"` + SourceTables []string `json:"source_tables"` + Messages []ListMessage `json:"messages"` + NextCursor string `json:"next_cursor"` +} + +type ListMessage struct { + MessageID string `json:"message_id"` + ID string `json:"id"` + ConversationID string `json:"conversation_id"` + ConversationType string `json:"conversation_type"` + SenderID string `json:"sender_id"` + SenderName string `json:"sender_name"` + ReceiverID string `json:"receiver_id"` + Text string `json:"text"` + ContentText string `json:"content_text"` + ContentType string `json:"content_type"` + Timestamp string `json:"timestamp"` + CreatedAt string `json:"created_at"` + Subject string `json:"subject"` + Read bool `json:"read"` + Source string `json:"source"` + RawRef string `json:"raw_ref"` + Attachments []ListMessageAttachment `json:"attachments"` +} + +type ListMessageAttachment struct { + FileID string `json:"file_id"` + FileName string `json:"file_name"` + SizeBytes int64 `json:"size_bytes"` + DownloadRef string `json:"download_ref"` +} + type DisplayEntitiesOptions struct { EntityType string Query string @@ -162,6 +213,46 @@ func (c *Client) MessageSources(ctx context.Context) ([]MessageSource, error) { return out.MessageSources, nil } +func (c *Client) ListMessages(ctx context.Context, opts ListMessagesOptions) (ListMessagesResult, error) { + if strings.TrimSpace(opts.Cursor) != "" { + return ListMessagesResult{}, errors.New("msglib list_messages cursor pagination is not available yet") + } + limit := opts.Limit + if limit <= 0 { + limit = DefaultListMessagesLimit + } + if limit > MaxListMessagesLimit { + limit = MaxListMessagesLimit + } + + args := map[string]any{ + "sqlite_dll_path": c.config.SQLiteDLLPath, + "db_path": c.config.DBPath, + "password": c.config.Password, + "limit": limit, + "include_body": opts.IncludeBody, + "include_attachment_metadata": opts.IncludeAttachmentMetadata, + } + if strings.TrimSpace(opts.ConversationID) != "" { + args["conversation_id"] = strings.TrimSpace(opts.ConversationID) + } + if strings.TrimSpace(opts.ConversationType) != "" { + args["conversation_type"] = strings.TrimSpace(opts.ConversationType) + } + if strings.TrimSpace(opts.Query) != "" { + args["query"] = strings.TrimSpace(opts.Query) + } + if strings.TrimSpace(opts.Since) != "" { + args["since"] = strings.TrimSpace(opts.Since) + } + + var out ListMessagesResult + if err := c.call(ctx, "list_messages", args, &out); err != nil { + return ListMessagesResult{}, err + } + return out, nil +} + func (c *Client) DisplayEntities(ctx context.Context, opts DisplayEntitiesOptions) ([]DisplayEntity, error) { entityType := strings.TrimSpace(opts.EntityType) if entityType != EntityTypeContacts && entityType != EntityTypeGroups { diff --git a/internal/msglib/sidecar_client_test.go b/internal/msglib/sidecar_client_test.go index dad7972..379043c 100644 --- a/internal/msglib/sidecar_client_test.go +++ b/internal/msglib/sidecar_client_test.go @@ -81,6 +81,46 @@ func TestMessageSourcesPassesConfiguredPathsAndDecodesMetadata(t *testing.T) { } } +func TestListMessagesPassesBoundedArgsAndDecodesNormalizedMessages(t *testing.T) { + cfg := Config{ + SQLiteDLLPath: `C:\fixture\System.Data.SQLite.dll`, + DBPath: `C:\fixture\MsgLib.db`, + Password: "123", + } + client := newFakeClient(t, "list_messages_ok", cfg) + + got, err := client.ListMessages(context.Background(), ListMessagesOptions{ + ConversationType: "direct", + Query: "redacted", + Since: "2026-07-10T00:00:00Z", + Limit: 99, + IncludeBody: false, + IncludeAttachmentMetadata: true, + }) + if err != nil { + t.Fatalf("ListMessages returned error: %v", err) + } + if !got.ReadOnly || got.MessageBodyValuesReturned || got.RawRowsReturned || got.FilePathsReturned { + t.Fatalf("unexpected safety flags: %+v", got) + } + if strings.Join(got.SourceTables, ",") != "tblPersonMsg" { + t.Fatalf("SourceTables = %#v", got.SourceTables) + } + if len(got.Messages) != 1 { + t.Fatalf("len(Messages) = %d, want 1", len(got.Messages)) + } + msg := got.Messages[0] + if msg.MessageID != "msg-redacted-1" || msg.ConversationType != "direct" || msg.RawRef != "msglib:tblPersonMsg" { + t.Fatalf("unexpected message identity: %+v", msg) + } + if msg.ContentText != "" || msg.Text != "" { + t.Fatalf("content should be omitted when include_body=false: %+v", msg) + } + if len(msg.Attachments) != 1 || msg.Attachments[0].FileName != "redacted.docx" || msg.Attachments[0].DownloadRef != "" { + t.Fatalf("unexpected attachments: %+v", msg.Attachments) + } +} + func TestDisplayEntitiesPassesBoundedArgsAndDecodesMetadata(t *testing.T) { cfg := Config{ SQLiteDLLPath: `C:\fixture\System.Data.SQLite.dll`, @@ -307,6 +347,49 @@ func fakeSidecarMain() int { "error": nil, "warnings": []any{}, }) + case "list_messages": + args, _ := req["args"].(map[string]any) + if args["sqlite_dll_path"] != `C:\fixture\System.Data.SQLite.dll` || args["db_path"] != `C:\fixture\MsgLib.db` || args["password"] != "123" || args["conversation_type"] != "direct" || args["query"] != "redacted" || args["since"] != "2026-07-10T00:00:00Z" || args["limit"] != float64(50) || args["include_body"] != false || args["include_attachment_metadata"] != true { + fmt.Fprintf(os.Stderr, "bad list_messages args: %#v\n", args) + return 2 + } + return writeFakeResponse(map[string]any{ + "protocol": Protocol, + "request_id": requestID, + "op": op, + "ok": true, + "data": map[string]any{ + "read_only": true, + "message_body_values_returned": false, + "raw_rows_returned": false, + "file_paths_returned": false, + "source_tables": []any{"tblPersonMsg"}, + "messages": []any{ + map[string]any{ + "message_id": "msg-redacted-1", + "id": "msg-redacted-1", + "conversation_id": "contact-redacted", + "conversation_type": "direct", + "sender_id": "sender-redacted", + "sender_name": "Sender Redacted", + "receiver_id": "receiver-redacted", + "text": "", + "content_text": "", + "content_type": "file", + "timestamp": "2026-07-10T00:01:02Z", + "created_at": "2026-07-10T00:01:02Z", + "read": true, + "source": "local_readonly", + "raw_ref": "msglib:tblPersonMsg", + "attachments": []any{ + map[string]any{"file_id": "msg-redacted-1:redacted.docx", "file_name": "redacted.docx", "size_bytes": float64(1234), "download_ref": ""}, + }, + }, + }, + }, + "error": nil, + "warnings": []any{}, + }) case "display_entities": args, _ := req["args"].(map[string]any) if args["sqlite_dll_path"] != `C:\fixture\System.Data.SQLite.dll` || args["db_path"] != `C:\fixture\MsgLib.db` || args["password"] != "123" || args["entity_type"] != "contacts" || args["query"] != "alice" || args["limit"] != float64(2) { diff --git a/native/MsgLibReadSidecar/Program.cs b/native/MsgLibReadSidecar/Program.cs index f50e98b..3da59c9 100644 --- a/native/MsgLibReadSidecar/Program.cs +++ b/native/MsgLibReadSidecar/Program.cs @@ -8,7 +8,7 @@ namespace MsgLibReadSidecar { internal static class Program { - private const string Version = "0.3.0"; + private const string Version = "0.4.0"; private const string DefaultPassword = "123"; private static readonly HashSet DefaultTargetTables = new HashSet(StringComparer.OrdinalIgnoreCase) @@ -49,6 +49,9 @@ namespace MsgLibReadSidecar case "message_sources": response = SidecarProtocol.Success(requestId, op, MessageSources(opArgs)); break; + case "list_messages": + response = SidecarProtocol.Success(requestId, op, ListMessages(opArgs)); + break; case "display_entities": response = SidecarProtocol.Success(requestId, op, DisplayEntities(opArgs)); break; @@ -80,7 +83,7 @@ namespace MsgLibReadSidecar { "read_only", true }, { "mutates_db", false }, { "db_mutation_allowed", false }, - { "message_body_reads_allowed", false } + { "message_body_reads_allowed", true } }; } @@ -160,6 +163,89 @@ namespace MsgLibReadSidecar return result; } + private static Dictionary ListMessages(Dictionary args) + { + string sqliteDllPath = SidecarProtocol.GetString(args, "sqlite_dll_path", ""); + string dbPath = SidecarProtocol.GetString(args, "db_path", ""); + string password = SidecarProtocol.GetString(args, "password", DefaultPassword); + string conversationId = SidecarProtocol.GetString(args, "conversation_id", "").Trim(); + string conversationType = SidecarProtocol.GetString(args, "conversation_type", "auto").Trim().ToLowerInvariant(); + string query = SidecarProtocol.GetString(args, "query", "").Trim(); + string since = SidecarProtocol.GetString(args, "since", "").Trim(); + string cursor = SidecarProtocol.GetString(args, "cursor", "").Trim(); + int limit = Clamp(SidecarProtocol.GetInt(args, "limit", 20), 1, 50); + bool includeBody = SidecarProtocol.GetBool(args, "include_body", false); + bool includeAttachmentMetadata = SidecarProtocol.GetBool(args, "include_attachment_metadata", false); + + if (!string.IsNullOrWhiteSpace(cursor)) + { + throw new ArgumentException("list_messages cursor pagination is not available yet"); + } + if (string.IsNullOrWhiteSpace(conversationType)) conversationType = "auto"; + if (!string.Equals(conversationType, "auto", StringComparison.OrdinalIgnoreCase) && + !string.Equals(conversationType, "direct", StringComparison.OrdinalIgnoreCase) && + !string.Equals(conversationType, "group", StringComparison.OrdinalIgnoreCase) && + !string.Equals(conversationType, "system", StringComparison.OrdinalIgnoreCase)) + { + throw new ArgumentException("conversation_type must be auto, direct, group, or system"); + } + + ValidateInputs(sqliteDllPath, dbPath); + + var result = new Dictionary + { + { "metadata_only", !includeBody }, + { "read_only", true }, + { "message_body_values_returned", includeBody }, + { "raw_rows_returned", false }, + { "file_paths_returned", false }, + { "limit", limit }, + { "provider", "System.Data.SQLite" }, + { "process_bits", IntPtr.Size * 8 }, + { "next_cursor", null } + }; + + using (DbConnection connection = OpenSqliteConnection(sqliteDllPath, dbPath, password)) + { + connection.Open(); + TryExecuteNonQuery(connection, "PRAGMA query_only=ON"); + List> tables = ReadMasterTables(connection, 256); + HashSet listTables = BuildListMessagesTableSet(); + List> columns = ReadColumns(connection, tables, listTables); + Dictionary> columnMap = BuildColumnMap(columns); + var sourceTables = new HashSet(StringComparer.OrdinalIgnoreCase); + var messages = new List>(); + + if (ShouldReadMessageRole(conversationType, "direct")) + { + ReadDirectMessages(connection, columnMap, conversationId, query, since, limit, includeBody, includeAttachmentMetadata, messages, sourceTables); + } + if (ShouldReadMessageRole(conversationType, "group")) + { + ReadGroupMessages(connection, columnMap, conversationId, query, since, limit, includeBody, includeAttachmentMetadata, messages, sourceTables); + } + if (ShouldReadMessageRole(conversationType, "system")) + { + ReadSystemMessages(connection, columnMap, conversationId, query, since, limit, includeBody, includeAttachmentMetadata, messages, sourceTables); + } + + messages.Sort(CompareMessagesDescending); + while (messages.Count > limit) + { + messages.RemoveAt(messages.Count - 1); + } + foreach (Dictionary message in messages) + { + message.Remove("sort_key"); + } + + result["source_tables"] = ToSortedArray(sourceTables); + result["messages"] = messages; + } + + return result; + } + private static Dictionary DisplayEntities(Dictionary args) { @@ -476,6 +562,511 @@ namespace MsgLibReadSidecar }); } + private static HashSet BuildListMessagesTableSet() + { + return new HashSet(StringComparer.OrdinalIgnoreCase) + { + "tblPersonMsg", + "tblMsgGroupPersonMsg", + "TD_SystemMessageRecord", + "TD_ReceiveFileRecord" + }; + } + + private static bool ShouldReadMessageRole(string requested, string role) + { + return string.Equals(requested, "auto", StringComparison.OrdinalIgnoreCase) || + string.Equals(requested, role, StringComparison.OrdinalIgnoreCase); + } + + private static void ReadDirectMessages( + DbConnection connection, + Dictionary> columnMap, + string conversationId, + string query, + string since, + int limit, + bool includeBody, + bool includeAttachmentMetadata, + List> messages, + HashSet sourceTables) + { + const string table = "tblPersonMsg"; + if (!HasColumns(columnMap, table, new string[] { "Id" })) return; + + string senderIdColumn = FirstExistingColumn(columnMap, table, new string[] { "SndPersonId", "SenderId" }); + string senderNameColumn = FirstExistingColumn(columnMap, table, new string[] { "SndPerson", "SenderName" }); + string receiverIdColumn = FirstExistingColumn(columnMap, table, new string[] { "RecvPersonId", "ReceiverId" }); + string receiverNameColumn = FirstExistingColumn(columnMap, table, new string[] { "RecvPerson", "ReceiverName" }); + string objectIdColumn = FirstExistingColumn(columnMap, table, new string[] { "ObjPersonId", "PersonId" }); + string objectNameColumn = FirstExistingColumn(columnMap, table, new string[] { "ObjPerson", "PersonName" }); + string timeColumn = FirstExistingColumn(columnMap, table, new string[] { "ActionTime", "MsgTime", "CreateTime", "SendTime" }); + string readColumn = FirstExistingColumn(columnMap, table, new string[] { "IsRead", "ReadState", "MsgReadState" }); + + using (DbCommand command = connection.CreateCommand()) + { + var where = new List(); + where.Add("coalesce(" + QuoteIdent("Id") + ", '') <> ''"); + AddListMessageWhere(command, where, table, columnMap, conversationId, query, since, timeColumn, + new string[] { senderIdColumn, receiverIdColumn, objectIdColumn }, + new string[] { "Id", senderIdColumn, senderNameColumn, receiverIdColumn, receiverNameColumn, objectIdColumn, objectNameColumn }, + includeBody); + + command.CommandText = + "select " + + QuoteIdent("Id") + " as message_id, " + + SelectColumn(senderIdColumn, "sender_id") + ", " + + SelectColumn(FirstNonEmpty(senderNameColumn, objectNameColumn), "sender_name") + ", " + + SelectColumn(receiverIdColumn, "receiver_id") + ", " + + SelectColumn(FirstNonEmpty(objectIdColumn, receiverIdColumn, senderIdColumn), "conversation_id") + ", " + + SelectColumn(timeColumn, "timestamp") + ", " + + SelectColumn(readColumn, "read_state") + ", " + + BodySelect(columnMap, table, includeBody) + + " from " + QuoteIdent(table) + + " where " + string.Join(" and ", where.ToArray()) + + " order by " + OrderByExpression(timeColumn) + " desc limit " + Convert.ToString(limit); + + using (DbDataReader reader = command.ExecuteReader()) + { + while (reader.Read() && messages.Count < limit * 3) + { + string messageId = ReaderString(reader, "message_id"); + string timestamp = ReaderString(reader, "timestamp"); + string contentText = includeBody ? ReaderString(reader, "content_text") : ""; + List> attachments = includeAttachmentMetadata ? ReadAttachmentsForMessage(connection, columnMap, messageId) : new List>(); + messages.Add(new Dictionary + { + { "message_id", messageId }, + { "id", messageId }, + { "conversation_id", ReaderString(reader, "conversation_id") }, + { "conversation_type", "direct" }, + { "sender_id", ReaderString(reader, "sender_id") }, + { "sender_name", ReaderString(reader, "sender_name") }, + { "receiver_id", ReaderString(reader, "receiver_id") }, + { "text", contentText }, + { "content_text", contentText }, + { "content_type", ContentType(contentText, attachments) }, + { "timestamp", timestamp }, + { "created_at", timestamp }, + { "subject", "" }, + { "read", ReaderBool(reader, "read_state") }, + { "source", "local_readonly" }, + { "raw_ref", "msglib:" + table }, + { "attachments", attachments }, + { "sort_key", timestamp + "|" + messageId } + }); + } + } + } + sourceTables.Add(table); + } + + private static void ReadGroupMessages( + DbConnection connection, + Dictionary> columnMap, + string conversationId, + string query, + string since, + int limit, + bool includeBody, + bool includeAttachmentMetadata, + List> messages, + HashSet sourceTables) + { + const string table = "tblMsgGroupPersonMsg"; + if (!HasColumns(columnMap, table, new string[] { "Id" })) return; + + string senderIdColumn = FirstExistingColumn(columnMap, table, new string[] { "SndPersonId", "SessionPersonId", "PersonId" }); + string senderNameColumn = FirstExistingColumn(columnMap, table, new string[] { "SndPerson", "SessionPersonName", "PersonName" }); + string groupIdColumn = FirstExistingColumn(columnMap, table, new string[] { "MsgGroupId", "GroupJID", "GroupId" }); + string groupNameColumn = FirstExistingColumn(columnMap, table, new string[] { "MsgGroupName", "GroupName", "SessionPersonName" }); + string timeColumn = FirstExistingColumn(columnMap, table, new string[] { "MsgTime", "ActionTime", "CreateTime", "SendTime" }); + string readColumn = FirstExistingColumn(columnMap, table, new string[] { "IsRead", "ReadState", "MsgReadState" }); + + using (DbCommand command = connection.CreateCommand()) + { + var where = new List(); + where.Add("coalesce(" + QuoteIdent("Id") + ", '') <> ''"); + AddListMessageWhere(command, where, table, columnMap, conversationId, query, since, timeColumn, + new string[] { groupIdColumn, senderIdColumn }, + new string[] { "Id", senderIdColumn, senderNameColumn, groupIdColumn, groupNameColumn }, + includeBody); + + command.CommandText = + "select " + + QuoteIdent("Id") + " as message_id, " + + SelectColumn(senderIdColumn, "sender_id") + ", " + + SelectColumn(senderNameColumn, "sender_name") + ", " + + SelectColumn(groupIdColumn, "conversation_id") + ", " + + SelectColumn(timeColumn, "timestamp") + ", " + + SelectColumn(readColumn, "read_state") + ", " + + BodySelect(columnMap, table, includeBody) + + " from " + QuoteIdent(table) + + " where " + string.Join(" and ", where.ToArray()) + + " order by " + OrderByExpression(timeColumn) + " desc limit " + Convert.ToString(limit); + + using (DbDataReader reader = command.ExecuteReader()) + { + while (reader.Read() && messages.Count < limit * 3) + { + string messageId = ReaderString(reader, "message_id"); + string timestamp = ReaderString(reader, "timestamp"); + string contentText = includeBody ? ReaderString(reader, "content_text") : ""; + List> attachments = includeAttachmentMetadata ? ReadAttachmentsForMessage(connection, columnMap, messageId) : new List>(); + messages.Add(new Dictionary + { + { "message_id", messageId }, + { "id", messageId }, + { "conversation_id", ReaderString(reader, "conversation_id") }, + { "conversation_type", "group" }, + { "sender_id", ReaderString(reader, "sender_id") }, + { "sender_name", ReaderString(reader, "sender_name") }, + { "receiver_id", "" }, + { "text", contentText }, + { "content_text", contentText }, + { "content_type", ContentType(contentText, attachments) }, + { "timestamp", timestamp }, + { "created_at", timestamp }, + { "subject", "" }, + { "read", ReaderBool(reader, "read_state") }, + { "source", "local_readonly" }, + { "raw_ref", "msglib:" + table }, + { "attachments", attachments }, + { "sort_key", timestamp + "|" + messageId } + }); + } + } + } + sourceTables.Add(table); + } + + private static void ReadSystemMessages( + DbConnection connection, + Dictionary> columnMap, + string conversationId, + string query, + string since, + int limit, + bool includeBody, + bool includeAttachmentMetadata, + List> messages, + HashSet sourceTables) + { + const string table = "TD_SystemMessageRecord"; + string idColumn = FirstExistingColumn(columnMap, table, new string[] { "MsgGuid", "Id" }); + if (string.IsNullOrWhiteSpace(idColumn)) return; + + string senderIdColumn = FirstExistingColumn(columnMap, table, new string[] { "SendPersonJid", "PersonID", "SourceID" }); + string senderNameColumn = FirstExistingColumn(columnMap, table, new string[] { "SendPersonName", "PersonName", "SourceName" }); + string titleColumn = FirstExistingColumn(columnMap, table, new string[] { "MsgTitle", "Title" }); + string timeColumn = FirstExistingColumn(columnMap, table, new string[] { "MsgTime", "ActionTime", "CreateTime", "SendTime" }); + string readColumn = FirstExistingColumn(columnMap, table, new string[] { "MsgReadState", "IsRead", "ReadState" }); + + using (DbCommand command = connection.CreateCommand()) + { + var where = new List(); + where.Add("coalesce(" + QuoteIdent(idColumn) + ", '') <> ''"); + AddListMessageWhere(command, where, table, columnMap, conversationId, query, since, timeColumn, + new string[] { senderIdColumn }, + new string[] { idColumn, senderIdColumn, senderNameColumn, titleColumn }, + includeBody); + + command.CommandText = + "select " + + QuoteIdent(idColumn) + " as message_id, " + + SelectColumn(senderIdColumn, "sender_id") + ", " + + SelectColumn(senderNameColumn, "sender_name") + ", " + + SelectColumn(titleColumn, "subject") + ", " + + SelectColumn(timeColumn, "timestamp") + ", " + + SelectColumn(readColumn, "read_state") + ", " + + BodySelect(columnMap, table, includeBody) + + " from " + QuoteIdent(table) + + " where " + string.Join(" and ", where.ToArray()) + + " order by " + OrderByExpression(timeColumn) + " desc limit " + Convert.ToString(limit); + + using (DbDataReader reader = command.ExecuteReader()) + { + while (reader.Read() && messages.Count < limit * 3) + { + string messageId = ReaderString(reader, "message_id"); + string timestamp = ReaderString(reader, "timestamp"); + string contentText = includeBody ? ReaderString(reader, "content_text") : ""; + List> attachments = includeAttachmentMetadata ? ReadAttachmentsForMessage(connection, columnMap, messageId) : new List>(); + messages.Add(new Dictionary + { + { "message_id", messageId }, + { "id", messageId }, + { "conversation_id", ReaderString(reader, "sender_id") }, + { "conversation_type", "system" }, + { "sender_id", ReaderString(reader, "sender_id") }, + { "sender_name", ReaderString(reader, "sender_name") }, + { "receiver_id", "" }, + { "text", contentText }, + { "content_text", contentText }, + { "content_type", ContentType(contentText, attachments) }, + { "timestamp", timestamp }, + { "created_at", timestamp }, + { "subject", ReaderString(reader, "subject") }, + { "read", ReaderBool(reader, "read_state") }, + { "source", "local_readonly" }, + { "raw_ref", "msglib:" + table }, + { "attachments", attachments }, + { "sort_key", timestamp + "|" + messageId } + }); + } + } + } + sourceTables.Add(table); + } + + private static void AddListMessageWhere( + DbCommand command, + List where, + string table, + Dictionary> columnMap, + string conversationId, + string query, + string since, + string timeColumn, + string[] conversationColumns, + string[] queryColumns, + bool includeBody) + { + if (!string.IsNullOrWhiteSpace(conversationId)) + { + var parts = new List(); + foreach (string column in conversationColumns) + { + if (!string.IsNullOrWhiteSpace(column) && HasColumns(columnMap, table, new string[] { column })) + { + parts.Add(QuoteIdent(column) + " = @conversation_id"); + } + } + if (parts.Count > 0) + { + where.Add("(" + string.Join(" or ", parts.ToArray()) + ")"); + AddParameter(command, "@conversation_id", conversationId); + } + } + + if (!string.IsNullOrWhiteSpace(query)) + { + var parts = new List(); + foreach (string column in queryColumns) + { + if (!string.IsNullOrWhiteSpace(column) && HasColumns(columnMap, table, new string[] { column })) + { + parts.Add(QuoteIdent(column) + " like @query"); + } + } + if (includeBody) + { + if (HasColumns(columnMap, table, new string[] { "MsgText" })) parts.Add(QuoteIdent("MsgText") + " like @query"); + if (HasColumns(columnMap, table, new string[] { "MessageBody" })) parts.Add(QuoteIdent("MessageBody") + " like @query"); + } + if (parts.Count > 0) + { + where.Add("(" + string.Join(" or ", parts.ToArray()) + ")"); + AddParameter(command, "@query", "%" + query + "%"); + } + } + + if (!string.IsNullOrWhiteSpace(since) && !string.IsNullOrWhiteSpace(timeColumn) && HasColumns(columnMap, table, new string[] { timeColumn })) + { + where.Add(QuoteIdent(timeColumn) + " >= @since"); + AddParameter(command, "@since", since); + } + } + + private static string BodySelect(Dictionary> columnMap, string table, bool includeBody) + { + if (!includeBody) + { + return "'' as content_text"; + } + + var expressions = new List(); + if (HasColumns(columnMap, table, new string[] { "MsgText" })) expressions.Add(QuoteIdent("MsgText")); + if (HasColumns(columnMap, table, new string[] { "MessageBody" })) expressions.Add(QuoteIdent("MessageBody")); + if (expressions.Count == 0) + { + return "'' as content_text"; + } + expressions.Add("''"); + return "coalesce(" + string.Join(", ", expressions.ToArray()) + ") as content_text"; + } + + private static List> ReadAttachmentsForMessage(DbConnection connection, Dictionary> columnMap, string messageId) + { + var attachments = new List>(); + const string table = "TD_ReceiveFileRecord"; + if (string.IsNullOrWhiteSpace(messageId)) return attachments; + if (!HasColumns(columnMap, table, new string[] { "ReceiveMsgGuid", "FileName" })) return attachments; + + string sizeColumn = FirstExistingColumn(columnMap, table, new string[] { "FileSize", "Size" }); + string sourceColumn = FirstExistingColumn(columnMap, table, new string[] { "SourceID", "FileGuid", "FileId" }); + using (DbCommand command = connection.CreateCommand()) + { + command.CommandText = + "select " + + SelectColumn(sourceColumn, "source_id") + ", " + + SelectColumn("FileName", "file_name") + ", " + + SelectColumn(sizeColumn, "size_bytes") + + " from " + QuoteIdent(table) + + " where " + QuoteIdent("ReceiveMsgGuid") + " = @message_id limit 20"; + AddParameter(command, "@message_id", messageId); + + using (DbDataReader reader = command.ExecuteReader()) + { + while (reader.Read()) + { + string fileName = SafeFileName(ReaderString(reader, "file_name")); + if (string.IsNullOrWhiteSpace(fileName)) continue; + string sourceId = ReaderString(reader, "source_id"); + attachments.Add(new Dictionary + { + { "file_id", FirstNonEmpty(sourceId, messageId + ":" + fileName) }, + { "file_name", fileName }, + { "size_bytes", ReaderInt64(reader, "size_bytes") }, + { "download_ref", "" } + }); + } + } + } + return attachments; + } + + private static bool HasColumns(Dictionary> columnMap, string table, string[] columns) + { + if (!columnMap.ContainsKey(table)) return false; + foreach (string column in columns) + { + if (string.IsNullOrWhiteSpace(column)) return false; + if (!columnMap[table].Contains(column)) return false; + } + return true; + } + + private static string FirstExistingColumn(Dictionary> columnMap, string table, string[] columns) + { + if (!columnMap.ContainsKey(table)) return ""; + foreach (string column in columns) + { + if (!string.IsNullOrWhiteSpace(column) && columnMap[table].Contains(column)) + { + return column; + } + } + return ""; + } + + private static string SelectColumn(string column, string alias) + { + if (string.IsNullOrWhiteSpace(column)) + { + return "'' as " + QuoteIdent(alias); + } + return QuoteIdent(column) + " as " + QuoteIdent(alias); + } + + private static string OrderByExpression(string column) + { + if (string.IsNullOrWhiteSpace(column)) + { + return "1"; + } + return QuoteIdent(column); + } + + private static string ReaderString(DbDataReader reader, string name) + { + try + { + object value = reader[name]; + if (value == null || value == DBNull.Value) return ""; + return Convert.ToString(value); + } + catch + { + return ""; + } + } + + private static bool ReaderBool(DbDataReader reader, string name) + { + string value = ReaderString(reader, name).Trim(); + if (string.IsNullOrWhiteSpace(value)) return false; + return string.Equals(value, "1", StringComparison.OrdinalIgnoreCase) || + string.Equals(value, "true", StringComparison.OrdinalIgnoreCase) || + string.Equals(value, "read", StringComparison.OrdinalIgnoreCase) || + string.Equals(value, "yes", StringComparison.OrdinalIgnoreCase); + } + + private static long ReaderInt64(DbDataReader reader, string name) + { + string value = ReaderString(reader, name).Trim(); + long parsed; + if (long.TryParse(value, out parsed)) return parsed; + return 0; + } + + private static string FirstNonEmpty(params string[] values) + { + foreach (string value in values) + { + if (!string.IsNullOrWhiteSpace(value)) return value; + } + return ""; + } + + private static string ContentType(string contentText, List> attachments) + { + if (attachments != null && attachments.Count > 0) return "file"; + if (string.IsNullOrWhiteSpace(contentText)) return "unknown"; + return "text"; + } + + private static int CompareMessagesDescending(Dictionary left, Dictionary right) + { + string leftKey = left.ContainsKey("sort_key") ? Convert.ToString(left["sort_key"]) : ""; + string rightKey = right.ContainsKey("sort_key") ? Convert.ToString(right["sort_key"]) : ""; + return string.Compare(rightKey, leftKey, StringComparison.Ordinal); + } + + private static string[] ToSortedArray(HashSet values) + { + var list = new List(); + foreach (string value in values) + { + list.Add(value); + } + list.Sort(StringComparer.OrdinalIgnoreCase); + return list.ToArray(); + } + + private static void AddParameter(DbCommand command, string name, object value) + { + DbParameter parameter = command.CreateParameter(); + parameter.ParameterName = name; + parameter.Value = value ?? DBNull.Value; + command.Parameters.Add(parameter); + } + + private static string SafeFileName(string value) + { + if (string.IsNullOrWhiteSpace(value)) return ""; + try + { + return Path.GetFileName(value); + } + catch + { + int slash = Math.Max(value.LastIndexOf('/'), value.LastIndexOf('\\')); + if (slash >= 0 && slash + 1 < value.Length) return value.Substring(slash + 1); + return value; + } + } + private sealed class DisplayEntitySpec { diff --git a/scripts/verify-msglib-sidecar.ps1 b/scripts/verify-msglib-sidecar.ps1 index 19fd88c..c42147f 100644 --- a/scripts/verify-msglib-sidecar.ps1 +++ b/scripts/verify-msglib-sidecar.ps1 @@ -49,6 +49,10 @@ $displaySources = $null $displayEntitySummaries = @() $messageSources = $null $messageSourceSummaries = @() +$messageList = $null +$messageListSources = @() +$messageListContentTextNonEmptyCount = 0 +$messageListDownloadRefNonEmptyCount = 0 if ($SQLiteDllPath -and $DbPath) { if (-not (Test-Path -LiteralPath $SQLiteDllPath)) { throw "SQLite DLL not found: $SQLiteDllPath" @@ -95,6 +99,31 @@ if ($SQLiteDllPath -and $DbPath) { } } + $messageList = Invoke-SidecarJson @{ + protocol = "isphere.msglib.v1" + request_id = "verify-list-messages" + op = "list_messages" + args = @{ + sqlite_dll_path = $SQLiteDllPath + db_path = $DbPath + password = "123" + conversation_type = "auto" + limit = 5 + include_body = $false + include_attachment_metadata = $true + } + } + if (-not $messageList.ok -or -not $messageList.data.read_only -or $messageList.data.message_body_values_returned -or $messageList.data.raw_rows_returned -or $messageList.data.file_paths_returned) { + throw "list_messages failed" + } + $messageListRows = @($messageList.data.messages) + $messageListSources = @($messageList.data.source_tables) + $messageListContentTextNonEmptyCount = @($messageListRows | Where-Object { $_.content_text -or $_.text }).Count + $messageListDownloadRefNonEmptyCount = @($messageListRows | ForEach-Object { @($_.attachments) } | Where-Object { $_.download_ref }).Count + if ($messageListContentTextNonEmptyCount -ne 0 -or $messageListDownloadRefNonEmptyCount -ne 0) { + throw "list_messages returned body text or download refs while include_body=false" + } + foreach ($entityType in @("contacts", "groups")) { $displayEntities = Invoke-SidecarJson @{ protocol = "isphere.msglib.v1" @@ -131,6 +160,13 @@ if ($SQLiteDllPath -and $DbPath) { display_entity_summaries = $displayEntitySummaries message_source_count = if ($messageSources) { $messageSources.data.message_sources.Count } else { 0 } message_source_summaries = $messageSourceSummaries + message_list_count = if ($messageList) { @($messageList.data.messages).Count } else { 0 } + message_list_sources = $messageListSources + message_list_body_values_returned = if ($messageList) { [bool]$messageList.data.message_body_values_returned } else { $false } + message_list_raw_rows_returned = if ($messageList) { [bool]$messageList.data.raw_rows_returned } else { $false } + message_list_file_paths_returned = if ($messageList) { [bool]$messageList.data.file_paths_returned } else { $false } + message_list_content_values_omitted = ($messageListContentTextNonEmptyCount -eq 0) + message_list_download_refs_omitted = ($messageListDownloadRefNonEmptyCount -eq 0) message_body_values_returned = $false raw_rows_returned = $false file_paths_returned = $false