feat: add msglib message source metadata

This commit is contained in:
zhaoyilun
2026-07-10 08:36:34 +08:00
parent 1a21a01184
commit c8ecf5b36e
8 changed files with 380 additions and 15 deletions

View File

@@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme
## Current loop ## Current loop
Current loop: `Stage C / Loop C36 - MsgLib message-source metadata sidecar`. Current loop: `Stage C / Loop C37 - bounded DB-backed receive-source design`.
Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`. Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.
@@ -104,10 +104,11 @@ Current loop output so far:
33. C33: optional real copied-DB MCP smoke now also constructs a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verifies receive-message display enrichment through MCP; output stays sanitized to counts, booleans, and `msglib:<source_table>` refs only. 33. C33: optional real copied-DB MCP smoke now also constructs a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verifies receive-message display enrichment through MCP; output stays sanitized to counts, booleans, and `msglib:<source_table>` refs only.
34. C34: `docs/go-mcp-runbook.md` now documents the operator setup path for optional MsgLib display-name enrichment, including the required env vars, partial-config failure rule, standard no-MsgLib verification, optional provider smoke, optional MCP enrichment smoke, and sanitized output boundaries. 34. C34: `docs/go-mcp-runbook.md` now documents the operator setup path for optional MsgLib display-name enrichment, including the required env vars, partial-config failure rule, standard no-MsgLib verification, optional provider smoke, optional MCP enrichment smoke, and sanitized output boundaries.
35. C35: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` maps `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, recent, and file-record schema columns to the future `isphere_receive_messages` contract; decision is metadata-only `message_sources` sidecar first, not body-returning DB listing yet. 35. C35: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` maps `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, recent, and file-record schema columns to the future `isphere_receive_messages` contract; decision is metadata-only `message_sources` sidecar first, not body-returning DB listing yet.
36. C36: added metadata-only MsgLib `message_sources` across the x86 sidecar, Go wrapper, and sidecar verification; evidence-backed smoke returned six message-source summaries with source/table/role/availability/row-count metadata only.
## Next business mainline ## Next business mainline
1. Run Stage C Loop C36: add a metadata-only MsgLib sidecar operation for message-source readiness, returning table availability, required/present columns, and row counts only. 1. Run Stage C Loop C37: design the bounded DB-backed receive-message source contract now that `message_sources` metadata is available; do not expose body rows until strict limits and output rules are specified.
2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated. 2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated.
3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated. 3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated.
4. Keep UIA selector/report work as fallback support, not as the primary roadmap. 4. Keep UIA selector/report work as fallback support, not as the primary roadmap.

View File

@@ -145,13 +145,52 @@ Initial allowlisted identity sources:
- Contacts: `TD_Roster(JId,Nick)`, `TD_CustomEffigy(PersonJid,PersonName)`, `tblRecent(PersonId,PersonName)`, `tblChatLevel(PersonId,PersonName)`, selected `tblPersonMsg` id/name pairs only. - Contacts: `TD_Roster(JId,Nick)`, `TD_CustomEffigy(PersonJid,PersonName)`, `tblRecent(PersonId,PersonName)`, `tblChatLevel(PersonId,PersonName)`, selected `tblPersonMsg` id/name pairs only.
- Groups: `TD_WorkGroupAuth(GroupJID,GroupName)`, `TD_WorkGroupAuth(ChildGroupJid,GroupName)`, selected `tblMsgGroupPersonMsg` group/session id/name pairs only. - Groups: `TD_WorkGroupAuth(GroupJID,GroupName)`, `TD_WorkGroupAuth(ChildGroupJid,GroupName)`, selected `tblMsgGroupPersonMsg` group/session id/name pairs only.
### `message_sources`
Same safe provider/DB validation as `display_sources`, but reports whether candidate message tables are ready for a future DB-backed `isphere_receive_messages` source.
Required args:
- `sqlite_dll_path`
- `db_path`
Optional args:
- `password`: default `123`.
Returned data adds:
- `metadata_only=true`
- `read_only=true`
- `message_body_values_returned=false`
- `raw_rows_returned=false`
- `file_paths_returned=false`
- `message_sources[]` entries with:
- `source`
- `table`
- `role`
- `available`
- `required_columns`
- `present_columns`
- `row_count`
Initial message source candidates:
- `direct_messages` from `tblPersonMsg`.
- `group_messages` from `tblMsgGroupPersonMsg`.
- `system_messages` from `TD_SystemMessageRecord`.
- `group_receipts` from `TD_GroupReceiptMessage`.
- `received_file_metadata` from `TD_ReceiveFileRecord`.
- `recent_conversation_index` from `tblRecent`.
## Security and scope rules ## Security and scope rules
- The sidecar must open DB files with read-only connection settings. - The sidecar must open DB files with read-only connection settings.
- `db_path` must point to a file named `MsgLib.db`. - `db_path` must point to a file named `MsgLib.db`.
- The schema/display-source operations return schema, columns, counts, and display-source availability only. - The schema/display-source operations return schema, columns, counts, and display-source availability only.
- `display_entities` may return bounded contact/group `jid` and `display_name` metadata only. - `display_entities` may return bounded contact/group `jid` and `display_name` metadata only.
- It does not return message bodies, message text, file paths, attachment contents, or raw row sets. - `message_sources` may return message-source table names, required/present column names, roles, availability flags, and row counts only.
- It does not return message bodies, message text, file path values, attachment contents, or raw row sets.
- It does not implement send, file download, mark-read, login, hooks, injection, or DB writes. - It does not implement send, file download, mark-read, login, hooks, injection, or DB writes.
- The Go MCP layer remains the coordinator; this process is a bounded local reader. - The Go MCP layer remains the coordinator; this process is a bounded local reader.
@@ -186,6 +225,13 @@ Committed C29 implementation:
- Evidence-backed C29 verification against the copied DB returned contact/group entity counts and source-table names without printing entity values. - Evidence-backed C29 verification against the copied DB returned contact/group entity counts and source-table names without printing entity values.
- C31 adds optional `scripts/verify-msglib-mcp-enrichment.ps1` to prove MCP-level contact/group enrichment through the real sidecar while printing only counts and `msglib:<source_table>` refs. C33 extends the same optional script to create a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verify receive-message `sender_name` plus `conversation.display_name` enrichment; the printed result remains limited to counts, booleans, and source refs. - C31 adds optional `scripts/verify-msglib-mcp-enrichment.ps1` to prove MCP-level contact/group enrichment through the real sidecar while printing only counts and `msglib:<source_table>` refs. C33 extends the same optional script to create a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verify receive-message `sender_name` plus `conversation.display_name` enrichment; the printed result remains limited to counts, booleans, and source refs.
Committed C36 implementation:
- Sidecar op: `message_sources`.
- Go wrapper: `MessageSources(ctx)`.
- Verification script calls `message_sources` only when copied DB env paths are supplied, and prints only sanitized `message_source_count` plus message-source summaries containing source/table/role/availability/row-count metadata.
- Evidence-backed C36 verification against the copied DB returned six message-source summaries without printing message body values, file path values, raw rows, or copied DB contents.
## Verification ## Verification
Default verification builds and checks `self_check` only: Default verification builds and checks `self_check` only:
@@ -210,4 +256,4 @@ $env:ISPHERE_MSGLIB_DB = "<path-to-copied-MsgLib.db>"
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-enrichment.ps1 powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-enrichment.ps1
``` ```
Expected printed evidence is sanitized: contact/group counts, `msglib:<source_table>` refs, `receive_display_smoke=true`, populated-field booleans, and no entity values, message bodies, file paths, or raw rows. Expected printed evidence is sanitized: contact/group counts, `msglib:<source_table>` refs, `message_source_count`, message-source table/role/row-count summaries, `receive_display_smoke=true`, populated-field booleans, and no entity values, message bodies, file path values, or raw rows.

View File

@@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice | | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
| --- | --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- | --- |
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables to the receive contract but keeps body/row reads out of scope | C36 metadata-only MsgLib message-source sidecar | | `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources` readiness with table/column/count summaries only | C37 bounded DB-backed receive-source design |
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar | | `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar | | `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence | | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
| `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration | | `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration |
| `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated | | `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated |
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
## Deferred source work ## Deferred source work
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, and C35 maps MsgLib message tables to receive-message contract fields without reading row values. The next step is a metadata-only sidecar operation for message-source readiness. `MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, and C36 adds metadata-only `message_sources` readiness. The next step is bounded DB-backed receive-source design.

View File

@@ -2600,17 +2600,55 @@ MsgLib message-table source precheck result:
- Return table availability, required/present columns, safe row counts, and source role metadata only. - Return table availability, required/present columns, safe row counts, and source role metadata only.
- Do not return `MsgText`, `MessageBody`, `FilePath`, entity values, raw rows, sends, downloads, hooks, injection, or DB writes. - Do not return `MsgText`, `MessageBody`, `FilePath`, entity values, raw rows, sends, downloads, hooks, injection, or DB writes.
- [ ] **Step 1: Write failing Go client/sidecar tests for message-source metadata** - [x] **Step 1: Write failing Go client/sidecar tests for message-source metadata**
Test request/response decoding and bounded metadata shape with a fake sidecar before implementation. Added `TestMessageSourcesPassesConfiguredPathsAndDecodesMetadata`; RED failed with `client.MessageSources undefined`.
- [ ] **Step 2: Implement sidecar metadata operation** - [x] **Step 2: Implement sidecar metadata operation**
Expose message-source readiness using existing read-only schema/count helpers and allowlisted table/column maps. Implemented Go `MessageSources(ctx)` plus x86 sidecar op `message_sources`. The op returns metadata-only source/table/role/availability/required/present columns/row counts for allowlisted message-source tables.
- [ ] **Step 3: Verify default and evidence-backed smokes** - [x] **Step 3: Verify default and evidence-backed smokes**
Update `scripts\verify-msglib-sidecar.ps1` to assert sanitized message-source metadata when copied DB env is configured, then run the full gate. Updated `scripts\verify-msglib-sidecar.ps1` to call `message_sources` when copied DB env is configured. Verification passed: `git diff --check`, `go test ./...`, default sidecar smoke, evidence-backed sidecar smoke, optional `scripts\verify-msglib-mcp-enrichment.ps1`, and deterministic `scripts\verify-go-mcp.ps1`. Default sidecar smoke returned `message_source_count=0`; evidence-backed smoke returned `message_source_count=6`, `message_body_values_returned=false`, `raw_rows_returned=false`, and `file_paths_returned=false`.
---
## Loop C36 Result
MsgLib message-source metadata sidecar result:
- Added sidecar op `message_sources` and Go wrapper `MessageSources(ctx)`.
- Added fake-sidecar unit coverage for request args and decoded metadata shape.
- Evidence-backed sidecar smoke returned six message-source summaries: direct, group, system, group receipt, received-file metadata, and recent conversation index.
- Output remains metadata-only: table/column/count summaries, no `MsgText`/`MessageBody` values, no file path values, no raw rows.
---
## Loop C37: Bounded DB-backed receive-source design
**Goal:** Design the smallest safe implementation slice for DB-backed `isphere_receive_messages` using C35 field mapping and C36 `message_sources` readiness metadata.
**Planned files:**
- Modify or create a design note under `docs/source-discovery/`.
- Modify matrix/status/plan.
- Do not implement body-returning DB listing until the contract and acceptance gate are explicit.
**Planned behavior:**
- Define direct/group source selection, ordering, limit, body-column policy, attachment metadata policy, and fallback behavior versus PacketReader.
- Decide whether C38 should add sidecar-level bounded row listing or first add a redacted synthetic DB fixture.
- [ ] **Step 1: Write bounded DB receive-source contract**
Define request args, returned fields, limits, and no-output boundaries for DB-backed receive listing.
- [ ] **Step 2: Choose implementation route**
Select between sidecar row listing, Go source abstraction first, or synthetic fixture-first tests.
- [ ] **Step 3: Update next implementation loop**
Record C37 decision and prepare C38.
--- ---

View File

@@ -87,6 +87,16 @@ type DisplaySource struct {
PresentColumns []string `json:"present_columns"` PresentColumns []string `json:"present_columns"`
} }
type MessageSource struct {
Source string `json:"source"`
Table string `json:"table"`
Role string `json:"role"`
Available bool `json:"available"`
RequiredColumns []string `json:"required_columns"`
PresentColumns []string `json:"present_columns"`
RowCount string `json:"row_count"`
}
type DisplayEntitiesOptions struct { type DisplayEntitiesOptions struct {
EntityType string EntityType string
Query string Query string
@@ -139,6 +149,19 @@ func (c *Client) DisplaySources(ctx context.Context) ([]DisplaySource, error) {
return out.DisplaySources, nil return out.DisplaySources, nil
} }
func (c *Client) MessageSources(ctx context.Context) ([]MessageSource, error) {
args := map[string]any{
"sqlite_dll_path": c.config.SQLiteDLLPath,
"db_path": c.config.DBPath,
"password": c.config.Password,
}
var out messageSourcesData
if err := c.call(ctx, "message_sources", args, &out); err != nil {
return nil, err
}
return out.MessageSources, nil
}
func (c *Client) DisplayEntities(ctx context.Context, opts DisplayEntitiesOptions) ([]DisplayEntity, error) { func (c *Client) DisplayEntities(ctx context.Context, opts DisplayEntitiesOptions) ([]DisplayEntity, error) {
entityType := strings.TrimSpace(opts.EntityType) entityType := strings.TrimSpace(opts.EntityType)
if entityType != EntityTypeContacts && entityType != EntityTypeGroups { if entityType != EntityTypeContacts && entityType != EntityTypeGroups {
@@ -174,6 +197,10 @@ type displaySourcesData struct {
DisplaySources []DisplaySource `json:"display_sources"` DisplaySources []DisplaySource `json:"display_sources"`
} }
type messageSourcesData struct {
MessageSources []MessageSource `json:"message_sources"`
}
type displayEntitiesData struct { type displayEntitiesData struct {
Entities []DisplayEntity `json:"entities"` Entities []DisplayEntity `json:"entities"`
} }

View File

@@ -55,6 +55,32 @@ func TestDisplaySourcesPassesConfiguredPathsAndDecodesMetadata(t *testing.T) {
} }
} }
func TestMessageSourcesPassesConfiguredPathsAndDecodesMetadata(t *testing.T) {
cfg := Config{
SQLiteDLLPath: `C:\fixture\System.Data.SQLite.dll`,
DBPath: `C:\fixture\MsgLib.db`,
Password: "123",
}
client := newFakeClient(t, "message_sources_ok", cfg)
got, err := client.MessageSources(context.Background())
if err != nil {
t.Fatalf("MessageSources returned error: %v", err)
}
if len(got) != 2 {
t.Fatalf("len(MessageSources) = %d, want 2", len(got))
}
if got[0].Source != "direct_messages" || got[0].Table != "tblPersonMsg" || got[0].Role != "direct" || !got[0].Available {
t.Fatalf("unexpected first message source: %+v", got[0])
}
if got[0].RowCount != "2132" || strings.Join(got[0].RequiredColumns, ",") != "Id,SndPersonId,RecvPersonId,ObjPersonId,ActionTime,MsgText,MessageBody" {
t.Fatalf("unexpected first message source metadata: %+v", got[0])
}
if got[1].Source != "group_messages" || got[1].Table != "tblMsgGroupPersonMsg" || got[1].Role != "group" || got[1].RowCount != "2884" {
t.Fatalf("unexpected second message source: %+v", got[1])
}
}
func TestDisplayEntitiesPassesBoundedArgsAndDecodesMetadata(t *testing.T) { func TestDisplayEntitiesPassesBoundedArgsAndDecodesMetadata(t *testing.T) {
cfg := Config{ cfg := Config{
SQLiteDLLPath: `C:\fixture\System.Data.SQLite.dll`, SQLiteDLLPath: `C:\fixture\System.Data.SQLite.dll`,
@@ -240,6 +266,47 @@ func fakeSidecarMain() int {
"error": nil, "error": nil,
"warnings": []any{}, "warnings": []any{},
}) })
case "message_sources":
args, _ := req["args"].(map[string]any)
if args["sqlite_dll_path"] != `C:\fixture\System.Data.SQLite.dll` || args["db_path"] != `C:\fixture\MsgLib.db` || args["password"] != "123" {
fmt.Fprintf(os.Stderr, "bad message_sources args: %#v\n", args)
return 2
}
return writeFakeResponse(map[string]any{
"protocol": Protocol,
"request_id": requestID,
"op": op,
"ok": true,
"data": map[string]any{
"metadata_only": true,
"read_only": true,
"message_body_values_returned": false,
"raw_rows_returned": false,
"file_paths_returned": false,
"message_sources": []any{
map[string]any{
"source": "direct_messages",
"table": "tblPersonMsg",
"role": "direct",
"available": true,
"required_columns": []any{"Id", "SndPersonId", "RecvPersonId", "ObjPersonId", "ActionTime", "MsgText", "MessageBody"},
"present_columns": []any{"Id", "SndPersonId", "RecvPersonId", "ObjPersonId", "ActionTime", "MsgText", "MessageBody"},
"row_count": "2132",
},
map[string]any{
"source": "group_messages",
"table": "tblMsgGroupPersonMsg",
"role": "group",
"available": true,
"required_columns": []any{"Id", "SndPersonId", "MsgGroupId", "MsgGroupName", "MsgTime", "MsgText", "MessageBody"},
"present_columns": []any{"Id", "SndPersonId", "MsgGroupId", "MsgGroupName", "MsgTime", "MsgText", "MessageBody"},
"row_count": "2884",
},
},
},
"error": nil,
"warnings": []any{},
})
case "display_entities": case "display_entities":
args, _ := req["args"].(map[string]any) args, _ := req["args"].(map[string]any)
if args["sqlite_dll_path"] != `C:\fixture\System.Data.SQLite.dll` || args["db_path"] != `C:\fixture\MsgLib.db` || args["password"] != "123" || args["entity_type"] != "contacts" || args["query"] != "alice" || args["limit"] != float64(2) { if args["sqlite_dll_path"] != `C:\fixture\System.Data.SQLite.dll` || args["db_path"] != `C:\fixture\MsgLib.db` || args["password"] != "123" || args["entity_type"] != "contacts" || args["query"] != "alice" || args["limit"] != float64(2) {

View File

@@ -8,7 +8,7 @@ namespace MsgLibReadSidecar
{ {
internal static class Program internal static class Program
{ {
private const string Version = "0.2.0"; private const string Version = "0.3.0";
private const string DefaultPassword = "123"; private const string DefaultPassword = "123";
private static readonly HashSet<string> DefaultTargetTables = new HashSet<string>(StringComparer.OrdinalIgnoreCase) private static readonly HashSet<string> DefaultTargetTables = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
@@ -46,6 +46,9 @@ namespace MsgLibReadSidecar
case "display_sources": case "display_sources":
response = SidecarProtocol.Success(requestId, op, SchemaSummary(opArgs, true)); response = SidecarProtocol.Success(requestId, op, SchemaSummary(opArgs, true));
break; break;
case "message_sources":
response = SidecarProtocol.Success(requestId, op, MessageSources(opArgs));
break;
case "display_entities": case "display_entities":
response = SidecarProtocol.Success(requestId, op, DisplayEntities(opArgs)); response = SidecarProtocol.Success(requestId, op, DisplayEntities(opArgs));
break; break;
@@ -123,6 +126,40 @@ namespace MsgLibReadSidecar
return result; return result;
} }
private static Dictionary<string, object> MessageSources(Dictionary<string, object> args)
{
string sqliteDllPath = SidecarProtocol.GetString(args, "sqlite_dll_path", "");
string dbPath = SidecarProtocol.GetString(args, "db_path", "");
string password = SidecarProtocol.GetString(args, "password", DefaultPassword);
ValidateInputs(sqliteDllPath, dbPath);
var result = new Dictionary<string, object>
{
{ "metadata_only", true },
{ "read_only", true },
{ "message_body_values_returned", false },
{ "raw_rows_returned", false },
{ "file_paths_returned", false },
{ "provider", "System.Data.SQLite" },
{ "process_bits", IntPtr.Size * 8 }
};
using (DbConnection connection = OpenSqliteConnection(sqliteDllPath, dbPath, password))
{
connection.Open();
TryExecuteNonQuery(connection, "PRAGMA query_only=ON");
List<Dictionary<string, object>> tables = ReadMasterTables(connection, 256);
HashSet<string> messageTables = BuildMessageSourceTableSet();
List<Dictionary<string, object>> columns = ReadColumns(connection, tables, messageTables);
Dictionary<string, HashSet<string>> columnMap = BuildColumnMap(columns);
Dictionary<string, string> rowCounts = ReadTargetCountMap(connection, tables, messageTables);
result["message_sources"] = BuildMessageSources(tables, columnMap, rowCounts);
}
return result;
}
private static Dictionary<string, object> DisplayEntities(Dictionary<string, object> args) private static Dictionary<string, object> DisplayEntities(Dictionary<string, object> args)
{ {
@@ -288,6 +325,16 @@ namespace MsgLibReadSidecar
return rows; return rows;
} }
private static Dictionary<string, string> ReadTargetCountMap(DbConnection connection, List<Dictionary<string, object>> tables, HashSet<string> targetTables)
{
var rowCounts = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
foreach (Dictionary<string, object> item in ReadTargetCounts(connection, tables, targetTables))
{
rowCounts[Convert.ToString(item["table"])] = Convert.ToString(item["row_count"]);
}
return rowCounts;
}
private static List<Dictionary<string, object>> BuildDisplaySources(List<Dictionary<string, object>> tables, List<Dictionary<string, object>> columns) private static List<Dictionary<string, object>> BuildDisplaySources(List<Dictionary<string, object>> tables, List<Dictionary<string, object>> columns)
{ {
var tableSet = new HashSet<string>(StringComparer.OrdinalIgnoreCase); var tableSet = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
@@ -320,6 +367,115 @@ namespace MsgLibReadSidecar
return sources; return sources;
} }
private sealed class MessageSourceSpec
{
public string Source;
public string Table;
public string Role;
public string[] RequiredColumns;
}
private static HashSet<string> BuildMessageSourceTableSet()
{
var tables = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (MessageSourceSpec spec in BuildMessageSourceSpecs())
{
tables.Add(spec.Table);
}
return tables;
}
private static List<MessageSourceSpec> BuildMessageSourceSpecs()
{
return new List<MessageSourceSpec>
{
new MessageSourceSpec
{
Source = "direct_messages",
Table = "tblPersonMsg",
Role = "direct",
RequiredColumns = new string[] { "Id", "SndPersonId", "RecvPersonId", "ObjPersonId", "ActionTime", "MsgText", "MessageBody" }
},
new MessageSourceSpec
{
Source = "group_messages",
Table = "tblMsgGroupPersonMsg",
Role = "group",
RequiredColumns = new string[] { "Id", "SndPersonId", "MsgGroupId", "MsgGroupName", "MsgTime", "MsgText", "MessageBody" }
},
new MessageSourceSpec
{
Source = "system_messages",
Table = "TD_SystemMessageRecord",
Role = "system",
RequiredColumns = new string[] { "MsgGuid", "MsgType", "MsgTitle", "MsgText", "SendPersonJid", "MsgTime", "MsgReadState" }
},
new MessageSourceSpec
{
Source = "group_receipts",
Table = "TD_GroupReceiptMessage",
Role = "receipt_metadata",
RequiredColumns = new string[] { "MsgGuid", "PersonID", "ReceiptTime" }
},
new MessageSourceSpec
{
Source = "received_file_metadata",
Table = "TD_ReceiveFileRecord",
Role = "attachment_metadata",
RequiredColumns = new string[] { "ReceiveMsgGuid", "FileName", "FileSize", "SourceID", "SourceName", "FileType", "SendPersonJid", "SendPersonName", "SendTime" }
},
new MessageSourceSpec
{
Source = "recent_conversation_index",
Table = "tblRecent",
Role = "conversation_index",
RequiredColumns = new string[] { "Id", "PersonId", "PersonName", "ActionTime", "FType" }
}
};
}
private static List<Dictionary<string, object>> BuildMessageSources(List<Dictionary<string, object>> tables, Dictionary<string, HashSet<string>> columnMap, Dictionary<string, string> rowCounts)
{
var tableSet = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (Dictionary<string, object> table in tables)
{
if (string.Equals(Convert.ToString(table["type"]), "table", StringComparison.OrdinalIgnoreCase))
{
tableSet.Add(Convert.ToString(table["name"]));
}
}
var sources = new List<Dictionary<string, object>>();
foreach (MessageSourceSpec spec in BuildMessageSourceSpecs())
{
AddMessageSource(sources, tableSet, columnMap, rowCounts, spec);
}
return sources;
}
private static void AddMessageSource(List<Dictionary<string, object>> sources, HashSet<string> tableSet, Dictionary<string, HashSet<string>> columnMap, Dictionary<string, string> rowCounts, MessageSourceSpec spec)
{
var presentColumns = new List<string>();
if (columnMap.ContainsKey(spec.Table))
{
foreach (string column in spec.RequiredColumns)
{
if (columnMap[spec.Table].Contains(column)) presentColumns.Add(column);
}
}
string rowCount = rowCounts.ContainsKey(spec.Table) ? rowCounts[spec.Table] : "0";
sources.Add(new Dictionary<string, object>
{
{ "source", spec.Source },
{ "table", spec.Table },
{ "role", spec.Role },
{ "available", tableSet.Contains(spec.Table) && presentColumns.Count == spec.RequiredColumns.Length },
{ "required_columns", spec.RequiredColumns },
{ "present_columns", presentColumns.ToArray() },
{ "row_count", rowCount }
});
}
private sealed class DisplayEntitySpec private sealed class DisplayEntitySpec
{ {

View File

@@ -47,6 +47,8 @@ if (-not $self.ok -or $self.data.sidecar_name -ne "MsgLibReadSidecar" -or $self.
$providerChecked = $false $providerChecked = $false
$displaySources = $null $displaySources = $null
$displayEntitySummaries = @() $displayEntitySummaries = @()
$messageSources = $null
$messageSourceSummaries = @()
if ($SQLiteDllPath -and $DbPath) { if ($SQLiteDllPath -and $DbPath) {
if (-not (Test-Path -LiteralPath $SQLiteDllPath)) { if (-not (Test-Path -LiteralPath $SQLiteDllPath)) {
throw "SQLite DLL not found: $SQLiteDllPath" throw "SQLite DLL not found: $SQLiteDllPath"
@@ -70,6 +72,29 @@ if ($SQLiteDllPath -and $DbPath) {
throw "display_sources failed" throw "display_sources failed"
} }
$messageSources = Invoke-SidecarJson @{
protocol = "isphere.msglib.v1"
request_id = "verify-message-sources"
op = "message_sources"
args = @{
sqlite_dll_path = $SQLiteDllPath
db_path = $DbPath
password = "123"
}
}
if (-not $messageSources.ok -or -not $messageSources.data.metadata_only -or -not $messageSources.data.read_only -or $messageSources.data.message_body_values_returned -or $messageSources.data.raw_rows_returned -or $messageSources.data.file_paths_returned) {
throw "message_sources failed"
}
foreach ($source in @($messageSources.data.message_sources)) {
$messageSourceSummaries += [ordered]@{
source = $source.source
table = $source.table
role = $source.role
available = $source.available
row_count = $source.row_count
}
}
foreach ($entityType in @("contacts", "groups")) { foreach ($entityType in @("contacts", "groups")) {
$displayEntities = Invoke-SidecarJson @{ $displayEntities = Invoke-SidecarJson @{
protocol = "isphere.msglib.v1" protocol = "isphere.msglib.v1"
@@ -104,4 +129,9 @@ if ($SQLiteDllPath -and $DbPath) {
provider_checked = $providerChecked provider_checked = $providerChecked
display_source_count = if ($displaySources) { $displaySources.data.display_sources.Count } else { 0 } display_source_count = if ($displaySources) { $displaySources.data.display_sources.Count } else { 0 }
display_entity_summaries = $displayEntitySummaries display_entity_summaries = $displayEntitySummaries
message_source_count = if ($messageSources) { $messageSources.data.message_sources.Count } else { 0 }
message_source_summaries = $messageSourceSummaries
message_body_values_returned = $false
raw_rows_returned = $false
file_paths_returned = $false
} | ConvertTo-Json -Depth 8 -Compress } | ConvertTo-Json -Depth 8 -Compress