admin/isphere-ai-bridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add msglib message source metadata

Browse Source
This commit is contained in:
zhaoyilun
2026-07-10 08:36:34 +08:00
parent 1a21a01184
commit c8ecf5b36e
8 changed files with 380 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/current-status-card.md
View File

@@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme
## Current loop
Current loop: `Stage C / Loop C36 - MsgLib message-source metadata sidecar`.
Current loop: `Stage C / Loop C37 - bounded DB-backed receive-source design`.
Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.
@@ -104,10 +104,11 @@ Current loop output so far:
33. C33: optional real copied-DB MCP smoke now also constructs a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verifies receive-message display enrichment through MCP; output stays sanitized to counts, booleans, and `msglib:<source_table>` refs only.
34. C34: `docs/go-mcp-runbook.md` now documents the operator setup path for optional MsgLib display-name enrichment, including the required env vars, partial-config failure rule, standard no-MsgLib verification, optional provider smoke, optional MCP enrichment smoke, and sanitized output boundaries.
35. C35: `docs/source-discovery/2026-07-10-msglib-message-source-precheck.md` maps `tblPersonMsg`, `tblMsgGroupPersonMsg`, `TD_SystemMessageRecord`, receipt, recent, and file-record schema columns to the future `isphere_receive_messages` contract; decision is metadata-only `message_sources` sidecar first, not body-returning DB listing yet.
36. C36: added metadata-only MsgLib `message_sources` across the x86 sidecar, Go wrapper, and sidecar verification; evidence-backed smoke returned six message-source summaries with source/table/role/availability/row-count metadata only.
## Next business mainline
1. Run Stage C Loop C36: add a metadata-only MsgLib sidecar operation for message-source readiness, returning table availability, required/present columns, and row counts only.
1. Run Stage C Loop C37: design the bounded DB-backed receive-message source contract now that `message_sources` metadata is available; do not expose body rows until strict limits and output rules are specified.
2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated.
3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated.
4. Keep UIA selector/report work as fallback support, not as the primary roadmap.

50
docs/msglib-read-sidecar-contract.md
View File

@@ -145,13 +145,52 @@ Initial allowlisted identity sources:
- Contacts: `TD_Roster(JId,Nick)`, `TD_CustomEffigy(PersonJid,PersonName)`, `tblRecent(PersonId,PersonName)`, `tblChatLevel(PersonId,PersonName)`, selected `tblPersonMsg` id/name pairs only.
- Groups: `TD_WorkGroupAuth(GroupJID,GroupName)`, `TD_WorkGroupAuth(ChildGroupJid,GroupName)`, selected `tblMsgGroupPersonMsg` group/session id/name pairs only.
### `message_sources`
Same safe provider/DB validation as `display_sources`, but reports whether candidate message tables are ready for a future DB-backed `isphere_receive_messages` source.
Required args:
- `sqlite_dll_path`
- `db_path`
Optional args:
- `password`: default `123`.
Returned data adds:
- `metadata_only=true`
- `read_only=true`
- `message_body_values_returned=false`
- `raw_rows_returned=false`
- `file_paths_returned=false`
- `message_sources[]` entries with:
- `source`
- `table`
- `role`
- `available`
- `required_columns`
- `present_columns`
- `row_count`
Initial message source candidates:
- `direct_messages` from `tblPersonMsg`.
- `group_messages` from `tblMsgGroupPersonMsg`.
- `system_messages` from `TD_SystemMessageRecord`.
- `group_receipts` from `TD_GroupReceiptMessage`.
- `received_file_metadata` from `TD_ReceiveFileRecord`.
- `recent_conversation_index` from `tblRecent`.
## Security and scope rules
- The sidecar must open DB files with read-only connection settings.
- `db_path` must point to a file named `MsgLib.db`.
- The schema/display-source operations return schema, columns, counts, and display-source availability only.
- `display_entities` may return bounded contact/group `jid` and `display_name` metadata only.
- It does not return message bodies, message text, file paths, attachment contents, or raw row sets.
- `message_sources` may return message-source table names, required/present column names, roles, availability flags, and row counts only.
- It does not return message bodies, message text, file path values, attachment contents, or raw row sets.
- It does not implement send, file download, mark-read, login, hooks, injection, or DB writes.
- The Go MCP layer remains the coordinator; this process is a bounded local reader.
@@ -186,6 +225,13 @@ Committed C29 implementation:
- Evidence-backed C29 verification against the copied DB returned contact/group entity counts and source-table names without printing entity values.
- C31 adds optional `scripts/verify-msglib-mcp-enrichment.ps1` to prove MCP-level contact/group enrichment through the real sidecar while printing only counts and `msglib:<source_table>` refs. C33 extends the same optional script to create a synthetic encrypted PacketReader fixture from internally selected MsgLib contact/group metadata and verify receive-message `sender_name` plus `conversation.display_name` enrichment; the printed result remains limited to counts, booleans, and source refs.
Committed C36 implementation:
- Sidecar op: `message_sources`.
- Go wrapper: `MessageSources(ctx)`.
- Verification script calls `message_sources` only when copied DB env paths are supplied, and prints only sanitized `message_source_count` plus message-source summaries containing source/table/role/availability/row-count metadata.
- Evidence-backed C36 verification against the copied DB returned six message-source summaries without printing message body values, file path values, raw rows, or copied DB contents.
## Verification
Default verification builds and checks `self_check` only:
@@ -210,4 +256,4 @@ $env:ISPHERE_MSGLIB_DB = "<path-to-copied-MsgLib.db>"
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-enrichment.ps1
```
Expected printed evidence is sanitized: contact/group counts, `msglib:<source_table>` refs, `receive_display_smoke=true`, populated-field booleans, and no entity values, message bodies, file paths, or raw rows.
Expected printed evidence is sanitized: contact/group counts, `msglib:<source_table>` refs, `message_source_count`, message-source table/role/row-count summaries, `receive_display_smoke=true`, populated-field booleans, and no entity values, message bodies, file path values, or raw rows.

8
docs/source-discovery/capability-source-matrix.md
View File

@@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
| --- | --- | --- | --- | --- | --- |
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables to the receive contract but keeps body/row reads out of scope | C36 metadata-only MsgLib message-source sidecar |
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar |
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C36 metadata-only MsgLib message-source sidecar |
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through future x86 read-only sidecar operations; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke with sanitized output; C34 documents the operator setup path; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources` readiness with table/column/count summaries only | C37 bounded DB-backed receive-source design |
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | C37 bounded DB-backed receive-source design |
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
| `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration |
| `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated |
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
## Deferred source work
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, and C35 maps MsgLib message tables to receive-message contract fields without reading row values. The next step is a metadata-only sidecar operation for message-source readiness.
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, and C36 adds metadata-only `message_sources` readiness. The next step is bounded DB-backed receive-source design.

50
docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md
View File

@@ -2600,17 +2600,55 @@ MsgLib message-table source precheck result:
- Return table availability, required/present columns, safe row counts, and source role metadata only.
- Do not return `MsgText`, `MessageBody`, `FilePath`, entity values, raw rows, sends, downloads, hooks, injection, or DB writes.
- [ ] **Step 1: Write failing Go client/sidecar tests for message-source metadata**
- [x] **Step 1: Write failing Go client/sidecar tests for message-source metadata**
Test request/response decoding and bounded metadata shape with a fake sidecar before implementation.
Added `TestMessageSourcesPassesConfiguredPathsAndDecodesMetadata`; RED failed with `client.MessageSources undefined`.
- [ ] **Step 2: Implement sidecar metadata operation**
- [x] **Step 2: Implement sidecar metadata operation**
Expose message-source readiness using existing read-only schema/count helpers and allowlisted table/column maps.
Implemented Go `MessageSources(ctx)` plus x86 sidecar op `message_sources`. The op returns metadata-only source/table/role/availability/required/present columns/row counts for allowlisted message-source tables.
- [ ] **Step 3: Verify default and evidence-backed smokes**
- [x] **Step 3: Verify default and evidence-backed smokes**
Update `scripts\verify-msglib-sidecar.ps1` to assert sanitized message-source metadata when copied DB env is configured, then run the full gate.
Updated `scripts\verify-msglib-sidecar.ps1` to call `message_sources` when copied DB env is configured. Verification passed: `git diff --check`, `go test ./...`, default sidecar smoke, evidence-backed sidecar smoke, optional `scripts\verify-msglib-mcp-enrichment.ps1`, and deterministic `scripts\verify-go-mcp.ps1`. Default sidecar smoke returned `message_source_count=0`; evidence-backed smoke returned `message_source_count=6`, `message_body_values_returned=false`, `raw_rows_returned=false`, and `file_paths_returned=false`.
---
## Loop C36 Result
MsgLib message-source metadata sidecar result:
- Added sidecar op `message_sources` and Go wrapper `MessageSources(ctx)`.
- Added fake-sidecar unit coverage for request args and decoded metadata shape.
- Evidence-backed sidecar smoke returned six message-source summaries: direct, group, system, group receipt, received-file metadata, and recent conversation index.
- Output remains metadata-only: table/column/count summaries, no `MsgText`/`MessageBody` values, no file path values, no raw rows.
---
## Loop C37: Bounded DB-backed receive-source design
**Goal:** Design the smallest safe implementation slice for DB-backed `isphere_receive_messages` using C35 field mapping and C36 `message_sources` readiness metadata.
**Planned files:**
- Modify or create a design note under `docs/source-discovery/`.
- Modify matrix/status/plan.
- Do not implement body-returning DB listing until the contract and acceptance gate are explicit.
**Planned behavior:**
- Define direct/group source selection, ordering, limit, body-column policy, attachment metadata policy, and fallback behavior versus PacketReader.
- Decide whether C38 should add sidecar-level bounded row listing or first add a redacted synthetic DB fixture.
- [ ] **Step 1: Write bounded DB receive-source contract**
Define request args, returned fields, limits, and no-output boundaries for DB-backed receive listing.
- [ ] **Step 2: Choose implementation route**
Select between sidecar row listing, Go source abstraction first, or synthetic fixture-first tests.
- [ ] **Step 3: Update next implementation loop**
Record C37 decision and prepare C38.
---