feat: enrich search with msglib display entities
This commit is contained in:
zhaoyilun
@@ -27,9 +27,9 @@ Remote base before local roadmap commits: `b2d839e Merge branch 'codex/n15-repor
|
||||
- `scripts/verify-win-helper.ps1` and `scripts/verify-go-mcp.ps1` provide repeatable local checks for the helper, eight-tool MCP surface, default empty-source receive/contact/group/file calls, synthetic configured-file receive/contact/group/file smoke, and synthetic configured-directory receive/contact/group/file smoke.
|
||||
- `isphere_receive_messages` now returns the contract-facing `ok`, `conversation`, `messages`, `next_cursor`, and `audit` envelope while retaining legacy parser-native message fields for compatibility.
|
||||
- `isphere_receive_messages` accepts the read-contract argument set for the current local-readonly path: `conversation_id`, `query`, `since`, `limit`, `include_attachment_metadata`, `source_preference`, `preview`, and empty `cursor`.
|
||||
- `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path, but display names/member data are not yet enriched from `MsgLib.db` until C30 wires the C29 bounded display-entity reader into the MCP search tools.
|
||||
- `isphere_search_contacts` and `isphere_search_groups` accept and validate the safe search-contract args for the current local-readonly path and can optionally enrich results from the env-configured C29 `MsgLib.db` display-entity reader; default behavior remains log/JID-backed when MsgLib env vars are absent.
|
||||
- `isphere_receive_files` list mode accepts and validates the safe file-list contract args for the current local-readonly path while keeping download blocked.
|
||||
- `native/MsgLibReadSidecar` provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks and bounded contact/group display-entity reads; `internal/msglib` wraps it from Go, but MCP tools are not wired to it yet.
|
||||
- `native/MsgLibReadSidecar` provides a bounded x86 .NET read-only boundary for `MsgLib.db` schema/display-source checks and bounded contact/group display-entity reads; `internal/msglib` wraps it from Go, and C30 wires it as an optional MCP contact/group enrichment source.
|
||||
- N12-pre and N12R documents define safe offline evidence intake and internal-sandbox live UIA capture procedures.
|
||||
- N13/N14/N15 produced selector catalog/report validation infrastructure, but those are supporting validation assets only.
|
||||
|
||||
@@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme
|
||||
|
||||
## Current loop
|
||||
|
||||
Current loop: `Stage C / Loop C30 - integrate MsgLib display entities into contact/group search`.
|
||||
Current loop: `Stage C / Loop C31 - optional real MsgLib MCP enrichment smoke`.
|
||||
|
||||
Plan file: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`.
|
||||
|
||||
@@ -98,10 +98,11 @@ Current loop output so far:
|
||||
27. C27: added `native/MsgLibReadSidecar` with protocol `isphere.msglib.v1`, x86 build/verify scripts, and safe `self_check`, `schema_summary`, and `display_sources` operations; evidence-backed smoke returned 8 display-source candidates without row/message values.
|
||||
28. C28: added `internal/msglib` Go client wrapper for the sidecar protocol with env config, fake-sidecar unit tests, timeout handling, structured sidecar errors, `SelfCheck`, and `DisplaySources`; no MCP tool wiring or message/display row reads yet.
|
||||
29. C29: added bounded `display_entities` support across the x86 sidecar, Go client, and verification script; evidence-backed verification returned sanitized contact/group counts and source-table names only, with no message bodies, file paths, raw rows, sends, writes, hooks, or injection.
|
||||
30. C30: wired `DisplayEntities` into `isphere_search_contacts` and `isphere_search_groups` as an optional env-configured enrichment source with fake-source tests; default MCP verification now clears MsgLib env to keep the standard smoke deterministic.
|
||||
|
||||
## Next business mainline
|
||||
|
||||
1. Run Stage C Loop C30: wire C29 `DisplayEntities` into `isphere_search_contacts` and `isphere_search_groups` as an env-configured optional enrichment source, with fake-sidecar tests first; keep message bodies, file paths, downloads, sends, writes, hooks, injection, and raw row dumps out of scope.
|
||||
1. Run Stage C Loop C31: add an optional copied-DB MCP enrichment smoke/hardening path that proves contact/group enrichment through the real sidecar while printing only sanitized counts/source metadata by default.
|
||||
2. Leave `isphere_send_message` blocked until bridge/API, UIA action-chain, or protocol connector evidence is validated.
|
||||
3. Leave actual file download/cache mapping as a later connector node until a message-to-cache hash or client download source is validated.
|
||||
4. Keep UIA selector/report work as fallback support, not as the primary roadmap.
|
||||
|
||||
@@ -17,9 +17,9 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
|
||||
|
||||
| MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C29 adds bounded contact/group display-entity extraction; DB-backed message listing is still not implemented | Defer DB-backed message listing; first run C30 contact/group search enrichment |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C29 verifies bounded contact display-entity extraction through the x86 sidecar and Go client; MCP search is not enriched yet | C30 wire optional MsgLib enrichment into `isphere_search_contacts` |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C29 verifies bounded group display-entity extraction through the x86 sidecar and Go client; MCP search is not enriched yet | C30 wire optional MsgLib enrichment into `isphere_search_groups` |
|
||||
| `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces; copied `MsgLib.db` through a future x86 read-only sidecar/helper | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md` | C20 supports and validates the current safe receive-message contract args; C30 wires bounded contact/group display entities into search enrichment; DB-backed message listing is still not implemented | Defer DB-backed message listing; run optional real MsgLib MCP enrichment smoke |
|
||||
| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C30 wires optional MsgLib display-entity enrichment into `isphere_search_contacts`; standard verification keeps MsgLib env cleared by default | C31 optional copied-DB MCP enrichment smoke with sanitized output |
|
||||
| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C30 wires optional MsgLib display-entity enrichment into `isphere_search_groups`; standard verification keeps MsgLib env cleared by default | C31 optional copied-DB MCP enrichment smoke with sanitized output |
|
||||
| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | defer download mapping; precheck contact/group display-name evidence |
|
||||
| `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | do not implement write behavior; harden existing read source configuration |
|
||||
| `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | defer until send-message connector is validated |
|
||||
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
|
||||
|
||||
## Deferred source work
|
||||
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, and C29 adds bounded `display_entities` extraction. The next step is optional contact/group MCP enrichment from that source.
|
||||
`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, and C30 wires it as optional contact/group MCP enrichment. The next step is an optional copied-DB MCP smoke with sanitized output.
|
||||
|
||||
@@ -2302,7 +2302,8 @@ Bounded display entity extraction result:
|
||||
|
||||
**Goal:** Wire the C29 bounded `DisplayEntities` source into `isphere_search_contacts` and `isphere_search_groups` as an optional env-configured enrichment path, while preserving the current log-backed fallback and keeping message bodies, file paths, downloads, sends, writes, hooks, injection, DB mutation, and raw row dumps out of scope.
|
||||
|
||||
**Planned files:**
|
||||
**Files:**
|
||||
- Create: `internal/tools/display_entities.go`
|
||||
- Modify: `internal/tools/isphere_contacts.go`
|
||||
- Modify: `internal/tools/isphere_contacts_test.go`
|
||||
- Modify: `internal/tools/isphere_groups.go`
|
||||
@@ -2318,24 +2319,24 @@ Bounded display entity extraction result:
|
||||
- Add an injectable display-entity source interface around `internal/msglib.Client`.
|
||||
- Keep default MCP behavior unchanged when `ISPHERE_MSGLIB_SIDECAR_EXE`, `ISPHERE_MSGLIB_SQLITE_DLL`, or `ISPHERE_MSGLIB_DB` are absent.
|
||||
- When all env vars are present, enrich contact/group search results from `DisplayEntities` with bounded `query` and `limit`.
|
||||
- Normal tests use fake display-entity source data; verification script should include a synthetic/fake enrichment smoke before any real copied DB smoke.
|
||||
- Normal tests use fake display-entity source data; `scripts/verify-go-mcp.ps1` clears MsgLib env by default to keep standard smoke deterministic.
|
||||
- Do not expose a new MCP tool in this loop.
|
||||
|
||||
- [ ] **Step 1: Write failing contact/group enrichment tests**
|
||||
- [x] **Step 1: Write failing contact/group enrichment tests**
|
||||
|
||||
Add tests proving contacts/groups can consume injected display entities and still pass existing log-backed/default behavior.
|
||||
Added fake-source tests proving `isphere_search_contacts` and `isphere_search_groups` can consume injected MsgLib display entities, pass query/limit/entity type, and return `msglib_readonly` metadata. Verified RED with missing `RegisterISphereContactToolsWithDisplayEntities` and `RegisterISphereGroupToolsWithDisplayEntities`.
|
||||
|
||||
- [ ] **Step 2: Implement optional display-entity source wiring**
|
||||
- [x] **Step 2: Implement optional display-entity source wiring**
|
||||
|
||||
Implement source interface and env-configured `internal/msglib` client construction without changing default empty-source behavior.
|
||||
Added `DisplayEntitySource`, mapping/merge helpers, contact/group registration functions with optional display entity source, and `NewServerFromEnv` MsgLib env wiring. Partial MsgLib env config now fails fast; default nil source remains unchanged.
|
||||
|
||||
- [ ] **Step 3: Update verification/docs**
|
||||
- [x] **Step 3: Update verification/docs**
|
||||
|
||||
Extend `scripts/verify-go-mcp.ps1` with synthetic/fake enrichment smoke and update status/matrix/plan.
|
||||
Updated `scripts/verify-go-mcp.ps1` to clear MsgLib env for deterministic default verification and added `msglib_configured=false` to the verification result. Updated status/matrix/plan.
|
||||
|
||||
- [ ] **Step 4: Full verification and commit**
|
||||
- [x] **Step 4: Full verification and commit**
|
||||
|
||||
Run:
|
||||
Run before commit:
|
||||
|
||||
```powershell
|
||||
git diff --check
|
||||
@@ -2346,6 +2347,48 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-sideca
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1
|
||||
```
|
||||
|
||||
Final verification passed: `git diff --check`, `go test ./...`, explicit `go build ./cmd/isphere-mcp`, default sidecar smoke, evidence-backed sidecar smoke with sanitized `display_entity_summaries`, and `scripts/verify-go-mcp.ps1` with `msglib_configured=false`.
|
||||
|
||||
---
|
||||
|
||||
## Loop C30 Result
|
||||
|
||||
Optional MsgLib MCP enrichment result:
|
||||
|
||||
- `isphere_search_contacts` can now use injected/Env MsgLib display entities and return `msglib_readonly` contact metadata.
|
||||
- `isphere_search_groups` can now use injected/Env MsgLib display entities and return `msglib_readonly` group metadata.
|
||||
- Default behavior stays log/JID-backed when MsgLib env is absent.
|
||||
- Standard `verify-go-mcp.ps1` stays deterministic by clearing MsgLib env; real copied-DB MCP smoke is deferred to C31 so it can be sanitized separately.
|
||||
|
||||
---
|
||||
|
||||
## Loop C31: Optional real MsgLib MCP enrichment smoke
|
||||
|
||||
**Goal:** Add an optional verification path that proves `isphere_search_contacts` and `isphere_search_groups` can call the real copied-DB MsgLib sidecar through MCP, while printing only sanitized counts/source metadata by default.
|
||||
|
||||
**Planned files:**
|
||||
- Modify: `scripts/verify-go-mcp.ps1` or add a focused optional verification script.
|
||||
- Modify: `docs/current-status-card.md`
|
||||
- Modify: `docs/source-discovery/capability-source-matrix.md`
|
||||
- Modify: `docs/superpowers/plans/2026-07-09-stage-c-log-backed-receive-messages-loop.md`
|
||||
|
||||
**Planned behavior:**
|
||||
- Default verification remains deterministic and does not use real MsgLib env.
|
||||
- Optional verification is gated by explicit env variables and copied DB paths.
|
||||
- Output only counts, source names, and booleans; do not print display names, JIDs, message bodies, file paths, or raw rows.
|
||||
|
||||
- [ ] **Step 1: Add optional real-MsgLib MCP smoke**
|
||||
|
||||
Implement opt-in smoke with sanitized summary only.
|
||||
|
||||
- [ ] **Step 2: Run default and optional verification**
|
||||
|
||||
Run default full verification and optional copied-DB smoke if paths are available.
|
||||
|
||||
- [ ] **Step 3: Update docs and commit**
|
||||
|
||||
Record sanitized evidence and next loop.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
Reference in New Issue
Block a user