feat: harden search ranking and deduplication

docs/current-status-card.md

@@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme
 
 ## Current loop
 
-Current loop: `Business Roadmap / R2 - Contact and group search quality hardening`.
+Current loop: `Business Roadmap / R3 - Receive-file download mapping precheck`.
 
 Plan file: `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`.
 
@@ -124,4 +124,12 @@ R1 receive-message source reconciliation precheck is complete:
 
 - Created `docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md`.
 - Decision: do not change default receive routing yet; empty/`auto` remains PacketReader/log-backed and `msglib_readonly` remains explicit.
-- Next node: R2 contact/group search quality hardening.
+- R1 next node was R2 contact/group search quality hardening.
+
+
+R2 contact/group search quality hardening is complete:
+
+- Search contacts now has deterministic exact-match-first ranking and case-insensitive de-duplication for log/JID candidates and optional MsgLib display enrichment.
+- Search groups now has deterministic exact-match-first ranking and case-insensitive de-duplication for groupchat/conference candidates and optional MsgLib display enrichment.
+- MCP output still preserves `source` and `raw_ref`; no send or download behavior was introduced.
+- Next node: R3 receive-file download mapping precheck.

docs/go-mcp-runbook.md

@@ -96,8 +96,8 @@ The verification confirms:
 - `isphere_receive_messages` returns the contract-facing `ok`, `conversation`, `messages`, `next_cursor`, and `audit` envelope, while preserving legacy message aliases such as `id`, `text`, and `timestamp`.
 - A synthetic configured-source smoke check proves `ISPHERE_PACKET_LOG_FILE` can drive `isphere_receive_messages` end-to-end with one redacted fixture message.
 - A synthetic configured-directory smoke check proves `ISPHERE_PACKET_LOG_DIR` can drive `isphere_receive_messages`, contact search, group search, and file-list extraction from a temporary directory fixture.
-- `isphere_search_contacts` is callable and returns JID-derived contact candidates from the same configured synthetic message fixture.
-- `isphere_search_groups` is callable and returns JID-derived group candidates from the configured synthetic groupchat fixture.
+- `isphere_search_contacts` is callable and returns JID-derived contact candidates from the same configured synthetic message fixture; exact ID/display/account matches rank before prefix/substring matches and case-insensitive duplicates are collapsed.
+- `isphere_search_groups` is callable and returns JID-derived group candidates from the configured synthetic groupchat fixture; exact ID/display-name matches rank before prefix/substring matches and case-insensitive duplicates are collapsed.
 - `isphere_receive_files` is callable in list mode and returns one file metadata candidate from the configured synthetic file-transfer fixture.
 - Verification uses synthetic/local helper checks and does not load raw N12-pre evidence.
 - The deterministic verification path clears MsgLib env variables, so it proves the default no-MsgLib behavior remains stable.
@@ -246,8 +246,8 @@ These eight tools are currently allowed:
 | `win_helper_scan_windows` | helper `scan_windows` | Read visible window metadata or likely iSphere candidates. |
 | `win_helper_dump_uia` | helper `dump_uia` | Read a UI Automation tree for a specified window handle. |
 | `isphere_receive_messages` | log-backed `EncryptedPacketLogSource` | Read normalized messages from the configured PacketReader log source with a contract-shaped response envelope. The default server source is empty unless `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` is configured. |
-| `isphere_search_contacts` | log-backed message JIDs | Search contact candidates extracted from sender/receiver bare JIDs in the configured message source. |
-| `isphere_search_groups` | log-backed groupchat JIDs | Search group candidates extracted from groupchat/conference/MUC bare JIDs in the configured message source. |
+| `isphere_search_contacts` | log-backed message JIDs plus optional MsgLib display enrichment | Search contact candidates extracted from sender/receiver bare JIDs in the configured message source, with deterministic exact-match-first ranking, case-insensitive de-duplication, and preserved `source`/`raw_ref`. |
+| `isphere_search_groups` | log-backed groupchat JIDs plus optional MsgLib display enrichment | Search group candidates extracted from groupchat/conference/MUC bare JIDs in the configured message source, with deterministic exact-match-first ranking, case-insensitive de-duplication, and preserved `source`/`raw_ref`. |
 | `isphere_receive_files` | log-backed file-transfer messages | List file metadata candidates extracted from file-transfer message bodies. Download/cache mapping is not implemented yet. |
 
 Allowed parameters:
@@ -274,7 +274,7 @@ The current runbook verifies eight tools:
 - `isphere_search_groups`
 - `isphere_receive_files`
 
-The current log-backed business read tools have parser/source/tool registration and an empty default server source. Raw N12-pre evidence remains under ignored `runs/` paths and is not committed. The current command entry point can read an operator-local encrypted PacketReader log-line file via `ISPHERE_PACKET_LOG_FILE` or a directory of `.log`/`.txt` files via `ISPHERE_PACKET_LOG_DIR`; this is still not a production ingestion claim because raw evidence remains local and uncommitted. `isphere_receive_messages` now exposes contract-facing fields for digital-employee consumption while preserving older aliases for compatibility, supports `conversation_id`, keyword `query`, and RFC3339 `since` filtering, and validates the remaining safe contract args without pretending unsupported connectors/pagination exist. Contact and group search also accept the safe local-readonly contract args and can optionally enrich display metadata from `MsgLibReadSidecar`; receive-message `sender_name` and `conversation.display_name` can use that same optional metadata when MsgLib env is configured. `isphere_receive_files` supports list mode with safe contract arg validation; download/cache mapping remains deferred.
+The current log-backed business read tools have parser/source/tool registration and an empty default server source. Raw N12-pre evidence remains under ignored `runs/` paths and is not committed. The current command entry point can read an operator-local encrypted PacketReader log-line file via `ISPHERE_PACKET_LOG_FILE` or a directory of `.log`/`.txt` files via `ISPHERE_PACKET_LOG_DIR`; this is still not a production ingestion claim because raw evidence remains local and uncommitted. `isphere_receive_messages` now exposes contract-facing fields for digital-employee consumption while preserving older aliases for compatibility, supports `conversation_id`, keyword `query`, and RFC3339 `since` filtering, and validates the remaining safe contract args without pretending unsupported connectors/pagination exist. Contact and group search also accept the safe local-readonly contract args, rank exact matches before prefix/substring matches, collapse case-insensitive duplicates across log-derived candidates and optional `MsgLibReadSidecar` display metadata, and preserve `source`/`raw_ref`; receive-message `sender_name` and `conversation.display_name` can use that same optional metadata when MsgLib env is configured. `isphere_receive_files` supports list mode with safe contract arg validation; download/cache mapping remains deferred.
 
 Core business tools for contacts, groups, messages, and files are defined in `docs/mcp-core-tools-contract.md`. Send capabilities enter the roadmap through that contract after source discovery, connector selection, preview metadata, execution metadata, and audit records are in place.
 

docs/source-discovery/capability-source-matrix.md

@@ -18,8 +18,8 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md`
 | MCP tool | Primary source | Fallback source | Evidence file | Decision status | Next implementation slice |
 | --- | --- | --- | --- | --- | --- |
 | `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `<message>` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through x86 read-only sidecar `list_messages` selected explicitly by `source_preference="msglib_readonly"`; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md`; `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md`; `docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design; C38 implements sidecar/Go-wrapper `list_messages` with sanitized copied-DB verification; C39 adds a Go adapter from `ListMessages` to the receive-message domain model; C40 adds tool-level explicit `msglib_readonly` source selection; C41 wires env-configured MsgLib receive source and optional copied-DB smoke; R1 precheck keeps `auto` PacketReader/log-backed until a future reconciliation helper is implemented | future receive reconciliation helper after R2/R3 |
-| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md` | C34 documents how to enable optional MsgLib contact display enrichment and verify sanitized MCP output; search enrichment remains available | R2 contact search quality hardening |
-| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents how to enable optional MsgLib group display enrichment and verify sanitized MCP output; search enrichment remains available | R2 group search quality hardening |
+| `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; `internal/isphere/contacts_test.go`; `internal/tools/isphere_contacts_test.go` | C34 documents optional MsgLib contact display enrichment; R2 adds deterministic exact-match-first ranking, case-insensitive de-duplication across log/MsgLib candidates, and preserves `source`/`raw_ref` | core contact search complete; optional richer fields later |
+| `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; `internal/isphere/groups_test.go`; `internal/tools/isphere_groups_test.go`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents optional MsgLib group display enrichment; R2 adds deterministic exact-match-first ranking, case-insensitive de-duplication across log/MsgLib candidates, and preserves `source`/`raw_ref` | core group search complete; optional member/owner hierarchy later |
 | `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\<hash>` cache entries remain unlinked | decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck` | C22 verifies safe file-list contract args; no log-to-cache/download mapping yet | R3 receive-file download mapping precheck |
 | `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | R5 send-message connector discovery |
 | `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | R7 send-file connector discovery |
@@ -39,4 +39,4 @@ Start Stage C with a narrow log-backed `isphere_receive_messages` source abstrac
 
 ## Deferred source work
 
-`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, C37 defines the bounded DB-backed receive-source design, C38 implements sidecar/Go-wrapper `list_messages` with sanitized verification, C39 adds a Go adapter from MsgLib list results to the receive-message domain model, C40 adds explicit tool-level `msglib_readonly` source selection, C41 wires env-configured explicit MsgLib receive with optional sanitized smoke, and R1 documents source reconciliation keys. The active roadmap is `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`; next step is R2 contact/group search quality hardening.
+`MsgLib.db` has a validated copied read-only open path through the bundled 32-bit `System.Data.SQLite.dll` and password `123`. C27 adds `MsgLibReadSidecar` as the bounded x86 .NET reader boundary, C28 adds the Go `internal/msglib` process client, C29 adds bounded `display_entities` extraction, C30 wires it as optional contact/group MCP enrichment, C31 proves real copied-DB MCP enrichment with sanitized output, C32 reuses it for receive-message display fields, C33 proves that receive display path through a real copied-DB MCP smoke without printing entity values, C34 documents the operator setup/verification path, C35 maps MsgLib message tables to receive-message contract fields without reading row values, C36 adds metadata-only `message_sources` readiness, C37 defines the bounded DB-backed receive-source design, C38 implements sidecar/Go-wrapper `list_messages` with sanitized verification, C39 adds a Go adapter from MsgLib list results to the receive-message domain model, C40 adds explicit tool-level `msglib_readonly` source selection, C41 wires env-configured explicit MsgLib receive with optional sanitized smoke, R1 documents source reconciliation keys, and R2 hardens contact/group search ranking and de-duplication. The active roadmap is `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`; next step is R3 receive-file download mapping precheck.

docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md

@@ -25,8 +25,8 @@
 
 | Business goal | Current capability | Remaining blockers | Next node |
 | --- | --- | --- | --- |
-| Search contacts | Registered MCP tool; log/JID candidates; optional copied-DB MsgLib display enrichment. | Result quality hardening: de-duplication, ranking, stable IDs, optional richer fields. | R2 |
-| Search groups | Registered MCP tool; groupchat/conference candidates; optional copied-DB MsgLib display enrichment. | Group member count, owner refs, nested/group hierarchy quality. | R2 |
+| Search contacts | Registered MCP tool; deterministic exact-match ranking and case-insensitive de-duplication across log/JID candidates and optional copied-DB MsgLib display enrichment. | Optional richer fields such as department/title remain future enrichment, not a core search blocker. | complete for core search; optional enrichment later |
+| Search groups | Registered MCP tool; deterministic exact-match ranking and case-insensitive de-duplication across groupchat/conference candidates and optional copied-DB MsgLib display enrichment. | Optional member count, owner refs, and nested/group hierarchy remain future enrichment, not a core search blocker. | complete for core search; optional enrichment later |
 | Receive messages | PacketReader source works; copied `MsgLib.db` explicit receive works with `source_preference="msglib_readonly"`; R1 precheck says default should remain PacketReader until a future reconciliation helper is implemented. | Future reconciliation helper, source confidence scoring, pagination policy. | future receive helper after R2/R3 |
 | Send messages | Contract exists; no validated write connector. | Need connector discovery, preview-only path, idempotency and audit before production send. | R5 then R6 |
 | Receive files | List mode exists from message-derived file metadata. | Need message-to-cache/download mapping and output-dir download behavior. | R3 then R4 |
@@ -150,7 +150,7 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-msglib-mcp-en
 
 **Steps:**
 
-- [ ] **Step 1: Add failing contact ranking test**
+- [x] **Step 1: Add failing contact ranking test**
 
 Target behavior:
 
@@ -168,11 +168,11 @@ go test ./internal/isphere -run TestSearchContactsRanksExactAndDeduplicates -v
 
 Expected before implementation: FAIL because ranking/dedup behavior is not implemented.
 
-- [ ] **Step 2: Implement contact ranking/dedup**
+- [x] **Step 2: Implement contact ranking/dedup**
 
 Implement minimal deterministic ranking in `internal/isphere/contacts.go`.
 
-- [ ] **Step 3: Add failing group ranking test**
+- [x] **Step 3: Add failing group ranking test**
 
 Run:
 
@@ -182,7 +182,7 @@ go test ./internal/isphere -run TestSearchGroupsRanksExactAndDeduplicates -v
 
 Expected before implementation: FAIL because ranking/dedup behavior is not implemented.
 
-- [ ] **Step 4: Implement group ranking/dedup**
+- [x] **Step 4: Implement group ranking/dedup**
 
 Implement minimal deterministic ranking in `internal/isphere/groups.go`.
 
@@ -198,6 +198,14 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1
 - Search contacts and search groups produce deterministic ranked results.
 - No send behavior is introduced.
 
+**R2 Result:**
+
+- Added failing-then-passing tests for contact and group exact-match ranking and case-insensitive de-duplication in `internal/isphere`.
+- Added failing-then-passing MCP tool tests proving optional MsgLib display entities and log-derived fallback entries merge into one de-duplicated result while preserving `source` and `raw_ref`.
+- Implemented deterministic ranking: exact ID/display/account match first, prefix match second, substring match after that, then stable case-insensitive ID ordering.
+- No send-message, send-file, file-download, hook, DB-write, or production side-effect behavior was introduced.
+- Decision: next business node is R3 receive-file download mapping precheck.
+
 ---
 
 ### R3: Receive-file download mapping precheck