diff --git a/docs/current-status-card.md b/docs/current-status-card.md index 9873aaf..39deced 100644 --- a/docs/current-status-card.md +++ b/docs/current-status-card.md @@ -56,7 +56,7 @@ N13/N14/N15 are pre-business validation results. They can help identify UI eleme ## Current loop -Current loop: `Business Roadmap / R3b - File cache mapping diagnostic`. +Current loop: `Business Roadmap / R5 - Send-message connector discovery`. Plan file: `docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md`. @@ -139,3 +139,10 @@ R3 receive-file download mapping precheck is complete: - Created `docs/source-discovery/2026-07-10-file-download-mapping-precheck.md`. - Decision: do not implement download mode yet; current evidence supports list mode and DB attachment metadata, but not a validated cache/download mapping. - Next node: R3b file cache mapping diagnostic with sanitized counts only. + +R3b file cache mapping diagnostic is complete: + +- Added `scripts/verify-file-cache-mapping.ps1` and a fixture test script. +- Real sanitized diagnostic found 166 MsgLib received-file records and 11 cache/archive candidates, but 0 filename/source-id/cache matches. +- Decision: R4 receive-file download implementation is blocked until better cache evidence or a UI/client download connector is available. +- Next node: R5 send-message connector discovery. diff --git a/docs/source-discovery/2026-07-10-file-cache-mapping-diagnostic.md b/docs/source-discovery/2026-07-10-file-cache-mapping-diagnostic.md new file mode 100644 index 0000000..ff0d463 --- /dev/null +++ b/docs/source-discovery/2026-07-10-file-cache-mapping-diagnostic.md @@ -0,0 +1,73 @@ +# File Cache Mapping Diagnostic + +Date: 2026-07-10 + +## Goal + +Run the R3b local-only diagnostic to determine whether copied MsgLib received-file metadata can be mapped to the available ignored cache/archive listing without printing file paths, raw rows, message bodies, or attachment contents. + +## Diagnostic command shape + +The diagnostic script is: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-file-cache-mapping.ps1 \ + -ArchiveListPath +``` + +The script can also accept `ISPHERE_MSGLIB_SQLITE_DLL` and `ISPHERE_MSGLIB_DB` from the environment. The committed note intentionally omits local DB, DLL, and archive-list path values. + +## Sanitized result + +Safe summary from the available copied DB plus ignored archive-list metadata: + +```json +{ + "ok": true, + "verification": "file_cache_mapping", + "db_checked": true, + "db_table_available": true, + "db_record_count": 166, + "db_filename_present_count": 166, + "db_size_present_count": 166, + "db_source_id_present_count": 166, + "db_receive_msg_guid_present_count": 166, + "cache_checked": true, + "cache_candidate_count": 11, + "hash_dir_length_counts": {"32": 11}, + "filename_only_match_count": 0, + "filename_size_match_count": 0, + "unique_filename_size_match_count": 0, + "source_id_hash_dir_match_count": 0, + "mapping_validated": false, + "recommended_next_node": "additional evidence request", + "raw_paths_printed": false, + "file_contents_read": false, + "file_paths_returned": false, + "raw_rows_returned": false, + "message_body_values_printed": false +} +``` + +Extension counts are allowed as coarse metadata. The diagnostic normalizes abnormal extensions to `.other` so file-name fragments are not printed. + +## Decision + +Do not implement R4 receive-file download mode from the current evidence. + +Reason: + +- MsgLib received-file metadata exists and is internally complete enough for metadata analysis: filename, size, source id, and receive message id are present for 166 inspected rows. +- The available cache/archive listing has 11 hash-shape cache candidates. +- The diagnostic found 0 filename-only matches, 0 filename+size matches, and 0 source-id-to-hash-dir matches. +- Therefore no unique or repeatable cache-file resolver can be claimed. + +## Next route + +File download remains blocked until one of these is available: + +1. A better operator-local cache root or fresh archive-list that actually contains the same received files as `TD_ReceiveFileRecord`. +2. A known recent received-file sample where the operator can confirm the message/file metadata and resulting local cache object, with committed output still limited to counts and booleans. +3. A UI/client download connector path that can trigger download without relying on cache guessing. + +The broader roadmap should skip R4 for now and continue with R5 send-message connector discovery. diff --git a/docs/source-discovery/capability-source-matrix.md b/docs/source-discovery/capability-source-matrix.md index 0b244b0..0778354 100644 --- a/docs/source-discovery/capability-source-matrix.md +++ b/docs/source-discovery/capability-source-matrix.md @@ -20,7 +20,7 @@ Schema notes: `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md` | `isphere_receive_messages` | decrypted `PacketReader.ProcessPacket` XMPP `` stanzas via `internal/isphere.EncryptedPacketLogSource`; configured by `ISPHERE_PACKET_LOG_FILE` or `ISPHERE_PACKET_LOG_DIR` | copied `MsgLib.db` message tables through x86 read-only sidecar `list_messages` selected explicitly by `source_preference="msglib_readonly"`; decrypted `Smark.SendReceive` transport stanzas; decrypted `SaveToDB` `Chat_OnMessageArrived` traces | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#field-mapping-evidence`; `docs/source-discovery/2026-07-10-dotnet-wrapper-static-analysis.md`; `docs/source-discovery/2026-07-10-db-backed-receive-source-design.md`; `docs/source-discovery/2026-07-10-receive-message-reconciliation-precheck.md` | C20 supports and validates the current safe receive-message contract args; C32 wires optional MsgLib display metadata into receive-message `sender_name` and `conversation.display_name`; C33 proves this path through a real copied-DB MCP smoke; C35 maps MsgLib message tables; C36 adds metadata-only `message_sources`; C37 defines bounded `list_messages` design; C38 implements sidecar/Go-wrapper `list_messages` with sanitized copied-DB verification; C39 adds a Go adapter from `ListMessages` to the receive-message domain model; C40 adds tool-level explicit `msglib_readonly` source selection; C41 wires env-configured MsgLib receive source and optional copied-DB smoke; R1 precheck keeps `auto` PacketReader/log-backed until a future reconciliation helper is implemented | future receive reconciliation helper after R2/R3 | | `isphere_search_contacts` | bare sender/receiver JIDs extracted from normalized log-backed messages loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `TD_Roster`, `TD_CustomEffigy`, `tblRecent`, `tblChatLevel`, `tblPersonMsg`; decrypted roster/contact stanzas from `Smark.SendReceive`; UIA helper source if display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; `internal/isphere/contacts_test.go`; `internal/tools/isphere_contacts_test.go` | C34 documents optional MsgLib contact display enrichment; R2 adds deterministic exact-match-first ranking, case-insensitive de-duplication across log/MsgLib candidates, and preserves `source`/`raw_ref` | core contact search complete; optional richer fields later | | `isphere_search_groups` | decrypted `PacketReader.ProcessPacket` `type="groupchat"` stanzas and conference/MUC JIDs loaded from configured PacketReader file or directory source | validated copied `MsgLib.db` tables: `tblMsgGroupPersonMsg`, `TD_WorkGroupAuth`, `tblRecent`; UIA helper source if group display names are still missing | `docs/source-discovery/2026-07-10-msglib-readonly-schema-extraction.md`; `internal/isphere/groups_test.go`; `internal/tools/isphere_groups_test.go`; C9 ignored-log scan: PacketReader 86 groupchat/86 conference hits; Smark 24 groupchat/658 conference/631 muc hits | C34 documents optional MsgLib group display enrichment; R2 adds deterministic exact-match-first ranking, case-insensitive de-duplication across log/MsgLib candidates, and preserves `source`/`raw_ref` | core group search complete; optional member/owner hierarchy later | -| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | MsgLib `TD_ReceiveFileRecord` safe metadata through explicit copied-DB path; decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck`; `docs/source-discovery/2026-07-10-file-download-mapping-precheck.md` | C22 verifies safe file-list contract args; R3 confirms list metadata is available but no validated message/file-record-to-cache-file mapping exists; R4 remains blocked | R3b file cache mapping diagnostic | +| `isphere_receive_files` | decrypted `PacketReader.ProcessPacket` file-transfer message stanzas via `internal/isphere.ListFilesFromMessages` and `isphere_receive_files` list mode; configured PacketReader file or directory source; `zyl\importal\` cache entries remain unlinked | MsgLib `TD_ReceiveFileRecord` safe metadata through explicit copied-DB path; decrypted `Smark.SendReceive` and `SaveToDB` traces for file-reference reconciliation; future UIA/client connector for download | `docs/source-discovery/2026-07-09-n12-pre-zyl-schema-notes.md#c12-file-transfer-evidence-precheck`; `docs/source-discovery/2026-07-10-file-download-mapping-precheck.md`; `docs/source-discovery/2026-07-10-file-cache-mapping-diagnostic.md` | C22 verifies safe file-list contract args; R3 confirms list metadata is available; R3b diagnostic finds 0 current cache/archive matches, so R4 remains blocked | blocked pending better cache evidence or UI/client download connector | | `isphere_send_message` | none validated yet | existing bridge/API/local service preferred; UIA write connector only after internal-sandbox action evidence; network/protocol connector only after protocol discovery | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | C15 blocked: contract exists, but no committed bridge/API/local service, helper write op, UIA action chain, or protocol connector is validated | R5 send-message connector discovery | | `isphere_send_file` | none validated yet | depends on send-message connector plus outbound file/upload evidence | `docs/source-discovery/2026-07-10-send-message-source-precheck.md` | blocked behind `isphere_send_message` connector selection and file upload/source policy | R7 send-file connector discovery | diff --git a/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md b/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md index 5c1dd87..7e87dc6 100644 --- a/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md +++ b/docs/superpowers/plans/2026-07-10-core-business-capabilities-roadmap.md @@ -29,7 +29,7 @@ | Search groups | Registered MCP tool; deterministic exact-match ranking and case-insensitive de-duplication across groupchat/conference candidates and optional copied-DB MsgLib display enrichment. | Optional member count, owner refs, and nested/group hierarchy remain future enrichment, not a core search blocker. | complete for core search; optional enrichment later | | Receive messages | PacketReader source works; copied `MsgLib.db` explicit receive works with `source_preference="msglib_readonly"`; R1 precheck says default should remain PacketReader until a future reconciliation helper is implemented. | Future reconciliation helper, source confidence scoring, pagination policy. | future receive helper after R2/R3 | | Send messages | Contract exists; no validated write connector. | Need connector discovery, preview-only path, idempotency and audit before production send. | R5 then R6 | -| Receive files | List mode exists from message-derived file metadata; R3 precheck confirms cache/download mapping is still unvalidated. | Need sanitized cache mapping diagnostic before output-dir download behavior. | R3b then R4 if validated | +| Receive files | List mode exists from message-derived file metadata; R3/R3b confirm DB file metadata exists but current cache/archive evidence does not map to local files. | R4 download is blocked until better cache evidence or a UI/client download connector is available. | blocked; continue R5 | | Send files | Contract exists; no validated upload/send connector. | Depends on send-message connector plus upload/file-transfer evidence. | R7 then R8 | ## Node Sequence @@ -259,6 +259,15 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts\verify-go-mcp.ps1 **Stop condition:** - If no unique/repeatable cache mapping exists, do not implement R4 download mode. +**R3b Result:** + +- Added `scripts/verify-file-cache-mapping.ps1` and `scripts/test-verify-file-cache-mapping.ps1`. +- Created `docs/source-discovery/2026-07-10-file-cache-mapping-diagnostic.md`. +- Real sanitized diagnostic checked 166 `TD_ReceiveFileRecord` rows and 11 ignored cache/archive candidates. +- Match counts were all zero: filename-only, filename+size, unique filename+size, and source-id-to-hash-dir. +- Safety booleans stayed false for raw paths, file contents, file paths returned, raw rows, and message body printing. +- Decision: R4 remains blocked; continue the broader roadmap at R5 send-message connector discovery. + --- ### R4: Receive-file download implementation diff --git a/scripts/test-verify-file-cache-mapping.ps1 b/scripts/test-verify-file-cache-mapping.ps1 new file mode 100644 index 0000000..7164b33 --- /dev/null +++ b/scripts/test-verify-file-cache-mapping.ps1 @@ -0,0 +1,43 @@ +param() + +$ErrorActionPreference = "Stop" +Set-StrictMode -Version Latest + +$repo = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..")).Path +$script = Join-Path $repo "scripts\verify-file-cache-mapping.ps1" +$tempDir = Join-Path $repo ("runs\file-cache-mapping-test-" + [guid]::NewGuid().ToString("N")) +$archiveList = Join-Path $tempDir "archive-list.txt" + +function Assert-True([bool]$Condition, [string]$Message) { + if (-not $Condition) { throw $Message } +} + +try { + New-Item -ItemType Directory -Force -Path $tempDir | Out-Null + @( + "C:\redacted\importal\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\redacted-report.docx", + "C:\redacted\importal\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\redacted-sheet.xlsx", + "C:\redacted\other\ignored.txt" + ) | Set-Content -LiteralPath $archiveList -Encoding UTF8 + + $output = & powershell -NoProfile -ExecutionPolicy Bypass -File $script -ArchiveListPath $archiveList -SkipDb + if ($LASTEXITCODE -ne 0) { + throw "verify-file-cache-mapping.ps1 failed with exit code $LASTEXITCODE" + } + $json = $output | ConvertFrom-Json + + Assert-True ($json.ok -eq $true) "ok should be true" + Assert-True ($json.cache_checked -eq $true) "cache should be checked" + Assert-True ($json.cache_candidate_count -eq 2) "cache candidate count should be 2" + Assert-True ($json.cache_extension_counts.".docx" -eq 1) "docx extension count should be 1" + Assert-True ($json.cache_extension_counts.".xlsx" -eq 1) "xlsx extension count should be 1" + Assert-True ($json.hash_dir_length_counts."32" -eq 2) "hash dir length count should be 2" + Assert-True ($json.raw_paths_printed -eq $false) "raw paths must not be printed" + Assert-True ($json.file_contents_read -eq $false) "file contents must not be read" + Assert-True ($json.file_paths_returned -eq $false) "file paths must not be returned" +} +finally { + if (Test-Path -LiteralPath $tempDir) { + Remove-Item -LiteralPath $tempDir -Recurse -Force + } +} diff --git a/scripts/verify-file-cache-mapping.ps1 b/scripts/verify-file-cache-mapping.ps1 new file mode 100644 index 0000000..f129d31 --- /dev/null +++ b/scripts/verify-file-cache-mapping.ps1 @@ -0,0 +1,552 @@ +param( + [string]$ArchiveListPath = "", + [string]$CacheRoot = "", + [string]$SQLiteDllPath = $env:ISPHERE_MSGLIB_SQLITE_DLL, + [string]$DbPath = $env:ISPHERE_MSGLIB_DB, + [string]$Password = "123", + [switch]$SkipDb +) + +$ErrorActionPreference = "Stop" +Set-StrictMode -Version Latest + +$repo = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..")).Path + +function Add-Count([hashtable]$Map, [string]$Key, [int]$Increment = 1) { + if ([string]::IsNullOrWhiteSpace($Key)) { return } + if ($Map.ContainsKey($Key)) { + $Map[$Key] = [int]$Map[$Key] + $Increment + } + else { + $Map[$Key] = $Increment + } +} + +function Get-Sha256Text([string]$Value) { + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + $bytes = [System.Text.Encoding]::UTF8.GetBytes($Value) + $hash = $sha.ComputeHash($bytes) + return (($hash | ForEach-Object { $_.ToString("x2") }) -join "") + } + finally { + $sha.Dispose() + } +} + +function Normalize-FileName([string]$Value) { + if ([string]::IsNullOrWhiteSpace($Value)) { return "" } + $leaf = ($Value -split '[\\/]')[-1] + return $leaf.Trim().ToLowerInvariant() +} + +function Normalize-Extension([string]$Value) { + if ([string]::IsNullOrWhiteSpace($Value)) { return "" } + $extension = $Value.Trim().ToLowerInvariant() + if ($extension -match '^\.[a-z0-9]{1,8}$') { return $extension } + return ".other" +} + +function New-EmptyCacheSummary { + return [ordered]@{ + checked = $false + candidate_count = 0 + extension_counts = @{} + hash_dir_length_counts = @{} + hash_dir_value_hash_counts = @{} + filename_hash_counts = @{} + filename_size_hash_counts = @{} + has_size_data = $false + } +} + +function Add-CacheCandidate($Summary, [string]$Name, [Nullable[Int64]]$Size, [string]$HashDir) { + $normalizedName = Normalize-FileName $Name + if ($normalizedName -eq "") { return } + $extension = Normalize-Extension ([System.IO.Path]::GetExtension($normalizedName)) + if ($extension -ne "") { + Add-Count $Summary.extension_counts $extension + } + if (-not [string]::IsNullOrWhiteSpace($HashDir)) { + Add-Count $Summary.hash_dir_length_counts ([string]$HashDir.Length) + Add-Count $Summary.hash_dir_value_hash_counts (Get-Sha256Text $HashDir.ToLowerInvariant()) + } + Add-Count $Summary.filename_hash_counts (Get-Sha256Text $normalizedName) + if ($null -ne $Size -and $Size.Value -ge 0) { + $Summary["has_size_data"] = $true + Add-Count $Summary.filename_size_hash_counts (Get-Sha256Text ($normalizedName + "|" + $Size.Value)) + } + $Summary["candidate_count"] = [int]$Summary["candidate_count"] + 1 +} + +function Get-HashDirFromPathText([string]$Value) { + $match = [regex]::Match($Value, '(?i)(?:^|[\\/])importal[\\/]([0-9a-f]{32,})(?:[\\/])') + if ($match.Success) { + return $match.Groups[1].Value.ToLowerInvariant() + } + return "" +} + +function Read-CacheSummaryFromArchiveList([string]$Path) { + if (-not (Test-Path -LiteralPath $Path)) { + throw "ArchiveListPath not found" + } + $summary = New-EmptyCacheSummary + $summary["checked"] = $true + foreach ($line in Get-Content -LiteralPath $Path) { + $text = [string]$line + $hashDir = Get-HashDirFromPathText $text + if ($hashDir -eq "") { continue } + Add-CacheCandidate $summary $text $null $hashDir + } + return $summary +} + +function Read-CacheSummaryFromRoot([string]$Path) { + if (-not (Test-Path -LiteralPath $Path)) { + throw "CacheRoot not found" + } + $summary = New-EmptyCacheSummary + $summary["checked"] = $true + foreach ($file in Get-ChildItem -LiteralPath $Path -File -Recurse) { + $hashDir = Get-HashDirFromPathText $file.FullName + if ($hashDir -eq "") { continue } + Add-CacheCandidate $summary $file.Name ([Int64]$file.Length) $hashDir + } + return $summary +} + +function Convert-CountObjectToHashtable($Value) { + $map = @{} + if ($null -eq $Value) { return $map } + if ($Value -is [System.Collections.IDictionary]) { + foreach ($key in $Value.Keys) { + $map[[string]$key] = [int]$Value[$key] + } + return $map + } + foreach ($property in $Value.PSObject.Properties) { + $map[$property.Name] = [int]$property.Value + } + return $map +} + +function Count-Intersections([hashtable]$Left, [hashtable]$Right) { + $count = 0 + foreach ($key in $Left.Keys) { + if ($Right.ContainsKey($key)) { + $count += [Math]::Min([int]$Left[$key], [int]$Right[$key]) + } + } + return $count +} + +function Count-UniqueIntersections([hashtable]$Left, [hashtable]$Right) { + $count = 0 + foreach ($key in $Left.Keys) { + if ($Right.ContainsKey($key) -and [int]$Left[$key] -eq 1 -and [int]$Right[$key] -eq 1) { + $count++ + } + } + return $count +} + +function Invoke-ReceiveFileRecordProbe([string]$SQLiteDll, [string]$Database, [string]$DbPassword) { + if (-not (Test-Path -LiteralPath $SQLiteDll)) { + throw "SQLite DLL not found" + } + if (-not (Test-Path -LiteralPath $Database)) { + throw "MsgLib DB not found" + } + + $tempDir = Join-Path $repo ("runs\file-cache-mapping-db-probe-" + [guid]::NewGuid().ToString("N")) + $source = Join-Path $tempDir "ReceiveFileRecordProbe.cs" + $exe = Join-Path $tempDir "ReceiveFileRecordProbe.exe" + $sqliteCopy = Join-Path $tempDir "System.Data.SQLite.dll" + $csc = "$env:WINDIR\Microsoft.NET\Framework\v4.0.30319\csc.exe" + if (-not (Test-Path -LiteralPath $csc)) { + throw "32-bit .NET Framework csc.exe v4.0.30319 not found" + } + + try { + New-Item -ItemType Directory -Force -Path $tempDir | Out-Null + Copy-Item -LiteralPath $SQLiteDll -Destination $sqliteCopy -Force + @' +using System; +using System.Collections.Generic; +using System.Data.Common; +using System.IO; +using System.Reflection; +using System.Security.Cryptography; +using System.Text; +using System.Web.Script.Serialization; + +internal static class ReceiveFileRecordProbe +{ + private static int Main(string[] args) + { + try + { + string sqliteDllPath = args[0]; + string dbPath = args[1]; + string password = args.Length > 2 ? args[2] : "123"; + AppDomain.CurrentDomain.AssemblyResolve += delegate(object sender, ResolveEventArgs eventArgs) + { + if (eventArgs.Name.StartsWith("System.Data.SQLite", StringComparison.OrdinalIgnoreCase)) + { + return Assembly.LoadFrom(sqliteDllPath); + } + return null; + }; + + Assembly sqliteAssembly = Assembly.LoadFrom(sqliteDllPath); + Type factoryType = sqliteAssembly.GetType("System.Data.SQLite.SQLiteFactory", true); + FieldInfo instanceField = factoryType.GetField("Instance", BindingFlags.Public | BindingFlags.Static); + DbProviderFactory factory = (DbProviderFactory)instanceField.GetValue(null); + + var result = NewResult(); + using (DbConnection connection = factory.CreateConnection()) + { + connection.ConnectionString = "Data Source=" + dbPath + ";Version=3;Read Only=True;Password=" + password + ";"; + connection.Open(); + ExecuteNonQuery(connection, "PRAGMA query_only=ON"); + if (!TableExists(connection, "TD_ReceiveFileRecord")) + { + PrintJson(result); + return 0; + } + result["table_available"] = true; + HashSet columns = ReadColumns(connection, "TD_ReceiveFileRecord"); + string msgColumn = ExistingColumn(columns, "ReceiveMsgGuid"); + string nameColumn = ExistingColumn(columns, "FileName"); + string sizeColumn = ExistingColumn(columns, "FileSize", "Size"); + string sourceColumn = ExistingColumn(columns, "SourceID", "FileGuid", "FileId"); + if (nameColumn == "") + { + PrintJson(result); + return 0; + } + + string sql = "select " + + SelectColumn(msgColumn, "receive_msg_guid") + ", " + + SelectColumn(nameColumn, "file_name") + ", " + + SelectColumn(sizeColumn, "size_bytes") + ", " + + SelectColumn(sourceColumn, "source_id") + + " from " + QuoteIdent("TD_ReceiveFileRecord") + " limit 100000"; + + using (DbCommand command = connection.CreateCommand()) + { + command.CommandText = sql; + using (DbDataReader reader = command.ExecuteReader()) + { + while (reader.Read()) + { + result["record_count"] = (int)result["record_count"] + 1; + string receiveMsgGuid = ReaderString(reader, "receive_msg_guid"); + string fileName = SafeFileName(ReaderString(reader, "file_name")); + long size = ReaderInt64(reader, "size_bytes"); + string sourceId = ReaderString(reader, "source_id").Trim().ToLowerInvariant(); + + if (receiveMsgGuid.Trim() != "") result["receive_msg_guid_present_count"] = (int)result["receive_msg_guid_present_count"] + 1; + if (fileName.Trim() != "") result["filename_present_count"] = (int)result["filename_present_count"] + 1; + if (size > 0) result["size_present_count"] = (int)result["size_present_count"] + 1; + if (sourceId != "") result["source_id_present_count"] = (int)result["source_id_present_count"] + 1; + + string normalizedName = fileName.Trim().ToLowerInvariant(); + if (normalizedName != "") + { + AddCount((Dictionary)result["extension_counts"], SafeExtension(Path.GetExtension(normalizedName))); + AddCount((Dictionary)result["filename_hash_counts"], Sha256(normalizedName)); + if (size > 0) AddCount((Dictionary)result["filename_size_hash_counts"], Sha256(normalizedName + "|" + size.ToString())); + } + if (sourceId != "") + { + AddCount((Dictionary)result["source_id_hash_counts"], Sha256(sourceId)); + if (IsHexLike(sourceId)) result["source_id_hash_shape_count"] = (int)result["source_id_hash_shape_count"] + 1; + } + } + } + } + } + PrintJson(result); + return 0; + } + catch (Exception ex) + { + Console.Error.WriteLine(ex.GetType().Name + ": " + ex.Message); + return 1; + } + } + + private static Dictionary NewResult() + { + return new Dictionary + { + { "table_available", false }, + { "record_count", 0 }, + { "filename_present_count", 0 }, + { "size_present_count", 0 }, + { "source_id_present_count", 0 }, + { "receive_msg_guid_present_count", 0 }, + { "source_id_hash_shape_count", 0 }, + { "extension_counts", new Dictionary() }, + { "filename_hash_counts", new Dictionary() }, + { "filename_size_hash_counts", new Dictionary() }, + { "source_id_hash_counts", new Dictionary() } + }; + } + + private static void PrintJson(Dictionary value) + { + Console.WriteLine(new JavaScriptSerializer().Serialize(value)); + } + + private static void AddCount(Dictionary map, string key) + { + if (string.IsNullOrWhiteSpace(key)) return; + if (map.ContainsKey(key)) map[key] = (int)map[key] + 1; + else map[key] = 1; + } + + private static bool TableExists(DbConnection connection, string table) + { + using (DbCommand command = connection.CreateCommand()) + { + command.CommandText = "select count(*) from sqlite_master where type='table' and name=@name"; + DbParameter parameter = command.CreateParameter(); + parameter.ParameterName = "@name"; + parameter.Value = table; + command.Parameters.Add(parameter); + return Convert.ToInt32(command.ExecuteScalar()) > 0; + } + } + + private static HashSet ReadColumns(DbConnection connection, string table) + { + var columns = new HashSet(StringComparer.OrdinalIgnoreCase); + using (DbCommand command = connection.CreateCommand()) + { + command.CommandText = "pragma table_info(" + QuoteIdent(table) + ")"; + using (DbDataReader reader = command.ExecuteReader()) + { + while (reader.Read()) columns.Add(Convert.ToString(reader["name"])); + } + } + return columns; + } + + private static string ExistingColumn(HashSet columns, params string[] names) + { + foreach (string name in names) + { + if (!string.IsNullOrWhiteSpace(name) && columns.Contains(name)) return name; + } + return ""; + } + + private static string SelectColumn(string column, string alias) + { + if (string.IsNullOrWhiteSpace(column)) return "'' as " + QuoteIdent(alias); + return QuoteIdent(column) + " as " + QuoteIdent(alias); + } + + private static string QuoteIdent(string value) + { + return "\"" + value.Replace("\"", "\"\"") + "\""; + } + + private static string ReaderString(DbDataReader reader, string column) + { + object value = reader[column]; + if (value == null || value == DBNull.Value) return ""; + return Convert.ToString(value); + } + + private static long ReaderInt64(DbDataReader reader, string column) + { + object value = reader[column]; + if (value == null || value == DBNull.Value) return 0; + long parsed; + if (long.TryParse(Convert.ToString(value), out parsed)) return parsed; + return 0; + } + + private static string SafeFileName(string value) + { + if (string.IsNullOrWhiteSpace(value)) return ""; + try { return Path.GetFileName(value); } + catch + { + int slash = Math.Max(value.LastIndexOf('/'), value.LastIndexOf('\\')); + if (slash >= 0 && slash + 1 < value.Length) return value.Substring(slash + 1); + return value; + } + } + + private static string SafeExtension(string value) + { + if (string.IsNullOrWhiteSpace(value)) return ""; + string extension = value.Trim().ToLowerInvariant(); + if (extension.Length < 2 || extension.Length > 9) return ".other"; + if (extension[0] != '.') return ".other"; + for (int i = 1; i < extension.Length; i++) + { + char c = extension[i]; + bool ok = (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9'); + if (!ok) return ".other"; + } + return extension; + } + + private static bool IsHexLike(string value) + { + if (value.Length < 16) return false; + foreach (char c in value) + { + bool ok = (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F'); + if (!ok) return false; + } + return true; + } + + private static string Sha256(string value) + { + using (SHA256 sha = SHA256.Create()) + { + byte[] hash = sha.ComputeHash(Encoding.UTF8.GetBytes(value)); + var builder = new StringBuilder(hash.Length * 2); + foreach (byte b in hash) builder.Append(b.ToString("x2")); + return builder.ToString(); + } + } + + private static void ExecuteNonQuery(DbConnection connection, string sql) + { + using (DbCommand command = connection.CreateCommand()) + { + command.CommandText = sql; + command.ExecuteNonQuery(); + } + } +} +'@ | Set-Content -LiteralPath $source -Encoding UTF8 + + & $csc ` + /nologo ` + /target:exe ` + /platform:x86 ` + /optimize+ ` + /warn:0 ` + /out:$exe ` + /reference:System.dll ` + /reference:System.Core.dll ` + /reference:System.Data.dll ` + /reference:System.Web.Extensions.dll ` + /reference:$sqliteCopy ` + $source | Out-Null + if ($LASTEXITCODE -ne 0) { + throw "ReceiveFileRecordProbe compilation failed" + } + @' + + + + + + + +'@ | Set-Content -LiteralPath ($exe + ".config") -Encoding UTF8 + + $output = & $exe $sqliteCopy $Database $DbPassword + if ($LASTEXITCODE -ne 0) { + throw "ReceiveFileRecordProbe failed" + } + return $output | ConvertFrom-Json + } + finally { + if (Test-Path -LiteralPath $tempDir) { + Remove-Item -LiteralPath $tempDir -Recurse -Force + } + } +} + +$cacheSummary = New-EmptyCacheSummary +if (-not [string]::IsNullOrWhiteSpace($ArchiveListPath)) { + $cacheSummary = Read-CacheSummaryFromArchiveList $ArchiveListPath +} +elseif (-not [string]::IsNullOrWhiteSpace($CacheRoot)) { + $cacheSummary = Read-CacheSummaryFromRoot $CacheRoot +} + +$dbChecked = $false +$dbProbe = [ordered]@{ + table_available = $false + record_count = 0 + filename_present_count = 0 + size_present_count = 0 + source_id_present_count = 0 + receive_msg_guid_present_count = 0 + source_id_hash_shape_count = 0 + extension_counts = @{} + filename_hash_counts = @{} + filename_size_hash_counts = @{} + source_id_hash_counts = @{} +} + +if (-not $SkipDb) { + if ([string]::IsNullOrWhiteSpace($SQLiteDllPath) -or [string]::IsNullOrWhiteSpace($DbPath)) { + throw "MsgLib DB probe requires SQLiteDllPath and DbPath, or pass -SkipDb" + } + $dbProbe = Invoke-ReceiveFileRecordProbe $SQLiteDllPath $DbPath $Password + $dbChecked = $true +} + +$dbFilenameHashes = Convert-CountObjectToHashtable $dbProbe.filename_hash_counts +$dbFilenameSizeHashes = Convert-CountObjectToHashtable $dbProbe.filename_size_hash_counts +$dbSourceIDHashes = Convert-CountObjectToHashtable $dbProbe.source_id_hash_counts +$cacheFilenameHashes = [hashtable]$cacheSummary["filename_hash_counts"] +$cacheFilenameSizeHashes = [hashtable]$cacheSummary["filename_size_hash_counts"] +$cacheHashDirValueHashes = [hashtable]$cacheSummary["hash_dir_value_hash_counts"] + +$filenameOnlyMatches = Count-Intersections $dbFilenameHashes $cacheFilenameHashes +$filenameSizeMatches = Count-Intersections $dbFilenameSizeHashes $cacheFilenameSizeHashes +$uniqueFilenameSizeMatches = Count-UniqueIntersections $dbFilenameSizeHashes $cacheFilenameSizeHashes +$sourceIDHashDirMatches = Count-Intersections $dbSourceIDHashes $cacheHashDirValueHashes + +$mappingValidated = ($uniqueFilenameSizeMatches -gt 0 -and [bool]$cacheSummary["has_size_data"]) +$recommendedNextNode = "additional evidence request" +if ($mappingValidated) { + $recommendedNextNode = "R4 cache-file resolver candidate" +} +elseif ($filenameOnlyMatches -gt 0 -or $sourceIDHashDirMatches -gt 0) { + $recommendedNextNode = "additional evidence request: collect file size or cache root metadata" +} + +[ordered]@{ + ok = $true + verification = "file_cache_mapping" + db_checked = $dbChecked + db_table_available = [bool]$dbProbe.table_available + db_record_count = [int]$dbProbe.record_count + db_filename_present_count = [int]$dbProbe.filename_present_count + db_size_present_count = [int]$dbProbe.size_present_count + db_source_id_present_count = [int]$dbProbe.source_id_present_count + db_receive_msg_guid_present_count = [int]$dbProbe.receive_msg_guid_present_count + db_source_id_hash_shape_count = [int]$dbProbe.source_id_hash_shape_count + db_extension_counts = $dbProbe.extension_counts + cache_checked = [bool]$cacheSummary["checked"] + cache_candidate_count = [int]$cacheSummary["candidate_count"] + cache_extension_counts = $cacheSummary["extension_counts"] + hash_dir_length_counts = $cacheSummary["hash_dir_length_counts"] + filename_only_match_count = $filenameOnlyMatches + filename_size_match_count = $filenameSizeMatches + unique_filename_size_match_count = $uniqueFilenameSizeMatches + source_id_hash_dir_match_count = $sourceIDHashDirMatches + mapping_validated = $mappingValidated + recommended_next_node = $recommendedNextNode + raw_paths_printed = $false + file_contents_read = $false + file_paths_returned = $false + raw_rows_returned = $false + message_body_values_printed = $false +} | ConvertTo-Json -Depth 8 -Compress