Add archive provenance and consolidation queries · 8f0603b

Add archive provenance and consolidation queries
Browse files

bogdan committed 2 weeks ago

main

1 parent 8d05fdb

commit 8f0603b

Showing 6 changed files with 482 additions and 19 deletions

+6 -3

HealthProbe/Doc/00-agent-guides/AGENTS.md

@@ -193,13 +193,16 @@ final class TypeDistributionBin {
 // Deletions are recorded by sampleUUIDHash because HKDeletedObject exposes UUIDs,
 // not complete sample payloads.
 
-// Interface updated 2026-05-23 — see AGENTS.md
+// Interface updated 2026-05-24 — see AGENTS.md
 // HealthArchiveStore exposes SQL-first observation diff APIs:
 // diffSummary(_:) returns appeared/disappeared/representationChanged counts and
 // diffRecords(_:) returns a paged record list for one change kind.
 // aggregateComparison(_:) compares materialized daily aggregates between two
-// observations. UI/cache agents should consume these APIs instead of loading full
-// observation record sets.
+// observations. sourceProvenanceBreakdown(_:) returns visible source/device
+// composition, and consolidationEvidence(_:) returns conservative evidence labels
+// with counts, aggregate sums, coverage, density, source compatibility, and
+// uncertainty text. UI/cache agents should consume these APIs instead of loading
+// full observation record sets.
 
 // Storage objective updated 2026-05-23 — see AGENTS.md
 // Recurring complete snapshots are out of scope for the target architecture.


+10 -11

HealthProbe/Doc/04-project/IMPLEMENTATION_STATUS.md

View

@@ -1,6 +1,6 @@
 # HealthProbe - Implementation Status
 
-**Last Updated:** 2026-05-23
+**Last Updated:** 2026-05-24
 
 ## Current Reality
 
@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab
 |------|----------------|--------------------|
 | Product docs | Updated | Keep `HealthProbe/Doc/README.md` as canonical index |
 | HealthKit capture | Prototype exists | Adapt capture to write differential SQLite observations first |
-| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, initial SQL diff/count/aggregate APIs, and XCTest coverage are in place; legacy write mirror still exists | Add provenance and consolidation-evidence SQL analysis, large synthetic-data tests, then retire `archive_samples` |
+| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, and XCTest coverage are in place; legacy write mirror still exists | Add large synthetic-data timing/memory tests, then retire `archive_samples` |
 | Core Data cache | Not implemented | Add rebuildable cache for expensive counts, summaries, report metadata, UI state |
 | SwiftData cache | Exists | Treat as disposable prototype data; reset/ignore during v2 transition |
 | UI | Prototype exists | Reframe screens around observations, diffs, export, archive status |
@@ -38,13 +38,12 @@ There are no real deployments, only test installations. Existing prototype datab
 
 Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.md).
 
-1. Add provenance and consolidation-evidence queries on top of the SQLite archive.
-2. Expand the synthetic large-data test harness for diff/export memory behavior.
-3. Add Core Data UI/report cache and rebuild pipeline.
-4. Replace SwiftData UI dependencies with Core Data/cache DTOs.
-5. Update UI language from anomaly/status to observation/diff/export.
-6. Add streaming exports with manifests.
-7. Validate on low-memory/legacy-class devices.
+1. Expand the synthetic large-data test harness for diff/export memory behavior.
+2. Add Core Data UI/report cache and rebuild pipeline.
+3. Replace SwiftData UI dependencies with Core Data/cache DTOs.
+4. Update UI language from anomaly/status to observation/diff/export.
+5. Add streaming exports with manifests.
+6. Validate on low-memory/legacy-class devices.
 
 ## Known Prototype Mismatches
 
@@ -54,13 +53,13 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m
 - Current archive schema is not sufficient as the long-term source of truth.
 - Existing implementation may decode or cache too much data for low-end devices.
 - Old prototype database compatibility is no longer required.
-- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, and materialized aggregate comparison, but not yet large-volume diff/export behavior.
+- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet large-volume diff/export behavior.
 
 ## Verification Checklist
 
 - [ ] SQLite archive v2 can reconstruct records visible at observation T.
 - [ ] No recurring complete snapshot copies are written for high-volume types.
-- [ ] SQL diff between two observations runs without loading full datasets into Swift arrays.
+- [x] SQL diff between two observations runs without loading full datasets into Swift arrays.
 - [ ] Expensive counts used by reports/UI are cached and rebuildable.
 - [ ] Deleting Core Data cache and rebuilding from SQLite restores UI/report summaries.
 - [ ] Export can stream large selected record sets.


+3 -3

HealthProbe/Doc/04-project/Refactoring-Plan.md

View

@@ -150,15 +150,15 @@ Checklist:
 - [x] Implement representationChanged query between observations.
 - [x] Implement diff counts using temp tables or equivalent SQL-first strategy.
 - [x] Implement aggregate comparison query.
-- [ ] Implement consolidation-likely evidence query.
-- [ ] Implement source/provenance breakdown query.
+- [x] Implement consolidation-likely evidence query.
+- [x] Implement source/provenance breakdown query.
 - [ ] Add query timing/memory tests on synthetic large datasets.
 
 Acceptance:
 - [x] Observation T can be reconstructed from ranges/events.
 - [ ] Large diff returns counts and first page without loading all rows.
 - [x] Query results are deterministic and ordered.
-- [ ] Consolidation evidence includes count, aggregate, coverage, density, and uncertainty data.
+- [x] Consolidation evidence includes count, aggregate, coverage, density, and uncertainty data.
 
 ## Milestone 6 - Core Data UI/Report Cache
 


+47 -0

HealthProbe/Services/Protocols/HealthArchiveStore.swift

View

+                         func diffSummary(_ request: HealthArchiveDiffRequest) async throws -> HealthArchiveDiffSummary
+                         func diffRecords(_ request: HealthArchiveDiffRecordRequest) async throws -> [ArchivedHealthRecord]
+                         func aggregateComparison(_ request: HealthArchiveAggregateComparisonRequest) async throws -> [HealthArchiveAggregateComparisonRow]
+                    +    func sourceProvenanceBreakdown(_ request: HealthArchiveSourceProvenanceRequest) async throws -> [HealthArchiveSourceProvenanceRow]
+                    +    func consolidationEvidence(_ request: HealthArchiveConsolidationEvidenceRequest) async throws -> [HealthArchiveConsolidationEvidence]
+                         func exportReport(_ request: HealthArchiveReportRequest) async throws -> URL
+                         func checkIntegrity() async throws -> HealthArchiveIntegrityReport
+                    +struct HealthArchiveSourceProvenanceRequest: Equatable, Sendable {
+                    +    let visibleAtObservationID: Int64?
+                    +    let sampleTypeIdentifier: String?
+                    +    let limit: Int?
+                    +}
+                    +struct HealthArchiveSourceProvenanceRow: Equatable, Sendable {
+                    +    let sampleTypeIdentifier: String
+                    +    let sourceNameHash: String?
+                    +    let sourceBundleIdentifier: String?
+                    +    let sourceProductType: String?
+                    +    let sourceVersion: String?
+                    +    let sourceOperatingSystemVersion: String?
+                    +    let deviceModel: String?
+                    +    let visibleRecordCount: Int
+                    +    let valueSum: Double?
+                    +    let earliestStartDate: Date?
+                    +    let latestEndDate: Date?
+                    +}
+                    +struct HealthArchiveConsolidationEvidenceRequest: Equatable, Sendable {
+                    +    let fromObservationID: Int64
+                    +    let toObservationID: Int64
+                    +    let sampleTypeIdentifier: String?
+                    +}
+                    +struct HealthArchiveConsolidationEvidence: Equatable, Sendable {
+                    +    let sampleTypeIdentifier: String
+                    +    let fromObservationID: Int64
+                    +    let toObservationID: Int64
+                    +    let disappearedCount: Int
+                    +    let appearedCount: Int
+                    +    let representationChangedCount: Int
+                    +    let fromVisibleRecordCount: Int
+                    +    let toVisibleRecordCount: Int
+                    +    let fromValueSum: Double?
+                    +    let toValueSum: Double?
+                    +    let coverageOverlapSeconds: Double?
+                    +    let densityBefore: Double?
+                    +    let densityAfter: Double?
+                    +    let sourceCompatible: Bool
+                    +    let label: String
+                    +    let uncertaintyReason: String?
+                    +}
+                     struct HealthArchiveReportRequest: Equatable, Sendable {
+                         let reportID: UUID
+                         let title: String

+327 -0

HealthProbe/Services/SQLiteHealthArchiveStore.swift

View

+                    +    func sourceProvenanceBreakdown(_ request: HealthArchiveSourceProvenanceRequest) async throws -> [HealthArchiveSourceProvenanceRow] {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        var clauses: [String] = []
+                    +        if request.sampleTypeIdentifier != nil {
+                    +            clauses.append("t.type_identifier = ?")
+                    +        }
+                    +        let whereClause = clauses.isEmpty ? "" : "WHERE \(clauses.joined(separator: " AND "))"
+                    +        let limitClause = request.limit.map { "LIMIT \(max($0, 0))" } ?? ""
+                    +        let sql = """
+                    +        WITH visible_ranges AS (
+                    +            SELECT sample_id, version_id
+                    +            FROM sample_visibility_ranges
+                    +            WHERE (? IS NULL OR (
+                    +                first_observation_id <= ?
+                    +                AND (last_observation_id IS NULL OR last_observation_id > ?)
+                    +            ))
+                    +        )
+                    +        SELECT
+                    +            t.type_identifier,
+                    +            src.source_name_hash,
+                    +            src.bundle_identifier,
+                    +            sr.product_type,
+                    +            sr.version,
+                    +            sr.operating_system_version,
+                    +            d.model,
+                    +            COUNT(*) AS visible_record_count,
+                    +            SUM(v.numeric_value) AS value_sum,
+                    +            MIN(v.start_date) AS earliest_start_date,
+                    +            MAX(v.end_date) AS latest_end_date
+                    +        FROM visible_ranges vr
+                    +        JOIN sample_versions v ON v.id = vr.version_id
+                    +        JOIN samples s ON s.id = vr.sample_id
+                    +        JOIN sample_types t ON t.id = s.sample_type_id
+                    +        LEFT JOIN source_revisions sr ON sr.id = v.source_revision_id
+                    +        LEFT JOIN sources src ON src.id = sr.source_id
+                    +        LEFT JOIN hk_devices d ON d.id = v.hk_device_id
+                    +        \(whereClause)
+                    +        GROUP BY
+                    +            t.type_identifier,
+                    +            src.source_name_hash,
+                    +            src.bundle_identifier,
+                    +            sr.product_type,
+                    +            sr.version,
+                    +            sr.operating_system_version,
+                    +            d.model
+                    +        ORDER BY visible_record_count DESC, t.type_identifier ASC, src.bundle_identifier ASC, d.model ASC
+                    +        \(limitClause)
+                    +        """
+                    +        return try withStatement(sql, db: db) { statement in
+                    +            var index: Int32 = 1
+                    +            bindInt64(request.visibleAtObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            bindInt64(request.visibleAtObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            bindInt64(request.visibleAtObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            if let sampleTypeIdentifier = request.sampleTypeIdentifier {
+                    +                bindText(sampleTypeIdentifier, to: index, in: statement)
+                    +                index += 1
+                    +            }
+                    +            var rows: [HealthArchiveSourceProvenanceRow] = []
+                    +            while sqlite3_step(statement) == SQLITE_ROW {
+                    +                rows.append(HealthArchiveSourceProvenanceRow(
+                    +                    sampleTypeIdentifier: columnText(statement, 0) ?? "",
+                    +                    sourceNameHash: columnText(statement, 1),
+                    +                    sourceBundleIdentifier: columnText(statement, 2),
+                    +                    sourceProductType: columnText(statement, 3),
+                    +                    sourceVersion: columnText(statement, 4),
+                    +                    sourceOperatingSystemVersion: columnText(statement, 5),
+                    +                    deviceModel: columnText(statement, 6),
+                    +                    visibleRecordCount: columnInt(statement, 7) ?? 0,
+                    +                    valueSum: columnDouble(statement, 8),
+                    +                    earliestStartDate: columnUnixDate(statement, 9),
+                    +                    latestEndDate: columnUnixDate(statement, 10)
+                    +                ))
+                    +            }
+                    +            return rows
+                    +        }
+                    +    }
+                    +    func consolidationEvidence(_ request: HealthArchiveConsolidationEvidenceRequest) async throws -> [HealthArchiveConsolidationEvidence] {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        let typeClause = request.sampleTypeIdentifier == nil ? "" : "AND t.type_identifier = ?"
+                    +        let topFilterClause = request.sampleTypeIdentifier == nil ? "" : "WHERE tk.type_identifier = ?"
+                    +        let sql = """
+                    +        WITH from_visible AS (
+                    +            SELECT sample_id, version_id
+                    +            FROM sample_visibility_ranges
+                    +            WHERE first_observation_id <= ?
+                    +              AND (last_observation_id IS NULL OR last_observation_id > ?)
+                    +        ),
+                    +        to_visible AS (
+                    +            SELECT sample_id, version_id
+                    +            FROM sample_visibility_ranges
+                    +            WHERE first_observation_id <= ?
+                    +              AND (last_observation_id IS NULL OR last_observation_id > ?)
+                    +        ),
+                    +        type_keys AS (
+                    +            SELECT DISTINCT sample_type_id, type_identifier
+                    +            FROM (
+                    +                SELECT s.sample_type_id, t.type_identifier
+                    +                FROM from_visible fv
+                    +                JOIN samples s ON s.id = fv.sample_id
+                    +                JOIN sample_types t ON t.id = s.sample_type_id
+                    +                UNION
+                    +                SELECT s.sample_type_id, t.type_identifier
+                    +                FROM to_visible tv
+                    +                JOIN samples s ON s.id = tv.sample_id
+                    +                JOIN sample_types t ON t.id = s.sample_type_id
+                    +            )
+                    +        ),
+                    +        from_type_metrics AS (
+                    +            SELECT
+                    +                s.sample_type_id,
+                    +                COUNT(*) AS from_visible_record_count,
+                    +                SUM(v.numeric_value) AS from_value_sum,
+                    +                MIN(v.start_date) AS from_start_date,
+                    +                MAX(v.end_date) AS from_end_date,
+                    +                COUNT(DISTINCT CASE WHEN src.bundle_identifier IS NULL THEN '__NULL__' ELSE src.bundle_identifier END) AS from_bundle_count,
+                    +                COUNT(DISTINCT CASE WHEN d.model IS NULL THEN '__NULL__' ELSE d.model END) AS from_model_count
+                    +            FROM from_visible fv
+                    +            JOIN samples s ON s.id = fv.sample_id
+                    +            JOIN sample_versions v ON v.id = fv.version_id
+                    +            LEFT JOIN source_revisions sr ON sr.id = v.source_revision_id
+                    +            LEFT JOIN sources src ON src.id = sr.source_id
+                    +            LEFT JOIN hk_devices d ON d.id = v.hk_device_id
+                    +            GROUP BY s.sample_type_id
+                    +        ),
+                    +        to_type_metrics AS (
+                    +            SELECT
+                    +                s.sample_type_id,
+                    +                COUNT(*) AS to_visible_record_count,
+                    +                SUM(v.numeric_value) AS to_value_sum,
+                    +                MIN(v.start_date) AS to_start_date,
+                    +                MAX(v.end_date) AS to_end_date,
+                    +                COUNT(DISTINCT CASE WHEN src.bundle_identifier IS NULL THEN '__NULL__' ELSE src.bundle_identifier END) AS to_bundle_count,
+                    +                COUNT(DISTINCT CASE WHEN d.model IS NULL THEN '__NULL__' ELSE d.model END) AS to_model_count
+                    +            FROM to_visible tv
+                    +            JOIN samples s ON s.id = tv.sample_id
+                    +            JOIN sample_versions v ON v.id = tv.version_id
+                    +            LEFT JOIN source_revisions sr ON sr.id = v.source_revision_id
+                    +            LEFT JOIN sources src ON src.id = sr.source_id
+                    +            LEFT JOIN hk_devices d ON d.id = v.hk_device_id
+                    +            GROUP BY s.sample_type_id
+                    +        ),
+                    +        appeared_by_type AS (
+                    +            SELECT s.sample_type_id, COUNT(*) AS appeared_count
+                    +            FROM to_visible tv
+                    +            LEFT JOIN from_visible fv ON fv.sample_id = tv.sample_id
+                    +            JOIN samples s ON s.id = tv.sample_id
+                    +            JOIN sample_types t ON t.id = s.sample_type_id
+                    +            WHERE fv.sample_id IS NULL \(typeClause)
+                    +            GROUP BY s.sample_type_id
+                    +        ),
+                    +        disappeared_by_type AS (
+                    +            SELECT s.sample_type_id, COUNT(*) AS disappeared_count
+                    +            FROM from_visible fv
+                    +            LEFT JOIN to_visible tv ON tv.sample_id = fv.sample_id
+                    +            JOIN samples s ON s.id = fv.sample_id
+                    +            JOIN sample_types t ON t.id = s.sample_type_id
+                    +            WHERE tv.sample_id IS NULL \(typeClause)
+                    +            GROUP BY s.sample_type_id
+                    +        ),
+                    +        changed_by_type AS (
+                    +            SELECT s.sample_type_id, COUNT(*) AS representation_changed_count
+                    +            FROM to_visible tv
+                    +            JOIN from_visible fv ON fv.sample_id = tv.sample_id
+                    +            JOIN samples s ON s.id = tv.sample_id
+                    +            JOIN sample_types t ON t.id = s.sample_type_id
+                    +            WHERE tv.version_id != fv.version_id \(typeClause)
+                    +            GROUP BY s.sample_type_id
+                    +        )
+                    +        SELECT
+                    +            tk.type_identifier,
+                    +            COALESCE(d.disappeared_count, 0) AS disappeared_count,
+                    +            COALESCE(a.appeared_count, 0) AS appeared_count,
+                    +            COALESCE(c.representation_changed_count, 0) AS representation_changed_count,
+                    +            COALESCE(f.from_visible_record_count, 0) AS from_visible_record_count,
+                    +            COALESCE(t.to_visible_record_count, 0) AS to_visible_record_count,
+                    +            f.from_value_sum,
+                    +            t.to_value_sum,
+                    +            f.from_start_date,
+                    +            f.from_end_date,
+                    +            t.to_start_date,
+                    +            t.to_end_date,
+                    +            CASE
+                    +                WHEN f.from_start_date IS NULL OR f.from_end_date IS NULL OR t.to_start_date IS NULL OR t.to_end_date IS NULL THEN NULL
+                    +                ELSE MAX(0, MIN(f.from_end_date, t.to_end_date) - MAX(f.from_start_date, t.to_start_date))
+                    +            END AS coverage_overlap_seconds,
+                    +            CASE
+                    +                WHEN COALESCE(f.from_visible_record_count, 0) = 0 OR f.from_start_date IS NULL OR f.from_end_date IS NULL OR f.from_end_date <= f.from_start_date THEN NULL
+                    +                ELSE CAST(f.from_visible_record_count AS REAL) / ((f.from_end_date - f.from_start_date) / 3600.0)
+                    +            END AS density_before,
+                    +            CASE
+                    +                WHEN COALESCE(t.to_visible_record_count, 0) = 0 OR t.to_start_date IS NULL OR t.to_end_date IS NULL OR t.to_end_date <= t.to_start_date THEN NULL
+                    +                ELSE CAST(t.to_visible_record_count AS REAL) / ((t.to_end_date - t.to_start_date) / 3600.0)
+                    +            END AS density_after,
+                    +            CASE
+                    +                WHEN COALESCE(f.from_bundle_count, 0) <= 1
+                    +                  AND COALESCE(f.from_model_count, 0) <= 1
+                    +                  AND COALESCE(t.to_bundle_count, 0) <= 1
+                    +                  AND COALESCE(t.to_model_count, 0) <= 1
+                    +                THEN 1 ELSE 0
+                    +            END AS source_compatible
+                    +        FROM type_keys tk
+                    +        LEFT JOIN from_type_metrics f ON f.sample_type_id = tk.sample_type_id
+                    +        LEFT JOIN to_type_metrics t ON t.sample_type_id = tk.sample_type_id
+                    +        LEFT JOIN appeared_by_type a ON a.sample_type_id = tk.sample_type_id
+                    +        LEFT JOIN disappeared_by_type d ON d.sample_type_id = tk.sample_type_id
+                    +        LEFT JOIN changed_by_type c ON c.sample_type_id = tk.sample_type_id
+                    +        \(topFilterClause)
+                    +        ORDER BY tk.type_identifier
+                    +        """
+                    +        return try withStatement(sql, db: db) { statement in
+                    +            var index: Int32 = 1
+                    +            bindInt64(request.fromObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            bindInt64(request.fromObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            bindInt64(request.toObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            bindInt64(request.toObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            if let sampleTypeIdentifier = request.sampleTypeIdentifier {
+                    +                bindText(sampleTypeIdentifier, to: index, in: statement)
+                    +                index += 1
+                    +            }
+                    +            if let sampleTypeIdentifier = request.sampleTypeIdentifier {
+                    +                bindText(sampleTypeIdentifier, to: index, in: statement)
+                    +                index += 1
+                    +            }
+                    +            if let sampleTypeIdentifier = request.sampleTypeIdentifier {
+                    +                bindText(sampleTypeIdentifier, to: index, in: statement)
+                    +                index += 1
+                    +            }
+                    +            if request.sampleTypeIdentifier != nil {
+                    +                bindText(request.sampleTypeIdentifier, to: index, in: statement)
+                    +                index += 1
+                    +            }
+                    +            var rows: [HealthArchiveConsolidationEvidence] = []
+                    +            while sqlite3_step(statement) == SQLITE_ROW {
+                    +                let sampleTypeIdentifier = columnText(statement, 0) ?? ""
+                    +                let disappearedCount = columnInt(statement, 1) ?? 0
+                    +                let appearedCount = columnInt(statement, 2) ?? 0
+                    +                let representationChangedCount = columnInt(statement, 3) ?? 0
+                    +                let fromVisibleRecordCount = columnInt(statement, 4) ?? 0
+                    +                let toVisibleRecordCount = columnInt(statement, 5) ?? 0
+                    +                let fromValueSum = columnDouble(statement, 6)
+                    +                let toValueSum = columnDouble(statement, 7)
+                    +                let coverageOverlapSeconds = columnDouble(statement, 12)
+                    +                let densityBefore = columnDouble(statement, 13)
+                    +                let densityAfter = columnDouble(statement, 14)
+                    +                let sourceCompatible = columnInt(statement, 15) == 1
+                    +                let stableValue: Bool
+                    +                if let fromValueSum, let toValueSum {
+                    +                    stableValue = abs(toValueSum - fromValueSum) <= max(1.0, abs(fromValueSum) * 0.1)
+                    +                } else {
+                    +                    stableValue = false
+                    +                }
+                    +                let hasCoverageOverlap = (coverageOverlapSeconds ?? 0) > 0
+                    +                let densityDoesNotIncrease: Bool
+                    +                if let densityBefore, let densityAfter {
+                    +                    densityDoesNotIncrease = densityAfter <= densityBefore
+                    +                } else {
+                    +                    densityDoesNotIncrease = false
+                    +                }
+                    +                let consolidationEvidencePresent = disappearedCount > 0
+                    +                    && sourceCompatible
+                    +                    && hasCoverageOverlap
+                    +                    && densityDoesNotIncrease
+                    +                    && stableValue
+                    +                let label: String
+                    +                let uncertaintyReason: String?
+                    +                if representationChangedCount > 0 && disappearedCount == 0 && appearedCount == 0 {
+                    +                    label = "representation_changed"
+                    +                    uncertaintyReason = nil
+                    +                } else if consolidationEvidencePresent {
+                    +                    label = "consolidation_likely"
+                    +                    uncertaintyReason = nil
+                    +                } else if appearedCount > 0 && disappearedCount == 0 && representationChangedCount == 0 {
+                    +                    label = "appeared"
+                    +                    uncertaintyReason = nil
+                    +                } else if appearedCount > 0 && disappearedCount > 0 && sourceCompatible && stableValue {
+                    +                    label = "aggregate_changed"
+                    +                    uncertaintyReason = nil
+                    +                } else if disappearedCount == 0 && appearedCount == 0 && representationChangedCount == 0 {
+                    +                    label = "uncertain"
+                    +                    uncertaintyReason = "no structural change detected"
+                    +                } else {
+                    +                    label = "uncertain"
+                    +                    uncertaintyReason = "insufficient evidence for a stable consolidation label"
+                    +                }
+                    +                rows.append(HealthArchiveConsolidationEvidence(
+                    +                    sampleTypeIdentifier: sampleTypeIdentifier,
+                    +                    fromObservationID: request.fromObservationID,
+                    +                    toObservationID: request.toObservationID,
+                    +                    disappearedCount: disappearedCount,
+                    +                    appearedCount: appearedCount,
+                    +                    representationChangedCount: representationChangedCount,
+                    +                    fromVisibleRecordCount: fromVisibleRecordCount,
+                    +                    toVisibleRecordCount: toVisibleRecordCount,
+                    +                    fromValueSum: fromValueSum,
+                    +                    toValueSum: toValueSum,
+                    +                    coverageOverlapSeconds: coverageOverlapSeconds,
+                    +                    densityBefore: densityBefore,
+                    +                    densityAfter: densityAfter,
+                    +                    sourceCompatible: sourceCompatible,
+                    +                    label: label,
+                    +                    uncertaintyReason: uncertaintyReason
+                    +                ))
+                    +            }
+                    +            return rows
+                    +        }
+                    +    }
+                         func exportReport(_ request: HealthArchiveReportRequest) async throws -> URL {
+                             let recordRequest = HealthArchiveRecordRequest(
+                                 sampleTypeIdentifier: request.typeIdentifierFilter,

+89 -2

HealthProbeTests/SQLiteHealthArchiveStoreTests.swift

View

+                             XCTAssertEqual(rows.first?.valueSumDelta, 7)
+                    +    func testSourceProvenanceBreakdownReturnsVisibleSourceRows() async throws {
+                    +        let url = databaseURL()
+                    +        let store = SQLiteHealthArchiveStore(databaseURL: url)
+                    +        let sample = makeStepCountSample(value: 42, start: 1_000)
+                    +        let typeIdentifier = sample.sampleType.identifier
+                    +        _ = try await store.upsertSamples([sample], observedAt: Date(timeIntervalSince1970: 3_000))
+                    +        let rows = try await store.sourceProvenanceBreakdown(HealthArchiveSourceProvenanceRequest(
+                    +            visibleAtObservationID: nil,
+                    +            sampleTypeIdentifier: typeIdentifier,
+                    +            limit: 10
+                    +        ))
+                    +        XCTAssertEqual(rows.count, 1)
+                    +        XCTAssertEqual(rows.first?.sampleTypeIdentifier, typeIdentifier)
+                    +        XCTAssertEqual(rows.first?.visibleRecordCount, 1)
+                    +        XCTAssertEqual(rows.first?.valueSum, 42)
+                    +        XCTAssertEqual(rows.first?.sourceBundleIdentifier, sample.sourceRevision.source.bundleIdentifier)
+                    +    }
+                    +    func testConsolidationEvidenceClassifiesStableCountDropAsConsolidationLikely() async throws {
+                    +        let url = databaseURL()
+                    +        let store = SQLiteHealthArchiveStore(databaseURL: url)
+                    +        let typeIdentifier = HKQuantityTypeIdentifier.stepCount.rawValue
+                    +        let firstOldSample = makeStepCountSample(value: 10, start: 1_000, end: 1_500)
+                    +        let secondOldSample = makeStepCountSample(value: 20, start: 1_600, end: 2_000)
+                    +        let consolidatedSample = makeStepCountSample(value: 30, start: 1_000, end: 2_500)
+                    +        _ = try await store.upsertSamples([firstOldSample, secondOldSample], observedAt: Date(timeIntervalSince1970: 3_000))
+                    +        _ = try await store.upsertSamples([consolidatedSample], observedAt: Date(timeIntervalSince1970: 3_060))
+                    +        try await store.recordDisappearance(sampleUUIDHash: HashService.sampleUUIDHash(firstOldSample.uuid.uuidString), sampleTypeIdentifier: typeIdentifier, observedMissingAt: Date(timeIntervalSince1970: 3_120))
+                    +        try await store.recordDisappearance(sampleUUIDHash: HashService.sampleUUIDHash(secondOldSample.uuid.uuidString), sampleTypeIdentifier: typeIdentifier, observedMissingAt: Date(timeIntervalSince1970: 3_180))
+                    +        let observationIDs = try observationIDs(at: url)
+                    +        XCTAssertEqual(observationIDs.count, 4)
+                    +        let rows = try await store.consolidationEvidence(HealthArchiveConsolidationEvidenceRequest(
+                    +            fromObservationID: observationIDs[0],
+                    +            toObservationID: observationIDs[observationIDs.count - 1],
+                    +            sampleTypeIdentifier: typeIdentifier
+                    +        ))
+                    +        XCTAssertEqual(rows.count, 1)
+                    +        XCTAssertEqual(rows.first?.sampleTypeIdentifier, typeIdentifier)
+                    +        XCTAssertEqual(rows.first?.disappearedCount, 2)
+                    +        XCTAssertEqual(rows.first?.appearedCount, 1)
+                    +        XCTAssertEqual(rows.first?.fromVisibleRecordCount, 2)
+                    +        XCTAssertEqual(rows.first?.toVisibleRecordCount, 1)
+                    +        XCTAssertEqual(rows.first?.fromValueSum, 30)
+                    +        XCTAssertEqual(rows.first?.toValueSum, 30)
+                    +        XCTAssertEqual(rows.first?.label, "consolidation_likely")
+                    +        XCTAssertTrue(rows.first?.sourceCompatible == true)
+                    +    }
+                    +    func testConsolidationEvidenceClassifiesDenseStableShiftAsAggregateChanged() async throws {
+                    +        let url = databaseURL()
+                    +        let store = SQLiteHealthArchiveStore(databaseURL: url)
+                    +        let typeIdentifier = HKQuantityTypeIdentifier.stepCount.rawValue
+                    +        let firstOldSample = makeStepCountSample(value: 10, start: 1_000, end: 1_500)
+                    +        let secondOldSample = makeStepCountSample(value: 20, start: 5_000, end: 6_000)
+                    +        let denseSample = makeStepCountSample(value: 30, start: 1_000, end: 1_200)
+                    +        _ = try await store.upsertSamples([firstOldSample, secondOldSample], observedAt: Date(timeIntervalSince1970: 3_000))
+                    +        _ = try await store.upsertSamples([denseSample], observedAt: Date(timeIntervalSince1970: 3_060))
+                    +        try await store.recordDisappearance(sampleUUIDHash: HashService.sampleUUIDHash(firstOldSample.uuid.uuidString), sampleTypeIdentifier: typeIdentifier, observedMissingAt: Date(timeIntervalSince1970: 3_120))
+                    +        try await store.recordDisappearance(sampleUUIDHash: HashService.sampleUUIDHash(secondOldSample.uuid.uuidString), sampleTypeIdentifier: typeIdentifier, observedMissingAt: Date(timeIntervalSince1970: 3_180))
+                    +        let observationIDs = try observationIDs(at: url)
+                    +        XCTAssertEqual(observationIDs.count, 4)
+                    +        let rows = try await store.consolidationEvidence(HealthArchiveConsolidationEvidenceRequest(
+                    +            fromObservationID: observationIDs[0],
+                    +            toObservationID: observationIDs[observationIDs.count - 1],
+                    +            sampleTypeIdentifier: typeIdentifier
+                    +        ))
+                    +        XCTAssertEqual(rows.count, 1)
+                    +        XCTAssertEqual(rows.first?.label, "aggregate_changed")
+                    +        XCTAssertEqual(rows.first?.fromVisibleRecordCount, 2)
+                    +        XCTAssertEqual(rows.first?.toVisibleRecordCount, 1)
+                    +        XCTAssertEqual(rows.first?.fromValueSum, 30)
+                    +        XCTAssertEqual(rows.first?.toValueSum, 30)
+                    +    }
+                         private func databaseURL() -> URL {
+                             temporaryDirectory.appending(path: "Archive.sqlite")
+                             makeStepCountSample(value: 42, start: 1_000)
+                    -    private func makeStepCountSample(value: Double, start: TimeInterval) -> HKQuantitySample {
+                    +    private func makeStepCountSample(value: Double, start: TimeInterval, end: TimeInterval? = nil) -> HKQuantitySample {
+                             let quantityType = HKQuantityType.quantityType(forIdentifier: .stepCount)!
+                             let quantity = HKQuantity(unit: .count(), doubleValue: value)
+                             let startDate = Date(timeIntervalSince1970: start)
+                    -        let endDate = Date(timeIntervalSince1970: start + 300)
+                    +        let endDate = Date(timeIntervalSince1970: end ?? (start + 300))
+                             return HKQuantitySample(type: quantityType, quantity: quantity, start: startDate, end: endDate)

	@@ -150,15 +150,15 @@ Checklist:
150	150	- [x] Implement representationChanged query between observations.
151	151	- [x] Implement diff counts using temp tables or equivalent SQL-first strategy.
152	152	- [x] Implement aggregate comparison query.
153		-- [ ] Implement consolidation-likely evidence query.
154		-- [ ] Implement source/provenance breakdown query.
	153	+- [x] Implement consolidation-likely evidence query.
	154	+- [x] Implement source/provenance breakdown query.
155	155	- [ ] Add query timing/memory tests on synthetic large datasets.
156	156
157	157	Acceptance:
158	158	- [x] Observation T can be reconstructed from ranges/events.
159	159	- [ ] Large diff returns counts and first page without loading all rows.
160	160	- [x] Query results are deterministic and ordered.
161		-- [ ] Consolidation evidence includes count, aggregate, coverage, density, and uncertainty data.
	161	+- [x] Consolidation evidence includes count, aggregate, coverage, density, and uncertainty data.
162	162
163	163	## Milestone 6 - Core Data UI/Report Cache
164	164

	@@ -193,13 +193,16 @@ final class TypeDistributionBin {
193	193	// Deletions are recorded by sampleUUIDHash because HKDeletedObject exposes UUIDs,
194	194	// not complete sample payloads.
195	195
196		-// Interface updated 2026-05-23 — see AGENTS.md
	196	+// Interface updated 2026-05-24 — see AGENTS.md
197	197	// HealthArchiveStore exposes SQL-first observation diff APIs:
198	198	// diffSummary(_:) returns appeared/disappeared/representationChanged counts and
199	199	// diffRecords(_:) returns a paged record list for one change kind.
200	200	// aggregateComparison(_:) compares materialized daily aggregates between two
201		-// observations. UI/cache agents should consume these APIs instead of loading full
202		-// observation record sets.
	201	+// observations. sourceProvenanceBreakdown(_:) returns visible source/device
	202	+// composition, and consolidationEvidence(_:) returns conservative evidence labels
	203	+// with counts, aggregate sums, coverage, density, source compatibility, and
	204	+// uncertainty text. UI/cache agents should consume these APIs instead of loading
	205	+// full observation record sets.
203	206
204	207	// Storage objective updated 2026-05-23 — see AGENTS.md
205	208	// Recurring complete snapshots are out of scope for the target architecture.

	@@ -1,6 +1,6 @@
1	1	# HealthProbe - Implementation Status
2	2
3		-Last Updated: 2026-05-23
	3	+Last Updated: 2026-05-24
4	4
5	5	## Current Reality
6	6
	@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab
25	25	\|------\|----------------\|--------------------\|
26	26	\| Product docs \| Updated \| Keep `HealthProbe/Doc/README.md` as canonical index \|
27	27	\| HealthKit capture \| Prototype exists \| Adapt capture to write differential SQLite observations first \|
28		-\| SQLite archive \| Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, initial SQL diff/count/aggregate APIs, and XCTest coverage are in place; legacy write mirror still exists \| Add provenance and consolidation-evidence SQL analysis, large synthetic-data tests, then retire `archive_samples` \|
	28	+\| SQLite archive \| Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, and XCTest coverage are in place; legacy write mirror still exists \| Add large synthetic-data timing/memory tests, then retire `archive_samples` \|
29	29	\| Core Data cache \| Not implemented \| Add rebuildable cache for expensive counts, summaries, report metadata, UI state \|
30	30	\| SwiftData cache \| Exists \| Treat as disposable prototype data; reset/ignore during v2 transition \|
31	31	\| UI \| Prototype exists \| Reframe screens around observations, diffs, export, archive status \|
	@@ -38,13 +38,12 @@ There are no real deployments, only test installations. Existing prototype datab
38	38
39	39	Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.md).
40	40
41		-1. Add provenance and consolidation-evidence queries on top of the SQLite archive.
42		-2. Expand the synthetic large-data test harness for diff/export memory behavior.
43		-3. Add Core Data UI/report cache and rebuild pipeline.
44		-4. Replace SwiftData UI dependencies with Core Data/cache DTOs.
45		-5. Update UI language from anomaly/status to observation/diff/export.
46		-6. Add streaming exports with manifests.
47		-7. Validate on low-memory/legacy-class devices.
	41	+1. Expand the synthetic large-data test harness for diff/export memory behavior.
	42	+2. Add Core Data UI/report cache and rebuild pipeline.
	43	+3. Replace SwiftData UI dependencies with Core Data/cache DTOs.
	44	+4. Update UI language from anomaly/status to observation/diff/export.
	45	+5. Add streaming exports with manifests.
	46	+6. Validate on low-memory/legacy-class devices.
48	47
49	48	## Known Prototype Mismatches
50	49
	@@ -54,13 +53,13 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m
54	53	- Current archive schema is not sufficient as the long-term source of truth.
55	54	- Existing implementation may decode or cache too much data for low-end devices.
56	55	- Old prototype database compatibility is no longer required.
57		-- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, and materialized aggregate comparison, but not yet large-volume diff/export behavior.
	56	+- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet large-volume diff/export behavior.
58	57
59	58	## Verification Checklist
60	59
61	60	- [ ] SQLite archive v2 can reconstruct records visible at observation T.
62	61	- [ ] No recurring complete snapshot copies are written for high-volume types.
63		-- [ ] SQL diff between two observations runs without loading full datasets into Swift arrays.
	62	+- [x] SQL diff between two observations runs without loading full datasets into Swift arrays.
64	63	- [ ] Expensive counts used by reports/UI are cached and rebuildable.
65	64	- [ ] Deleting Core Data cache and rebuilding from SQLite restores UI/report summaries.
66	65	- [ ] Export can stream large selected record sets.