Add SQL archive diff queries · 9ceeb21

Add SQL archive diff queries
Browse files

bogdan committed 2 weeks ago

main

1 parent 7406fb5

commit 9ceeb21

Showing 6 changed files with 389 additions and 35 deletions

+6 -0

HealthProbe/Doc/00-agent-guides/AGENTS.md

@@ -193,6 +193,12 @@ final class TypeDistributionBin {
 // Deletions are recorded by sampleUUIDHash because HKDeletedObject exposes UUIDs,
 // not complete sample payloads.
 
+// Interface updated 2026-05-23 — see AGENTS.md
+// HealthArchiveStore exposes SQL-first observation diff APIs:
+// diffSummary(_:) returns appeared/disappeared/representationChanged counts and
+// diffRecords(_:) returns a paged record list for one change kind. UI/cache agents
+// should consume these APIs instead of loading full observation record sets.
+
 // Storage objective updated 2026-05-23 — see AGENTS.md
 // Recurring complete snapshots are out of scope for the target architecture.
 // Store differential observations, versioned sample payloads, observation ranges,


+3 -3

HealthProbe/Doc/04-project/IMPLEMENTATION_STATUS.md

View

@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab
 |------|----------------|--------------------|
 | Product docs | Updated | Keep `HealthProbe/Doc/README.md` as canonical index |
 | HealthKit capture | Prototype exists | Adapt capture to write differential SQLite observations first |
-| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, and initial XCTest coverage are in place; legacy write mirror still exists | Add SQL diff/count queries, large synthetic-data tests, then retire `archive_samples` |
+| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, initial SQL diff/count APIs, and XCTest coverage are in place; legacy write mirror still exists | Add aggregate/provenance SQL analysis, large synthetic-data tests, then retire `archive_samples` |
 | Core Data cache | Not implemented | Add rebuildable cache for expensive counts, summaries, report metadata, UI state |
 | SwiftData cache | Exists | Treat as disposable prototype data; reset/ignore during v2 transition |
 | UI | Prototype exists | Reframe screens around observations, diffs, export, archive status |
@@ -38,7 +38,7 @@ There are no real deployments, only test installations. Existing prototype datab
 
 Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.md).
 
-1. Move large diffs/counts into SQL queries with indexes/temp tables/paged results.
+1. Add aggregate/provenance/consolidation evidence queries on top of the SQLite archive.
 2. Expand the synthetic large-data test harness for diff/export memory behavior.
 3. Add Core Data UI/report cache and rebuild pipeline.
 4. Replace SwiftData UI dependencies with Core Data/cache DTOs.
@@ -54,7 +54,7 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m
 - Current archive schema is not sufficient as the long-term source of truth.
 - Existing implementation may decode or cache too much data for low-end devices.
 - Old prototype database compatibility is no longer required.
-- Initial SQLite archive tests cover open/init/reset/idempotency, but not yet large-volume diff/export behavior.
+- Initial SQLite archive tests cover open/init/reset/idempotency and small observation diffs, but not yet large-volume diff/export behavior.
 
 ## Verification Checklist
 


+5 -5

HealthProbe/Doc/04-project/Refactoring-Plan.md

View

@@ -145,10 +145,10 @@ Acceptance:
 Checklist:
 - [x] Implement point-in-time visible-record query.
 - [x] Implement paged record table query.
-- [ ] Implement appeared query between observations.
-- [ ] Implement disappeared query between observations.
-- [ ] Implement representationChanged query between observations.
-- [ ] Implement diff counts using temp tables or equivalent SQL-first strategy.
+- [x] Implement appeared query between observations.
+- [x] Implement disappeared query between observations.
+- [x] Implement representationChanged query between observations.
+- [x] Implement diff counts using temp tables or equivalent SQL-first strategy.
 - [ ] Implement aggregate comparison query.
 - [ ] Implement consolidation-likely evidence query.
 - [ ] Implement source/provenance breakdown query.
@@ -157,7 +157,7 @@ Checklist:
 Acceptance:
 - [x] Observation T can be reconstructed from ranges/events.
 - [ ] Large diff returns counts and first page without loading all rows.
-- [ ] Query results are deterministic and ordered.
+- [x] Query results are deterministic and ordered.
 - [ ] Consolidation evidence includes count, aggregate, coverage, density, and uncertainty data.
 
 ## Milestone 6 - Core Data UI/Report Cache


+59 -1

HealthProbe/Services/Protocols/HealthArchiveStore.swift

View

+                     import Foundation
+                     import HealthKit
+                    -// Interface updated 2026-05-18 — see AGENTS.md
+                    +// Interface updated 2026-05-23 — see AGENTS.md
+                     protocol HealthArchiveStore {
+                         func upsertSamples(_ samples: [HKSample], observedAt: Date) async throws -> HealthArchiveWriteSummary
+                         func markVerification(sampleType: HKSampleType, verifiedAt: Date) async throws
+                         func recordDisappearance(sampleUUIDHash: String, sampleTypeIdentifier: String, observedMissingAt: Date) async throws
+                         func records(for request: HealthArchiveRecordRequest) async throws -> [ArchivedHealthRecord]
+                    +    func diffSummary(_ request: HealthArchiveDiffRequest) async throws -> HealthArchiveDiffSummary
+                    +    func diffRecords(_ request: HealthArchiveDiffRecordRequest) async throws -> [ArchivedHealthRecord]
+                         func exportReport(_ request: HealthArchiveReportRequest) async throws -> URL
+                         func checkIntegrity() async throws -> HealthArchiveIntegrityReport
+                         let strictFingerprint: String
+                    +enum HealthArchiveDiffKind: String, Codable, Sendable {
+                    +    case appeared
+                    +    case disappeared
+                    +    case representationChanged
+                    +}
+                    +struct HealthArchiveDiffRequest: Equatable, Sendable {
+                    +    let fromObservationID: Int64
+                    +    let toObservationID: Int64
+                    +    let sampleTypeIdentifier: String?
+                    +    init(fromObservationID: Int64, toObservationID: Int64, sampleTypeIdentifier: String? = nil) {
+                    +        self.fromObservationID = fromObservationID
+                    +        self.toObservationID = toObservationID
+                    +        self.sampleTypeIdentifier = sampleTypeIdentifier
+                    +    }
+                    +}
+                    +struct HealthArchiveDiffSummary: Equatable, Sendable {
+                    +    let fromObservationID: Int64
+                    +    let toObservationID: Int64
+                    +    let sampleTypeIdentifier: String?
+                    +    let appearedCount: Int
+                    +    let disappearedCount: Int
+                    +    let representationChangedCount: Int
+                    +    var totalChangeCount: Int {
+                    +        appearedCount + disappearedCount + representationChangedCount
+                    +    }
+                    +}
+                    +struct HealthArchiveDiffRecordRequest: Equatable, Sendable {
+                    +    let fromObservationID: Int64
+                    +    let toObservationID: Int64
+                    +    let sampleTypeIdentifier: String?
+                    +    let kind: HealthArchiveDiffKind
+                    +    let afterCursor: RecordCursor?
+                    +    let limit: Int?
+                    +    init(
+                    +        fromObservationID: Int64,
+                    +        toObservationID: Int64,
+                    +        sampleTypeIdentifier: String? = nil,
+                    +        kind: HealthArchiveDiffKind,
+                    +        afterCursor: RecordCursor? = nil,
+                    +        limit: Int? = nil
+                    +    ) {
+                    +        self.fromObservationID = fromObservationID
+                    +        self.toObservationID = toObservationID
+                    +        self.sampleTypeIdentifier = sampleTypeIdentifier
+                    +        self.kind = kind
+                    +        self.afterCursor = afterCursor
+                    +        self.limit = limit
+                    +    }
+                    +}
+                     struct HealthArchiveReportRequest: Equatable, Sendable {
+                         let reportID: UUID
+                         let title: String

+231 -22

HealthProbe/Services/SQLiteHealthArchiveStore.swift

View

+                                 var records: [ArchivedHealthRecord] = []
+                                 while sqlite3_step(statement) == SQLITE_ROW {
+                    -                records.append(ArchivedHealthRecord(
+                    -                    id: columnText(statement, 0) ?? "",
+                    -                    sampleTypeIdentifier: columnText(statement, 1) ?? "",
+                    -                    strictFingerprint: columnText(statement, 2) ?? "",
+                    -                    semanticFingerprint: columnText(statement, 3),
+                    -                    healthKitUUIDHash: columnText(statement, 4),
+                    -                    startDate: columnUnixDate(statement, 5) ?? Date(timeIntervalSince1970: 0),
+                    -                    endDate: columnUnixDate(statement, 6) ?? Date(timeIntervalSince1970: 0),
+                    -                    firstSeenAt: columnUnixDate(statement, 7) ?? Date(timeIntervalSince1970: 0),
+                    -                    lastSeenAt: columnUnixDate(statement, 8),
+                    -                    lastVerifiedAt: columnUnixDate(statement, 9),
+                    -                    disappearedAt: columnUnixDate(statement, 10),
+                    -                    valueKind: columnText(statement, 11),
+                    -                    value: columnDouble(statement, 12),
+                    -                    unit: columnText(statement, 13),
+                    -                    categoryValue: columnInt(statement, 14),
+                    -                    workoutActivityType: columnInt(statement, 15),
+                    -                    durationSeconds: columnDouble(statement, 16),
+                    -                    sourceName: nil,
+                    -                    sourceBundleIdentifier: columnText(statement, 17),
+                    -                    deviceName: nil
+                    -                ))
+                    +                records.append(archiveRecord(from: statement))
+                    +            }
+                    +            return records
+                    +        }
+                    +    }
+                    +    func diffSummary(_ request: HealthArchiveDiffRequest) async throws -> HealthArchiveDiffSummary {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        let typeClause = request.sampleTypeIdentifier == nil ? "" : "AND t.type_identifier = ?"
+                    +        let sql = """
+                    +        WITH from_visible AS (
+                    +            SELECT sample_id, version_id
+                    +            FROM sample_visibility_ranges
+                    +            WHERE first_observation_id <= ?
+                    +              AND (last_observation_id IS NULL OR last_observation_id > ?)
+                    +        ),
+                    +        to_visible AS (
+                    +            SELECT sample_id, version_id
+                    +            FROM sample_visibility_ranges
+                    +            WHERE first_observation_id <= ?
+                    +              AND (last_observation_id IS NULL OR last_observation_id > ?)
+                    +        )
+                    +        SELECT
+                    +            (
+                    +                SELECT COUNT(*)
+                    +                FROM to_visible tv
+                    +                LEFT JOIN from_visible fv ON fv.sample_id = tv.sample_id
+                    +                JOIN samples s ON s.id = tv.sample_id
+                    +                JOIN sample_types t ON t.id = s.sample_type_id
+                    +                WHERE fv.sample_id IS NULL \(typeClause)
+                    +            ) AS appeared_count,
+                    +            (
+                    +                SELECT COUNT(*)
+                    +                FROM from_visible fv
+                    +                LEFT JOIN to_visible tv ON tv.sample_id = fv.sample_id
+                    +                JOIN samples s ON s.id = fv.sample_id
+                    +                JOIN sample_types t ON t.id = s.sample_type_id
+                    +                WHERE tv.sample_id IS NULL \(typeClause)
+                    +            ) AS disappeared_count,
+                    +            (
+                    +                SELECT COUNT(*)
+                    +                FROM to_visible tv
+                    +                JOIN from_visible fv ON fv.sample_id = tv.sample_id
+                    +                JOIN samples s ON s.id = tv.sample_id
+                    +                JOIN sample_types t ON t.id = s.sample_type_id
+                    +                WHERE tv.version_id != fv.version_id \(typeClause)
+                    +            ) AS representation_changed_count
+                    +        """
+                    +        return try withStatement(sql, db: db) { statement in
+                    +            var index: Int32 = 1
+                    +            bindDiffObservationIDs(request.fromObservationID, request.toObservationID, to: statement, startingAt: &index)
+                    +            for _ in 0..<3 {
+                    +                if let sampleTypeIdentifier = request.sampleTypeIdentifier {
+                    +                    bindText(sampleTypeIdentifier, to: index, in: statement)
+                    +                    index += 1
+                    +                }
+                    +            }
+                    +            guard sqlite3_step(statement) == SQLITE_ROW else {
+                    +                return HealthArchiveDiffSummary(
+                    +                    fromObservationID: request.fromObservationID,
+                    +                    toObservationID: request.toObservationID,
+                    +                    sampleTypeIdentifier: request.sampleTypeIdentifier,
+                    +                    appearedCount: 0,
+                    +                    disappearedCount: 0,
+                    +                    representationChangedCount: 0
+                    +                )
+                    +            }
+                    +            return HealthArchiveDiffSummary(
+                    +                fromObservationID: request.fromObservationID,
+                    +                toObservationID: request.toObservationID,
+                    +                sampleTypeIdentifier: request.sampleTypeIdentifier,
+                    +                appearedCount: columnInt(statement, 0) ?? 0,
+                    +                disappearedCount: columnInt(statement, 1) ?? 0,
+                    +                representationChangedCount: columnInt(statement, 2) ?? 0
+                    +            )
+                    +        }
+                    +    }
+                    +    func diffRecords(_ request: HealthArchiveDiffRecordRequest) async throws -> [ArchivedHealthRecord] {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        let selectedRangeSQL: String
+                    +        switch request.kind {
+                    +        case .appeared:
+                    +            selectedRangeSQL = """
+                    +            SELECT tv.sample_id, tv.version_id
+                    +            FROM to_visible tv
+                    +            LEFT JOIN from_visible fv ON fv.sample_id = tv.sample_id
+                    +            WHERE fv.sample_id IS NULL
+                    +            """
+                    +        case .disappeared:
+                    +            selectedRangeSQL = """
+                    +            SELECT fv.sample_id, fv.version_id
+                    +            FROM from_visible fv
+                    +            LEFT JOIN to_visible tv ON tv.sample_id = fv.sample_id
+                    +            WHERE tv.sample_id IS NULL
+                    +            """
+                    +        case .representationChanged:
+                    +            selectedRangeSQL = """
+                    +            SELECT tv.sample_id, tv.version_id
+                    +            FROM to_visible tv
+                    +            JOIN from_visible fv ON fv.sample_id = tv.sample_id
+                    +            WHERE tv.version_id != fv.version_id
+                    +            """
+                    +        }
+                    +        var clauses: [String] = []
+                    +        if request.sampleTypeIdentifier != nil {
+                    +            clauses.append("t.type_identifier = ?")
+                    +        }
+                    +        if request.afterCursor != nil {
+                    +            clauses.append("(v.start_date > ? OR (v.start_date = ? AND s.strict_fingerprint > ?))")
+                    +        }
+                    +        let whereClause = clauses.isEmpty ? "" : "WHERE \(clauses.joined(separator: " AND "))"
+                    +        let limitClause = request.limit.map { "LIMIT \(max($0, 0))" } ?? ""
+                    +        let sql = """
+                    +        WITH from_visible AS (
+                    +            SELECT sample_id, version_id
+                    +            FROM sample_visibility_ranges
+                    +            WHERE first_observation_id <= ?
+                    +              AND (last_observation_id IS NULL OR last_observation_id > ?)
+                    +        ),
+                    +        to_visible AS (
+                    +            SELECT sample_id, version_id
+                    +            FROM sample_visibility_ranges
+                    +            WHERE first_observation_id <= ?
+                    +              AND (last_observation_id IS NULL OR last_observation_id > ?)
+                    +        ),
+                    +        selected_ranges AS (
+                    +            \(selectedRangeSQL)
+                    +        ),
+                    +        event_summary AS (
+                    +            SELECT
+                    +                sample_id,
+                    +                MAX(CASE WHEN event_kind != 'disappeared' THEN observed_at END) AS last_seen_at,
+                    +                MAX(observed_at) AS last_verified_at,
+                    +                MAX(CASE WHEN event_kind = 'disappeared' THEN observed_at END) AS disappeared_at
+                    +            FROM sample_observation_events
+                    +            WHERE observation_id <= ?
+                    +            GROUP BY sample_id
+                    +        )
+                    +        SELECT
+                    +               COALESCE(s.sample_uuid_hash, s.strict_fingerprint) AS record_id,
+                    +               t.type_identifier, s.strict_fingerprint, s.semantic_fingerprint, s.sample_uuid_hash,
+                    +               v.start_date, v.end_date, s.first_seen_at,
+                    +               COALESCE(es.last_seen_at, s.first_seen_at) AS last_seen_at,
+                    +               es.last_verified_at,
+                    +               es.disappeared_at,
+                    +               v.value_kind, v.numeric_value, v.unit, v.category_value, v.workout_activity_type, v.duration_seconds,
+                    +               src.bundle_identifier
+                    +        FROM selected_ranges srng
+                    +        JOIN samples s ON s.id = srng.sample_id
+                    +        JOIN sample_types t ON t.id = s.sample_type_id
+                    +        JOIN sample_versions v ON v.id = srng.version_id
+                    +        LEFT JOIN source_revisions sr ON sr.id = v.source_revision_id
+                    +        LEFT JOIN sources src ON src.id = sr.source_id
+                    +        LEFT JOIN event_summary es ON es.sample_id = s.id
+                    +        \(whereClause)
+                    +        ORDER BY v.start_date ASC, s.strict_fingerprint ASC
+                    +        \(limitClause)
+                    +        """
+                    +        return try withStatement(sql, db: db) { statement in
+                    +            var index: Int32 = 1
+                    +            bindDiffObservationIDs(request.fromObservationID, request.toObservationID, to: statement, startingAt: &index)
+                    +            bindInt64(request.toObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            if let sampleTypeIdentifier = request.sampleTypeIdentifier {
+                    +                bindText(sampleTypeIdentifier, to: index, in: statement)
+                    +                index += 1
+                    +            }
+                    +            if let cursor = request.afterCursor {
+                    +                sqlite3_bind_double(statement, index, cursor.startDate.timeIntervalSince1970)
+                    +                index += 1
+                    +                sqlite3_bind_double(statement, index, cursor.startDate.timeIntervalSince1970)
+                    +                index += 1
+                    +                bindText(cursor.strictFingerprint, to: index, in: statement)
+                    +                index += 1
+                    +            }
+                    +            var records: [ArchivedHealthRecord] = []
+                    +            while sqlite3_step(statement) == SQLITE_ROW {
+                    +                records.append(archiveRecord(from: statement))
+                                 return records
+                    +    private func bindDiffObservationIDs(
+                    +        _ fromObservationID: Int64,
+                    +        _ toObservationID: Int64,
+                    +        to statement: OpaquePointer?,
+                    +        startingAt index: inout Int32
+                    +    ) {
+                    +        bindInt64(fromObservationID, to: index, in: statement)
+                    +        index += 1
+                    +        bindInt64(fromObservationID, to: index, in: statement)
+                    +        index += 1
+                    +        bindInt64(toObservationID, to: index, in: statement)
+                    +        index += 1
+                    +        bindInt64(toObservationID, to: index, in: statement)
+                    +        index += 1
+                    +    }
+                    +    private func archiveRecord(from statement: OpaquePointer?) -> ArchivedHealthRecord {
+                    +        ArchivedHealthRecord(
+                    +            id: columnText(statement, 0) ?? "",
+                    +            sampleTypeIdentifier: columnText(statement, 1) ?? "",
+                    +            strictFingerprint: columnText(statement, 2) ?? "",
+                    +            semanticFingerprint: columnText(statement, 3),
+                    +            healthKitUUIDHash: columnText(statement, 4),
+                    +            startDate: columnUnixDate(statement, 5) ?? Date(timeIntervalSince1970: 0),
+                    +            endDate: columnUnixDate(statement, 6) ?? Date(timeIntervalSince1970: 0),
+                    +            firstSeenAt: columnUnixDate(statement, 7) ?? Date(timeIntervalSince1970: 0),
+                    +            lastSeenAt: columnUnixDate(statement, 8),
+                    +            lastVerifiedAt: columnUnixDate(statement, 9),
+                    +            disappearedAt: columnUnixDate(statement, 10),
+                    +            valueKind: columnText(statement, 11),
+                    +            value: columnDouble(statement, 12),
+                    +            unit: columnText(statement, 13),
+                    +            categoryValue: columnInt(statement, 14),
+                    +            workoutActivityType: columnInt(statement, 15),
+                    +            durationSeconds: columnDouble(statement, 16),
+                    +            sourceName: nil,
+                    +            sourceBundleIdentifier: columnText(statement, 17),
+                    +            deviceName: nil
+                    +        )
+                    +    }
+                         private func requiredInt64(
+                             _ sql: String,
+                             db: OpaquePointer?,

+85 -4

HealthProbeTests/SQLiteHealthArchiveStoreTests.swift

View

+                             XCTAssertTrue(report.passed)
+                    +    func testDiffSummaryAndRecordsBetweenObservationsUseSQLVisibility() async throws {
+                    +        let url = databaseURL()
+                    +        let store = SQLiteHealthArchiveStore(databaseURL: url)
+                    +        let firstSample = makeStepCountSample(value: 42, start: 1_000)
+                    +        let secondSample = makeStepCountSample(value: 7, start: 2_000)
+                    +        let typeIdentifier = HKQuantityTypeIdentifier.stepCount.rawValue
+                    +        _ = try await store.upsertSamples([firstSample], observedAt: Date(timeIntervalSince1970: 3_000))
+                    +        _ = try await store.upsertSamples([firstSample, secondSample], observedAt: Date(timeIntervalSince1970: 3_060))
+                    +        try await store.recordDisappearance(
+                    +            sampleUUIDHash: HashService.sampleUUIDHash(firstSample.uuid.uuidString),
+                    +            sampleTypeIdentifier: typeIdentifier,
+                    +            observedMissingAt: Date(timeIntervalSince1970: 3_120)
+                    +        )
+                    +        let observationIDs = try observationIDs(at: url)
+                    +        XCTAssertEqual(observationIDs.count, 3)
+                    +        let appearedSummary = try await store.diffSummary(HealthArchiveDiffRequest(
+                    +            fromObservationID: observationIDs[0],
+                    +            toObservationID: observationIDs[1],
+                    +            sampleTypeIdentifier: typeIdentifier
+                    +        ))
+                    +        let appearedRecords = try await store.diffRecords(HealthArchiveDiffRecordRequest(
+                    +            fromObservationID: observationIDs[0],
+                    +            toObservationID: observationIDs[1],
+                    +            sampleTypeIdentifier: typeIdentifier,
+                    +            kind: .appeared,
+                    +            limit: 10
+                    +        ))
+                    +        let disappearedSummary = try await store.diffSummary(HealthArchiveDiffRequest(
+                    +            fromObservationID: observationIDs[1],
+                    +            toObservationID: observationIDs[2],
+                    +            sampleTypeIdentifier: typeIdentifier
+                    +        ))
+                    +        let disappearedRecords = try await store.diffRecords(HealthArchiveDiffRecordRequest(
+                    +            fromObservationID: observationIDs[1],
+                    +            toObservationID: observationIDs[2],
+                    +            sampleTypeIdentifier: typeIdentifier,
+                    +            kind: .disappeared,
+                    +            limit: 10
+                    +        ))
+                    +        XCTAssertEqual(appearedSummary.appearedCount, 1)
+                    +        XCTAssertEqual(appearedSummary.disappearedCount, 0)
+                    +        XCTAssertEqual(appearedSummary.representationChangedCount, 0)
+                    +        XCTAssertEqual(appearedRecords.map(\.displayValue), ["7.0 count"])
+                    +        XCTAssertEqual(disappearedSummary.appearedCount, 0)
+                    +        XCTAssertEqual(disappearedSummary.disappearedCount, 1)
+                    +        XCTAssertEqual(disappearedSummary.representationChangedCount, 0)
+                    +        XCTAssertEqual(disappearedRecords.map(\.displayValue), ["42.0 count"])
+                    +        XCTAssertNotNil(disappearedRecords.first?.disappearedAt)
+                    +    }
+                         private func databaseURL() -> URL {
+                             temporaryDirectory.appending(path: "Archive.sqlite")
+                         private func makeStepCountSample() -> HKQuantitySample {
+                    +        makeStepCountSample(value: 42, start: 1_000)
+                    +    }
+                    +    private func makeStepCountSample(value: Double, start: TimeInterval) -> HKQuantitySample {
+                             let quantityType = HKQuantityType.quantityType(forIdentifier: .stepCount)!
+                    -        let quantity = HKQuantity(unit: .count(), doubleValue: 42)
+                    -        let start = Date(timeIntervalSince1970: 1_000)
+                    -        let end = Date(timeIntervalSince1970: 1_300)
+                    -        return HKQuantitySample(type: quantityType, quantity: quantity, start: start, end: end)
+                    +        let quantity = HKQuantity(unit: .count(), doubleValue: value)
+                    +        let startDate = Date(timeIntervalSince1970: start)
+                    +        let endDate = Date(timeIntervalSince1970: start + 300)
+                    +        return HKQuantitySample(type: quantityType, quantity: quantity, start: startDate, end: endDate)
+                         private func countRows(in tableName: String, at url: URL) throws -> Int {
+                             return Int(sqlite3_column_int(statement, 0))
+                    +    private func observationIDs(at url: URL) throws -> [Int64] {
+                    +        var db: OpaquePointer?
+                    +        guard sqlite3_open_v2(url.path, &db, SQLITE_OPEN_READONLY | SQLITE_OPEN_FULLMUTEX, nil) == SQLITE_OK else {
+                    +            sqlite3_close(db)
+                    +            XCTFail("Could not open test database")
+                    +            return []
+                    +        }
+                    +        defer { sqlite3_close(db) }
+                    +        var statement: OpaquePointer?
+                    +        guard sqlite3_prepare_v2(db, "SELECT id FROM observations ORDER BY id", -1, &statement, nil) == SQLITE_OK else {
+                    +            sqlite3_finalize(statement)
+                    +            XCTFail("Could not prepare observation query")
+                    +            return []
+                    +        }
+                    +        defer { sqlite3_finalize(statement) }
+                    +        var ids: [Int64] = []
+                    +        while sqlite3_step(statement) == SQLITE_ROW {
+                    +            ids.append(sqlite3_column_int64(statement, 0))
+                    +        }
+                    +        return ids
+                    +    }
+                         private func sampleVersionDebugRows(at url: URL) throws -> String {
+                             var db: OpaquePointer?
+                             guard sqlite3_open_v2(url.path, &db, SQLITE_OPEN_READONLY | SQLITE_OPEN_FULLMUTEX, nil) == SQLITE_OK else {

	@@ -145,10 +145,10 @@ Acceptance:
145	145	Checklist:
146	146	- [x] Implement point-in-time visible-record query.
147	147	- [x] Implement paged record table query.
148		-- [ ] Implement appeared query between observations.
149		-- [ ] Implement disappeared query between observations.
150		-- [ ] Implement representationChanged query between observations.
151		-- [ ] Implement diff counts using temp tables or equivalent SQL-first strategy.
	148	+- [x] Implement appeared query between observations.
	149	+- [x] Implement disappeared query between observations.
	150	+- [x] Implement representationChanged query between observations.
	151	+- [x] Implement diff counts using temp tables or equivalent SQL-first strategy.
152	152	- [ ] Implement aggregate comparison query.
153	153	- [ ] Implement consolidation-likely evidence query.
154	154	- [ ] Implement source/provenance breakdown query.
	@@ -157,7 +157,7 @@ Checklist:
157	157	Acceptance:
158	158	- [x] Observation T can be reconstructed from ranges/events.
159	159	- [ ] Large diff returns counts and first page without loading all rows.
160		-- [ ] Query results are deterministic and ordered.
	160	+- [x] Query results are deterministic and ordered.
161	161	- [ ] Consolidation evidence includes count, aggregate, coverage, density, and uncertainty data.
162	162
163	163	## Milestone 6 - Core Data UI/Report Cache

	@@ -193,6 +193,12 @@ final class TypeDistributionBin {
193	193	// Deletions are recorded by sampleUUIDHash because HKDeletedObject exposes UUIDs,
194	194	// not complete sample payloads.
195	195
	196	+// Interface updated 2026-05-23 — see AGENTS.md
	197	+// HealthArchiveStore exposes SQL-first observation diff APIs:
	198	+// diffSummary(_:) returns appeared/disappeared/representationChanged counts and
	199	+// diffRecords(_:) returns a paged record list for one change kind. UI/cache agents
	200	+// should consume these APIs instead of loading full observation record sets.
	201	+
196	202	// Storage objective updated 2026-05-23 — see AGENTS.md
197	203	// Recurring complete snapshots are out of scope for the target architecture.
198	204	// Store differential observations, versioned sample payloads, observation ranges,

	@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab
25	25	\|------\|----------------\|--------------------\|
26	26	\| Product docs \| Updated \| Keep `HealthProbe/Doc/README.md` as canonical index \|
27	27	\| HealthKit capture \| Prototype exists \| Adapt capture to write differential SQLite observations first \|
28		-\| SQLite archive \| Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, and initial XCTest coverage are in place; legacy write mirror still exists \| Add SQL diff/count queries, large synthetic-data tests, then retire `archive_samples` \|
	28	+\| SQLite archive \| Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, initial SQL diff/count APIs, and XCTest coverage are in place; legacy write mirror still exists \| Add aggregate/provenance SQL analysis, large synthetic-data tests, then retire `archive_samples` \|
29	29	\| Core Data cache \| Not implemented \| Add rebuildable cache for expensive counts, summaries, report metadata, UI state \|
30	30	\| SwiftData cache \| Exists \| Treat as disposable prototype data; reset/ignore during v2 transition \|
31	31	\| UI \| Prototype exists \| Reframe screens around observations, diffs, export, archive status \|
	@@ -38,7 +38,7 @@ There are no real deployments, only test installations. Existing prototype datab
38	38
39	39	Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.md).
40	40
41		-1. Move large diffs/counts into SQL queries with indexes/temp tables/paged results.
	41	+1. Add aggregate/provenance/consolidation evidence queries on top of the SQLite archive.
42	42	2. Expand the synthetic large-data test harness for diff/export memory behavior.
43	43	3. Add Core Data UI/report cache and rebuild pipeline.
44	44	4. Replace SwiftData UI dependencies with Core Data/cache DTOs.
	@@ -54,7 +54,7 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m
54	54	- Current archive schema is not sufficient as the long-term source of truth.
55	55	- Existing implementation may decode or cache too much data for low-end devices.
56	56	- Old prototype database compatibility is no longer required.
57		-- Initial SQLite archive tests cover open/init/reset/idempotency, but not yet large-volume diff/export behavior.
	57	+- Initial SQLite archive tests cover open/init/reset/idempotency and small observation diffs, but not yet large-volume diff/export behavior.
58	58
59	59	## Verification Checklist
60	60