Add archive aggregate comparison query · 8d05fdb

Add archive aggregate comparison query
Browse files

bogdan committed 2 weeks ago

main

1 parent 9ceeb21

commit 8d05fdb

Showing 6 changed files with 187 additions and 6 deletions

+4 -2

HealthProbe/Doc/00-agent-guides/AGENTS.md

@@ -196,8 +196,10 @@ final class TypeDistributionBin {
 // Interface updated 2026-05-23 — see AGENTS.md
 // HealthArchiveStore exposes SQL-first observation diff APIs:
 // diffSummary(_:) returns appeared/disappeared/representationChanged counts and
-// diffRecords(_:) returns a paged record list for one change kind. UI/cache agents
-// should consume these APIs instead of loading full observation record sets.
+// diffRecords(_:) returns a paged record list for one change kind.
+// aggregateComparison(_:) compares materialized daily aggregates between two
+// observations. UI/cache agents should consume these APIs instead of loading full
+// observation record sets.
 
 // Storage objective updated 2026-05-23 — see AGENTS.md
 // Recurring complete snapshots are out of scope for the target architecture.


+3 -3

HealthProbe/Doc/04-project/IMPLEMENTATION_STATUS.md

View

@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab
 |------|----------------|--------------------|
 | Product docs | Updated | Keep `HealthProbe/Doc/README.md` as canonical index |
 | HealthKit capture | Prototype exists | Adapt capture to write differential SQLite observations first |
-| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, initial SQL diff/count APIs, and XCTest coverage are in place; legacy write mirror still exists | Add aggregate/provenance SQL analysis, large synthetic-data tests, then retire `archive_samples` |
+| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, initial SQL diff/count/aggregate APIs, and XCTest coverage are in place; legacy write mirror still exists | Add provenance and consolidation-evidence SQL analysis, large synthetic-data tests, then retire `archive_samples` |
 | Core Data cache | Not implemented | Add rebuildable cache for expensive counts, summaries, report metadata, UI state |
 | SwiftData cache | Exists | Treat as disposable prototype data; reset/ignore during v2 transition |
 | UI | Prototype exists | Reframe screens around observations, diffs, export, archive status |
@@ -38,7 +38,7 @@ There are no real deployments, only test installations. Existing prototype datab
 
 Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.md).
 
-1. Add aggregate/provenance/consolidation evidence queries on top of the SQLite archive.
+1. Add provenance and consolidation-evidence queries on top of the SQLite archive.
 2. Expand the synthetic large-data test harness for diff/export memory behavior.
 3. Add Core Data UI/report cache and rebuild pipeline.
 4. Replace SwiftData UI dependencies with Core Data/cache DTOs.
@@ -54,7 +54,7 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m
 - Current archive schema is not sufficient as the long-term source of truth.
 - Existing implementation may decode or cache too much data for low-end devices.
 - Old prototype database compatibility is no longer required.
-- Initial SQLite archive tests cover open/init/reset/idempotency and small observation diffs, but not yet large-volume diff/export behavior.
+- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, and materialized aggregate comparison, but not yet large-volume diff/export behavior.
 
 ## Verification Checklist
 


+1 -1

HealthProbe/Doc/04-project/Refactoring-Plan.md

View

@@ -149,7 +149,7 @@ Checklist:
 - [x] Implement disappeared query between observations.
 - [x] Implement representationChanged query between observations.
 - [x] Implement diff counts using temp tables or equivalent SQL-first strategy.
-- [ ] Implement aggregate comparison query.
+- [x] Implement aggregate comparison query.
 - [ ] Implement consolidation-likely evidence query.
 - [ ] Implement source/provenance breakdown query.
 - [ ] Add query timing/memory tests on synthetic large datasets.


+50 -0

HealthProbe/Services/Protocols/HealthArchiveStore.swift

View

+                         func records(for request: HealthArchiveRecordRequest) async throws -> [ArchivedHealthRecord]
+                         func diffSummary(_ request: HealthArchiveDiffRequest) async throws -> HealthArchiveDiffSummary
+                         func diffRecords(_ request: HealthArchiveDiffRecordRequest) async throws -> [ArchivedHealthRecord]
+                    +    func aggregateComparison(_ request: HealthArchiveAggregateComparisonRequest) async throws -> [HealthArchiveAggregateComparisonRow]
+                         func exportReport(_ request: HealthArchiveReportRequest) async throws -> URL
+                         func checkIntegrity() async throws -> HealthArchiveIntegrityReport
+                    +struct HealthArchiveAggregateComparisonRequest: Equatable, Sendable {
+                    +    let fromObservationID: Int64
+                    +    let toObservationID: Int64
+                    +    let sampleTypeIdentifier: String?
+                    +    let afterBucketStart: Date?
+                    +    let limit: Int?
+                    +    init(
+                    +        fromObservationID: Int64,
+                    +        toObservationID: Int64,
+                    +        sampleTypeIdentifier: String? = nil,
+                    +        afterBucketStart: Date? = nil,
+                    +        limit: Int? = nil
+                    +    ) {
+                    +        self.fromObservationID = fromObservationID
+                    +        self.toObservationID = toObservationID
+                    +        self.sampleTypeIdentifier = sampleTypeIdentifier
+                    +        self.afterBucketStart = afterBucketStart
+                    +        self.limit = limit
+                    +    }
+                    +}
+                    +struct HealthArchiveAggregateComparisonRow: Equatable, Sendable {
+                    +    let sampleTypeIdentifier: String
+                    +    let bucketStart: Date
+                    +    let bucketEnd: Date
+                    +    let fromVisibleRecordCount: Int
+                    +    let toVisibleRecordCount: Int
+                    +    let fromValueSum: Double?
+                    +    let toValueSum: Double?
+                    +    var visibleRecordDelta: Int {
+                    +        toVisibleRecordCount - fromVisibleRecordCount
+                    +    }
+                    +    var valueSumDelta: Double? {
+                    +        switch (fromValueSum, toValueSum) {
+                    +        case let (fromValueSum?, toValueSum?):
+                    +            return toValueSum - fromValueSum
+                    +        case let (nil, toValueSum?):
+                    +            return toValueSum
+                    +        case let (fromValueSum?, nil):
+                    +            return -fromValueSum
+                    +        case (nil, nil):
+                    +            return nil
+                    +        }
+                    +    }
+                    +}
+                     struct HealthArchiveReportRequest: Equatable, Sendable {
+                         let reportID: UUID
+                         let title: String

+100 -0

HealthProbe/Services/SQLiteHealthArchiveStore.swift

View

+                    +    func aggregateComparison(_ request: HealthArchiveAggregateComparisonRequest) async throws -> [HealthArchiveAggregateComparisonRow] {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        var clauses: [String] = []
+                    +        if request.sampleTypeIdentifier != nil {
+                    +            clauses.append("t.type_identifier = ?")
+                    +        }
+                    +        if request.afterBucketStart != nil {
+                    +            clauses.append("k.bucket_start > ?")
+                    +        }
+                    +        let whereClause = clauses.isEmpty ? "" : "WHERE \(clauses.joined(separator: " AND "))"
+                    +        let limitClause = request.limit.map { "LIMIT \(max($0, 0))" } ?? ""
+                    +        let sql = """
+                    +        WITH from_aggregates AS (
+                    +            SELECT
+                    +                sample_type_id,
+                    +                bucket_start,
+                    +                bucket_end,
+                    +                SUM(visible_record_count) AS visible_record_count,
+                    +                SUM(value_sum) AS value_sum
+                    +            FROM daily_type_aggregates
+                    +            WHERE observation_id = ?
+                    +            GROUP BY sample_type_id, bucket_start, bucket_end
+                    +        ),
+                    +        to_aggregates AS (
+                    +            SELECT
+                    +                sample_type_id,
+                    +                bucket_start,
+                    +                bucket_end,
+                    +                SUM(visible_record_count) AS visible_record_count,
+                    +                SUM(value_sum) AS value_sum
+                    +            FROM daily_type_aggregates
+                    +            WHERE observation_id = ?
+                    +            GROUP BY sample_type_id, bucket_start, bucket_end
+                    +        ),
+                    +        aggregate_keys AS (
+                    +            SELECT sample_type_id, bucket_start, bucket_end FROM from_aggregates
+                    +            UNION
+                    +            SELECT sample_type_id, bucket_start, bucket_end FROM to_aggregates
+                    +        )
+                    +        SELECT
+                    +            t.type_identifier,
+                    +            k.bucket_start,
+                    +            k.bucket_end,
+                    +            COALESCE(f.visible_record_count, 0) AS from_visible_record_count,
+                    +            COALESCE(ta.visible_record_count, 0) AS to_visible_record_count,
+                    +            f.value_sum,
+                    +            ta.value_sum
+                    +        FROM aggregate_keys k
+                    +        JOIN sample_types t ON t.id = k.sample_type_id
+                    +        LEFT JOIN from_aggregates f
+                    +          ON f.sample_type_id = k.sample_type_id
+                    +         AND f.bucket_start = k.bucket_start
+                    +         AND f.bucket_end = k.bucket_end
+                    +        LEFT JOIN to_aggregates ta
+                    +          ON ta.sample_type_id = k.sample_type_id
+                    +         AND ta.bucket_start = k.bucket_start
+                    +         AND ta.bucket_end = k.bucket_end
+                    +        \(whereClause)
+                    +        ORDER BY k.bucket_start ASC, t.type_identifier ASC
+                    +        \(limitClause)
+                    +        """
+                    +        return try withStatement(sql, db: db) { statement in
+                    +            var index: Int32 = 1
+                    +            bindInt64(request.fromObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            bindInt64(request.toObservationID, to: index, in: statement)
+                    +            index += 1
+                    +            if let sampleTypeIdentifier = request.sampleTypeIdentifier {
+                    +                bindText(sampleTypeIdentifier, to: index, in: statement)
+                    +                index += 1
+                    +            }
+                    +            if let afterBucketStart = request.afterBucketStart {
+                    +                sqlite3_bind_double(statement, index, afterBucketStart.timeIntervalSince1970)
+                    +                index += 1
+                    +            }
+                    +            var rows: [HealthArchiveAggregateComparisonRow] = []
+                    +            while sqlite3_step(statement) == SQLITE_ROW {
+                    +                guard let bucketStart = columnUnixDate(statement, 1),
+                    +                      let bucketEnd = columnUnixDate(statement, 2) else {
+                    +                    continue
+                    +                }
+                    +                rows.append(HealthArchiveAggregateComparisonRow(
+                    +                    sampleTypeIdentifier: columnText(statement, 0) ?? "",
+                    +                    bucketStart: bucketStart,
+                    +                    bucketEnd: bucketEnd,
+                    +                    fromVisibleRecordCount: columnInt(statement, 3) ?? 0,
+                    +                    toVisibleRecordCount: columnInt(statement, 4) ?? 0,
+                    +                    fromValueSum: columnDouble(statement, 5),
+                    +                    toValueSum: columnDouble(statement, 6)
+                    +                ))
+                    +            }
+                    +            return rows
+                    +        }
+                    +    }
+                         func exportReport(_ request: HealthArchiveReportRequest) async throws -> URL {
+                             let recordRequest = HealthArchiveRecordRequest(
+                                 sampleTypeIdentifier: request.typeIdentifierFilter,

+29 -0

HealthProbeTests/SQLiteHealthArchiveStoreTests.swift

View

+                             XCTAssertNotNil(disappearedRecords.first?.disappearedAt)
+                    +    func testAggregateComparisonUsesMaterializedDailyAggregates() async throws {
+                    +        let url = databaseURL()
+                    +        let store = SQLiteHealthArchiveStore(databaseURL: url)
+                    +        let firstSample = makeStepCountSample(value: 42, start: 1_000)
+                    +        let secondSample = makeStepCountSample(value: 7, start: 2_000)
+                    +        let typeIdentifier = HKQuantityTypeIdentifier.stepCount.rawValue
+                    +        _ = try await store.upsertSamples([firstSample], observedAt: Date(timeIntervalSince1970: 3_000))
+                    +        _ = try await store.upsertSamples([firstSample, secondSample], observedAt: Date(timeIntervalSince1970: 3_060))
+                    +        let observationIDs = try observationIDs(at: url)
+                    +        XCTAssertEqual(observationIDs.count, 2)
+                    +        let rows = try await store.aggregateComparison(HealthArchiveAggregateComparisonRequest(
+                    +            fromObservationID: observationIDs[0],
+                    +            toObservationID: observationIDs[1],
+                    +            sampleTypeIdentifier: typeIdentifier,
+                    +            limit: 10
+                    +        ))
+                    +        XCTAssertEqual(rows.count, 1)
+                    +        XCTAssertEqual(rows.first?.sampleTypeIdentifier, typeIdentifier)
+                    +        XCTAssertEqual(rows.first?.fromVisibleRecordCount, 1)
+                    +        XCTAssertEqual(rows.first?.toVisibleRecordCount, 2)
+                    +        XCTAssertEqual(rows.first?.visibleRecordDelta, 1)
+                    +        XCTAssertEqual(rows.first?.fromValueSum, 42)
+                    +        XCTAssertEqual(rows.first?.toValueSum, 49)
+                    +        XCTAssertEqual(rows.first?.valueSumDelta, 7)
+                    +    }
+                         private func databaseURL() -> URL {
+                             temporaryDirectory.appending(path: "Archive.sqlite")

	@@ -149,7 +149,7 @@ Checklist:
149	149	- [x] Implement disappeared query between observations.
150	150	- [x] Implement representationChanged query between observations.
151	151	- [x] Implement diff counts using temp tables or equivalent SQL-first strategy.
152		-- [ ] Implement aggregate comparison query.
	152	+- [x] Implement aggregate comparison query.
153	153	- [ ] Implement consolidation-likely evidence query.
154	154	- [ ] Implement source/provenance breakdown query.
155	155	- [ ] Add query timing/memory tests on synthetic large datasets.

	@@ -196,8 +196,10 @@ final class TypeDistributionBin {
196	196	// Interface updated 2026-05-23 — see AGENTS.md
197	197	// HealthArchiveStore exposes SQL-first observation diff APIs:
198	198	// diffSummary(_:) returns appeared/disappeared/representationChanged counts and
199		-// diffRecords(_:) returns a paged record list for one change kind. UI/cache agents
200		-// should consume these APIs instead of loading full observation record sets.
	199	+// diffRecords(_:) returns a paged record list for one change kind.
	200	+// aggregateComparison(_:) compares materialized daily aggregates between two
	201	+// observations. UI/cache agents should consume these APIs instead of loading full
	202	+// observation record sets.
201	203
202	204	// Storage objective updated 2026-05-23 — see AGENTS.md
203	205	// Recurring complete snapshots are out of scope for the target architecture.

	@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab
25	25	\|------\|----------------\|--------------------\|
26	26	\| Product docs \| Updated \| Keep `HealthProbe/Doc/README.md` as canonical index \|
27	27	\| HealthKit capture \| Prototype exists \| Adapt capture to write differential SQLite observations first \|
28		-\| SQLite archive \| Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, initial SQL diff/count APIs, and XCTest coverage are in place; legacy write mirror still exists \| Add aggregate/provenance SQL analysis, large synthetic-data tests, then retire `archive_samples` \|
	28	+\| SQLite archive \| Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, initial SQL diff/count/aggregate APIs, and XCTest coverage are in place; legacy write mirror still exists \| Add provenance and consolidation-evidence SQL analysis, large synthetic-data tests, then retire `archive_samples` \|
29	29	\| Core Data cache \| Not implemented \| Add rebuildable cache for expensive counts, summaries, report metadata, UI state \|
30	30	\| SwiftData cache \| Exists \| Treat as disposable prototype data; reset/ignore during v2 transition \|
31	31	\| UI \| Prototype exists \| Reframe screens around observations, diffs, export, archive status \|
	@@ -38,7 +38,7 @@ There are no real deployments, only test installations. Existing prototype datab
38	38
39	39	Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.md).
40	40
41		-1. Add aggregate/provenance/consolidation evidence queries on top of the SQLite archive.
	41	+1. Add provenance and consolidation-evidence queries on top of the SQLite archive.
42	42	2. Expand the synthetic large-data test harness for diff/export memory behavior.
43	43	3. Add Core Data UI/report cache and rebuild pipeline.
44	44	4. Replace SwiftData UI dependencies with Core Data/cache DTOs.
	@@ -54,7 +54,7 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m
54	54	- Current archive schema is not sufficient as the long-term source of truth.
55	55	- Existing implementation may decode or cache too much data for low-end devices.
56	56	- Old prototype database compatibility is no longer required.
57		-- Initial SQLite archive tests cover open/init/reset/idempotency and small observation diffs, but not yet large-volume diff/export behavior.
	57	+- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, and materialized aggregate comparison, but not yet large-volume diff/export behavior.
58	58
59	59	## Verification Checklist
60	60