- Add archive query performance metrics
- Browse files
-
bogdan committed 2 weeks ago
Showing 3 changed files
with
85 additions
and
10 deletions
+8
-9
HealthProbe/Doc/04-project/IMPLEMENTATION_STATUS.md
@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab |
||
| 25 | 25 |
|------|----------------|--------------------| |
| 26 | 26 |
| Product docs | Updated | Keep `HealthProbe/Doc/README.md` as canonical index | |
| 27 | 27 |
| HealthKit capture | Prototype exists | Adapt capture to write differential SQLite observations first | |
| 28 |
-| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, and XCTest coverage are in place; legacy write mirror still exists | Add formal timing/memory metrics, then retire `archive_samples` | |
|
| 28 |
+| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; legacy write mirror still exists | Investigate and retire `archive_samples`, then start Core Data cache work | |
|
| 29 | 29 |
| Core Data cache | Not implemented | Add rebuildable cache for expensive counts, summaries, report metadata, UI state | |
| 30 | 30 |
| SwiftData cache | Exists | Treat as disposable prototype data; reset/ignore during v2 transition | |
| 31 | 31 |
| UI | Prototype exists | Reframe screens around observations, diffs, export, archive status | |
@@ -38,13 +38,12 @@ There are no real deployments, only test installations. Existing prototype datab |
||
| 38 | 38 |
|
| 39 | 39 |
Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.md). |
| 40 | 40 |
|
| 41 |
-1. Add formal timing/memory metrics for synthetic SQLite queries. |
|
| 42 |
-2. Retire the legacy `archive_samples` write mirror. |
|
| 43 |
-3. Add Core Data UI/report cache and rebuild pipeline. |
|
| 44 |
-4. Replace SwiftData UI dependencies with Core Data/cache DTOs. |
|
| 45 |
-5. Update UI language from anomaly/status to observation/diff/export. |
|
| 46 |
-6. Add streaming exports with manifests. |
|
| 47 |
-7. Validate on low-memory/legacy-class devices. |
|
| 41 |
+1. Investigate why removing the legacy `archive_samples` write mirror changes v2 idempotency in tests, then retire it safely. |
|
| 42 |
+2. Add Core Data UI/report cache and rebuild pipeline. |
|
| 43 |
+3. Replace SwiftData UI dependencies with Core Data/cache DTOs. |
|
| 44 |
+4. Update UI language from anomaly/status to observation/diff/export. |
|
| 45 |
+5. Add streaming exports with manifests. |
|
| 46 |
+6. Validate on low-memory/legacy-class devices. |
|
| 48 | 47 |
|
| 49 | 48 |
## Known Prototype Mismatches |
| 50 | 49 |
|
@@ -54,7 +53,7 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m |
||
| 54 | 53 |
- Current archive schema is not sufficient as the long-term source of truth. |
| 55 | 54 |
- Existing implementation may decode or cache too much data for low-end devices. |
| 56 | 55 |
- Old prototype database compatibility is no longer required. |
| 57 |
-- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, large synthetic diff pagination, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet export behavior or formal timing/memory metrics. |
|
| 56 |
+- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, large synthetic diff pagination, formal timing/memory metrics, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet export behavior. |
|
| 58 | 57 |
|
| 59 | 58 |
## Verification Checklist |
| 60 | 59 |
|
+1
-1
HealthProbe/Doc/04-project/Refactoring-Plan.md
@@ -153,7 +153,7 @@ Checklist: |
||
| 153 | 153 |
- [x] Implement consolidation-likely evidence query. |
| 154 | 154 |
- [x] Implement source/provenance breakdown query. |
| 155 | 155 |
- [x] Add large synthetic diff/pagination regression. |
| 156 |
-- [ ] Add formal query timing/memory metrics on synthetic large datasets. |
|
| 156 |
+- [x] Add formal query timing/memory metrics on synthetic large datasets. |
|
| 157 | 157 |
|
| 158 | 158 |
Acceptance: |
| 159 | 159 |
- [x] Observation T can be reconstructed from ranges/events. |
+76
-0
HealthProbeTests/SQLiteHealthArchiveStoreTests.swift
@@ -4,6 +4,10 @@ import XCTest |
||
| 4 | 4 |
@testable import HealthProbe |
| 5 | 5 |
|
| 6 | 6 |
final class SQLiteHealthArchiveStoreTests: XCTestCase {
|
| 7 |
+ private final class AsyncResultBox<T>: @unchecked Sendable {
|
|
| 8 |
+ var result: Result<T, Error>? |
|
| 9 |
+ } |
|
| 10 |
+ |
|
| 7 | 11 |
private var temporaryDirectory: URL! |
| 8 | 12 |
|
| 9 | 13 |
override func setUpWithError() throws {
|
@@ -190,6 +194,61 @@ final class SQLiteHealthArchiveStoreTests: XCTestCase {
|
||
| 190 | 194 |
XCTAssertLessThan(queryElapsedSeconds, 10) |
| 191 | 195 |
} |
| 192 | 196 |
|
| 197 |
+ func testLargeSyntheticDiffQueryMetrics() throws {
|
|
| 198 |
+ let url = databaseURL() |
|
| 199 |
+ let store = SQLiteHealthArchiveStore(databaseURL: url) |
|
| 200 |
+ let typeIdentifier = HKQuantityTypeIdentifier.stepCount.rawValue |
|
| 201 |
+ let initialCount = 900 |
|
| 202 |
+ let appearedCount = 120 |
|
| 203 |
+ let disappearedCount = 100 |
|
| 204 |
+ let pageSize = 25 |
|
| 205 |
+ let initialSamples = makeStepCountSamples(count: initialCount, startIndex: 0) |
|
| 206 |
+ let appearedSamples = makeStepCountSamples(count: appearedCount, startIndex: initialCount) |
|
| 207 |
+ |
|
| 208 |
+ try waitForArchiveOperation {
|
|
| 209 |
+ _ = try await store.upsertSamples(initialSamples, observedAt: Date(timeIntervalSince1970: 1_200_000)) |
|
| 210 |
+ _ = try await store.upsertSamples(initialSamples + appearedSamples, observedAt: Date(timeIntervalSince1970: 1_200_060)) |
|
| 211 |
+ for (offset, sample) in initialSamples.prefix(disappearedCount).enumerated() {
|
|
| 212 |
+ try await store.recordDisappearance( |
|
| 213 |
+ sampleUUIDHash: HashService.sampleUUIDHash(sample.uuid.uuidString), |
|
| 214 |
+ sampleTypeIdentifier: typeIdentifier, |
|
| 215 |
+ observedMissingAt: Date(timeIntervalSince1970: 1_200_120 + Double(offset)) |
|
| 216 |
+ ) |
|
| 217 |
+ } |
|
| 218 |
+ } |
|
| 219 |
+ |
|
| 220 |
+ let observationIDs = try observationIDs(at: url) |
|
| 221 |
+ let firstObservationID = try XCTUnwrap(observationIDs.first) |
|
| 222 |
+ let lastObservationID = try XCTUnwrap(observationIDs.last) |
|
| 223 |
+ let options = XCTMeasureOptions() |
|
| 224 |
+ options.iterationCount = 3 |
|
| 225 |
+ |
|
| 226 |
+ measure(metrics: [XCTClockMetric(), XCTMemoryMetric()], options: options) {
|
|
| 227 |
+ do {
|
|
| 228 |
+ let result = try waitForArchiveOperation {
|
|
| 229 |
+ let summary = try await store.diffSummary(HealthArchiveDiffRequest( |
|
| 230 |
+ fromObservationID: firstObservationID, |
|
| 231 |
+ toObservationID: lastObservationID, |
|
| 232 |
+ sampleTypeIdentifier: typeIdentifier |
|
| 233 |
+ )) |
|
| 234 |
+ let records = try await store.diffRecords(HealthArchiveDiffRecordRequest( |
|
| 235 |
+ fromObservationID: firstObservationID, |
|
| 236 |
+ toObservationID: lastObservationID, |
|
| 237 |
+ sampleTypeIdentifier: typeIdentifier, |
|
| 238 |
+ kind: .appeared, |
|
| 239 |
+ limit: pageSize |
|
| 240 |
+ )) |
|
| 241 |
+ return (summary, records) |
|
| 242 |
+ } |
|
| 243 |
+ XCTAssertEqual(result.0.appearedCount, appearedCount) |
|
| 244 |
+ XCTAssertEqual(result.0.disappearedCount, disappearedCount) |
|
| 245 |
+ XCTAssertEqual(result.1.count, pageSize) |
|
| 246 |
+ } catch {
|
|
| 247 |
+ XCTFail("Measured archive query failed: \(error)")
|
|
| 248 |
+ } |
|
| 249 |
+ } |
|
| 250 |
+ } |
|
| 251 |
+ |
|
| 193 | 252 |
func testAggregateComparisonUsesMaterializedDailyAggregates() async throws {
|
| 194 | 253 |
let url = databaseURL() |
| 195 | 254 |
let store = SQLiteHealthArchiveStore(databaseURL: url) |
@@ -344,6 +403,23 @@ final class SQLiteHealthArchiveStoreTests: XCTestCase {
|
||
| 344 | 403 |
} |
| 345 | 404 |
} |
| 346 | 405 |
|
| 406 |
+ private func waitForArchiveOperation<T>(_ operation: @escaping () async throws -> T) throws -> T {
|
|
| 407 |
+ let expectation = expectation(description: "archive operation") |
|
| 408 |
+ let box = AsyncResultBox<T>() |
|
| 409 |
+ |
|
| 410 |
+ Task {
|
|
| 411 |
+ do {
|
|
| 412 |
+ box.result = .success(try await operation()) |
|
| 413 |
+ } catch {
|
|
| 414 |
+ box.result = .failure(error) |
|
| 415 |
+ } |
|
| 416 |
+ expectation.fulfill() |
|
| 417 |
+ } |
|
| 418 |
+ |
|
| 419 |
+ wait(for: [expectation], timeout: 20) |
|
| 420 |
+ return try XCTUnwrap(box.result).get() |
|
| 421 |
+ } |
|
| 422 |
+ |
|
| 347 | 423 |
private func countRows(in tableName: String, at url: URL) throws -> Int {
|
| 348 | 424 |
var db: OpaquePointer? |
| 349 | 425 |
guard sqlite3_open_v2(url.path, &db, SQLITE_OPEN_READONLY | SQLITE_OPEN_FULLMUTEX, nil) == SQLITE_OK else {
|