- Add large archive diff regression
- Browse files
-
bogdan committed 2 weeks ago
Showing 3 changed files
with
82 additions
and
10 deletions
+9
-8
HealthProbe/Doc/04-project/IMPLEMENTATION_STATUS.md
@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab |
||
| 25 | 25 |
|------|----------------|--------------------| |
| 26 | 26 |
| Product docs | Updated | Keep `HealthProbe/Doc/README.md` as canonical index | |
| 27 | 27 |
| HealthKit capture | Prototype exists | Adapt capture to write differential SQLite observations first | |
| 28 |
-| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, and XCTest coverage are in place; legacy write mirror still exists | Add large synthetic-data timing/memory tests, then retire `archive_samples` | |
|
| 28 |
+| SQLite archive | Archive v2 schema, differential write path, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, and XCTest coverage are in place; legacy write mirror still exists | Add formal timing/memory metrics, then retire `archive_samples` | |
|
| 29 | 29 |
| Core Data cache | Not implemented | Add rebuildable cache for expensive counts, summaries, report metadata, UI state | |
| 30 | 30 |
| SwiftData cache | Exists | Treat as disposable prototype data; reset/ignore during v2 transition | |
| 31 | 31 |
| UI | Prototype exists | Reframe screens around observations, diffs, export, archive status | |
@@ -38,12 +38,13 @@ There are no real deployments, only test installations. Existing prototype datab |
||
| 38 | 38 |
|
| 39 | 39 |
Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.md). |
| 40 | 40 |
|
| 41 |
-1. Expand the synthetic large-data test harness for diff/export memory behavior. |
|
| 42 |
-2. Add Core Data UI/report cache and rebuild pipeline. |
|
| 43 |
-3. Replace SwiftData UI dependencies with Core Data/cache DTOs. |
|
| 44 |
-4. Update UI language from anomaly/status to observation/diff/export. |
|
| 45 |
-5. Add streaming exports with manifests. |
|
| 46 |
-6. Validate on low-memory/legacy-class devices. |
|
| 41 |
+1. Add formal timing/memory metrics for synthetic SQLite queries. |
|
| 42 |
+2. Retire the legacy `archive_samples` write mirror. |
|
| 43 |
+3. Add Core Data UI/report cache and rebuild pipeline. |
|
| 44 |
+4. Replace SwiftData UI dependencies with Core Data/cache DTOs. |
|
| 45 |
+5. Update UI language from anomaly/status to observation/diff/export. |
|
| 46 |
+6. Add streaming exports with manifests. |
|
| 47 |
+7. Validate on low-memory/legacy-class devices. |
|
| 47 | 48 |
|
| 48 | 49 |
## Known Prototype Mismatches |
| 49 | 50 |
|
@@ -53,7 +54,7 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m |
||
| 53 | 54 |
- Current archive schema is not sufficient as the long-term source of truth. |
| 54 | 55 |
- Existing implementation may decode or cache too much data for low-end devices. |
| 55 | 56 |
- Old prototype database compatibility is no longer required. |
| 56 |
-- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet large-volume diff/export behavior. |
|
| 57 |
+- Initial SQLite archive tests cover open/init/reset/idempotency, small observation diffs, large synthetic diff pagination, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet export behavior or formal timing/memory metrics. |
|
| 57 | 58 |
|
| 58 | 59 |
## Verification Checklist |
| 59 | 60 |
|
+3
-2
HealthProbe/Doc/04-project/Refactoring-Plan.md
@@ -152,11 +152,12 @@ Checklist: |
||
| 152 | 152 |
- [x] Implement aggregate comparison query. |
| 153 | 153 |
- [x] Implement consolidation-likely evidence query. |
| 154 | 154 |
- [x] Implement source/provenance breakdown query. |
| 155 |
-- [ ] Add query timing/memory tests on synthetic large datasets. |
|
| 155 |
+- [x] Add large synthetic diff/pagination regression. |
|
| 156 |
+- [ ] Add formal query timing/memory metrics on synthetic large datasets. |
|
| 156 | 157 |
|
| 157 | 158 |
Acceptance: |
| 158 | 159 |
- [x] Observation T can be reconstructed from ranges/events. |
| 159 |
-- [ ] Large diff returns counts and first page without loading all rows. |
|
| 160 |
+- [x] Large diff returns counts and first page without loading all rows. |
|
| 160 | 161 |
- [x] Query results are deterministic and ordered. |
| 161 | 162 |
- [x] Consolidation evidence includes count, aggregate, coverage, density, and uncertainty data. |
| 162 | 163 |
|
+70
-0
HealthProbeTests/SQLiteHealthArchiveStoreTests.swift
@@ -130,6 +130,66 @@ final class SQLiteHealthArchiveStoreTests: XCTestCase {
|
||
| 130 | 130 |
XCTAssertNotNil(disappearedRecords.first?.disappearedAt) |
| 131 | 131 |
} |
| 132 | 132 |
|
| 133 |
+ func testLargeSyntheticDiffReturnsCountsAndBoundedPages() async throws {
|
|
| 134 |
+ let url = databaseURL() |
|
| 135 |
+ let store = SQLiteHealthArchiveStore(databaseURL: url) |
|
| 136 |
+ let typeIdentifier = HKQuantityTypeIdentifier.stepCount.rawValue |
|
| 137 |
+ let initialCount = 1_200 |
|
| 138 |
+ let appearedCount = 180 |
|
| 139 |
+ let disappearedCount = 160 |
|
| 140 |
+ let pageSize = 25 |
|
| 141 |
+ let initialSamples = makeStepCountSamples(count: initialCount, startIndex: 0) |
|
| 142 |
+ let appearedSamples = makeStepCountSamples(count: appearedCount, startIndex: initialCount) |
|
| 143 |
+ |
|
| 144 |
+ _ = try await store.upsertSamples(initialSamples, observedAt: Date(timeIntervalSince1970: 1_000_000)) |
|
| 145 |
+ _ = try await store.upsertSamples(initialSamples + appearedSamples, observedAt: Date(timeIntervalSince1970: 1_000_060)) |
|
| 146 |
+ for (offset, sample) in initialSamples.prefix(disappearedCount).enumerated() {
|
|
| 147 |
+ try await store.recordDisappearance( |
|
| 148 |
+ sampleUUIDHash: HashService.sampleUUIDHash(sample.uuid.uuidString), |
|
| 149 |
+ sampleTypeIdentifier: typeIdentifier, |
|
| 150 |
+ observedMissingAt: Date(timeIntervalSince1970: 1_000_120 + Double(offset)) |
|
| 151 |
+ ) |
|
| 152 |
+ } |
|
| 153 |
+ |
|
| 154 |
+ let observationIDs = try observationIDs(at: url) |
|
| 155 |
+ let firstObservationID = try XCTUnwrap(observationIDs.first) |
|
| 156 |
+ let lastObservationID = try XCTUnwrap(observationIDs.last) |
|
| 157 |
+ let queryStartedAt = Date() |
|
| 158 |
+ let summary = try await store.diffSummary(HealthArchiveDiffRequest( |
|
| 159 |
+ fromObservationID: firstObservationID, |
|
| 160 |
+ toObservationID: lastObservationID, |
|
| 161 |
+ sampleTypeIdentifier: typeIdentifier |
|
| 162 |
+ )) |
|
| 163 |
+ let firstPage = try await store.diffRecords(HealthArchiveDiffRecordRequest( |
|
| 164 |
+ fromObservationID: firstObservationID, |
|
| 165 |
+ toObservationID: lastObservationID, |
|
| 166 |
+ sampleTypeIdentifier: typeIdentifier, |
|
| 167 |
+ kind: .appeared, |
|
| 168 |
+ limit: pageSize |
|
| 169 |
+ )) |
|
| 170 |
+ let firstPageLastRecord = try XCTUnwrap(firstPage.last) |
|
| 171 |
+ let secondPage = try await store.diffRecords(HealthArchiveDiffRecordRequest( |
|
| 172 |
+ fromObservationID: firstObservationID, |
|
| 173 |
+ toObservationID: lastObservationID, |
|
| 174 |
+ sampleTypeIdentifier: typeIdentifier, |
|
| 175 |
+ kind: .appeared, |
|
| 176 |
+ afterCursor: RecordCursor( |
|
| 177 |
+ startDate: firstPageLastRecord.startDate, |
|
| 178 |
+ strictFingerprint: firstPageLastRecord.strictFingerprint |
|
| 179 |
+ ), |
|
| 180 |
+ limit: pageSize |
|
| 181 |
+ )) |
|
| 182 |
+ let queryElapsedSeconds = Date().timeIntervalSince(queryStartedAt) |
|
| 183 |
+ |
|
| 184 |
+ XCTAssertEqual(summary.appearedCount, appearedCount) |
|
| 185 |
+ XCTAssertEqual(summary.disappearedCount, disappearedCount) |
|
| 186 |
+ XCTAssertEqual(summary.representationChangedCount, 0) |
|
| 187 |
+ XCTAssertEqual(firstPage.count, pageSize) |
|
| 188 |
+ XCTAssertEqual(secondPage.count, pageSize) |
|
| 189 |
+ XCTAssertTrue(Set(firstPage.map(\.id)).isDisjoint(with: Set(secondPage.map(\.id)))) |
|
| 190 |
+ XCTAssertLessThan(queryElapsedSeconds, 10) |
|
| 191 |
+ } |
|
| 192 |
+ |
|
| 133 | 193 |
func testAggregateComparisonUsesMaterializedDailyAggregates() async throws {
|
| 134 | 194 |
let url = databaseURL() |
| 135 | 195 |
let store = SQLiteHealthArchiveStore(databaseURL: url) |
@@ -274,6 +334,16 @@ final class SQLiteHealthArchiveStoreTests: XCTestCase {
|
||
| 274 | 334 |
return HKQuantitySample(type: quantityType, quantity: quantity, start: startDate, end: endDate) |
| 275 | 335 |
} |
| 276 | 336 |
|
| 337 |
+ private func makeStepCountSamples(count: Int, startIndex: Int) -> [HKQuantitySample] {
|
|
| 338 |
+ (0..<count).map { offset in
|
|
| 339 |
+ let index = startIndex + offset |
|
| 340 |
+ return makeStepCountSample( |
|
| 341 |
+ value: Double((index % 97) + 1), |
|
| 342 |
+ start: 10_000 + Double(index * 600) |
|
| 343 |
+ ) |
|
| 344 |
+ } |
|
| 345 |
+ } |
|
| 346 |
+ |
|
| 277 | 347 |
private func countRows(in tableName: String, at url: URL) throws -> Int {
|
| 278 | 348 |
var db: OpaquePointer? |
| 279 | 349 |
guard sqlite3_open_v2(url.path, &db, SQLITE_OPEN_READONLY | SQLITE_OPEN_FULLMUTEX, nil) == SQLITE_OK else {
|