- Fast-path unchanged archive verification
- Browse files
-
bogdan committed 4 days ago
Showing 5 changed files
with
150 additions
and
7 deletions
+5
-0
HealthProbe/Doc/04-project/Import-Optimization-Log.md
@@ -578,6 +578,7 @@ rows exist". |
||
| 578 | 578 |
| 2026-06-03 | pending | Migrate legacy core-profile selections to full available capture by default. | A follow-up real-device report still showed the old `4907...` monitored type-set hash and `Types: 15/15 processed`, proving the running app still used the old persisted selected type set. New installs and pre-profile settings that exactly match the old core profile now migrate to `All available`; only an explicit `Select Core Profile` action persists the core subset. Settings also shows the active profile label (`All available`, `Core`, or `Custom`) for quick verification before capture. | |
| 579 | 579 |
| 2026-06-03 | pending | Harden quantity unit conversion for full-profile imports. | The first 127-metric run crashed while archiving `HKQuantityTypeIdentifierDietaryWater`: the previous fallback converted unknown quantities with `.count()`, and HealthKit raised `NSInvalidArgumentException` for incompatible `mL` to `count`. The archive now maps known extended units, stores quantity rows with nil numeric/unit when a future type is unmapped, and removes the unsafe display fallback. Next run should pass Water and reveal any remaining type-specific unit gaps. | |
| 580 | 580 |
| 2026-06-03 | pending | Preserve completed import diagnostics and render large diagnostic reports lazily. | A copied full-profile reimport report completed successfully: `Types: 127/127 processed`, `Records: 2,646,527`, `WallClock: 40.3s`, `SummedProcessingElapsed: 21.7s`, `SummedFinalizeElapsed: 15.1s`, `SummedFetchElapsed: 1.7s`, `SummedInsertElapsed: 0.1s`, and `0` degraded metrics. The Diagnostics sheet copied the report text but displayed a blank body for the huge report; completed/partial/review reports are now persisted under Application Support and displayed in chunked lazy text blocks. | |
| 581 |
+| 2026-06-03 | pending | Fast-path unchanged verification after an empty HealthKit delta page. | The successful full-profile reimport spent `15.1s` in finalize despite `0.1s` insert time; Heart Rate alone spent `9.0s` finalizing. The empty-delta path now calls an explicit unchanged-verification store method that copies the previous type summary and daily aggregates without first scanning sample events to prove zero changes. Expected signal: repeated no-delta/full-profile `SummedFinalizeElapsed` and Heart Rate finalize time drop sharply. | |
|
| 581 | 582 |
|
| 582 | 583 |
## Current Diagnosis |
| 583 | 584 |
|
@@ -616,6 +617,10 @@ The likely bottleneck is per-row SQLite work: |
||
| 616 | 617 |
metrics and 2.6M archived rows. This is not comparable to clean first-import |
| 617 | 618 |
timing, but it confirms that the expanded registry can pass the previous Water |
| 618 | 619 |
unit crash point. |
| 620 |
+- On unchanged full-profile reimports, `markVerification` can still be expensive |
|
| 621 |
+ if it counts nonexistent events for high-volume types. The capture layer knows |
|
| 622 |
+ the first anchored delta page is empty, so unchanged verification should skip |
|
| 623 |
+ event-count scans and copy previous materialized summaries directly. |
|
| 619 | 624 |
|
| 620 | 625 |
## Open Issues / Observations |
| 621 | 626 |
|
+39
-7
HealthProbe/Services/HealthKitService.swift
@@ -1089,7 +1089,7 @@ final class HealthKitService {
|
||
| 1089 | 1089 |
earliestDate: earliestDate, |
| 1090 | 1090 |
latestDate: latestDate |
| 1091 | 1091 |
) {
|
| 1092 |
- captureTimings.finalizeElapsedSeconds += try await finalizeArchiveVerification( |
|
| 1092 |
+ captureTimings.finalizeElapsedSeconds += try await finalizeUnchangedArchiveVerification( |
|
| 1093 | 1093 |
sampleType: sampleType, |
| 1094 | 1094 |
typeIdentifier: typeIdentifier, |
| 1095 | 1095 |
recordCount: unchanged.totalCount, |
@@ -1554,7 +1554,7 @@ final class HealthKitService {
|
||
| 1554 | 1554 |
return Double(processedCount) / elapsedSeconds |
| 1555 | 1555 |
} |
| 1556 | 1556 |
|
| 1557 |
- private func finalizeArchiveVerification( |
|
| 1557 |
+ private func finalizeUnchangedArchiveVerification( |
|
| 1558 | 1558 |
sampleType: HKSampleType, |
| 1559 | 1559 |
typeIdentifier: String, |
| 1560 | 1560 |
recordCount: Int, |
@@ -1562,6 +1562,34 @@ final class HealthKitService {
|
||
| 1562 | 1562 |
processedEventCount: Int, |
| 1563 | 1563 |
archiveObservationID: Int64, |
| 1564 | 1564 |
progress: SnapshotFetchProgress? |
| 1565 |
+ ) async throws -> TimeInterval {
|
|
| 1566 |
+ try await finalizeArchiveVerification( |
|
| 1567 |
+ sampleType: sampleType, |
|
| 1568 |
+ typeIdentifier: typeIdentifier, |
|
| 1569 |
+ recordCount: recordCount, |
|
| 1570 |
+ progressStarted: progressStarted, |
|
| 1571 |
+ processedEventCount: processedEventCount, |
|
| 1572 |
+ archiveObservationID: archiveObservationID, |
|
| 1573 |
+ progress: progress, |
|
| 1574 |
+ markVerification: {
|
|
| 1575 |
+ try await self.archiveStore.markUnchangedVerification( |
|
| 1576 |
+ sampleType: sampleType, |
|
| 1577 |
+ verifiedAt: Date(), |
|
| 1578 |
+ observationID: archiveObservationID |
|
| 1579 |
+ ) |
|
| 1580 |
+ } |
|
| 1581 |
+ ) |
|
| 1582 |
+ } |
|
| 1583 |
+ |
|
| 1584 |
+ private func finalizeArchiveVerification( |
|
| 1585 |
+ sampleType: HKSampleType, |
|
| 1586 |
+ typeIdentifier: String, |
|
| 1587 |
+ recordCount: Int, |
|
| 1588 |
+ progressStarted: Date, |
|
| 1589 |
+ processedEventCount: Int, |
|
| 1590 |
+ archiveObservationID: Int64, |
|
| 1591 |
+ progress: SnapshotFetchProgress?, |
|
| 1592 |
+ markVerification: (() async throws -> Void)? = nil |
|
| 1565 | 1593 |
) async throws -> TimeInterval {
|
| 1566 | 1594 |
let elapsedBeforeFinalize = Date().timeIntervalSince(progressStarted) |
| 1567 | 1595 |
progress?.updateBlockProgress( |
@@ -1576,11 +1604,15 @@ final class HealthKitService {
|
||
| 1576 | 1604 |
) |
| 1577 | 1605 |
|
| 1578 | 1606 |
let verificationStartedAt = Date() |
| 1579 |
- try await archiveStore.markVerification( |
|
| 1580 |
- sampleType: sampleType, |
|
| 1581 |
- verifiedAt: Date(), |
|
| 1582 |
- observationID: archiveObservationID |
|
| 1583 |
- ) |
|
| 1607 |
+ if let markVerification {
|
|
| 1608 |
+ try await markVerification() |
|
| 1609 |
+ } else {
|
|
| 1610 |
+ try await archiveStore.markVerification( |
|
| 1611 |
+ sampleType: sampleType, |
|
| 1612 |
+ verifiedAt: Date(), |
|
| 1613 |
+ observationID: archiveObservationID |
|
| 1614 |
+ ) |
|
| 1615 |
+ } |
|
| 1584 | 1616 |
let finalizeElapsed = Date().timeIntervalSince(verificationStartedAt) |
| 1585 | 1617 |
|
| 1586 | 1618 |
let elapsedAfterFinalize = Date().timeIntervalSince(progressStarted) |
+1
-0
HealthProbe/Services/Protocols/HealthArchiveStore.swift
@@ -9,6 +9,7 @@ protocol HealthArchiveStore {
|
||
| 9 | 9 |
func upsertSamples(_ samples: [HKSample], observedAt: Date, observationID: Int64) async throws -> HealthArchiveWriteSummary |
| 10 | 10 |
func markVerification(sampleType: HKSampleType, verifiedAt: Date) async throws |
| 11 | 11 |
func markVerification(sampleType: HKSampleType, verifiedAt: Date, observationID: Int64) async throws |
| 12 |
+ func markUnchangedVerification(sampleType: HKSampleType, verifiedAt: Date, observationID: Int64) async throws |
|
| 12 | 13 |
func recordDisappearance(sampleUUIDHash: String, sampleTypeIdentifier: String, observedMissingAt: Date) async throws |
| 13 | 14 |
func recordDisappearance(sampleUUIDHash: String, sampleTypeIdentifier: String, observedMissingAt: Date, observationID: Int64) async throws |
| 14 | 15 |
func recordDisappearances(sampleUUIDHashes: [String], sampleTypeIdentifier: String, observedMissingAt: Date, observationID: Int64) async throws -> Int |
+75
-0
HealthProbe/Services/SQLiteHealthArchiveStore.swift
@@ -191,6 +191,25 @@ actor SQLiteHealthArchiveStore: HealthArchiveStore {
|
||
| 191 | 191 |
} |
| 192 | 192 |
} |
| 193 | 193 |
|
| 194 |
+ func markUnchangedVerification(sampleType: HKSampleType, verifiedAt: Date, observationID: Int64) async throws {
|
|
| 195 |
+ let db = try openDatabase() |
|
| 196 |
+ defer { sqlite3_close(db) }
|
|
| 197 |
+ try prepareSchemaIfNeeded(db) |
|
| 198 |
+ try execute("BEGIN IMMEDIATE TRANSACTION", db: db)
|
|
| 199 |
+ do {
|
|
| 200 |
+ try markUnchangedVerification( |
|
| 201 |
+ sampleType: sampleType, |
|
| 202 |
+ verifiedAt: verifiedAt, |
|
| 203 |
+ observationID: observationID, |
|
| 204 |
+ db: db |
|
| 205 |
+ ) |
|
| 206 |
+ try execute("COMMIT", db: db)
|
|
| 207 |
+ } catch {
|
|
| 208 |
+ try? execute("ROLLBACK", db: db)
|
|
| 209 |
+ throw error |
|
| 210 |
+ } |
|
| 211 |
+ } |
|
| 212 |
+ |
|
| 194 | 213 |
func recordDisappearance(sampleUUIDHash: String, sampleTypeIdentifier: String, observedMissingAt: Date) async throws {
|
| 195 | 214 |
let db = try openDatabase() |
| 196 | 215 |
defer { sqlite3_close(db) }
|
@@ -2016,6 +2035,62 @@ actor SQLiteHealthArchiveStore: HealthArchiveStore {
|
||
| 2016 | 2035 |
) |
| 2017 | 2036 |
} |
| 2018 | 2037 |
|
| 2038 |
+ private func markUnchangedVerification( |
|
| 2039 |
+ sampleType: HKSampleType, |
|
| 2040 |
+ verifiedAt: Date, |
|
| 2041 |
+ observationID: Int64, |
|
| 2042 |
+ db: OpaquePointer? |
|
| 2043 |
+ ) throws {
|
|
| 2044 |
+ let statementCache = SQLiteStatementCache(db: db) |
|
| 2045 |
+ defer { statementCache.finalizeAll() }
|
|
| 2046 |
+ let lookupCache = SQLiteWriteLookupCache() |
|
| 2047 |
+ let sampleTypeID = try upsertSampleType( |
|
| 2048 |
+ typeIdentifier: sampleType.identifier, |
|
| 2049 |
+ db: db, |
|
| 2050 |
+ statementCache: statementCache, |
|
| 2051 |
+ lookupCache: lookupCache |
|
| 2052 |
+ ) |
|
| 2053 |
+ guard let previousSummary = try previousTypeSummary( |
|
| 2054 |
+ sampleTypeID: sampleTypeID, |
|
| 2055 |
+ beforeObservationID: observationID, |
|
| 2056 |
+ db: db, |
|
| 2057 |
+ statementCache: statementCache |
|
| 2058 |
+ ) else {
|
|
| 2059 |
+ try markVerification( |
|
| 2060 |
+ sampleType: sampleType, |
|
| 2061 |
+ verifiedAt: verifiedAt, |
|
| 2062 |
+ observationID: observationID, |
|
| 2063 |
+ db: db |
|
| 2064 |
+ ) |
|
| 2065 |
+ return |
|
| 2066 |
+ } |
|
| 2067 |
+ |
|
| 2068 |
+ try insertObservationTypeRun( |
|
| 2069 |
+ observationID: observationID, |
|
| 2070 |
+ sampleTypeID: sampleTypeID, |
|
| 2071 |
+ status: "completed", |
|
| 2072 |
+ observedAt: verifiedAt, |
|
| 2073 |
+ insertedEventCount: 0, |
|
| 2074 |
+ deletedEventCount: 0, |
|
| 2075 |
+ verifiedVisibleCount: previousSummary.aggregate.visibleRecordCount, |
|
| 2076 |
+ db: db, |
|
| 2077 |
+ statementCache: statementCache |
|
| 2078 |
+ ) |
|
| 2079 |
+ try upsertTypeSummary( |
|
| 2080 |
+ observationID: observationID, |
|
| 2081 |
+ sampleTypeID: sampleTypeID, |
|
| 2082 |
+ counts: (appeared: 0, disappeared: 0, representationChanged: 0), |
|
| 2083 |
+ aggregate: previousSummary.aggregate, |
|
| 2084 |
+ db: db |
|
| 2085 |
+ ) |
|
| 2086 |
+ try copyDailyAggregates( |
|
| 2087 |
+ fromObservationID: previousSummary.observationID, |
|
| 2088 |
+ toObservationID: observationID, |
|
| 2089 |
+ sampleTypeID: sampleTypeID, |
|
| 2090 |
+ db: db |
|
| 2091 |
+ ) |
|
| 2092 |
+ } |
|
| 2093 |
+ |
|
| 2019 | 2094 |
private func recordDisappearance( |
| 2020 | 2095 |
sampleUUIDHash: String, |
| 2021 | 2096 |
sampleTypeIdentifier: String, |
+30
-0
HealthProbeTests/SQLiteHealthArchiveStoreTests.swift
@@ -103,6 +103,36 @@ final class SQLiteHealthArchiveStoreTests: XCTestCase {
|
||
| 103 | 103 |
XCTAssertFalse(try tableExists("archive_samples", at: url))
|
| 104 | 104 |
} |
| 105 | 105 |
|
| 106 |
+ func testUnchangedVerificationCopiesPreviousSummaryWithoutNewEvents() async throws {
|
|
| 107 |
+ let url = databaseURL() |
|
| 108 |
+ let store = SQLiteHealthArchiveStore(databaseURL: url) |
|
| 109 |
+ let sample = makeStepCountSample() |
|
| 110 |
+ |
|
| 111 |
+ _ = try await store.upsertSamples([sample], observedAt: Date(timeIntervalSince1970: 2_000)) |
|
| 112 |
+ try await store.markVerification(sampleType: sample.sampleType, verifiedAt: Date(timeIntervalSince1970: 2_060)) |
|
| 113 |
+ let unchangedObservationID = try await store.beginObservation( |
|
| 114 |
+ observedAt: Date(timeIntervalSince1970: 2_120), |
|
| 115 |
+ triggerReason: "manual", |
|
| 116 |
+ selectedTypeSetHash: "selected-types" |
|
| 117 |
+ ) |
|
| 118 |
+ try await store.markUnchangedVerification( |
|
| 119 |
+ sampleType: sample.sampleType, |
|
| 120 |
+ verifiedAt: Date(timeIntervalSince1970: 2_120), |
|
| 121 |
+ observationID: unchangedObservationID |
|
| 122 |
+ ) |
|
| 123 |
+ try await store.finishObservation( |
|
| 124 |
+ observationID: unchangedObservationID, |
|
| 125 |
+ status: "completed", |
|
| 126 |
+ endedAt: Date(timeIntervalSince1970: 2_121) |
|
| 127 |
+ ) |
|
| 128 |
+ |
|
| 129 |
+ XCTAssertEqual(try countRows(in: "sample_observation_events", at: url), 1) |
|
| 130 |
+ XCTAssertEqual(try countRows(in: "observation_type_runs WHERE observation_id = \(unchangedObservationID) AND inserted_event_count = 0 AND deleted_event_count = 0 AND verified_visible_count = 1", at: url), 1) |
|
| 131 |
+ XCTAssertEqual(try countRows(in: "observation_type_summaries WHERE observation_id = \(unchangedObservationID) AND visible_record_count = 1 AND appeared_count = 0 AND disappeared_count = 0 AND representation_changed_count = 0", at: url), 1) |
|
| 132 |
+ XCTAssertEqual(try countRows(in: "daily_type_aggregates WHERE observation_id = \(unchangedObservationID) AND visible_record_count = 1", at: url), 1) |
|
| 133 |
+ XCTAssertFalse(try tableExists("archive_samples", at: url))
|
|
| 134 |
+ } |
|
| 135 |
+ |
|
| 106 | 136 |
func testDiffSummaryAndRecordsBetweenObservationsUseSQLVisibility() async throws {
|
| 107 | 137 |
let url = databaseURL() |
| 108 | 138 |
let store = SQLiteHealthArchiveStore(databaseURL: url) |