Persist HealthKit capture anchors in SQLite · bb61aa6

Persist HealthKit capture anchors in SQLite
Browse files

bogdan committed 4 days ago

main

1 parent 08e59a2

commit bb61aa6

Showing 5 changed files with 323 additions and 19 deletions

+10 -1

HealthProbe/Doc/04-project/Import-Optimization-Log.md

@@ -579,6 +579,7 @@ rows exist".
 | 2026-06-03 | pending | Harden quantity unit conversion for full-profile imports. | The first 127-metric run crashed while archiving `HKQuantityTypeIdentifierDietaryWater`: the previous fallback converted unknown quantities with `.count()`, and HealthKit raised `NSInvalidArgumentException` for incompatible `mL` to `count`. The archive now maps known extended units, stores quantity rows with nil numeric/unit when a future type is unmapped, and removes the unsafe display fallback. Next run should pass Water and reveal any remaining type-specific unit gaps. |
 | 2026-06-03 | pending | Preserve completed import diagnostics and render large diagnostic reports lazily. | A copied full-profile reimport report completed successfully: `Types: 127/127 processed`, `Records: 2,646,527`, `WallClock: 40.3s`, `SummedProcessingElapsed: 21.7s`, `SummedFinalizeElapsed: 15.1s`, `SummedFetchElapsed: 1.7s`, `SummedInsertElapsed: 0.1s`, and `0` degraded metrics. The Diagnostics sheet copied the report text but displayed a blank body for the huge report; completed/partial/review reports are now persisted under Application Support and displayed in chunked lazy text blocks. |
 | 2026-06-03 | pending | Fast-path unchanged verification after an empty HealthKit delta page. | The successful full-profile reimport spent `15.1s` in finalize despite `0.1s` insert time; Heart Rate alone spent `9.0s` finalizing. The empty-delta path now calls an explicit unchanged-verification store method that copies the previous type summary and daily aggregates without first scanning sample events to prove zero changes. Expected signal: repeated no-delta/full-profile `SummedFinalizeElapsed` and Heart Rate finalize time drop sharply. |
+| 2026-06-03 | pending | Persist HealthKit capture anchors in SQLite. | A later no-delta attempt still reimported full high-volume types (`Records: 2,646,590`, `WallClock: 45.8s`, Heart Rate `922,521`, Active Energy `346,473`) because incremental anchors were still read from legacy SwiftData `TypeCount`, while the refactored app now chains observations from SQLite. SQLite now stores per-type capture state (`anchor_data`, count, content hash, yearly counts, range) and the import path uses it when legacy `TypeCount` state is missing. Expected signal: the first run after this change may seed anchors; the following no-delta run should report empty HealthKit delta pages instead of full `record_import` counts for large stable types. |
 
 ## Current Diagnosis
 
@@ -621,6 +622,10 @@ The likely bottleneck is per-row SQLite work:
   if it counts nonexistent events for high-volume types. The capture layer knows
   the first anchored delta page is empty, so unchanged verification should skip
   event-count scans and copy previous materialized summaries directly.
+- Incremental anchors must be persisted in SQLite, not only in legacy SwiftData
+  `TypeCount.distributionBinsData`. Otherwise refactored SQLite-driven snapshots
+  lose their HealthKit query anchor across launches/refactors and every
+  high-volume metric becomes a full scan again.
 
 ## Open Issues / Observations
 
@@ -653,7 +658,11 @@ Prioritize experiments in this order:
    diagnostic report does not freeze the app, and Dashboard/Snapshots show the
    latest observation from SQLite. Also verify Snapshot detail and Data Types
    show per-type summaries without a manual cache rebuild.
-3. Run a repeated no-delta benchmark after copying unchanged metric summaries and daily aggregates. Compare `SummedFinalizeElapsed`, `Heart Rate finalizeElapsed`, `Active Energy finalizeElapsed`, and wall clock.
+3. Run two repeated no-delta benchmarks after SQLite capture-state persistence:
+   first to seed missing SQLite anchors if needed, second to confirm `record_import`
+   for large stable types uses empty delta pages. Compare `WallClockDuration`,
+   `SummedProcessingElapsed`, `SummedFinalizeElapsed`, Heart Rate count/timing,
+   and Active Energy count/timing.
 4. Add or inspect timing around per-record processing for changed high-volume metrics, especially Heart Rate, to separate sample DTO/fingerprint work from SQLite idempotency checks.
 5. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
 6. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.


+100 -18

HealthProbe/Services/HealthKitService.swift

View

@@ -907,7 +907,11 @@ final class HealthKitService {
 
         let earliest = earliestResult.value ?? nil
         let latest = latestResult.value ?? nil
-        let previousDistribution = PreviousDistributionState(typeCount: previousTypeCount)
+        let previousArchiveState = try? await archiveStore.typeCaptureState(sampleTypeIdentifier: monitoredType.id)
+        let previousDistribution = PreviousDistributionState(
+            typeCount: previousTypeCount,
+            archiveState: previousArchiveState
+        )
         let distributionResult = await measureAPICall(
             queryType: "record_import",
             timeoutSeconds: nil
@@ -922,7 +926,12 @@ final class HealthKitService {
                 progress: progress
             )
         } resultDescription: { distribution in
-            "\(distribution.totalCount) samples in \(distribution.bins.count) anchored segment(s)"
+            if distribution.recordArchiveData == nil,
+               distribution.records.isEmpty,
+               distribution.totalCount > 0 {
+                return "\(distribution.totalCount) unchanged samples via empty HealthKit delta"
+            }
+            return "\(distribution.totalCount) samples in \(distribution.bins.count) anchored segment(s)"
         }
         apiCalls.insert(distributionResult.apiCall, at: 0)
 
@@ -983,7 +992,14 @@ final class HealthKitService {
         var timingBreakdown = distribution.timingBreakdown
         timingBreakdown.fetchElapsedSeconds += dateFetchElapsedSeconds
         let recordArchiveStartedAt = Date()
-        let recordArchiveData = distribution.recordArchiveData ?? HealthRecordArchive.encode(distribution.records)
+        let recordArchiveData: Data?
+        if let distributionArchiveData = distribution.recordArchiveData {
+            recordArchiveData = distributionArchiveData
+        } else if distribution.records.isEmpty {
+            recordArchiveData = nil
+        } else {
+            recordArchiveData = HealthRecordArchive.encode(distribution.records)
+        }
         timingBreakdown.processingElapsedSeconds += Date().timeIntervalSince(recordArchiveStartedAt)
 
         var result = TypeCountFetchResult(
@@ -1012,9 +1028,33 @@ final class HealthKitService {
             recordArchiveData: recordArchiveData
         )
         result.timingBreakdown = timingBreakdown
+        await persistTypeCaptureState(from: result, observationID: archiveObservationID)
         return result
     }
 
+    private func persistTypeCaptureState(from result: TypeCountFetchResult, observationID: Int64) async {
+        guard result.quality == .complete,
+              result.count >= 0,
+              let anchorData = result.distributionBins.last?.anchorData else {
+            return
+        }
+        let yearlyCounts = Dictionary(uniqueKeysWithValues: result.yearlyCounts.map { ($0.year, $0.count) })
+        let state = HealthArchiveTypeCaptureState(
+            sampleTypeIdentifier: result.typeIdentifier,
+            count: result.count,
+            contentHash: result.contentHash,
+            earliestDate: result.earliestDate,
+            latestDate: result.latestDate,
+            yearlyCounts: yearlyCounts,
+            anchorData: anchorData
+        )
+        do {
+            try await archiveStore.upsertTypeCaptureState(state, observationID: observationID)
+        } catch {
+            print("[HealthProbeImport] capture_state_persist_failed type=\(result.typeIdentifier) error=\(error)")
+        }
+    }
+
     // MARK: - HealthKit queries
 
     private func fetchDistribution(
@@ -1066,6 +1106,28 @@ final class HealthKitService {
                 )
             }
             captureTimings.fetchElapsedSeconds += Date().timeIntervalSince(firstPageFetchStartedAt)
+            anchor = page.anchor
+
+            if !page.samples.isEmpty || !page.deletedObjects.isEmpty,
+               !previousDistribution.canApplyDeltaChanges {
+                progress?.updateBlockProgress(
+                    typeIdentifier,
+                    detail: "Delta requires full archive refresh",
+                    recordCount: previousDistribution.count,
+                    elapsedSeconds: Date().timeIntervalSince(progressStarted),
+                    samplesPerSecond: 0
+                )
+                return try await fetchInitialDistributionStreaming(
+                    for: sampleType,
+                    typeIdentifier: typeIdentifier,
+                    earliestDate: earliestDate,
+                    latestDate: latestDate,
+                    progressStarted: progressStarted,
+                    archiveObservationID: archiveObservationID,
+                    progress: progress
+                )
+            }
+
             let archiveResult = try await archivePage(
                 page,
                 sampleType: sampleType,
@@ -1080,7 +1142,6 @@ final class HealthKitService {
             )
             persistenceState = archiveResult.persistenceState
             captureTimings.insertElapsedSeconds += archiveResult.insertElapsedSeconds
-            anchor = page.anchor
 
             if page.samples.isEmpty, page.deletedObjects.isEmpty,
                let unchanged = previousDistribution.unchangedDistribution(
@@ -2524,22 +2585,44 @@ private struct PreviousDistributionState: Sendable {
         return bins[0].anchor
     }
 
-    init(typeCount: TypeCount?) {
-        self.typeIdentifier = typeCount?.typeIdentifier ?? ""
-        self.count = typeCount?.count ?? 0
-        self.contentHash = typeCount?.contentHash ?? ""
-        self.earliestRecordDate = typeCount?.earliestDate
-        self.latestRecordDate = typeCount?.latestDate
+    var canApplyDeltaChanges: Bool {
+        count == 0 || recordArchiveData != nil
+    }
+
+    init(typeCount: TypeCount?, archiveState: HealthArchiveTypeCaptureState?) {
+        let typeCountBins = typeCount?.distributionBins ?? []
+        self.typeIdentifier = typeCount?.typeIdentifier ?? archiveState?.sampleTypeIdentifier ?? ""
+        self.count = typeCount?.count ?? archiveState?.count ?? 0
+        self.contentHash = typeCount?.contentHash ?? archiveState?.contentHash ?? ""
+        self.earliestRecordDate = typeCount?.earliestDate ?? archiveState?.earliestDate
+        self.latestRecordDate = typeCount?.latestDate ?? archiveState?.latestDate
         self.recordArchiveData = typeCount?.recordArchiveData
-        self.yearlyCounts = Dictionary(
+        let typeCountYearlyCounts = Dictionary(
             uniqueKeysWithValues: (typeCount?.yearlyCounts ?? []).map { ($0.year, $0.count) }
         )
-        self.bins = (typeCount?.distributionBins ?? []).map {
-            Bin(
-                bucketStart: $0.bucketStart,
-                bucketEnd: $0.bucketEnd,
-                anchorData: $0.anchorData
-            )
+        self.yearlyCounts = typeCountYearlyCounts.isEmpty
+            ? archiveState?.yearlyCounts ?? [:]
+            : typeCountYearlyCounts
+        if typeCountBins.count == 1, typeCountBins[0].anchorData != nil {
+            self.bins = typeCountBins.map {
+                Bin(
+                    bucketStart: $0.bucketStart,
+                    bucketEnd: $0.bucketEnd,
+                    anchorData: $0.anchorData
+                )
+            }
+        } else if let archiveState, let anchorData = archiveState.anchorData {
+            let bucketStart = archiveState.earliestDate ?? Date()
+            let rawBucketEnd = archiveState.latestDate ?? bucketStart
+            self.bins = [
+                Bin(
+                    bucketStart: bucketStart,
+                    bucketEnd: rawBucketEnd > bucketStart ? rawBucketEnd : bucketStart.addingTimeInterval(1),
+                    anchorData: anchorData
+                )
+            ]
+        } else {
+            self.bins = []
         }
     }
 
@@ -2550,7 +2633,6 @@ private struct PreviousDistributionState: Sendable {
         latestDate: Date?
     ) -> SampleDistribution? {
         guard count >= 0,
-              count == 0 || recordArchiveData != nil,
               count == 0 || !contentHash.isEmpty else {
             return nil
         }


+33 -0

HealthProbe/Services/Protocols/HealthArchiveStore.swift

View

	@@ -5,6 +5,8 @@ import HealthKit
5	5	protocol HealthArchiveStore {
6	6	func beginObservation(observedAt: Date, triggerReason: String, selectedTypeSetHash: String?) async throws -> Int64
7	7	func finishObservation(observationID: Int64, status: String, endedAt: Date) async throws
	8	+ func typeCaptureState(sampleTypeIdentifier: String) async throws -> HealthArchiveTypeCaptureState?
	9	+ func upsertTypeCaptureState(_ state: HealthArchiveTypeCaptureState, observationID: Int64) async throws
8	10	func upsertSamples(_ samples: [HKSample], observedAt: Date) async throws -> HealthArchiveWriteSummary
9	11	func upsertSamples(_ samples: [HKSample], observedAt: Date, observationID: Int64) async throws -> HealthArchiveWriteSummary
10	12	func markVerification(sampleType: HKSampleType, verifiedAt: Date) async throws
	@@ -30,6 +32,37 @@ struct HealthArchiveWriteSummary: Equatable, Sendable {
30	32	let unchangedCount: Int
31	33	}
32	34
	35	+struct HealthArchiveTypeCaptureState: Equatable, Sendable {
	36	+ let sampleTypeIdentifier: String
	37	+ let observationID: Int64?
	38	+ let count: Int
	39	+ let contentHash: String
	40	+ let earliestDate: Date?
	41	+ let latestDate: Date?
	42	+ let yearlyCounts: [Int: Int]
	43	+ let anchorData: Data?
	44	+
	45	+ init(
	46	+ sampleTypeIdentifier: String,
	47	+ observationID: Int64? = nil,
	48	+ count: Int,
	49	+ contentHash: String,
	50	+ earliestDate: Date?,
	51	+ latestDate: Date?,
	52	+ yearlyCounts: [Int: Int],
	53	+ anchorData: Data?
	54	+ ) {
	55	+ self.sampleTypeIdentifier = sampleTypeIdentifier
	56	+ self.observationID = observationID
	57	+ self.count = count
	58	+ self.contentHash = contentHash
	59	+ self.earliestDate = earliestDate
	60	+ self.latestDate = latestDate
	61	+ self.yearlyCounts = yearlyCounts
	62	+ self.anchorData = anchorData
	63	+ }
	64	+}
	65	+
33	66	struct HealthArchiveIntegrityReport: Equatable, Sendable {
34	67	let schemaVersion: Int?
35	68	let sqliteIntegrityStatus: String

+144 -0

HealthProbe/Services/SQLiteHealthArchiveStore.swift

View

+                             "sample_visibility_ranges",
+                             "sample_relationships",
+                             "observation_type_summaries",
+                    +        "type_capture_states",
+                             "daily_type_aggregates",
+                             "export_manifests",
+                             "export_items"
+                         private let databaseURL: URL
+                         private var didPrepareSchema = false
+                    +    private nonisolated static let captureStateEncoder = PropertyListEncoder()
+                    +    private nonisolated static let captureStateDecoder = PropertyListDecoder()
+                         private static let monitoredTypeMetadataByIdentifier: [String: (displayName: String, category: String)] = {
+                             Dictionary(uniqueKeysWithValues: HealthKitService.allTypes.map {
+                             return try typeSummaries(observationID: observationID, limit: limit, db: db)
+                    +    func typeCaptureState(sampleTypeIdentifier: String) async throws -> HealthArchiveTypeCaptureState? {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        return try typeCaptureState(sampleTypeIdentifier: sampleTypeIdentifier, db: db)
+                    +    }
+                    +    func upsertTypeCaptureState(_ state: HealthArchiveTypeCaptureState, observationID: Int64) async throws {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        try execute("BEGIN IMMEDIATE TRANSACTION", db: db)
+                    +        do {
+                    +            try upsertTypeCaptureState(state, observationID: observationID, db: db)
+                    +            try execute("COMMIT", db: db)
+                    +        } catch {
+                    +            try? execute("ROLLBACK", db: db)
+                    +            throw error
+                    +        }
+                    +    }
+                         func upsertSamples(_ samples: [HKSample], observedAt: Date) async throws -> HealthArchiveWriteSummary {
+                             guard !samples.isEmpty else {
+                                 return HealthArchiveWriteSummary(insertedCount: 0, updatedCount: 0, unchangedCount: 0)
+                             """, db: db)
+                             try execute("""
+                    +        CREATE TABLE IF NOT EXISTS type_capture_states (
+                    +            sample_type_id INTEGER PRIMARY KEY REFERENCES sample_types(id),
+                    +            observation_id INTEGER REFERENCES observations(id),
+                    +            anchor_data BLOB,
+                    +            record_count INTEGER NOT NULL,
+                    +            content_hash TEXT NOT NULL,
+                    +            earliest_start_date REAL,
+                    +            latest_end_date REAL,
+                    +            yearly_counts_data BLOB,
+                    +            updated_at REAL NOT NULL
+                    +        )
+                    +        """, db: db)
+                    +        try execute("CREATE INDEX IF NOT EXISTS idx_type_capture_states_observation ON type_capture_states(observation_id)", db: db)
+                    +        try execute("""
+                             CREATE TABLE IF NOT EXISTS daily_type_aggregates (
+                                 observation_id INTEGER NOT NULL REFERENCES observations(id),
+                                 sample_type_id INTEGER NOT NULL REFERENCES sample_types(id),
+                             return rows
+                    +    private func typeCaptureState(
+                    +        sampleTypeIdentifier: String,
+                    +        db: OpaquePointer?
+                    +    ) throws -> HealthArchiveTypeCaptureState? {
+                    +        let sql = """
+                    +        SELECT
+                    +            t.type_identifier,
+                    +            s.observation_id,
+                    +            s.record_count,
+                    +            s.content_hash,
+                    +            s.earliest_start_date,
+                    +            s.latest_end_date,
+                    +            s.yearly_counts_data,
+                    +            s.anchor_data
+                    +        FROM type_capture_states s
+                    +        JOIN sample_types t ON t.id = s.sample_type_id
+                    +        WHERE t.type_identifier = ?
+                    +        LIMIT 1
+                    +        """
+                    +        return try withStatement(sql, db: db) { statement in
+                    +            bindText(sampleTypeIdentifier, to: 1, in: statement)
+                    +            guard sqlite3_step(statement) == SQLITE_ROW else { return nil }
+                    +            let yearlyCountsData = columnData(statement, 6)
+                    +            let yearlyCounts = yearlyCountsData.flatMap {
+                    +                try? Self.captureStateDecoder.decode([Int: Int].self, from: $0)
+                    +            } ?? [:]
+                    +            return HealthArchiveTypeCaptureState(
+                    +                sampleTypeIdentifier: columnText(statement, 0) ?? sampleTypeIdentifier,
+                    +                observationID: columnInt64(statement, 1),
+                    +                count: columnInt(statement, 2) ?? 0,
+                    +                contentHash: columnText(statement, 3) ?? "",
+                    +                earliestDate: columnUnixDate(statement, 4),
+                    +                latestDate: columnUnixDate(statement, 5),
+                    +                yearlyCounts: yearlyCounts,
+                    +                anchorData: columnData(statement, 7)
+                    +            )
+                    +        }
+                    +    }
+                    +    private func upsertTypeCaptureState(
+                    +        _ state: HealthArchiveTypeCaptureState,
+                    +        observationID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws {
+                    +        let sampleTypeID = try upsertSampleType(typeIdentifier: state.sampleTypeIdentifier, db: db)
+                    +        let yearlyCountsData = try? Self.captureStateEncoder.encode(state.yearlyCounts)
+                    +        let sql = """
+                    +        INSERT INTO type_capture_states (
+                    +            sample_type_id,
+                    +            observation_id,
+                    +            anchor_data,
+                    +            record_count,
+                    +            content_hash,
+                    +            earliest_start_date,
+                    +            latest_end_date,
+                    +            yearly_counts_data,
+                    +            updated_at
+                    +        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
+                    +        ON CONFLICT(sample_type_id) DO UPDATE SET
+                    +            observation_id = excluded.observation_id,
+                    +            anchor_data = excluded.anchor_data,
+                    +            record_count = excluded.record_count,
+                    +            content_hash = excluded.content_hash,
+                    +            earliest_start_date = excluded.earliest_start_date,
+                    +            latest_end_date = excluded.latest_end_date,
+                    +            yearly_counts_data = excluded.yearly_counts_data,
+                    +            updated_at = excluded.updated_at
+                    +        """
+                    +        try withStatement(sql, db: db) { statement in
+                    +            bindInt64(sampleTypeID, to: 1, in: statement)
+                    +            bindInt64(observationID, to: 2, in: statement)
+                    +            bindData(state.anchorData, to: 3, in: statement)
+                    +            bindInt(state.count, to: 4, in: statement)
+                    +            bindText(state.contentHash, to: 5, in: statement)
+                    +            bindDouble(state.earliestDate?.timeIntervalSince1970, to: 6, in: statement)
+                    +            bindDouble(state.latestDate?.timeIntervalSince1970, to: 7, in: statement)
+                    +            bindData(yearlyCountsData, to: 8, in: statement)
+                    +            sqlite3_bind_double(statement, 9, Date().timeIntervalSince1970)
+                    +            guard sqlite3_step(statement) == SQLITE_DONE else {
+                    +                throw SQLiteHealthArchiveStoreError.stepFailed(lastErrorMessage(db))
+                    +            }
+                    +        }
+                    +    }
+                         private func updateObservationStatus(
+                             observationID: Int64,
+                             status: String,
+                         sqlite3_bind_text(statement, index, value, -1, unsafeBitCast(-1, to: sqlite3_destructor_type.self))
+                    +nonisolated private func bindData(_ value: Data?, to index: Int32, in statement: OpaquePointer?) {
+                    +    guard let value else {
+                    +        sqlite3_bind_null(statement, index)
+                    +        return
+                    +    }
+                    +    value.withUnsafeBytes { rawBuffer in
+                    +        let pointer = rawBuffer.baseAddress
+                    +        sqlite3_bind_blob(statement, index, pointer, Int32(value.count), unsafeBitCast(-1, to: sqlite3_destructor_type.self))
+                    +    }
+                    +}
+                     nonisolated private func bindDouble(_ value: Double?, to index: Int32, in statement: OpaquePointer?) {
+                         guard let value else {
+                             sqlite3_bind_null(statement, index)
+                         return String(cString: pointer)
+                    +nonisolated private func columnData(_ statement: OpaquePointer?, _ index: Int32) -> Data? {
+                    +    guard sqlite3_column_type(statement, index) != SQLITE_NULL,
+                    +          let bytes = sqlite3_column_blob(statement, index) else {
+                    +        return nil
+                    +    }
+                    +    let count = Int(sqlite3_column_bytes(statement, index))
+                    +    return Data(bytes: bytes, count: count)
+                    +}
+                     nonisolated private func columnDate(_ statement: OpaquePointer?, _ index: Int32) -> Date? {
+                         guard sqlite3_column_type(statement, index) != SQLITE_NULL else { return nil }
+                         return Date(timeIntervalSinceReferenceDate: sqlite3_column_double(statement, index))

+36 -0

HealthProbeTests/SQLiteHealthArchiveStoreTests.swift

View

+                             XCTAssertTrue(records.isEmpty)
+                    +    func testTypeCaptureStateRoundTripsAnchorAndSummary() async throws {
+                    +        let url = databaseURL()
+                    +        let store = SQLiteHealthArchiveStore(databaseURL: url)
+                    +        let observationID = try await store.beginObservation(
+                    +            observedAt: Date(timeIntervalSince1970: 2_000),
+                    +            triggerReason: "manual",
+                    +            selectedTypeSetHash: "selected-types"
+                    +        )
+                    +        let expected = HealthArchiveTypeCaptureState(
+                    +            sampleTypeIdentifier: HKQuantityTypeIdentifier.stepCount.rawValue,
+                    +            count: 42,
+                    +            contentHash: "content-hash",
+                    +            earliestDate: Date(timeIntervalSince1970: 1_000),
+                    +            latestDate: Date(timeIntervalSince1970: 1_500),
+                    +            yearlyCounts: [2026: 42],
+                    +            anchorData: Data([0xde, 0xad, 0xbe, 0xef])
+                    +        )
+                    +        try await store.upsertTypeCaptureState(expected, observationID: observationID)
+                    +        let actual = try await store.typeCaptureState(sampleTypeIdentifier: HKQuantityTypeIdentifier.stepCount.rawValue)
+                    +        let report = try await store.checkIntegrity()
+                    +        XCTAssertEqual(actual, HealthArchiveTypeCaptureState(
+                    +            sampleTypeIdentifier: expected.sampleTypeIdentifier,
+                    +            observationID: observationID,
+                    +            count: expected.count,
+                    +            contentHash: expected.contentHash,
+                    +            earliestDate: expected.earliestDate,
+                    +            latestDate: expected.latestDate,
+                    +            yearlyCounts: expected.yearlyCounts,
+                    +            anchorData: expected.anchorData
+                    +        ))
+                    +        XCTAssertTrue(report.passed)
+                    +        XCTAssertEqual(try countRows(in: "type_capture_states", at: url), 1)
+                    +    }
+                         func testPrototypeArchiveIsResetAndReinitializedAsV2() async throws {
+                             let url = databaseURL()
+                             try createPrototypeDatabase(at: url)

	@@ -579,6 +579,7 @@ rows exist".
579	579	\| 2026-06-03 \| pending \| Harden quantity unit conversion for full-profile imports. \| The first 127-metric run crashed while archiving `HKQuantityTypeIdentifierDietaryWater`: the previous fallback converted unknown quantities with `.count()`, and HealthKit raised `NSInvalidArgumentException` for incompatible `mL` to `count`. The archive now maps known extended units, stores quantity rows with nil numeric/unit when a future type is unmapped, and removes the unsafe display fallback. Next run should pass Water and reveal any remaining type-specific unit gaps. \|
580	580	\| 2026-06-03 \| pending \| Preserve completed import diagnostics and render large diagnostic reports lazily. \| A copied full-profile reimport report completed successfully: `Types: 127/127 processed`, `Records: 2,646,527`, `WallClock: 40.3s`, `SummedProcessingElapsed: 21.7s`, `SummedFinalizeElapsed: 15.1s`, `SummedFetchElapsed: 1.7s`, `SummedInsertElapsed: 0.1s`, and `0` degraded metrics. The Diagnostics sheet copied the report text but displayed a blank body for the huge report; completed/partial/review reports are now persisted under Application Support and displayed in chunked lazy text blocks. \|
581	581	\| 2026-06-03 \| pending \| Fast-path unchanged verification after an empty HealthKit delta page. \| The successful full-profile reimport spent `15.1s` in finalize despite `0.1s` insert time; Heart Rate alone spent `9.0s` finalizing. The empty-delta path now calls an explicit unchanged-verification store method that copies the previous type summary and daily aggregates without first scanning sample events to prove zero changes. Expected signal: repeated no-delta/full-profile `SummedFinalizeElapsed` and Heart Rate finalize time drop sharply. \|
	582	+\| 2026-06-03 \| pending \| Persist HealthKit capture anchors in SQLite. \| A later no-delta attempt still reimported full high-volume types (`Records: 2,646,590`, `WallClock: 45.8s`, Heart Rate `922,521`, Active Energy `346,473`) because incremental anchors were still read from legacy SwiftData `TypeCount`, while the refactored app now chains observations from SQLite. SQLite now stores per-type capture state (`anchor_data`, count, content hash, yearly counts, range) and the import path uses it when legacy `TypeCount` state is missing. Expected signal: the first run after this change may seed anchors; the following no-delta run should report empty HealthKit delta pages instead of full `record_import` counts for large stable types. \|
582	583
583	584	## Current Diagnosis
584	585
	@@ -621,6 +622,10 @@ The likely bottleneck is per-row SQLite work:
621	622	if it counts nonexistent events for high-volume types. The capture layer knows
622	623	the first anchored delta page is empty, so unchanged verification should skip
623	624	event-count scans and copy previous materialized summaries directly.
	625	+- Incremental anchors must be persisted in SQLite, not only in legacy SwiftData
	626	+ `TypeCount.distributionBinsData`. Otherwise refactored SQLite-driven snapshots
	627	+ lose their HealthKit query anchor across launches/refactors and every
	628	+ high-volume metric becomes a full scan again.
624	629
625	630	## Open Issues / Observations
626	631
	@@ -653,7 +658,11 @@ Prioritize experiments in this order:
653	658	diagnostic report does not freeze the app, and Dashboard/Snapshots show the
654	659	latest observation from SQLite. Also verify Snapshot detail and Data Types
655	660	show per-type summaries without a manual cache rebuild.
656		-3. Run a repeated no-delta benchmark after copying unchanged metric summaries and daily aggregates. Compare `SummedFinalizeElapsed`, `Heart Rate finalizeElapsed`, `Active Energy finalizeElapsed`, and wall clock.
	661	+3. Run two repeated no-delta benchmarks after SQLite capture-state persistence:
	662	+ first to seed missing SQLite anchors if needed, second to confirm `record_import`
	663	+ for large stable types uses empty delta pages. Compare `WallClockDuration`,
	664	+ `SummedProcessingElapsed`, `SummedFinalizeElapsed`, Heart Rate count/timing,
	665	+ and Active Energy count/timing.
657	666	4. Add or inspect timing around per-record processing for changed high-volume metrics, especially Heart Rate, to separate sample DTO/fingerprint work from SQLite idempotency checks.
658	667	5. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
659	668	6. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.

	@@ -907,7 +907,11 @@ final class HealthKitService {
907	907
908	908	let earliest = earliestResult.value ?? nil
909	909	let latest = latestResult.value ?? nil
910		- let previousDistribution = PreviousDistributionState(typeCount: previousTypeCount)
	910	+ let previousArchiveState = try? await archiveStore.typeCaptureState(sampleTypeIdentifier: monitoredType.id)
	911	+ let previousDistribution = PreviousDistributionState(
	912	+ typeCount: previousTypeCount,
	913	+ archiveState: previousArchiveState
	914	+ )
911	915	let distributionResult = await measureAPICall(
912	916	queryType: "record_import",
913	917	timeoutSeconds: nil
	@@ -922,7 +926,12 @@ final class HealthKitService {
922	926	progress: progress
923	927	)
924	928	} resultDescription: { distribution in
925		- "\(distribution.totalCount) samples in \(distribution.bins.count) anchored segment(s)"
	929	+ if distribution.recordArchiveData == nil,
	930	+ distribution.records.isEmpty,
	931	+ distribution.totalCount > 0 {
	932	+ return "\(distribution.totalCount) unchanged samples via empty HealthKit delta"
	933	+ }
	934	+ return "\(distribution.totalCount) samples in \(distribution.bins.count) anchored segment(s)"
926	935	}
927	936	apiCalls.insert(distributionResult.apiCall, at: 0)
928	937
	@@ -983,7 +992,14 @@ final class HealthKitService {
983	992	var timingBreakdown = distribution.timingBreakdown
984	993	timingBreakdown.fetchElapsedSeconds += dateFetchElapsedSeconds
985	994	let recordArchiveStartedAt = Date()
986		- let recordArchiveData = distribution.recordArchiveData ?? HealthRecordArchive.encode(distribution.records)
	995	+ let recordArchiveData: Data?
	996	+ if let distributionArchiveData = distribution.recordArchiveData {
	997	+ recordArchiveData = distributionArchiveData
	998	+ } else if distribution.records.isEmpty {
	999	+ recordArchiveData = nil
	1000	+ } else {
	1001	+ recordArchiveData = HealthRecordArchive.encode(distribution.records)
	1002	+ }
987	1003	timingBreakdown.processingElapsedSeconds += Date().timeIntervalSince(recordArchiveStartedAt)
988	1004
989	1005	var result = TypeCountFetchResult(
	@@ -1012,9 +1028,33 @@ final class HealthKitService {
1012	1028	recordArchiveData: recordArchiveData
1013	1029	)
1014	1030	result.timingBreakdown = timingBreakdown
	1031	+ await persistTypeCaptureState(from: result, observationID: archiveObservationID)
1015	1032	return result
1016	1033	}
1017	1034
	1035	+ private func persistTypeCaptureState(from result: TypeCountFetchResult, observationID: Int64) async {
	1036	+ guard result.quality == .complete,
	1037	+ result.count >= 0,
	1038	+ let anchorData = result.distributionBins.last?.anchorData else {
	1039	+ return
	1040	+ }
	1041	+ let yearlyCounts = Dictionary(uniqueKeysWithValues: result.yearlyCounts.map { ($0.year, $0.count) })
	1042	+ let state = HealthArchiveTypeCaptureState(
	1043	+ sampleTypeIdentifier: result.typeIdentifier,
	1044	+ count: result.count,
	1045	+ contentHash: result.contentHash,
	1046	+ earliestDate: result.earliestDate,
	1047	+ latestDate: result.latestDate,
	1048	+ yearlyCounts: yearlyCounts,
	1049	+ anchorData: anchorData
	1050	+ )
	1051	+ do {
	1052	+ try await archiveStore.upsertTypeCaptureState(state, observationID: observationID)
	1053	+ } catch {
	1054	+ print("[HealthProbeImport] capture_state_persist_failed type=\(result.typeIdentifier) error=\(error)")
	1055	+ }
	1056	+ }
	1057	+
1018	1058	// MARK: - HealthKit queries
1019	1059
1020	1060	private func fetchDistribution(
	@@ -1066,6 +1106,28 @@ final class HealthKitService {
1066	1106	)
1067	1107	}
1068	1108	captureTimings.fetchElapsedSeconds += Date().timeIntervalSince(firstPageFetchStartedAt)
	1109	+ anchor = page.anchor
	1110	+
	1111	+ if !page.samples.isEmpty \|\| !page.deletedObjects.isEmpty,
	1112	+ !previousDistribution.canApplyDeltaChanges {
	1113	+ progress?.updateBlockProgress(
	1114	+ typeIdentifier,
	1115	+ detail: "Delta requires full archive refresh",
	1116	+ recordCount: previousDistribution.count,
	1117	+ elapsedSeconds: Date().timeIntervalSince(progressStarted),
	1118	+ samplesPerSecond: 0
	1119	+ )
	1120	+ return try await fetchInitialDistributionStreaming(
	1121	+ for: sampleType,
	1122	+ typeIdentifier: typeIdentifier,
	1123	+ earliestDate: earliestDate,
	1124	+ latestDate: latestDate,
	1125	+ progressStarted: progressStarted,
	1126	+ archiveObservationID: archiveObservationID,
	1127	+ progress: progress
	1128	+ )
	1129	+ }
	1130	+
1069	1131	let archiveResult = try await archivePage(
1070	1132	page,
1071	1133	sampleType: sampleType,
	@@ -1080,7 +1142,6 @@ final class HealthKitService {
1080	1142	)
1081	1143	persistenceState = archiveResult.persistenceState
1082	1144	captureTimings.insertElapsedSeconds += archiveResult.insertElapsedSeconds
1083		- anchor = page.anchor
1084	1145
1085	1146	if page.samples.isEmpty, page.deletedObjects.isEmpty,
1086	1147	let unchanged = previousDistribution.unchangedDistribution(
	@@ -2524,22 +2585,44 @@ private struct PreviousDistributionState: Sendable {
2524	2585	return bins[0].anchor
2525	2586	}
2526	2587
2527		- init(typeCount: TypeCount?) {
2528		- self.typeIdentifier = typeCount?.typeIdentifier ?? ""
2529		- self.count = typeCount?.count ?? 0
2530		- self.contentHash = typeCount?.contentHash ?? ""
2531		- self.earliestRecordDate = typeCount?.earliestDate
2532		- self.latestRecordDate = typeCount?.latestDate
	2588	+ var canApplyDeltaChanges: Bool {
	2589	+ count == 0 \|\| recordArchiveData != nil
	2590	+ }
	2591	+
	2592	+ init(typeCount: TypeCount?, archiveState: HealthArchiveTypeCaptureState?) {
	2593	+ let typeCountBins = typeCount?.distributionBins ?? []
	2594	+ self.typeIdentifier = typeCount?.typeIdentifier ?? archiveState?.sampleTypeIdentifier ?? ""
	2595	+ self.count = typeCount?.count ?? archiveState?.count ?? 0
	2596	+ self.contentHash = typeCount?.contentHash ?? archiveState?.contentHash ?? ""
	2597	+ self.earliestRecordDate = typeCount?.earliestDate ?? archiveState?.earliestDate
	2598	+ self.latestRecordDate = typeCount?.latestDate ?? archiveState?.latestDate
2533	2599	self.recordArchiveData = typeCount?.recordArchiveData
2534		- self.yearlyCounts = Dictionary(
	2600	+ let typeCountYearlyCounts = Dictionary(
2535	2601	uniqueKeysWithValues: (typeCount?.yearlyCounts ?? []).map { ($0.year, $0.count) }
2536	2602	)
2537		- self.bins = (typeCount?.distributionBins ?? []).map {
2538		- Bin(
2539		- bucketStart: $0.bucketStart,
2540		- bucketEnd: $0.bucketEnd,
2541		- anchorData: $0.anchorData
2542		- )
	2603	+ self.yearlyCounts = typeCountYearlyCounts.isEmpty
	2604	+ ? archiveState?.yearlyCounts ?? [:]
	2605	+ : typeCountYearlyCounts
	2606	+ if typeCountBins.count == 1, typeCountBins[0].anchorData != nil {
	2607	+ self.bins = typeCountBins.map {
	2608	+ Bin(
	2609	+ bucketStart: $0.bucketStart,
	2610	+ bucketEnd: $0.bucketEnd,
	2611	+ anchorData: $0.anchorData
	2612	+ )
	2613	+ }
	2614	+ } else if let archiveState, let anchorData = archiveState.anchorData {
	2615	+ let bucketStart = archiveState.earliestDate ?? Date()
	2616	+ let rawBucketEnd = archiveState.latestDate ?? bucketStart
	2617	+ self.bins = [
	2618	+ Bin(
	2619	+ bucketStart: bucketStart,
	2620	+ bucketEnd: rawBucketEnd > bucketStart ? rawBucketEnd : bucketStart.addingTimeInterval(1),
	2621	+ anchorData: anchorData
	2622	+ )
	2623	+ ]
	2624	+ } else {
	2625	+ self.bins = []
2543	2626	}
2544	2627	}
2545	2628
	@@ -2550,7 +2633,6 @@ private struct PreviousDistributionState: Sendable {
2550	2633	latestDate: Date?
2551	2634	) -> SampleDistribution? {
2552	2635	guard count >= 0,
2553		- count == 0 \|\| recordArchiveData != nil,
2554	2636	count == 0 \|\| !contentHash.isEmpty else {
2555	2637	return nil
2556	2638	}