Group archive writes by snapshot observation · 21f0b20

Group archive writes by snapshot observation
Browse files

bogdan committed 2 weeks ago

main

1 parent c0aa933

commit 21f0b20

Showing 10 changed files with 446 additions and 99 deletions

+7 -5

HealthProbe/Doc/00-agent-guides/AGENTS.md

@@ -186,12 +186,14 @@ final class TypeDistributionBin {
 // SQLite archive store, in one schema that can preserve relationships across data
 // types, sources, devices, workouts, and metadata.
 
-// Interface updated 2026-05-18 — see AGENTS.md
+// Interface updated 2026-05-24 — see AGENTS.md
 // Services/Protocols/HealthArchiveStore.swift defines the local archive boundary.
-// SQLiteHealthArchiveStore is the current implementation. HealthKit anchored-query
-// pages must be written to this archive before UI/cache rows are saved.
-// Deletions are recorded by sampleUUIDHash because HKDeletedObject exposes UUIDs,
-// not complete sample payloads.
+// SQLiteHealthArchiveStore is the current implementation. Each user-visible
+// capture creates one archive observation; HealthKit anchored-query pages,
+// verification runs, and deleted-object evidence for that capture must attach to
+// that observation id before UI/cache rows are saved. Deletions are recorded by
+// sampleUUIDHash because HKDeletedObject exposes UUIDs, not complete sample
+// payloads.
 
 // Interface updated 2026-05-24 — see AGENTS.md
 // HealthArchiveStore exposes SQL-first observation diff APIs:


+17 -6

HealthProbe/Doc/02-architecture/Database-Design.md

View

@@ -91,6 +91,12 @@ An observation records:
 - HealthKit anchors;
 - events and aggregate changes observed during the capture.
 
+Implementation note, 2026-05-24: the app creates one archive `observations`
+row for each user-visible capture/snapshot. Anchored HealthKit pages,
+`HKDeletedObject` evidence, and the final type verification rows for that
+capture attach to that same observation id so SQL diffs, cache rows, and
+transition UI can reference a stable capture boundary.
+
 ### Terminology
 
 - **Capture**: the act of querying HealthKit and writing results to the archive.
@@ -538,10 +544,10 @@ Manifest hashes must cover exported content through item hashes, not only counts
 
 ### 7.1 Capture Transaction Shape
 
-For each observation/type run:
+For each user-visible capture:
 
-1. Open SQLite transaction.
-2. Insert/update `observations` and `observation_type_runs`.
+1. Create one `observations` row with status `running`.
+2. For each observation/type run, update `observation_type_runs`.
 3. For each added/visible sample page:
    - upsert source/source revision/device/metadata;
    - upsert `samples`;
@@ -552,9 +558,14 @@ For each observation/type run:
    - find sample by UUID hash;
    - insert deleted/disappeared event;
    - close open visibility ranges.
-5. Recompute affected materialized aggregates.
-6. Commit SQLite.
-7. Update/rebuild Core Data cache after SQLite commit.
+5. Record final per-type verification counts for the same observation id.
+6. Recompute affected materialized aggregates.
+7. Mark the observation `completed`, `needs_review`, `partial_*`, or `failed`.
+
+SQLite writes may be committed per page/type to keep memory and lock time
+bounded, but they must share the capture's observation id.
+
+8. Update/rebuild Core Data cache after SQLite commit.
 
 SQLite commit must happen before Core Data cache update. Cache rebuild failures must not corrupt archive truth.
 


+2 -2

HealthProbe/Doc/02-architecture/Implementation-Guide.md

View

@@ -62,9 +62,9 @@ Use:
 
 Capture flow:
 1. Resolve the current local device chain ID.
-2. Start an observation record.
+2. Start one archive observation record for the user-visible capture and keep its id.
 3. For each selected sample type, run anchored queries.
-4. Write HealthKit samples and deleted-object evidence to the local archive first.
+4. Write HealthKit samples, deleted-object evidence, and final per-type verification to the local archive first, all under that same observation id.
 5. Update materialized aggregate tables in SQLite.
 6. Save/rebuild derived Core Data cache rows only after archive writes succeed.
 7. Compute summary/diff caches for UI and reports.


+4 -4

HealthProbe/Doc/04-project/IMPLEMENTATION_STATUS.md

View

@@ -24,8 +24,8 @@ There are no real deployments, only test installations. Existing prototype datab
 | Area | Current Status | Target / Next Work |
 |------|----------------|--------------------|
 | Product docs | Updated | Keep `HealthProbe/Doc/README.md` as canonical index |
-| HealthKit capture | Prototype exists | Adapt capture to write differential SQLite observations first |
-| SQLite archive | Archive v2 schema, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed | Start Core Data cache work |
+| HealthKit capture | Capture now opens one archive observation per user-visible snapshot and attaches HealthKit pages, deleted-object evidence, and type verification to that observation id before finishing it | Continue moving UI/cache reads to archive-backed observation ids |
+| SQLite archive | Archive v2 schema, snapshot-level observation grouping, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed | Move Snapshots/Data Types from SwiftData previews to archive/cache DTOs |
 | Core Data cache | Initial programmatic Core Data model, full-cache rebuild service, read DTOs for observation/type/health rows, and Dashboard archive-cache status wiring are in place | Move Snapshots/Data Types to cache DTOs and add targeted partial invalidation |
 | SwiftData cache | Exists; test builds now reset legacy prototype UI/archive/cache stores once for archive v2 so old SwiftData-only snapshots are not treated as backed-up observations | Treat as disposable prototype data; reset/ignore during v2 transition |
 | UI | Prototype exists; Snapshots/Data Types now default to the local device timeline instead of a multi-device picker, and record-change detail uses a separate preview/paged list view with archive-value enrichment and scoped export action | Reframe remaining screens around observations, diffs, export, archive status |
@@ -53,12 +53,12 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m
 - Legacy SwiftData-only snapshots can show diffs without archive-backed values; they are now reset for archive v2 test installs rather than migrated.
 - Existing implementation may decode or cache too much data for low-end devices.
 - Old prototype database compatibility is no longer required.
-- Initial SQLite archive tests cover open/init/reset/idempotency, legacy mirror removal, small observation diffs, large synthetic diff pagination, formal timing/memory metrics, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet export behavior.
+- Initial SQLite archive tests cover open/init/reset/idempotency, snapshot-level observation grouping, legacy mirror removal, small observation diffs, large synthetic diff pagination, formal timing/memory metrics, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet export behavior.
 - Initial Core Data cache tests cover full rebuild from SQLite and delete-cache-then-rebuild without losing archive data.
 
 ## Verification Checklist
 
-- [ ] SQLite archive v2 can reconstruct records visible at observation T.
+- [x] SQLite archive v2 can reconstruct records visible at observation T.
 - [ ] No recurring complete snapshot copies are written for high-volume types.
 - [x] SQL diff between two observations runs without loading full datasets into Swift arrays.
 - [x] Expensive counts used by reports/UI are cached and rebuildable.


+4 -1

HealthProbe/Doc/04-project/Refactoring-Plan.md

View

@@ -1,6 +1,6 @@
 # HealthProbe - Database-Led Refactoring Plan
 
-**Last Updated:** 2026-05-23
+**Last Updated:** 2026-05-24
 **Status:** Active planning document
 
 ## Goal
@@ -119,6 +119,8 @@ Acceptance:
 
 Checklist:
 - [x] Create observation transaction wrapper.
+- [x] Attach HealthKit pages, deleted-object evidence, and final type verification to the same user-visible archive observation.
+- [x] Store the archive observation id on the legacy `HealthSnapshot` bridge model for transition screens.
 - [x] Upsert sample types.
 - [x] Upsert source/source revision/device/metadata rows.
 - [x] Upsert sample identity.
@@ -140,6 +142,7 @@ Acceptance:
 - [x] Representation change creates a new version, not a new logical sample.
 - [x] Disappearance closes visibility range.
 - [x] No full observation copy table is created or written.
+- [x] A user-visible capture has one archive observation id that SQL diff/cache/UI layers can reference.
 
 ## Milestone 5 - SQL Analysis Layer
 


+1 -0

HealthProbe/Models/HealthSnapshot.swift

View

@@ -24,6 +24,7 @@ import SwiftData
     var appBuildVersion: String = ""
     var monitoredTypeSetHash: String = ""
     var monitoredRegistryVersion: Int = 0
+    var archiveObservationID: Int64?
     var yearlyCountTimezoneIdentifier: String = ""
     var cachedSummaryVersion: Int = 0
     var cachedTypeCount: Int = 0


+55 -12

HealthProbe/Services/HealthKitService.swift

View

@@ -96,6 +96,15 @@ final class HealthKitService {
             deviceName: UIDevice.current.name,
             deviceID: deviceResolution.id
         )
+        let intendedTypeIDs = active.map { $0.id }
+        let monitoredTypeSetHash = HashService.typeSetHash(typeIDs: intendedTypeIDs)
+        let archiveObservationID = try await archiveStore.beginObservation(
+            observedAt: snapshot.timestamp,
+            triggerReason: triggerReason,
+            selectedTypeSetHash: monitoredTypeSetHash
+        )
+        snapshot.archiveObservationID = archiveObservationID
+        snapshot.monitoredTypeSetHash = monitoredTypeSetHash
         snapshot.recoveredDeviceID = deviceResolution.isRecovered
         snapshot.triggerReason = triggerReason
         snapshot.retryOfSnapshotID = retryOfSnapshotID
@@ -108,6 +117,7 @@ final class HealthKitService {
             context: context,
             snapshot: snapshot,
             previousSnapshot: previousSnapshot,
+            archiveObservationID: archiveObservationID,
             adaptiveTimeoutsEnabled: adaptiveTimeoutsEnabled,
             timeoutMultiplier: timeoutMultiplier,
             progress: progress
@@ -143,7 +153,7 @@ final class HealthKitService {
         configureSnapshotMetadata(
             snapshot,
             typeCounts: typeCounts,
-            intendedTypeIDs: active.map { $0.id },
+            intendedTypeIDs: intendedTypeIDs,
             context: context
         )
         markContentEquivalenceIfNeeded(
@@ -160,11 +170,30 @@ final class HealthKitService {
         if snapshot.snapshotQuality == .complete,
            reviewAmbiguousCompleteDisappearedTypes,
            hasAmbiguousCompleteDisappearance(snapshot: snapshot, typeCounts: typeCounts, context: context) {
+            try await archiveStore.finishObservation(
+                observationID: archiveObservationID,
+                status: "needs_review",
+                endedAt: Date()
+            )
             return snapshot
         }
 
-        if snapshot.snapshotQuality == .complete {
-            try await persistSnapshot(snapshot, typeCounts: typeCounts, context: context)
+        do {
+            if snapshot.snapshotQuality == .complete {
+                try await persistSnapshot(snapshot, typeCounts: typeCounts, context: context)
+            }
+            try await archiveStore.finishObservation(
+                observationID: archiveObservationID,
+                status: snapshot.snapshotQuality == .complete ? "completed" : "partial_\(snapshot.snapshotQuality.rawValue)",
+                endedAt: Date()
+            )
+        } catch {
+            try? await archiveStore.finishObservation(
+                observationID: archiveObservationID,
+                status: "failed",
+                endedAt: Date()
+            )
+            throw error
         }
 
         return snapshot
@@ -573,6 +602,7 @@ final class HealthKitService {
         context: ModelContext,
         snapshot: HealthSnapshot,
         previousSnapshot: HealthSnapshot?,
+        archiveObservationID: Int64,
         adaptiveTimeoutsEnabled: Bool,
         timeoutMultiplier: Double,
         progress: SnapshotFetchProgress? = nil
@@ -592,6 +622,7 @@ final class HealthKitService {
                 timeoutProfile: profile,
                 timeoutSeconds: timeout,
                 previousTypeCount: previousSnapshot?.typeCounts?.first { $0.typeIdentifier == monitoredType.id },
+                archiveObservationID: archiveObservationID,
                 progress: progress
             )
             updateTimeoutProfile(profile, with: result, monitoredType: monitoredType)
@@ -609,6 +640,7 @@ final class HealthKitService {
         timeoutProfile: MetricTimeoutProfile,
         timeoutSeconds: TimeInterval,
         previousTypeCount: TypeCount?,
+        archiveObservationID: Int64,
         progress: SnapshotFetchProgress? = nil
     ) async -> TypeCountFetchResult {
         let started = Date()
@@ -647,6 +679,7 @@ final class HealthKitService {
             sampleType: sampleType,
             timeoutSeconds: timeoutSeconds,
             previousTypeCount: previousTypeCount,
+            archiveObservationID: archiveObservationID,
             progress: progress
         )
         result.totalElapsedSeconds = Date().timeIntervalSince(started)
@@ -672,6 +705,7 @@ final class HealthKitService {
         sampleType: HKSampleType,
         timeoutSeconds: TimeInterval,
         previousTypeCount: TypeCount?,
+        archiveObservationID: Int64,
         progress: SnapshotFetchProgress?
     ) async -> TypeCountFetchResult {
         let dateDeadline = Date().addingTimeInterval(timeoutSeconds)
@@ -730,6 +764,7 @@ final class HealthKitService {
                 earliestDate: earliest,
                 latestDate: latest,
                 previousDistribution: previousDistribution,
+                archiveObservationID: archiveObservationID,
                 progress: progress
             )
         } resultDescription: { distribution in
@@ -826,6 +861,7 @@ final class HealthKitService {
         earliestDate: Date?,
         latestDate: Date?,
         previousDistribution: PreviousDistributionState,
+        archiveObservationID: Int64,
         progress: SnapshotFetchProgress?
     ) async throws -> SampleDistribution {
         var anchor = previousDistribution.globalAnchor
@@ -859,7 +895,7 @@ final class HealthKitService {
                     anchor: anchor
                 )
             }
-            try await archivePage(page, sampleType: sampleType)
+            try await archivePage(page, sampleType: sampleType, observationID: archiveObservationID)
             anchor = page.anchor
 
             if page.samples.isEmpty, page.deletedObjects.isEmpty,
@@ -869,7 +905,11 @@ final class HealthKitService {
                     earliestDate: earliestDate,
                     latestDate: latestDate
                ) {
-                try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date())
+                try await archiveStore.markVerification(
+                    sampleType: sampleType,
+                    verifiedAt: Date(),
+                    observationID: archiveObservationID
+                )
                 progress?.updateBlockProgress(
                     typeIdentifier,
                     detail: "No HealthKit delta",
@@ -890,6 +930,7 @@ final class HealthKitService {
                 earliestDate: earliestDate,
                 latestDate: latestDate,
                 progressStarted: progressStarted,
+                archiveObservationID: archiveObservationID,
                 progress: progress
             )
         }
@@ -941,7 +982,7 @@ final class HealthKitService {
                     anchor: anchor
                 )
             }
-            try await archivePage(page, sampleType: sampleType)
+            try await archivePage(page, sampleType: sampleType, observationID: archiveObservationID)
             anchor = page.anchor
 
             applyDistributionPage(page, sampleType: sampleType, to: &recordMap)
@@ -963,7 +1004,7 @@ final class HealthKitService {
             )
         }
 
-        try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date())
+        try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date(), observationID: archiveObservationID)
 
         let sortedKeys = recordMap.keys.sorted {
             guard let left = recordMap[$0],
@@ -1040,6 +1081,7 @@ final class HealthKitService {
         earliestDate: Date?,
         latestDate: Date?,
         progressStarted: Date,
+        archiveObservationID: Int64,
         progress: SnapshotFetchProgress?
     ) async throws -> SampleDistribution {
         var anchor: HKQueryAnchor?
@@ -1078,7 +1120,7 @@ final class HealthKitService {
                     anchor: anchor
                 )
             }
-            try await archivePage(page, sampleType: sampleType)
+            try await archivePage(page, sampleType: sampleType, observationID: archiveObservationID)
             anchor = page.anchor
 
             for sample in page.samples {
@@ -1122,7 +1164,7 @@ final class HealthKitService {
             )
         )
 
-        try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date())
+        try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date(), observationID: archiveObservationID)
 
         guard recordCount > 0 || anchor != nil else {
             return SampleDistribution(
@@ -1266,14 +1308,15 @@ final class HealthKitService {
         }
     }
 
-    private func archivePage(_ page: SampleDistributionPage, sampleType: HKSampleType) async throws {
+    private func archivePage(_ page: SampleDistributionPage, sampleType: HKSampleType, observationID: Int64) async throws {
         let observedAt = Date()
-        _ = try await archiveStore.upsertSamples(page.samples, observedAt: observedAt)
+        _ = try await archiveStore.upsertSamples(page.samples, observedAt: observedAt, observationID: observationID)
         for deletedObject in page.deletedObjects {
             try await archiveStore.recordDisappearance(
                 sampleUUIDHash: HashService.sampleUUIDHash(deletedObject.uuid.uuidString),
                 sampleTypeIdentifier: sampleType.identifier,
-                observedMissingAt: observedAt
+                observedMissingAt: observedAt,
+                observationID: observationID
             )
         }
     }


+6 -1

HealthProbe/Services/Protocols/HealthArchiveStore.swift

View

@@ -1,11 +1,16 @@
 import Foundation
 import HealthKit
 
-// Interface updated 2026-05-23 — see AGENTS.md
+// Interface updated 2026-05-24 — see AGENTS.md
 protocol HealthArchiveStore {
+    func beginObservation(observedAt: Date, triggerReason: String, selectedTypeSetHash: String?) async throws -> Int64
+    func finishObservation(observationID: Int64, status: String, endedAt: Date) async throws
     func upsertSamples(_ samples: [HKSample], observedAt: Date) async throws -> HealthArchiveWriteSummary
+    func upsertSamples(_ samples: [HKSample], observedAt: Date, observationID: Int64) async throws -> HealthArchiveWriteSummary
     func markVerification(sampleType: HKSampleType, verifiedAt: Date) async throws
+    func markVerification(sampleType: HKSampleType, verifiedAt: Date, observationID: Int64) async throws
     func recordDisappearance(sampleUUIDHash: String, sampleTypeIdentifier: String, observedMissingAt: Date) async throws
+    func recordDisappearance(sampleUUIDHash: String, sampleTypeIdentifier: String, observedMissingAt: Date, observationID: Int64) async throws
     func records(for request: HealthArchiveRecordRequest) async throws -> [ArchivedHealthRecord]
     func diffSummary(_ request: HealthArchiveDiffRequest) async throws -> HealthArchiveDiffSummary
     func diffRecords(_ request: HealthArchiveDiffRecordRequest) async throws -> [ArchivedHealthRecord]


+298 -67

HealthProbe/Services/SQLiteHealthArchiveStore.swift

View

+                         case exportEncodingFailed
+                    -// Interface updated 2026-05-18 — see AGENTS.md
+                    +// Interface updated 2026-05-24 — see AGENTS.md
+                     actor SQLiteHealthArchiveStore: HealthArchiveStore {
+                         static let shared = SQLiteHealthArchiveStore()
+                         nonisolated static let defaultDatabaseURL = URL.applicationSupportDirectory.appending(path: "HealthProbeArchive.sqlite")
+                             self.databaseURL = databaseURL ?? Self.defaultDatabaseURL
+                    +    func beginObservation(observedAt: Date, triggerReason: String, selectedTypeSetHash: String?) async throws -> Int64 {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        try execute("BEGIN IMMEDIATE TRANSACTION", db: db)
+                    +        do {
+                    +            let observationID = try createObservation(
+                    +                observedAt: observedAt,
+                    +                triggerReason: triggerReason,
+                    +                status: "running",
+                    +                selectedTypeSetHash: selectedTypeSetHash,
+                    +                db: db
+                    +            )
+                    +            try execute("COMMIT", db: db)
+                    +            return observationID
+                    +        } catch {
+                    +            try? execute("ROLLBACK", db: db)
+                    +            throw error
+                    +        }
+                    +    }
+                    +    func finishObservation(observationID: Int64, status: String, endedAt: Date) async throws {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        try execute("BEGIN IMMEDIATE TRANSACTION", db: db)
+                    +        do {
+                    +            try updateObservationStatus(
+                    +                observationID: observationID,
+                    +                status: status,
+                    +                endedAt: endedAt,
+                    +                db: db
+                    +            )
+                    +            try execute("COMMIT", db: db)
+                    +        } catch {
+                    +            try? execute("ROLLBACK", db: db)
+                    +            throw error
+                    +        }
+                    +    }
+                         func upsertSamples(_ samples: [HKSample], observedAt: Date) async throws -> HealthArchiveWriteSummary {
+                             guard !samples.isEmpty else {
+                                 return HealthArchiveWriteSummary(insertedCount: 0, updatedCount: 0, unchangedCount: 0)
+                    +    func upsertSamples(_ samples: [HKSample], observedAt: Date, observationID: Int64) async throws -> HealthArchiveWriteSummary {
+                    +        guard !samples.isEmpty else {
+                    +            return HealthArchiveWriteSummary(insertedCount: 0, updatedCount: 0, unchangedCount: 0)
+                    +        }
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        try execute("BEGIN IMMEDIATE TRANSACTION", db: db)
+                    +        do {
+                    +            let summary = try upsertSamples(samples, observedAt: observedAt, observationID: observationID, db: db)
+                    +            try execute("PRAGMA foreign_key_check", db: db)
+                    +            try execute("COMMIT", db: db)
+                    +            return summary
+                    +        } catch {
+                    +            try? execute("ROLLBACK", db: db)
+                    +            throw error
+                    +        }
+                    +    }
+                         func markVerification(sampleType: HKSampleType, verifiedAt: Date) async throws {
+                             let db = try openDatabase()
+                             defer { sqlite3_close(db) }
+                                     status: "completed",
+                                     db: db
+                    -            let sampleTypeID = try upsertSampleType(typeIdentifier: sampleType.identifier, db: db)
+                    -            let visibleCount = try visibleAggregate(sampleTypeID: sampleTypeID, db: db).visibleRecordCount
+                    -            try insertObservationTypeRun(
+                    -                observationID: observationID,
+                    -                sampleTypeID: sampleTypeID,
+                    -                status: "completed",
+                    -                observedAt: verifiedAt,
+                    -                insertedEventCount: 0,
+                    -                deletedEventCount: 0,
+                    -                verifiedVisibleCount: visibleCount,
+                    -                db: db
+                    -            )
+                    -            try rebuildTypeSummary(observationID: observationID, sampleTypeID: sampleTypeID, db: db)
+                    -            try rebuildDailyAggregates(
+                    -                observationID: observationID,
+                    -                sampleTypeID: sampleTypeID,
+                    -                observedAt: verifiedAt,
+                    -                db: db
+                    -            )
+                    +            try markVerification(sampleType: sampleType, verifiedAt: verifiedAt, observationID: observationID, db: db)
+                    +            try execute("COMMIT", db: db)
+                    +        } catch {
+                    +            try? execute("ROLLBACK", db: db)
+                    +            throw error
+                    +        }
+                    +    }
+                    +    func markVerification(sampleType: HKSampleType, verifiedAt: Date, observationID: Int64) async throws {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        try execute("BEGIN IMMEDIATE TRANSACTION", db: db)
+                    +        do {
+                    +            try markVerification(sampleType: sampleType, verifiedAt: verifiedAt, observationID: observationID, db: db)
+                                 try execute("COMMIT", db: db)
+                             } catch {
+                                 try? execute("ROLLBACK", db: db)
+                                     status: "completed",
+                                     db: db
+                    -            if let sampleTypeID = try sampleTypeID(typeIdentifier: sampleTypeIdentifier, db: db),
+                    -               let sampleID = try sampleID(sampleUUIDHash: sampleUUIDHash, sampleTypeID: sampleTypeID, db: db) {
+                    -                try insertObservationEvent(
+                    -                    observationID: observationID,
+                    -                    sampleID: sampleID,
+                    -                    versionID: nil,
+                    -                    eventKind: "disappeared",
+                    -                    evidenceKind: "deleted_object",
+                    -                    observedAt: observedMissingAt,
+                    -                    db: db
+                    -                )
+                    -                try closeOpenVisibilityRanges(
+                    -                    sampleID: sampleID,
+                    -                    excludingVersionID: nil,
+                    -                    closedAtObservationID: observationID,
+                    -                    observedAt: observedMissingAt,
+                    -                    db: db
+                    -                )
+                    -                try rebuildTypeSummary(observationID: observationID, sampleTypeID: sampleTypeID, db: db)
+                    -                try rebuildDailyAggregates(
+                    -                    observationID: observationID,
+                    -                    sampleTypeID: sampleTypeID,
+                    -                    observedAt: observedMissingAt,
+                    -                    db: db
+                    -                )
+                    -            }
+                    +            try recordDisappearance(
+                    +                sampleUUIDHash: sampleUUIDHash,
+                    +                sampleTypeIdentifier: sampleTypeIdentifier,
+                    +                observedMissingAt: observedMissingAt,
+                    +                observationID: observationID,
+                    +                db: db
+                    +            )
+                    +            try execute("COMMIT", db: db)
+                    +        } catch {
+                    +            try? execute("ROLLBACK", db: db)
+                    +            throw error
+                    +        }
+                    +    }
+                    +    func recordDisappearance(
+                    +        sampleUUIDHash: String,
+                    +        sampleTypeIdentifier: String,
+                    +        observedMissingAt: Date,
+                    +        observationID: Int64
+                    +    ) async throws {
+                    +        let db = try openDatabase()
+                    +        defer { sqlite3_close(db) }
+                    +        try prepareSchemaIfNeeded(db)
+                    +        try execute("BEGIN IMMEDIATE TRANSACTION", db: db)
+                    +        do {
+                    +            try recordDisappearance(
+                    +                sampleUUIDHash: sampleUUIDHash,
+                    +                sampleTypeIdentifier: sampleTypeIdentifier,
+                    +                observedMissingAt: observedMissingAt,
+                    +                observationID: observationID,
+                    +                db: db
+                    +            )
+                                 try execute("COMMIT", db: db)
+                             } catch {
+                                 try? execute("ROLLBACK", db: db)
+                         private func upsertSamples(_ samples: [HKSample], observedAt: Date, db: OpaquePointer?) throws -> HealthArchiveWriteSummary {
+                    -        let rows = samples.map { ArchiveSampleRow(sample: $0, observedAt: observedAt) }
+                             let observationID = try createObservation(
+                                 observedAt: observedAt,
+                                 triggerReason: "anchored_page",
+                                 status: "completed",
+                                 db: db
+                    +        return try upsertSamples(samples, observedAt: observedAt, observationID: observationID, db: db)
+                    +    }
+                    +    private func upsertSamples(
+                    +        _ samples: [HKSample],
+                    +        observedAt: Date,
+                    +        observationID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws -> HealthArchiveWriteSummary {
+                    +        let rows = samples.map { ArchiveSampleRow(sample: $0, observedAt: observedAt) }
+                             var inserted = 0
+                             var updated = 0
+                             var unchanged = 0
+                    -        var touchedTypeIDs = Set<Int64>()
+                    +        var typeEventCounts: [Int64: (inserted: Int, deleted: Int)] = [:]
+                             for row in rows {
+                                 let result = try upsertArchiveV2Sample(row, observationID: observationID, db: db)
+                    -            touchedTypeIDs.insert(result.sampleTypeID)
+                    +            var counts = typeEventCounts[result.sampleTypeID, default: (inserted: 0, deleted: 0)]
+                                 switch result.kind {
+                                 case .inserted:
+                                     inserted += 1
+                    +                counts.inserted += 1
+                                 case .updated:
+                                     updated += 1
+                    +                counts.inserted += 1
+                                 case .unchanged:
+                                     unchanged += 1
+                    +            typeEventCounts[result.sampleTypeID] = counts
+                    -        for sampleTypeID in touchedTypeIDs {
+                    +        for (sampleTypeID, eventCounts) in typeEventCounts {
+                    +            try insertObservationTypeRun(
+                    +                observationID: observationID,
+                    +                sampleTypeID: sampleTypeID,
+                    +                status: "running",
+                    +                observedAt: observedAt,
+                    +                insertedEventCount: eventCounts.inserted,
+                    +                deletedEventCount: eventCounts.deleted,
+                    +                verifiedVisibleCount: nil,
+                    +                db: db
+                    +            )
+                                 try rebuildTypeSummary(observationID: observationID, sampleTypeID: sampleTypeID, db: db)
+                                 try rebuildDailyAggregates(
+                                     observationID: observationID,
+                    +    private func markVerification(
+                    +        sampleType: HKSampleType,
+                    +        verifiedAt: Date,
+                    +        observationID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws {
+                    +        let sampleTypeID = try upsertSampleType(typeIdentifier: sampleType.identifier, db: db)
+                    +        let visibleCount = try visibleAggregate(sampleTypeID: sampleTypeID, db: db).visibleRecordCount
+                    +        try insertObservationTypeRun(
+                    +            observationID: observationID,
+                    +            sampleTypeID: sampleTypeID,
+                    +            status: "completed",
+                    +            observedAt: verifiedAt,
+                    +            insertedEventCount: 0,
+                    +            deletedEventCount: 0,
+                    +            verifiedVisibleCount: visibleCount,
+                    +            db: db
+                    +        )
+                    +        try rebuildTypeSummary(observationID: observationID, sampleTypeID: sampleTypeID, db: db)
+                    +        try rebuildDailyAggregates(
+                    +            observationID: observationID,
+                    +            sampleTypeID: sampleTypeID,
+                    +            observedAt: verifiedAt,
+                    +            db: db
+                    +        )
+                    +    }
+                    +    private func recordDisappearance(
+                    +        sampleUUIDHash: String,
+                    +        sampleTypeIdentifier: String,
+                    +        observedMissingAt: Date,
+                    +        observationID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws {
+                    +        guard let sampleTypeID = try sampleTypeID(typeIdentifier: sampleTypeIdentifier, db: db),
+                    +              let sampleID = try sampleID(sampleUUIDHash: sampleUUIDHash, sampleTypeID: sampleTypeID, db: db) else {
+                    +            return
+                    +        }
+                    +        try insertObservationEvent(
+                    +            observationID: observationID,
+                    +            sampleID: sampleID,
+                    +            versionID: nil,
+                    +            eventKind: "disappeared",
+                    +            evidenceKind: "deleted_object",
+                    +            observedAt: observedMissingAt,
+                    +            db: db
+                    +        )
+                    +        try closeOpenVisibilityRanges(
+                    +            sampleID: sampleID,
+                    +            excludingVersionID: nil,
+                    +            closedAtObservationID: observationID,
+                    +            observedAt: observedMissingAt,
+                    +            db: db
+                    +        )
+                    +        try insertObservationTypeRun(
+                    +            observationID: observationID,
+                    +            sampleTypeID: sampleTypeID,
+                    +            status: "running",
+                    +            observedAt: observedMissingAt,
+                    +            insertedEventCount: 0,
+                    +            deletedEventCount: 1,
+                    +            verifiedVisibleCount: nil,
+                    +            db: db
+                    +        )
+                    +        try rebuildTypeSummary(observationID: observationID, sampleTypeID: sampleTypeID, db: db)
+                    +        try rebuildDailyAggregates(
+                    +            observationID: observationID,
+                    +            sampleTypeID: sampleTypeID,
+                    +            observedAt: observedMissingAt,
+                    +            db: db
+                    +        )
+                    +    }
+                         private func upsertArchiveV2Sample(
+                             _ row: ArchiveSampleRow,
+                             observationID: Int64,
+                             observedAt: Date,
+                             triggerReason: String,
+                             status: String,
+                    +        selectedTypeSetHash: String? = nil,
+                             db: OpaquePointer?
+                         ) throws -> Int64 {
+                             let deviceChainID = try upsertCurrentDeviceChain(db)
+                                 device_chain_id, observed_at, started_at, ended_at, status, trigger_reason,
+                                 app_version, os_version, time_zone_identifier, time_zone_seconds_from_gmt,
+                                 schema_version, selected_type_set_hash, notes
+                    -        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL, NULL)
+                    +        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL)
+                             """
+                             try withStatement(sql, db: db) { statement in
+                                 bindInt64(deviceChainID, to: 1, in: statement)
+                                 bindText(timeZone.identifier, to: 9, in: statement)
+                                 sqlite3_bind_int(statement, 10, Int32(timeZone.secondsFromGMT(for: observedAt)))
+                                 bindInt(Self.archiveSchemaVersion, to: 11, in: statement)
+                    +            bindText(selectedTypeSetHash, to: 12, in: statement)
+                                 guard sqlite3_step(statement) == SQLITE_DONE else {
+                                     throw SQLiteHealthArchiveStoreError.stepFailed(lastErrorMessage(db))
+                             return sqlite3_last_insert_rowid(db)
+                    +    private func updateObservationStatus(
+                    +        observationID: Int64,
+                    +        status: String,
+                    +        endedAt: Date,
+                    +        db: OpaquePointer?
+                    +    ) throws {
+                    +        try withStatement(
+                    +            "UPDATE observations SET status = ?, ended_at = ? WHERE id = ?",
+                    +            db: db
+                    +        ) { statement in
+                    +            bindText(status, to: 1, in: statement)
+                    +            sqlite3_bind_double(statement, 2, endedAt.timeIntervalSince1970)
+                    +            bindInt64(observationID, to: 3, in: statement)
+                    +            guard sqlite3_step(statement) == SQLITE_DONE else {
+                    +                throw SQLiteHealthArchiveStoreError.stepFailed(lastErrorMessage(db))
+                    +            }
+                    +        }
+                    +    }
+                         private func insertObservationTypeRun(
+                             observationID: Int64,
+                             sampleTypeID: Int64,
+                             verifiedVisibleCount: Int?,
+                             db: OpaquePointer?
+                         ) throws {
+                    -        let sql = """
+                    -        INSERT OR REPLACE INTO observation_type_runs (
+                    -            observation_id, sample_type_id, status, started_at, ended_at,
+                    -            inserted_event_count, deleted_event_count, verified_visible_count
+                    -        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+                    -        """
+                    -        try withStatement(sql, db: db) { statement in
+                    +        if let existing = try observationTypeRunCounts(
+                    +            observationID: observationID,
+                    +            sampleTypeID: sampleTypeID,
+                    +            db: db
+                    +        ) {
+                    +            try withStatement(
+                    +                """
+                    +                UPDATE observation_type_runs
+                    +                SET status = ?,
+                    +                    started_at = COALESCE(started_at, ?),
+                    +                    ended_at = ?,
+                    +                    inserted_event_count = ?,
+                    +                    deleted_event_count = ?,
+                    +                    verified_visible_count = COALESCE(?, verified_visible_count)
+                    +                WHERE observation_id = ? AND sample_type_id = ?
+                    +                """,
+                    +                db: db
+                    +            ) { statement in
+                    +                bindText(status, to: 1, in: statement)
+                    +                sqlite3_bind_double(statement, 2, observedAt.timeIntervalSince1970)
+                    +                sqlite3_bind_double(statement, 3, observedAt.timeIntervalSince1970)
+                    +                bindInt(existing.inserted + insertedEventCount, to: 4, in: statement)
+                    +                bindInt(existing.deleted + deletedEventCount, to: 5, in: statement)
+                    +                bindInt(verifiedVisibleCount, to: 6, in: statement)
+                    +                bindInt64(observationID, to: 7, in: statement)
+                    +                bindInt64(sampleTypeID, to: 8, in: statement)
+                    +                guard sqlite3_step(statement) == SQLITE_DONE else {
+                    +                    throw SQLiteHealthArchiveStoreError.stepFailed(lastErrorMessage(db))
+                    +                }
+                    +            }
+                    +        } else {
+                    +            let sql = """
+                    +            INSERT INTO observation_type_runs (
+                    +                observation_id, sample_type_id, status, started_at, ended_at,
+                    +                inserted_event_count, deleted_event_count, verified_visible_count
+                    +            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+                    +            """
+                    +            try withStatement(sql, db: db) { statement in
+                    +                bindInt64(observationID, to: 1, in: statement)
+                    +                bindInt64(sampleTypeID, to: 2, in: statement)
+                    +                bindText(status, to: 3, in: statement)
+                    +                sqlite3_bind_double(statement, 4, observedAt.timeIntervalSince1970)
+                    +                sqlite3_bind_double(statement, 5, observedAt.timeIntervalSince1970)
+                    +                bindInt(insertedEventCount, to: 6, in: statement)
+                    +                bindInt(deletedEventCount, to: 7, in: statement)
+                    +                bindInt(verifiedVisibleCount, to: 8, in: statement)
+                    +                guard sqlite3_step(statement) == SQLITE_DONE else {
+                    +                    throw SQLiteHealthArchiveStoreError.stepFailed(lastErrorMessage(db))
+                    +                }
+                    +            }
+                    +        }
+                    +    }
+                    +    private func observationTypeRunCounts(
+                    +        observationID: Int64,
+                    +        sampleTypeID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws -> (inserted: Int, deleted: Int)? {
+                    +        try withStatement(
+                    +            """
+                    +            SELECT inserted_event_count, deleted_event_count
+                    +            FROM observation_type_runs
+                    +            WHERE observation_id = ? AND sample_type_id = ?
+                    +            LIMIT 1
+                    +            """,
+                    +            db: db
+                    +        ) { statement in
+                                 bindInt64(observationID, to: 1, in: statement)
+                                 bindInt64(sampleTypeID, to: 2, in: statement)
+                    -            bindText(status, to: 3, in: statement)
+                    -            sqlite3_bind_double(statement, 4, observedAt.timeIntervalSince1970)
+                    -            sqlite3_bind_double(statement, 5, observedAt.timeIntervalSince1970)
+                    -            bindInt(insertedEventCount, to: 6, in: statement)
+                    -            bindInt(deletedEventCount, to: 7, in: statement)
+                    -            bindInt(verifiedVisibleCount, to: 8, in: statement)
+                    -            guard sqlite3_step(statement) == SQLITE_DONE else {
+                    -                throw SQLiteHealthArchiveStoreError.stepFailed(lastErrorMessage(db))
+                    -            }
+                    +            guard sqlite3_step(statement) == SQLITE_ROW else { return nil }
+                    +            return (Int(sqlite3_column_int(statement, 0)), Int(sqlite3_column_int(statement, 1)))

+52 -1

HealthProbeTests/SQLiteHealthArchiveStoreTests.swift

View

+                             let observationIDs = try observationIDs(at: url)
+                             XCTAssertEqual(observationIDs.count, 2)
+                    -        XCTAssertEqual(try countRows(in: "observation_type_runs", at: url), 1)
+                    +        XCTAssertEqual(try countRows(in: "observation_type_runs", at: url), 2)
+                    +        XCTAssertEqual(try countRows(in: "observation_type_runs WHERE observation_id = \(observationIDs[0]) AND inserted_event_count = 1", at: url), 1)
+                             XCTAssertEqual(try countRows(in: "observation_type_summaries WHERE observation_id = \(observationIDs[1]) AND visible_record_count = 1", at: url), 1)
+                             XCTAssertFalse(try tableExists("archive_samples", at: url))
+                             XCTAssertNotNil(disappearedRecords.first?.disappearedAt)
+                    +    func testGroupedObservationKeepsPageWritesDeletesAndVerificationTogether() async throws {
+                    +        let url = databaseURL()
+                    +        let store = SQLiteHealthArchiveStore(databaseURL: url)
+                    +        let firstSample = makeStepCountSample(value: 42, start: 1_000)
+                    +        let secondSample = makeStepCountSample(value: 7, start: 2_000)
+                    +        let typeIdentifier = HKQuantityTypeIdentifier.stepCount.rawValue
+                    +        _ = try await store.upsertSamples([firstSample], observedAt: Date(timeIntervalSince1970: 3_000))
+                    +        let observationID = try await store.beginObservation(
+                    +            observedAt: Date(timeIntervalSince1970: 3_060),
+                    +            triggerReason: "manual",
+                    +            selectedTypeSetHash: "selected-types"
+                    +        )
+                    +        _ = try await store.upsertSamples(
+                    +            [secondSample],
+                    +            observedAt: Date(timeIntervalSince1970: 3_060),
+                    +            observationID: observationID
+                    +        )
+                    +        try await store.recordDisappearance(
+                    +            sampleUUIDHash: HashService.sampleUUIDHash(firstSample.uuid.uuidString),
+                    +            sampleTypeIdentifier: typeIdentifier,
+                    +            observedMissingAt: Date(timeIntervalSince1970: 3_060),
+                    +            observationID: observationID
+                    +        )
+                    +        try await store.markVerification(
+                    +            sampleType: secondSample.sampleType,
+                    +            verifiedAt: Date(timeIntervalSince1970: 3_060),
+                    +            observationID: observationID
+                    +        )
+                    +        try await store.finishObservation(
+                    +            observationID: observationID,
+                    +            status: "completed",
+                    +            endedAt: Date(timeIntervalSince1970: 3_070)
+                    +        )
+                    +        let observationIDs = try observationIDs(at: url)
+                    +        let summary = try await store.diffSummary(HealthArchiveDiffRequest(
+                    +            fromObservationID: observationIDs[0],
+                    +            toObservationID: observationID,
+                    +            sampleTypeIdentifier: typeIdentifier
+                    +        ))
+                    +        XCTAssertEqual(observationIDs.count, 2)
+                    +        XCTAssertEqual(summary.appearedCount, 1)
+                    +        XCTAssertEqual(summary.disappearedCount, 1)
+                    +        XCTAssertEqual(try countRows(in: "observations WHERE id = \(observationID) AND status = 'completed' AND selected_type_set_hash = 'selected-types'", at: url), 1)
+                    +        XCTAssertEqual(try countRows(in: "sample_observation_events WHERE observation_id = \(observationID)", at: url), 2)
+                    +        XCTAssertEqual(try countRows(in: "observation_type_runs WHERE observation_id = \(observationID) AND inserted_event_count = 1 AND deleted_event_count = 1 AND verified_visible_count = 1 AND status = 'completed'", at: url), 1)
+                    +    }
+                         func testLargeSyntheticDiffReturnsCountsAndBoundedPages() async throws {
+                             let url = databaseURL()
+                             let store = SQLiteHealthArchiveStore(databaseURL: url)

	@@ -186,12 +186,14 @@ final class TypeDistributionBin {
186	186	// SQLite archive store, in one schema that can preserve relationships across data
187	187	// types, sources, devices, workouts, and metadata.
188	188
189		-// Interface updated 2026-05-18 — see AGENTS.md
	189	+// Interface updated 2026-05-24 — see AGENTS.md
190	190	// Services/Protocols/HealthArchiveStore.swift defines the local archive boundary.
191		-// SQLiteHealthArchiveStore is the current implementation. HealthKit anchored-query
192		-// pages must be written to this archive before UI/cache rows are saved.
193		-// Deletions are recorded by sampleUUIDHash because HKDeletedObject exposes UUIDs,
194		-// not complete sample payloads.
	191	+// SQLiteHealthArchiveStore is the current implementation. Each user-visible
	192	+// capture creates one archive observation; HealthKit anchored-query pages,
	193	+// verification runs, and deleted-object evidence for that capture must attach to
	194	+// that observation id before UI/cache rows are saved. Deletions are recorded by
	195	+// sampleUUIDHash because HKDeletedObject exposes UUIDs, not complete sample
	196	+// payloads.
195	197
196	198	// Interface updated 2026-05-24 — see AGENTS.md
197	199	// HealthArchiveStore exposes SQL-first observation diff APIs:

	@@ -91,6 +91,12 @@ An observation records:
91	91	- HealthKit anchors;
92	92	- events and aggregate changes observed during the capture.
93	93
	94	+Implementation note, 2026-05-24: the app creates one archive `observations`
	95	+row for each user-visible capture/snapshot. Anchored HealthKit pages,
	96	+`HKDeletedObject` evidence, and the final type verification rows for that
	97	+capture attach to that same observation id so SQL diffs, cache rows, and
	98	+transition UI can reference a stable capture boundary.
	99	+
94	100	### Terminology
95	101
96	102	- Capture: the act of querying HealthKit and writing results to the archive.
	@@ -538,10 +544,10 @@ Manifest hashes must cover exported content through item hashes, not only counts
538	544
539	545	### 7.1 Capture Transaction Shape
540	546
541		-For each observation/type run:
	547	+For each user-visible capture:
542	548
543		-1. Open SQLite transaction.
544		-2. Insert/update `observations` and `observation_type_runs`.
	549	+1. Create one `observations` row with status `running`.
	550	+2. For each observation/type run, update `observation_type_runs`.
545	551	3. For each added/visible sample page:
546	552	- upsert source/source revision/device/metadata;
547	553	- upsert `samples`;
	@@ -552,9 +558,14 @@ For each observation/type run:
552	558	- find sample by UUID hash;
553	559	- insert deleted/disappeared event;
554	560	- close open visibility ranges.
555		-5. Recompute affected materialized aggregates.
556		-6. Commit SQLite.
557		-7. Update/rebuild Core Data cache after SQLite commit.
	561	+5. Record final per-type verification counts for the same observation id.
	562	+6. Recompute affected materialized aggregates.
	563	+7. Mark the observation `completed`, `needs_review`, `partial_*`, or `failed`.
	564	+
	565	+SQLite writes may be committed per page/type to keep memory and lock time
	566	+bounded, but they must share the capture's observation id.
	567	+
	568	+8. Update/rebuild Core Data cache after SQLite commit.
558	569
559	570	SQLite commit must happen before Core Data cache update. Cache rebuild failures must not corrupt archive truth.
560	571

	@@ -62,9 +62,9 @@ Use:
62	62
63	63	Capture flow:
64	64	1. Resolve the current local device chain ID.
65		-2. Start an observation record.
	65	+2. Start one archive observation record for the user-visible capture and keep its id.
66	66	3. For each selected sample type, run anchored queries.
67		-4. Write HealthKit samples and deleted-object evidence to the local archive first.
	67	+4. Write HealthKit samples, deleted-object evidence, and final per-type verification to the local archive first, all under that same observation id.
68	68	5. Update materialized aggregate tables in SQLite.
69	69	6. Save/rebuild derived Core Data cache rows only after archive writes succeed.
70	70	7. Compute summary/diff caches for UI and reports.

	@@ -24,8 +24,8 @@ There are no real deployments, only test installations. Existing prototype datab
24	24	\| Area \| Current Status \| Target / Next Work \|
25	25	\|------\|----------------\|--------------------\|
26	26	\| Product docs \| Updated \| Keep `HealthProbe/Doc/README.md` as canonical index \|
27		-\| HealthKit capture \| Prototype exists \| Adapt capture to write differential SQLite observations first \|
28		-\| SQLite archive \| Archive v2 schema, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed \| Start Core Data cache work \|
	27	+\| HealthKit capture \| Capture now opens one archive observation per user-visible snapshot and attaches HealthKit pages, deleted-object evidence, and type verification to that observation id before finishing it \| Continue moving UI/cache reads to archive-backed observation ids \|
	28	+\| SQLite archive \| Archive v2 schema, snapshot-level observation grouping, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed \| Move Snapshots/Data Types from SwiftData previews to archive/cache DTOs \|
29	29	\| Core Data cache \| Initial programmatic Core Data model, full-cache rebuild service, read DTOs for observation/type/health rows, and Dashboard archive-cache status wiring are in place \| Move Snapshots/Data Types to cache DTOs and add targeted partial invalidation \|
30	30	\| SwiftData cache \| Exists; test builds now reset legacy prototype UI/archive/cache stores once for archive v2 so old SwiftData-only snapshots are not treated as backed-up observations \| Treat as disposable prototype data; reset/ignore during v2 transition \|
31	31	\| UI \| Prototype exists; Snapshots/Data Types now default to the local device timeline instead of a multi-device picker, and record-change detail uses a separate preview/paged list view with archive-value enrichment and scoped export action \| Reframe remaining screens around observations, diffs, export, archive status \|
	@@ -53,12 +53,12 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m
53	53	- Legacy SwiftData-only snapshots can show diffs without archive-backed values; they are now reset for archive v2 test installs rather than migrated.
54	54	- Existing implementation may decode or cache too much data for low-end devices.
55	55	- Old prototype database compatibility is no longer required.
56		-- Initial SQLite archive tests cover open/init/reset/idempotency, legacy mirror removal, small observation diffs, large synthetic diff pagination, formal timing/memory metrics, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet export behavior.
	56	+- Initial SQLite archive tests cover open/init/reset/idempotency, snapshot-level observation grouping, legacy mirror removal, small observation diffs, large synthetic diff pagination, formal timing/memory metrics, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet export behavior.
57	57	- Initial Core Data cache tests cover full rebuild from SQLite and delete-cache-then-rebuild without losing archive data.
58	58
59	59	## Verification Checklist
60	60
61		-- [ ] SQLite archive v2 can reconstruct records visible at observation T.
	61	+- [x] SQLite archive v2 can reconstruct records visible at observation T.
62	62	- [ ] No recurring complete snapshot copies are written for high-volume types.
63	63	- [x] SQL diff between two observations runs without loading full datasets into Swift arrays.
64	64	- [x] Expensive counts used by reports/UI are cached and rebuildable.

	@@ -1,6 +1,6 @@
1	1	# HealthProbe - Database-Led Refactoring Plan
2	2
3		-Last Updated: 2026-05-23
	3	+Last Updated: 2026-05-24
4	4	Status: Active planning document
5	5
6	6	## Goal
	@@ -119,6 +119,8 @@ Acceptance:
119	119
120	120	Checklist:
121	121	- [x] Create observation transaction wrapper.
	122	+- [x] Attach HealthKit pages, deleted-object evidence, and final type verification to the same user-visible archive observation.
	123	+- [x] Store the archive observation id on the legacy `HealthSnapshot` bridge model for transition screens.
122	124	- [x] Upsert sample types.
123	125	- [x] Upsert source/source revision/device/metadata rows.
124	126	- [x] Upsert sample identity.
	@@ -140,6 +142,7 @@ Acceptance:
140	142	- [x] Representation change creates a new version, not a new logical sample.
141	143	- [x] Disappearance closes visibility range.
142	144	- [x] No full observation copy table is created or written.
	145	+- [x] A user-visible capture has one archive observation id that SQL diff/cache/UI layers can reference.
143	146
144	147	## Milestone 5 - SQL Analysis Layer
145	148

	@@ -24,6 +24,7 @@ import SwiftData
24	24	var appBuildVersion: String = ""
25	25	var monitoredTypeSetHash: String = ""
26	26	var monitoredRegistryVersion: Int = 0
	27	+ var archiveObservationID: Int64?
27	28	var yearlyCountTimezoneIdentifier: String = ""
28	29	var cachedSummaryVersion: Int = 0
29	30	var cachedTypeCount: Int = 0

	@@ -96,6 +96,15 @@ final class HealthKitService {
96	96	deviceName: UIDevice.current.name,
97	97	deviceID: deviceResolution.id
98	98	)
	99	+ let intendedTypeIDs = active.map { $0.id }
	100	+ let monitoredTypeSetHash = HashService.typeSetHash(typeIDs: intendedTypeIDs)
	101	+ let archiveObservationID = try await archiveStore.beginObservation(
	102	+ observedAt: snapshot.timestamp,
	103	+ triggerReason: triggerReason,
	104	+ selectedTypeSetHash: monitoredTypeSetHash
	105	+ )
	106	+ snapshot.archiveObservationID = archiveObservationID
	107	+ snapshot.monitoredTypeSetHash = monitoredTypeSetHash
99	108	snapshot.recoveredDeviceID = deviceResolution.isRecovered
100	109	snapshot.triggerReason = triggerReason
101	110	snapshot.retryOfSnapshotID = retryOfSnapshotID
	@@ -108,6 +117,7 @@ final class HealthKitService {
108	117	context: context,
109	118	snapshot: snapshot,
110	119	previousSnapshot: previousSnapshot,
	120	+ archiveObservationID: archiveObservationID,
111	121	adaptiveTimeoutsEnabled: adaptiveTimeoutsEnabled,
112	122	timeoutMultiplier: timeoutMultiplier,
113	123	progress: progress
	@@ -143,7 +153,7 @@ final class HealthKitService {
143	153	configureSnapshotMetadata(
144	154	snapshot,
145	155	typeCounts: typeCounts,
146		- intendedTypeIDs: active.map { $0.id },
	156	+ intendedTypeIDs: intendedTypeIDs,
147	157	context: context
148	158	)
149	159	markContentEquivalenceIfNeeded(
	@@ -160,11 +170,30 @@ final class HealthKitService {
160	170	if snapshot.snapshotQuality == .complete,
161	171	reviewAmbiguousCompleteDisappearedTypes,
162	172	hasAmbiguousCompleteDisappearance(snapshot: snapshot, typeCounts: typeCounts, context: context) {
	173	+ try await archiveStore.finishObservation(
	174	+ observationID: archiveObservationID,
	175	+ status: "needs_review",
	176	+ endedAt: Date()
	177	+ )
163	178	return snapshot
164	179	}
165	180
166		- if snapshot.snapshotQuality == .complete {
167		- try await persistSnapshot(snapshot, typeCounts: typeCounts, context: context)
	181	+ do {
	182	+ if snapshot.snapshotQuality == .complete {
	183	+ try await persistSnapshot(snapshot, typeCounts: typeCounts, context: context)
	184	+ }
	185	+ try await archiveStore.finishObservation(
	186	+ observationID: archiveObservationID,
	187	+ status: snapshot.snapshotQuality == .complete ? "completed" : "partial_\(snapshot.snapshotQuality.rawValue)",
	188	+ endedAt: Date()
	189	+ )
	190	+ } catch {
	191	+ try? await archiveStore.finishObservation(
	192	+ observationID: archiveObservationID,
	193	+ status: "failed",
	194	+ endedAt: Date()
	195	+ )
	196	+ throw error
168	197	}
169	198
170	199	return snapshot
	@@ -573,6 +602,7 @@ final class HealthKitService {
573	602	context: ModelContext,
574	603	snapshot: HealthSnapshot,
575	604	previousSnapshot: HealthSnapshot?,
	605	+ archiveObservationID: Int64,
576	606	adaptiveTimeoutsEnabled: Bool,
577	607	timeoutMultiplier: Double,
578	608	progress: SnapshotFetchProgress? = nil
	@@ -592,6 +622,7 @@ final class HealthKitService {
592	622	timeoutProfile: profile,
593	623	timeoutSeconds: timeout,
594	624	previousTypeCount: previousSnapshot?.typeCounts?.first { $0.typeIdentifier == monitoredType.id },
	625	+ archiveObservationID: archiveObservationID,
595	626	progress: progress
596	627	)
597	628	updateTimeoutProfile(profile, with: result, monitoredType: monitoredType)
	@@ -609,6 +640,7 @@ final class HealthKitService {
609	640	timeoutProfile: MetricTimeoutProfile,
610	641	timeoutSeconds: TimeInterval,
611	642	previousTypeCount: TypeCount?,
	643	+ archiveObservationID: Int64,
612	644	progress: SnapshotFetchProgress? = nil
613	645	) async -> TypeCountFetchResult {
614	646	let started = Date()
	@@ -647,6 +679,7 @@ final class HealthKitService {
647	679	sampleType: sampleType,
648	680	timeoutSeconds: timeoutSeconds,
649	681	previousTypeCount: previousTypeCount,
	682	+ archiveObservationID: archiveObservationID,
650	683	progress: progress
651	684	)
652	685	result.totalElapsedSeconds = Date().timeIntervalSince(started)
	@@ -672,6 +705,7 @@ final class HealthKitService {
672	705	sampleType: HKSampleType,
673	706	timeoutSeconds: TimeInterval,
674	707	previousTypeCount: TypeCount?,
	708	+ archiveObservationID: Int64,
675	709	progress: SnapshotFetchProgress?
676	710	) async -> TypeCountFetchResult {
677	711	let dateDeadline = Date().addingTimeInterval(timeoutSeconds)
	@@ -730,6 +764,7 @@ final class HealthKitService {
730	764	earliestDate: earliest,
731	765	latestDate: latest,
732	766	previousDistribution: previousDistribution,
	767	+ archiveObservationID: archiveObservationID,
733	768	progress: progress
734	769	)
735	770	} resultDescription: { distribution in
	@@ -826,6 +861,7 @@ final class HealthKitService {
826	861	earliestDate: Date?,
827	862	latestDate: Date?,
828	863	previousDistribution: PreviousDistributionState,
	864	+ archiveObservationID: Int64,
829	865	progress: SnapshotFetchProgress?
830	866	) async throws -> SampleDistribution {
831	867	var anchor = previousDistribution.globalAnchor
	@@ -859,7 +895,7 @@ final class HealthKitService {
859	895	anchor: anchor
860	896	)
861	897	}
862		- try await archivePage(page, sampleType: sampleType)
	898	+ try await archivePage(page, sampleType: sampleType, observationID: archiveObservationID)
863	899	anchor = page.anchor
864	900
865	901	if page.samples.isEmpty, page.deletedObjects.isEmpty,
	@@ -869,7 +905,11 @@ final class HealthKitService {
869	905	earliestDate: earliestDate,
870	906	latestDate: latestDate
871	907	) {
872		- try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date())
	908	+ try await archiveStore.markVerification(
	909	+ sampleType: sampleType,
	910	+ verifiedAt: Date(),
	911	+ observationID: archiveObservationID
	912	+ )
873	913	progress?.updateBlockProgress(
874	914	typeIdentifier,
875	915	detail: "No HealthKit delta",
	@@ -890,6 +930,7 @@ final class HealthKitService {
890	930	earliestDate: earliestDate,
891	931	latestDate: latestDate,
892	932	progressStarted: progressStarted,
	933	+ archiveObservationID: archiveObservationID,
893	934	progress: progress
894	935	)
895	936	}
	@@ -941,7 +982,7 @@ final class HealthKitService {
941	982	anchor: anchor
942	983	)
943	984	}
944		- try await archivePage(page, sampleType: sampleType)
	985	+ try await archivePage(page, sampleType: sampleType, observationID: archiveObservationID)
945	986	anchor = page.anchor
946	987
947	988	applyDistributionPage(page, sampleType: sampleType, to: &recordMap)
	@@ -963,7 +1004,7 @@ final class HealthKitService {
963	1004	)
964	1005	}
965	1006
966		- try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date())
	1007	+ try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date(), observationID: archiveObservationID)
967	1008
968	1009	let sortedKeys = recordMap.keys.sorted {
969	1010	guard let left = recordMap[$0],
	@@ -1040,6 +1081,7 @@ final class HealthKitService {
1040	1081	earliestDate: Date?,
1041	1082	latestDate: Date?,
1042	1083	progressStarted: Date,
	1084	+ archiveObservationID: Int64,
1043	1085	progress: SnapshotFetchProgress?
1044	1086	) async throws -> SampleDistribution {
1045	1087	var anchor: HKQueryAnchor?
	@@ -1078,7 +1120,7 @@ final class HealthKitService {
1078	1120	anchor: anchor
1079	1121	)
1080	1122	}
1081		- try await archivePage(page, sampleType: sampleType)
	1123	+ try await archivePage(page, sampleType: sampleType, observationID: archiveObservationID)
1082	1124	anchor = page.anchor
1083	1125
1084	1126	for sample in page.samples {
	@@ -1122,7 +1164,7 @@ final class HealthKitService {
1122	1164	)
1123	1165	)
1124	1166
1125		- try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date())
	1167	+ try await archiveStore.markVerification(sampleType: sampleType, verifiedAt: Date(), observationID: archiveObservationID)
1126	1168
1127	1169	guard recordCount > 0 \|\| anchor != nil else {
1128	1170	return SampleDistribution(
	@@ -1266,14 +1308,15 @@ final class HealthKitService {
1266	1308	}
1267	1309	}
1268	1310
1269		- private func archivePage(_ page: SampleDistributionPage, sampleType: HKSampleType) async throws {
	1311	+ private func archivePage(_ page: SampleDistributionPage, sampleType: HKSampleType, observationID: Int64) async throws {
1270	1312	let observedAt = Date()
1271		- _ = try await archiveStore.upsertSamples(page.samples, observedAt: observedAt)
	1313	+ _ = try await archiveStore.upsertSamples(page.samples, observedAt: observedAt, observationID: observationID)
1272	1314	for deletedObject in page.deletedObjects {
1273	1315	try await archiveStore.recordDisappearance(
1274	1316	sampleUUIDHash: HashService.sampleUUIDHash(deletedObject.uuid.uuidString),
1275	1317	sampleTypeIdentifier: sampleType.identifier,
1276		- observedMissingAt: observedAt
	1318	+ observedMissingAt: observedAt,
	1319	+ observationID: observationID
1277	1320	)
1278	1321	}
1279	1322	}

	@@ -1,11 +1,16 @@
1	1	import Foundation
2	2	import HealthKit
3	3
4		-// Interface updated 2026-05-23 — see AGENTS.md
	4	+// Interface updated 2026-05-24 — see AGENTS.md
5	5	protocol HealthArchiveStore {
	6	+ func beginObservation(observedAt: Date, triggerReason: String, selectedTypeSetHash: String?) async throws -> Int64
	7	+ func finishObservation(observationID: Int64, status: String, endedAt: Date) async throws
6	8	func upsertSamples(_ samples: [HKSample], observedAt: Date) async throws -> HealthArchiveWriteSummary
	9	+ func upsertSamples(_ samples: [HKSample], observedAt: Date, observationID: Int64) async throws -> HealthArchiveWriteSummary
7	10	func markVerification(sampleType: HKSampleType, verifiedAt: Date) async throws
	11	+ func markVerification(sampleType: HKSampleType, verifiedAt: Date, observationID: Int64) async throws
8	12	func recordDisappearance(sampleUUIDHash: String, sampleTypeIdentifier: String, observedMissingAt: Date) async throws
	13	+ func recordDisappearance(sampleUUIDHash: String, sampleTypeIdentifier: String, observedMissingAt: Date, observationID: Int64) async throws
9	14	func records(for request: HealthArchiveRecordRequest) async throws -> [ArchivedHealthRecord]
10	15	func diffSummary(_ request: HealthArchiveDiffRequest) async throws -> HealthArchiveDiffSummary
11	16	func diffRecords(_ request: HealthArchiveDiffRecordRequest) async throws -> [ArchivedHealthRecord]