Showing 6 changed files with 723 additions and 23 deletions
+8 -1
HealthProbe/Doc/00-agent-guides/AGENTS.md
@@ -210,6 +210,13 @@ final class TypeDistributionBin {
210 210
 // and materialized aggregates. Expensive counts used by reports/UI should be
211 211
 // cached in Core Data and be rebuildable from SQLite.
212 212
 
213
+// Interface updated 2026-05-24 — see AGENTS.md
214
+// Services/CoreDataArchiveCacheStore.swift defines the rebuildable Core Data
215
+// UI/report cache boundary. It owns programmatic cache entities for observation
216
+// rows, type summaries, daily aggregates, diff summaries, export manifests, and
217
+// archive health. Cache rows carry source observation ids plus archive/cache
218
+// schema versions; deleting this cache must never delete the SQLite archive.
219
+
213 220
 // Objective updated 2026-05-23 — see AGENTS.md
214 221
 // HealthProbe is a local Health DB Time Machine. Snapshot/device identifiers are
215 222
 // retained only to preserve local provenance and keep comparisons within one
@@ -338,7 +345,7 @@ When one agent needs to communicate a decision or change to another:
338 345
 
339 346
 | Module | Status | Owner |
340 347
 |--------|--------|-------|
341
-| Core Data UI/Report Cache | ⏳ Planned replacement of SwiftData | Models agent |
348
+| Core Data UI/Report Cache | ⏳ Started: programmatic model + full rebuild service | Models agent |
342 349
 | HealthKit Integration | ✅ Done | Services agent |
343 350
 | Snapshot Diff Service | ✅ Done | Services agent |
344 351
 | Service Protocols | ⏳ Not started | Services agent |
+3 -2
HealthProbe/Doc/02-architecture/Core-Data-Cache-Design.md
@@ -1,7 +1,7 @@
1 1
 # HealthProbe - Core Data Cache Design
2 2
 
3 3
 **Last Updated:** 2026-05-23
4
-**Status:** Target design for UI/report cache
4
+**Status:** Initial programmatic model and full-cache rebuild implemented; UI wiring and targeted invalidation pending
5 5
 
6 6
 ## 1. Purpose
7 7
 
@@ -172,6 +172,8 @@ Rebuild order:
172 172
 
173 173
 Partial rebuild is allowed when SQLite can identify affected observations/types. Full rebuild must remain available for repair and tests.
174 174
 
175
+Implementation note, 2026-05-24: `CoreDataArchiveCacheStore` defines the cache model programmatically and can rebuild all cache entities from the SQLite archive. The first implementation rebuilds observation rows, type summaries, daily aggregates, adjacent-observation diff summaries, export manifest rows, and archive health rows. It is intentionally rebuildable: deleting the Core Data cache must not touch the SQLite archive.
176
+
175 177
 ## 5. Legacy Device Mode
176 178
 
177 179
 Legacy or low-memory UI should still use the same Core Data cache. It may reduce:
@@ -185,4 +187,3 @@ It must preserve:
185 187
 - cached summaries;
186 188
 - report generation;
187 189
 - paged SQLite detail/export access.
188
-
+9 -7
HealthProbe/Doc/04-project/IMPLEMENTATION_STATUS.md
@@ -26,7 +26,7 @@ There are no real deployments, only test installations. Existing prototype datab
26 26
 | Product docs | Updated | Keep `HealthProbe/Doc/README.md` as canonical index |
27 27
 | HealthKit capture | Prototype exists | Adapt capture to write differential SQLite observations first |
28 28
 | SQLite archive | Archive v2 schema, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed | Start Core Data cache work |
29
-| Core Data cache | Not implemented | Add rebuildable cache for expensive counts, summaries, report metadata, UI state |
29
+| Core Data cache | Initial programmatic Core Data model and full-cache rebuild service are in place for observation rows, type summaries, daily aggregates, diff summaries, export manifest rows, and archive health | Wire cache reads into UI-facing view models and add targeted partial invalidation |
30 30
 | SwiftData cache | Exists | Treat as disposable prototype data; reset/ignore during v2 transition |
31 31
 | UI | Prototype exists | Reframe screens around observations, diffs, export, archive status |
32 32
 | Diff/change explanation | Prototype/legacy anomaly logic exists | Move heavy diffing into SQLite and use neutral change classifications |
@@ -38,11 +38,12 @@ There are no real deployments, only test installations. Existing prototype datab
38 38
 
39 39
 Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.md).
40 40
 
41
-1. Add Core Data UI/report cache and rebuild pipeline.
41
+1. Wire Core Data cache reads into UI-facing view models.
42 42
 2. Replace SwiftData UI dependencies with Core Data/cache DTOs.
43
-3. Update UI language from anomaly/status to observation/diff/export.
44
-4. Add streaming exports with manifests.
45
-5. Validate on low-memory/legacy-class devices.
43
+3. Add targeted cache invalidation for affected observation/type ranges.
44
+4. Update UI language from anomaly/status to observation/diff/export.
45
+5. Add streaming exports with manifests.
46
+6. Validate on low-memory/legacy-class devices.
46 47
 
47 48
 ## Known Prototype Mismatches
48 49
 
@@ -53,14 +54,15 @@ Detailed checkable milestones live in [`Refactoring-Plan.md`](Refactoring-Plan.m
53 54
 - Existing implementation may decode or cache too much data for low-end devices.
54 55
 - Old prototype database compatibility is no longer required.
55 56
 - Initial SQLite archive tests cover open/init/reset/idempotency, legacy mirror removal, small observation diffs, large synthetic diff pagination, formal timing/memory metrics, materialized aggregate comparison, source/provenance breakdowns, and consolidation-evidence labels, but not yet export behavior.
57
+- Initial Core Data cache tests cover full rebuild from SQLite and delete-cache-then-rebuild without losing archive data.
56 58
 
57 59
 ## Verification Checklist
58 60
 
59 61
 - [ ] SQLite archive v2 can reconstruct records visible at observation T.
60 62
 - [ ] No recurring complete snapshot copies are written for high-volume types.
61 63
 - [x] SQL diff between two observations runs without loading full datasets into Swift arrays.
62
-- [ ] Expensive counts used by reports/UI are cached and rebuildable.
63
-- [ ] Deleting Core Data cache and rebuilding from SQLite restores UI/report summaries.
64
+- [x] Expensive counts used by reports/UI are cached and rebuildable.
65
+- [x] Deleting Core Data cache and rebuilding from SQLite restores UI/report summaries.
64 66
 - [ ] Export can stream large selected record sets.
65 67
 - [ ] Export manifests include hashes and observation metadata.
66 68
 - [ ] iOS app remains read-only with respect to HealthKit.
+15 -13
HealthProbe/Doc/04-project/Refactoring-Plan.md
@@ -169,21 +169,23 @@ Acceptance:
169 169
 **Purpose:** Cache expensive presentation/report values while keeping SQLite authoritative.
170 170
 
171 171
 Checklist:
172
-- [ ] Define Core Data model for observation rows.
173
-- [ ] Define type summary cache entity.
174
-- [ ] Define daily/monthly aggregate cache entity.
175
-- [ ] Define diff summary cache entity.
176
-- [ ] Define export manifest/status cache entity.
177
-- [ ] Define archive health/status cache entity.
178
-- [ ] Implement cache rebuild from SQLite.
179
-- [ ] Implement cache invalidation by archive schema/cache schema/version/hash.
180
-- [ ] Implement delete-cache-and-rebuild flow.
181
-- [ ] Add cache schema/version and rebuild tests.
172
+- [x] Define Core Data model for observation rows.
173
+- [x] Define type summary cache entity.
174
+- [x] Define daily/monthly aggregate cache entity.
175
+- [x] Define diff summary cache entity.
176
+- [x] Define export manifest/status cache entity.
177
+- [x] Define archive health/status cache entity.
178
+- [x] Implement initial cache rebuild from SQLite for observation/type/daily/diff/export/health rows.
179
+- [x] Include archive schema/cache schema/version/hash fields on rebuilt rows.
180
+- [x] Implement delete-cache-and-rebuild flow.
181
+- [x] Add cache schema/version and rebuild tests.
182
+- [ ] Wire Core Data cache into UI-facing view models.
183
+- [ ] Add targeted partial invalidation for affected observation/type ranges.
182 184
 
183 185
 Acceptance:
184
-- [ ] Deleting Core Data cache does not lose forensic data.
185
-- [ ] Cache rebuild restores dashboard/timeline/report summaries.
186
-- [ ] Cache rows include source observation ids and archive/cache schema versions.
186
+- [x] Deleting Core Data cache does not lose forensic data.
187
+- [x] Cache rebuild restores dashboard/timeline/report summaries.
188
+- [x] Cache rows include source observation ids and archive/cache schema versions.
187 189
 - [ ] SQLite wins on disagreement.
188 190
 
189 191
 ## Milestone 7 - Export Layer
+585 -0
HealthProbe/Services/CoreDataArchiveCacheStore.swift
@@ -0,0 +1,585 @@
1
+import CoreData
2
+import Foundation
3
+import SQLite3
4
+
5
+enum CoreDataArchiveCacheStoreError: Error {
6
+    case persistentStoreLoadFailed(Error)
7
+    case openArchiveFailed(String)
8
+    case prepareFailed(String)
9
+    case stepFailed(String)
10
+}
11
+
12
+struct CoreDataArchiveCacheRebuildSummary: Equatable, Sendable {
13
+    let observationRows: Int
14
+    let typeSummaryRows: Int
15
+    let dailyAggregateRows: Int
16
+    let diffSummaryRows: Int
17
+    let exportManifestRows: Int
18
+    let archiveHealthRows: Int
19
+}
20
+
21
+// Interface updated 2026-05-24 — see AGENTS.md
22
+final class CoreDataArchiveCacheStore {
23
+    static let cacheSchemaVersion = 1
24
+
25
+    let container: NSPersistentContainer
26
+
27
+    init(storeURL: URL? = nil, inMemory: Bool = false) throws {
28
+        let model = Self.makeModel()
29
+        container = NSPersistentContainer(name: "HealthProbeCache", managedObjectModel: model)
30
+
31
+        let description = NSPersistentStoreDescription()
32
+        if inMemory {
33
+            description.type = NSInMemoryStoreType
34
+        } else {
35
+            let resolvedURL = storeURL ?? URL.applicationSupportDirectory.appending(path: "HealthProbeCache.sqlite")
36
+            description.url = resolvedURL
37
+        }
38
+        description.shouldMigrateStoreAutomatically = true
39
+        description.shouldInferMappingModelAutomatically = true
40
+        container.persistentStoreDescriptions = [description]
41
+
42
+        var loadError: Error?
43
+        container.loadPersistentStores { _, error in
44
+            loadError = error
45
+        }
46
+        if let loadError {
47
+            throw CoreDataArchiveCacheStoreError.persistentStoreLoadFailed(loadError)
48
+        }
49
+        container.viewContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
50
+    }
51
+
52
+    func rebuild(fromArchiveAt archiveURL: URL) throws -> CoreDataArchiveCacheRebuildSummary {
53
+        let archive = try openArchive(at: archiveURL)
54
+        defer { sqlite3_close(archive) }
55
+
56
+        let archiveSchemaVersion = try archiveSchemaVersion(db: archive)
57
+        let integrityStatus = try firstText("PRAGMA integrity_check", db: archive) ?? "missing"
58
+        let computedAt = Date()
59
+        let context = container.viewContext
60
+
61
+        try resetCache(context: context)
62
+
63
+        let observationCount = try insertObservationRows(db: archive, context: context, archiveSchemaVersion: archiveSchemaVersion, computedAt: computedAt)
64
+        let typeSummaryCount = try insertTypeSummaryRows(db: archive, context: context, computedAt: computedAt)
65
+        let dailyAggregateCount = try insertDailyAggregateRows(db: archive, context: context, computedAt: computedAt)
66
+        let diffSummaryCount = try insertDiffSummaryRows(db: archive, context: context, computedAt: computedAt)
67
+        let exportManifestCount = try insertExportManifestRows(db: archive, context: context, computedAt: computedAt)
68
+        try insertArchiveHealthRow(
69
+            context: context,
70
+            archiveSchemaVersion: archiveSchemaVersion,
71
+            integrityStatus: integrityStatus,
72
+            computedAt: computedAt
73
+        )
74
+
75
+        if context.hasChanges {
76
+            try context.save()
77
+        }
78
+
79
+        return CoreDataArchiveCacheRebuildSummary(
80
+            observationRows: observationCount,
81
+            typeSummaryRows: typeSummaryCount,
82
+            dailyAggregateRows: dailyAggregateCount,
83
+            diffSummaryRows: diffSummaryCount,
84
+            exportManifestRows: exportManifestCount,
85
+            archiveHealthRows: 1
86
+        )
87
+    }
88
+
89
+    func deleteCache() throws {
90
+        let context = container.viewContext
91
+        try resetCache(context: context)
92
+        if context.hasChanges {
93
+            try context.save()
94
+        }
95
+    }
96
+
97
+    private func resetCache(context: NSManagedObjectContext) throws {
98
+        for entityName in Self.cacheEntityNames {
99
+            let request = NSFetchRequest<NSManagedObject>(entityName: entityName)
100
+            let objects = try context.fetch(request)
101
+            for object in objects {
102
+                context.delete(object)
103
+            }
104
+        }
105
+    }
106
+
107
+    private func insertObservationRows(
108
+        db: OpaquePointer?,
109
+        context: NSManagedObjectContext,
110
+        archiveSchemaVersion: Int,
111
+        computedAt: Date
112
+    ) throws -> Int {
113
+        let sql = """
114
+        SELECT
115
+            o.id,
116
+            o.observed_at,
117
+            o.status,
118
+            o.trigger_reason,
119
+            o.time_zone_identifier,
120
+            COUNT(s.sample_type_id),
121
+            COALESCE(SUM(s.visible_record_count), 0),
122
+            COALESCE(SUM(s.appeared_count), 0),
123
+            COALESCE(SUM(s.disappeared_count), 0),
124
+            COALESCE(SUM(s.representation_changed_count), 0),
125
+            COALESCE(GROUP_CONCAT(s.aggregate_hash, '|'), '')
126
+        FROM observations o
127
+        LEFT JOIN observation_type_summaries s ON s.observation_id = o.id
128
+        GROUP BY o.id, o.observed_at, o.status, o.trigger_reason, o.time_zone_identifier
129
+        ORDER BY o.id
130
+        """
131
+        return try withStatement(sql, db: db) { statement in
132
+            var count = 0
133
+            while sqlite3_step(statement) == SQLITE_ROW {
134
+                let observationID = sqlite3_column_int64(statement, 0)
135
+                let sourceHash = HashService.archiveContentHash(
136
+                    domain: "hp:cache:observation_row",
137
+                    parts: [
138
+                        String(observationID),
139
+                        columnText(statement, 10)
140
+                    ]
141
+                )
142
+                let object = NSEntityDescription.insertNewObject(forEntityName: "CachedObservationRow", into: context)
143
+                object.setValue(observationID, forKey: "observationID")
144
+                object.setValue(columnUnixDate(statement, 1), forKey: "observedAt")
145
+                object.setValue(columnText(statement, 2) ?? "unknown", forKey: "status")
146
+                object.setValue(columnText(statement, 3) ?? "unknown", forKey: "triggerReason")
147
+                object.setValue(columnText(statement, 4), forKey: "timeZoneIdentifier")
148
+                object.setValue(Int64(sqlite3_column_int64(statement, 5)), forKey: "trackedTypeCount")
149
+                object.setValue(Int64(sqlite3_column_int64(statement, 6)), forKey: "visibleRecordCount")
150
+                object.setValue(Int64(sqlite3_column_int64(statement, 7)), forKey: "appearedCount")
151
+                object.setValue(Int64(sqlite3_column_int64(statement, 8)), forKey: "disappearedCount")
152
+                object.setValue(Int64(sqlite3_column_int64(statement, 9)), forKey: "representationChangedCount")
153
+                object.setValue(Int64(archiveSchemaVersion), forKey: "archiveSchemaVersion")
154
+                object.setValue(Int64(Self.cacheSchemaVersion), forKey: "cacheSchemaVersion")
155
+                object.setValue(sourceHash, forKey: "sourceAggregateHash")
156
+                object.setValue(computedAt, forKey: "computedAt")
157
+                count += 1
158
+            }
159
+            return count
160
+        }
161
+    }
162
+
163
+    private func insertTypeSummaryRows(
164
+        db: OpaquePointer?,
165
+        context: NSManagedObjectContext,
166
+        computedAt: Date
167
+    ) throws -> Int {
168
+        let sql = """
169
+        SELECT
170
+            s.observation_id,
171
+            t.type_identifier,
172
+            t.display_name,
173
+            s.visible_record_count,
174
+            s.appeared_count,
175
+            s.disappeared_count,
176
+            s.representation_changed_count,
177
+            s.earliest_start_date,
178
+            s.latest_end_date,
179
+            s.value_sum,
180
+            s.value_max,
181
+            s.aggregate_hash
182
+        FROM observation_type_summaries s
183
+        JOIN sample_types t ON t.id = s.sample_type_id
184
+        ORDER BY s.observation_id, t.type_identifier
185
+        """
186
+        return try withStatement(sql, db: db) { statement in
187
+            var count = 0
188
+            while sqlite3_step(statement) == SQLITE_ROW {
189
+                let object = NSEntityDescription.insertNewObject(forEntityName: "CachedTypeSummary", into: context)
190
+                object.setValue(sqlite3_column_int64(statement, 0), forKey: "observationID")
191
+                object.setValue(columnText(statement, 1) ?? "", forKey: "sampleTypeIdentifier")
192
+                object.setValue(columnText(statement, 2), forKey: "displayName")
193
+                object.setValue(Int64(sqlite3_column_int64(statement, 3)), forKey: "visibleRecordCount")
194
+                object.setValue(Int64(sqlite3_column_int64(statement, 4)), forKey: "appearedCount")
195
+                object.setValue(Int64(sqlite3_column_int64(statement, 5)), forKey: "disappearedCount")
196
+                object.setValue(Int64(sqlite3_column_int64(statement, 6)), forKey: "representationChangedCount")
197
+                object.setValue(columnUnixDate(statement, 7), forKey: "earliestStartDate")
198
+                object.setValue(columnUnixDate(statement, 8), forKey: "latestEndDate")
199
+                object.setValue(columnDouble(statement, 9), forKey: "valueSum")
200
+                object.setValue(columnDouble(statement, 10), forKey: "valueMax")
201
+                object.setValue(columnText(statement, 11), forKey: "aggregateHash")
202
+                object.setValue(computedAt, forKey: "computedAt")
203
+                count += 1
204
+            }
205
+            return count
206
+        }
207
+    }
208
+
209
+    private func insertDailyAggregateRows(
210
+        db: OpaquePointer?,
211
+        context: NSManagedObjectContext,
212
+        computedAt: Date
213
+    ) throws -> Int {
214
+        let sql = """
215
+        SELECT
216
+            d.observation_id,
217
+            t.type_identifier,
218
+            d.bucket_start,
219
+            d.bucket_end,
220
+            o.time_zone_identifier,
221
+            d.visible_record_count,
222
+            d.value_sum,
223
+            d.value_max,
224
+            sr.id,
225
+            d.aggregate_hash
226
+        FROM daily_type_aggregates d
227
+        JOIN sample_types t ON t.id = d.sample_type_id
228
+        JOIN observations o ON o.id = d.observation_id
229
+        LEFT JOIN source_revisions sr ON sr.id = d.source_revision_id
230
+        ORDER BY d.observation_id, t.type_identifier, d.bucket_start, sr.id
231
+        """
232
+        return try withStatement(sql, db: db) { statement in
233
+            var count = 0
234
+            while sqlite3_step(statement) == SQLITE_ROW {
235
+                let sourceRevisionHash = columnInt64(statement, 8).map {
236
+                    HashService.archiveContentHash(domain: "hp:cache:source_revision", parts: [String($0)])
237
+                }
238
+                let object = NSEntityDescription.insertNewObject(forEntityName: "CachedDailyAggregate", into: context)
239
+                object.setValue(sqlite3_column_int64(statement, 0), forKey: "observationID")
240
+                object.setValue(columnText(statement, 1) ?? "", forKey: "sampleTypeIdentifier")
241
+                object.setValue(columnUnixDate(statement, 2), forKey: "bucketStart")
242
+                object.setValue(columnUnixDate(statement, 3), forKey: "bucketEnd")
243
+                object.setValue(columnText(statement, 4), forKey: "timeZoneIdentifier")
244
+                object.setValue(Int64(sqlite3_column_int64(statement, 5)), forKey: "visibleRecordCount")
245
+                object.setValue(columnDouble(statement, 6), forKey: "valueSum")
246
+                object.setValue(columnDouble(statement, 7), forKey: "valueMax")
247
+                object.setValue(sourceRevisionHash, forKey: "sourceRevisionDisplayHash")
248
+                object.setValue(columnText(statement, 9), forKey: "aggregateHash")
249
+                object.setValue(computedAt, forKey: "computedAt")
250
+                count += 1
251
+            }
252
+            return count
253
+        }
254
+    }
255
+
256
+    private func insertDiffSummaryRows(
257
+        db: OpaquePointer?,
258
+        context: NSManagedObjectContext,
259
+        computedAt: Date
260
+    ) throws -> Int {
261
+        let sql = """
262
+        WITH pairs AS (
263
+            SELECT
264
+                o.id AS to_id,
265
+                (SELECT MAX(p.id) FROM observations p WHERE p.id < o.id) AS from_id
266
+            FROM observations o
267
+        )
268
+        SELECT
269
+            pairs.from_id,
270
+            pairs.to_id,
271
+            t.type_identifier,
272
+            SUM(CASE WHEN e.event_kind = 'appeared' THEN 1 ELSE 0 END),
273
+            SUM(CASE WHEN e.event_kind = 'disappeared' THEN 1 ELSE 0 END),
274
+            SUM(CASE WHEN e.event_kind = 'representationChanged' THEN 1 ELSE 0 END)
275
+        FROM pairs
276
+        JOIN sample_observation_events e ON e.observation_id = pairs.to_id
277
+        JOIN samples s ON s.id = e.sample_id
278
+        JOIN sample_types t ON t.id = s.sample_type_id
279
+        WHERE pairs.from_id IS NOT NULL
280
+        GROUP BY pairs.from_id, pairs.to_id, t.type_identifier
281
+        ORDER BY pairs.from_id, pairs.to_id, t.type_identifier
282
+        """
283
+        return try withStatement(sql, db: db) { statement in
284
+            var count = 0
285
+            while sqlite3_step(statement) == SQLITE_ROW {
286
+                let fromObservationID = sqlite3_column_int64(statement, 0)
287
+                let toObservationID = sqlite3_column_int64(statement, 1)
288
+                let sampleTypeIdentifier = columnText(statement, 2) ?? ""
289
+                let appearedCount = Int64(sqlite3_column_int64(statement, 3))
290
+                let disappearedCount = Int64(sqlite3_column_int64(statement, 4))
291
+                let representationChangedCount = Int64(sqlite3_column_int64(statement, 5))
292
+                let sourceHash = HashService.archiveContentHash(
293
+                    domain: "hp:cache:diff_summary",
294
+                    parts: [
295
+                        String(fromObservationID),
296
+                        String(toObservationID),
297
+                        sampleTypeIdentifier,
298
+                        String(appearedCount),
299
+                        String(disappearedCount),
300
+                        String(representationChangedCount)
301
+                    ]
302
+                )
303
+                let object = NSEntityDescription.insertNewObject(forEntityName: "CachedDiffSummary", into: context)
304
+                object.setValue(fromObservationID, forKey: "fromObservationID")
305
+                object.setValue(toObservationID, forKey: "toObservationID")
306
+                object.setValue(sampleTypeIdentifier, forKey: "sampleTypeIdentifier")
307
+                object.setValue(appearedCount, forKey: "appearedCount")
308
+                object.setValue(disappearedCount, forKey: "disappearedCount")
309
+                object.setValue(representationChangedCount, forKey: "representationChangedCount")
310
+                object.setValue(false, forKey: "consolidationLikely")
311
+                object.setValue(nil, forKey: "uncertaintyReason")
312
+                object.setValue(sourceHash, forKey: "sourceAggregateHash")
313
+                object.setValue(computedAt, forKey: "computedAt")
314
+                count += 1
315
+            }
316
+            return count
317
+        }
318
+    }
319
+
320
+    private func insertExportManifestRows(
321
+        db: OpaquePointer?,
322
+        context: NSManagedObjectContext,
323
+        computedAt: Date
324
+    ) throws -> Int {
325
+        let sql = """
326
+        SELECT
327
+            export_id,
328
+            export_kind,
329
+            created_at,
330
+            from_observation_id,
331
+            to_observation_id,
332
+            filter_json,
333
+            record_count,
334
+            manifest_hash
335
+        FROM export_manifests
336
+        ORDER BY created_at, id
337
+        """
338
+        return try withStatement(sql, db: db) { statement in
339
+            var count = 0
340
+            while sqlite3_step(statement) == SQLITE_ROW {
341
+                let object = NSEntityDescription.insertNewObject(forEntityName: "CachedExportManifest", into: context)
342
+                object.setValue(columnText(statement, 0) ?? "", forKey: "exportID")
343
+                object.setValue(columnText(statement, 1) ?? "unknown", forKey: "exportKind")
344
+                object.setValue(columnUnixDate(statement, 2), forKey: "createdAt")
345
+                object.setValue(columnInt64(statement, 3), forKey: "fromObservationID")
346
+                object.setValue(columnInt64(statement, 4), forKey: "toObservationID")
347
+                object.setValue(columnText(statement, 5), forKey: "filterSummary")
348
+                object.setValue(Int64(sqlite3_column_int64(statement, 6)), forKey: "recordCount")
349
+                object.setValue(columnText(statement, 7) ?? "", forKey: "manifestHash")
350
+                object.setValue(nil, forKey: "fileURLBookmarkData")
351
+                object.setValue("available", forKey: "status")
352
+                object.setValue(computedAt, forKey: "computedAt")
353
+                count += 1
354
+            }
355
+            return count
356
+        }
357
+    }
358
+
359
+    private func insertArchiveHealthRow(
360
+        context: NSManagedObjectContext,
361
+        archiveSchemaVersion: Int,
362
+        integrityStatus: String,
363
+        computedAt: Date
364
+    ) throws {
365
+        let object = NSEntityDescription.insertNewObject(forEntityName: "CachedArchiveHealth", into: context)
366
+        object.setValue(Int64(archiveSchemaVersion), forKey: "archiveSchemaVersion")
367
+        object.setValue(Int64(Self.cacheSchemaVersion), forKey: "cacheSchemaVersion")
368
+        object.setValue(computedAt, forKey: "lastIntegrityCheckAt")
369
+        object.setValue(integrityStatus, forKey: "lastIntegrityStatus")
370
+        object.setValue(nil, forKey: "lastErrorKind")
371
+        object.setValue(nil, forKey: "lastErrorMessageHash")
372
+        object.setValue(UUID().uuidString, forKey: "cacheBuildID")
373
+        object.setValue(computedAt, forKey: "computedAt")
374
+    }
375
+}
376
+
377
+private extension CoreDataArchiveCacheStore {
378
+    static let cacheEntityNames = [
379
+        "CachedObservationRow",
380
+        "CachedTypeSummary",
381
+        "CachedDailyAggregate",
382
+        "CachedDiffSummary",
383
+        "CachedExportManifest",
384
+        "CachedArchiveHealth"
385
+    ]
386
+
387
+    static func makeModel() -> NSManagedObjectModel {
388
+        let model = NSManagedObjectModel()
389
+        model.entities = [
390
+            cachedObservationRowEntity(),
391
+            cachedTypeSummaryEntity(),
392
+            cachedDailyAggregateEntity(),
393
+            cachedDiffSummaryEntity(),
394
+            cachedExportManifestEntity(),
395
+            cachedArchiveHealthEntity()
396
+        ]
397
+        return model
398
+    }
399
+
400
+    static func cachedObservationRowEntity() -> NSEntityDescription {
401
+        entity("CachedObservationRow", attributes: [
402
+            attribute("observationID", .integer64AttributeType),
403
+            attribute("observedAt", .dateAttributeType),
404
+            attribute("status", .stringAttributeType),
405
+            attribute("triggerReason", .stringAttributeType),
406
+            attribute("timeZoneIdentifier", .stringAttributeType, optional: true),
407
+            attribute("trackedTypeCount", .integer64AttributeType),
408
+            attribute("visibleRecordCount", .integer64AttributeType),
409
+            attribute("appearedCount", .integer64AttributeType),
410
+            attribute("disappearedCount", .integer64AttributeType),
411
+            attribute("representationChangedCount", .integer64AttributeType),
412
+            attribute("archiveSchemaVersion", .integer64AttributeType),
413
+            attribute("cacheSchemaVersion", .integer64AttributeType),
414
+            attribute("sourceAggregateHash", .stringAttributeType),
415
+            attribute("computedAt", .dateAttributeType)
416
+        ])
417
+    }
418
+
419
+    static func cachedTypeSummaryEntity() -> NSEntityDescription {
420
+        entity("CachedTypeSummary", attributes: [
421
+            attribute("observationID", .integer64AttributeType),
422
+            attribute("sampleTypeIdentifier", .stringAttributeType),
423
+            attribute("displayName", .stringAttributeType, optional: true),
424
+            attribute("visibleRecordCount", .integer64AttributeType),
425
+            attribute("appearedCount", .integer64AttributeType),
426
+            attribute("disappearedCount", .integer64AttributeType),
427
+            attribute("representationChangedCount", .integer64AttributeType),
428
+            attribute("earliestStartDate", .dateAttributeType, optional: true),
429
+            attribute("latestEndDate", .dateAttributeType, optional: true),
430
+            attribute("valueSum", .doubleAttributeType, optional: true),
431
+            attribute("valueMax", .doubleAttributeType, optional: true),
432
+            attribute("aggregateHash", .stringAttributeType, optional: true),
433
+            attribute("computedAt", .dateAttributeType)
434
+        ])
435
+    }
436
+
437
+    static func cachedDailyAggregateEntity() -> NSEntityDescription {
438
+        entity("CachedDailyAggregate", attributes: [
439
+            attribute("observationID", .integer64AttributeType),
440
+            attribute("sampleTypeIdentifier", .stringAttributeType),
441
+            attribute("bucketStart", .dateAttributeType),
442
+            attribute("bucketEnd", .dateAttributeType),
443
+            attribute("timeZoneIdentifier", .stringAttributeType, optional: true),
444
+            attribute("visibleRecordCount", .integer64AttributeType),
445
+            attribute("valueSum", .doubleAttributeType, optional: true),
446
+            attribute("valueMax", .doubleAttributeType, optional: true),
447
+            attribute("sourceRevisionDisplayHash", .stringAttributeType, optional: true),
448
+            attribute("aggregateHash", .stringAttributeType, optional: true),
449
+            attribute("computedAt", .dateAttributeType)
450
+        ])
451
+    }
452
+
453
+    static func cachedDiffSummaryEntity() -> NSEntityDescription {
454
+        entity("CachedDiffSummary", attributes: [
455
+            attribute("fromObservationID", .integer64AttributeType),
456
+            attribute("toObservationID", .integer64AttributeType),
457
+            attribute("sampleTypeIdentifier", .stringAttributeType, optional: true),
458
+            attribute("appearedCount", .integer64AttributeType),
459
+            attribute("disappearedCount", .integer64AttributeType),
460
+            attribute("representationChangedCount", .integer64AttributeType),
461
+            attribute("consolidationLikely", .booleanAttributeType),
462
+            attribute("uncertaintyReason", .stringAttributeType, optional: true),
463
+            attribute("sourceAggregateHash", .stringAttributeType),
464
+            attribute("computedAt", .dateAttributeType)
465
+        ])
466
+    }
467
+
468
+    static func cachedExportManifestEntity() -> NSEntityDescription {
469
+        entity("CachedExportManifest", attributes: [
470
+            attribute("exportID", .stringAttributeType),
471
+            attribute("exportKind", .stringAttributeType),
472
+            attribute("createdAt", .dateAttributeType),
473
+            attribute("fromObservationID", .integer64AttributeType, optional: true),
474
+            attribute("toObservationID", .integer64AttributeType, optional: true),
475
+            attribute("filterSummary", .stringAttributeType, optional: true),
476
+            attribute("recordCount", .integer64AttributeType),
477
+            attribute("manifestHash", .stringAttributeType),
478
+            attribute("fileURLBookmarkData", .binaryDataAttributeType, optional: true),
479
+            attribute("status", .stringAttributeType),
480
+            attribute("computedAt", .dateAttributeType)
481
+        ])
482
+    }
483
+
484
+    static func cachedArchiveHealthEntity() -> NSEntityDescription {
485
+        entity("CachedArchiveHealth", attributes: [
486
+            attribute("archiveSchemaVersion", .integer64AttributeType),
487
+            attribute("cacheSchemaVersion", .integer64AttributeType),
488
+            attribute("lastIntegrityCheckAt", .dateAttributeType),
489
+            attribute("lastIntegrityStatus", .stringAttributeType),
490
+            attribute("lastErrorKind", .stringAttributeType, optional: true),
491
+            attribute("lastErrorMessageHash", .stringAttributeType, optional: true),
492
+            attribute("cacheBuildID", .stringAttributeType),
493
+            attribute("computedAt", .dateAttributeType)
494
+        ])
495
+    }
496
+
497
+    static func entity(_ name: String, attributes: [NSAttributeDescription]) -> NSEntityDescription {
498
+        let entity = NSEntityDescription()
499
+        entity.name = name
500
+        entity.managedObjectClassName = "NSManagedObject"
501
+        entity.properties = attributes
502
+        return entity
503
+    }
504
+
505
+    static func attribute(
506
+        _ name: String,
507
+        _ type: NSAttributeType,
508
+        optional: Bool = false
509
+    ) -> NSAttributeDescription {
510
+        let attribute = NSAttributeDescription()
511
+        attribute.name = name
512
+        attribute.attributeType = type
513
+        attribute.isOptional = optional
514
+        return attribute
515
+    }
516
+}
517
+
518
+private extension CoreDataArchiveCacheStore {
519
+    func openArchive(at archiveURL: URL) throws -> OpaquePointer? {
520
+        var db: OpaquePointer?
521
+        guard sqlite3_open_v2(archiveURL.path, &db, SQLITE_OPEN_READONLY | SQLITE_OPEN_FULLMUTEX, nil) == SQLITE_OK else {
522
+            let message = db.map(lastErrorMessage) ?? "unable to open archive database"
523
+            sqlite3_close(db)
524
+            throw CoreDataArchiveCacheStoreError.openArchiveFailed(message)
525
+        }
526
+        return db
527
+    }
528
+
529
+    func archiveSchemaVersion(db: OpaquePointer?) throws -> Int {
530
+        let sql = "SELECT value FROM archive_metadata WHERE key = 'schema_version' LIMIT 1"
531
+        return try withStatement(sql, db: db) { statement in
532
+            guard sqlite3_step(statement) == SQLITE_ROW,
533
+                  let value = columnText(statement, 0),
534
+                  let version = Int(value) else {
535
+                return 0
536
+            }
537
+            return version
538
+        }
539
+    }
540
+
541
+    func firstText(_ sql: String, db: OpaquePointer?) throws -> String? {
542
+        try withStatement(sql, db: db) { statement in
543
+            guard sqlite3_step(statement) == SQLITE_ROW else {
544
+                return nil
545
+            }
546
+            return columnText(statement, 0)
547
+        }
548
+    }
549
+
550
+    func withStatement<T>(_ sql: String, db: OpaquePointer?, body: (OpaquePointer?) throws -> T) throws -> T {
551
+        var statement: OpaquePointer?
552
+        guard sqlite3_prepare_v2(db, sql, -1, &statement, nil) == SQLITE_OK else {
553
+            throw CoreDataArchiveCacheStoreError.prepareFailed(lastErrorMessage(db))
554
+        }
555
+        defer { sqlite3_finalize(statement) }
556
+        return try body(statement)
557
+    }
558
+}
559
+
560
+private func lastErrorMessage(_ db: OpaquePointer?) -> String {
561
+    guard let message = sqlite3_errmsg(db) else { return "unknown SQLite error" }
562
+    return String(cString: message)
563
+}
564
+
565
+private func columnText(_ statement: OpaquePointer?, _ index: Int32) -> String? {
566
+    guard sqlite3_column_type(statement, index) != SQLITE_NULL,
567
+          let pointer = sqlite3_column_text(statement, index) else {
568
+        return nil
569
+    }
570
+    return String(cString: pointer)
571
+}
572
+
573
+private func columnDouble(_ statement: OpaquePointer?, _ index: Int32) -> Double? {
574
+    guard sqlite3_column_type(statement, index) != SQLITE_NULL else { return nil }
575
+    return sqlite3_column_double(statement, index)
576
+}
577
+
578
+private func columnInt64(_ statement: OpaquePointer?, _ index: Int32) -> Int64? {
579
+    guard sqlite3_column_type(statement, index) != SQLITE_NULL else { return nil }
580
+    return sqlite3_column_int64(statement, index)
581
+}
582
+
583
+private func columnUnixDate(_ statement: OpaquePointer?, _ index: Int32) -> Date? {
584
+    columnDouble(statement, index).map { Date(timeIntervalSince1970: $0) }
585
+}
+103 -0
HealthProbeTests/CoreDataArchiveCacheStoreTests.swift
@@ -0,0 +1,103 @@
1
+import CoreData
2
+import HealthKit
3
+import XCTest
4
+@testable import HealthProbe
5
+
6
+final class CoreDataArchiveCacheStoreTests: XCTestCase {
7
+    private var temporaryDirectory: URL!
8
+
9
+    override func setUpWithError() throws {
10
+        temporaryDirectory = FileManager.default.temporaryDirectory
11
+            .appending(path: "HealthProbeCacheTests-\(UUID().uuidString)", directoryHint: .isDirectory)
12
+        try FileManager.default.createDirectory(at: temporaryDirectory, withIntermediateDirectories: true)
13
+    }
14
+
15
+    override func tearDownWithError() throws {
16
+        if let temporaryDirectory {
17
+            try? FileManager.default.removeItem(at: temporaryDirectory)
18
+        }
19
+        temporaryDirectory = nil
20
+    }
21
+
22
+    func testRebuildCreatesCoreDataRowsFromSQLiteArchive() async throws {
23
+        let archiveURL = temporaryDirectory.appending(path: "Archive.sqlite")
24
+        let archive = SQLiteHealthArchiveStore(databaseURL: archiveURL)
25
+        let firstSample = makeStepCountSample(value: 42, start: 1_000)
26
+        let secondSample = makeStepCountSample(value: 7, start: 2_000)
27
+
28
+        _ = try await archive.upsertSamples([firstSample], observedAt: Date(timeIntervalSince1970: 3_000))
29
+        _ = try await archive.upsertSamples([firstSample, secondSample], observedAt: Date(timeIntervalSince1970: 3_060))
30
+        try await archive.recordDisappearance(
31
+            sampleUUIDHash: HashService.sampleUUIDHash(firstSample.uuid.uuidString),
32
+            sampleTypeIdentifier: HKQuantityTypeIdentifier.stepCount.rawValue,
33
+            observedMissingAt: Date(timeIntervalSince1970: 3_120)
34
+        )
35
+        try await archive.markVerification(
36
+            sampleType: firstSample.sampleType,
37
+            verifiedAt: Date(timeIntervalSince1970: 3_180)
38
+        )
39
+
40
+        let cache = try CoreDataArchiveCacheStore(inMemory: true)
41
+        let summary = try cache.rebuild(fromArchiveAt: archiveURL)
42
+        let context = cache.container.viewContext
43
+
44
+        XCTAssertEqual(summary.observationRows, 4)
45
+        XCTAssertEqual(summary.typeSummaryRows, 4)
46
+        XCTAssertGreaterThanOrEqual(summary.dailyAggregateRows, 1)
47
+        XCTAssertEqual(summary.archiveHealthRows, 1)
48
+        XCTAssertEqual(try count("CachedObservationRow", in: context), 4)
49
+        XCTAssertEqual(try count("CachedTypeSummary", in: context), 4)
50
+        XCTAssertEqual(try count("CachedArchiveHealth", in: context), 1)
51
+
52
+        let latestObservation = try fetchFirst(
53
+            "CachedObservationRow",
54
+            predicate: NSPredicate(format: "observationID == %d", 4),
55
+            in: context
56
+        )
57
+        XCTAssertEqual(latestObservation?.value(forKey: "visibleRecordCount") as? Int64, 1)
58
+        XCTAssertEqual(latestObservation?.value(forKey: "cacheSchemaVersion") as? Int64, Int64(CoreDataArchiveCacheStore.cacheSchemaVersion))
59
+    }
60
+
61
+    func testDeletingCacheDoesNotDeleteSQLiteArchiveAndRebuildRestoresRows() async throws {
62
+        let archiveURL = temporaryDirectory.appending(path: "Archive.sqlite")
63
+        let archive = SQLiteHealthArchiveStore(databaseURL: archiveURL)
64
+        let sample = makeStepCountSample(value: 10, start: 1_000)
65
+        _ = try await archive.upsertSamples([sample], observedAt: Date(timeIntervalSince1970: 2_000))
66
+
67
+        let cache = try CoreDataArchiveCacheStore(inMemory: true)
68
+        _ = try cache.rebuild(fromArchiveAt: archiveURL)
69
+        try cache.deleteCache()
70
+
71
+        XCTAssertEqual(try count("CachedObservationRow", in: cache.container.viewContext), 0)
72
+        let integrityReport = try await archive.checkIntegrity()
73
+        XCTAssertTrue(integrityReport.passed)
74
+
75
+        let rebuilt = try cache.rebuild(fromArchiveAt: archiveURL)
76
+        XCTAssertEqual(rebuilt.observationRows, 1)
77
+        XCTAssertEqual(try count("CachedObservationRow", in: cache.container.viewContext), 1)
78
+    }
79
+
80
+    private func makeStepCountSample(value: Double, start: TimeInterval, end: TimeInterval? = nil) -> HKQuantitySample {
81
+        let quantityType = HKQuantityType.quantityType(forIdentifier: .stepCount)!
82
+        let quantity = HKQuantity(unit: .count(), doubleValue: value)
83
+        let startDate = Date(timeIntervalSince1970: start)
84
+        let endDate = Date(timeIntervalSince1970: end ?? (start + 300))
85
+        return HKQuantitySample(type: quantityType, quantity: quantity, start: startDate, end: endDate)
86
+    }
87
+
88
+    private func count(_ entityName: String, in context: NSManagedObjectContext) throws -> Int {
89
+        let request = NSFetchRequest<NSManagedObject>(entityName: entityName)
90
+        return try context.count(for: request)
91
+    }
92
+
93
+    private func fetchFirst(
94
+        _ entityName: String,
95
+        predicate: NSPredicate,
96
+        in context: NSManagedObjectContext
97
+    ) throws -> NSManagedObject? {
98
+        let request = NSFetchRequest<NSManagedObject>(entityName: entityName)
99
+        request.predicate = predicate
100
+        request.fetchLimit = 1
101
+        return try context.fetch(request).first
102
+    }
103
+}