- Disable legacy detail cache during snapshot save
- Browse files
-
bogdan committed 4 days ago
Showing 5 changed files
with
102 additions
and
116 deletions
+4
-6
HealthProbe/Doc/00-agent-guides/AGENTS.md
@@ -223,12 +223,10 @@ struct TypeDistributionBin: Codable, Sendable {
|
||
| 223 | 223 |
|
| 224 | 224 |
// Interface updated 2026-05-17 — see AGENTS.md |
| 225 | 225 |
// Models/TypeCount.detailCacheData is legacy SwiftData UI cache for comparing a |
| 226 |
-// TypeCount with the immediately previous snapshot on the same device. It contains |
|
| 227 |
-// aggregate added/disappeared counts, capped preview records, and daily change bins. |
|
| 228 |
-// It may be precomputed only for bounded archive sizes; high-volume types must skip |
|
| 229 |
-// this legacy cache and rely on SQLite/Core Data archive views instead. Existing |
|
| 230 |
-// stores are backfilled incrementally with strict caps to avoid decoding many large |
|
| 231 |
-// archives in one run. |
|
| 226 |
+// TypeCount with the immediately previous snapshot on the same device. It must not |
|
| 227 |
+// be precomputed during snapshot save: real-device imports showed Core Data |
|
| 228 |
+// mutated-while-enumerated crashes even with small detail caches. Current UI should |
|
| 229 |
+// rely on SQLite archive queries and rebuildable Core Data cache rows instead. |
|
| 232 | 230 |
|
| 233 | 231 |
// Interface updated 2026-05-17 — see AGENTS.md |
| 234 | 232 |
// Models/HealthSnapshot.contentEquivalentSnapshotID marks snapshots whose TypeCount |
+32
-5
HealthProbe/Doc/04-project/Import-Optimization-Log.md
@@ -423,6 +423,29 @@ refresh because the view model still awaited cache rebuild before releasing the |
||
| 423 | 423 |
post-snapshot UI path. That refresh should run fire-and-forget and publish its |
| 424 | 424 |
result back to the main actor only when complete. |
| 425 | 425 |
|
| 426 |
+### 2026-06-03 Final Freeze Log: Small Detail Cache Still Crashes |
|
| 427 |
+ |
|
| 428 |
+Source: user-provided overnight freeze/crash log after the large-cache skip. |
|
| 429 |
+ |
|
| 430 |
+The log shows `healthKit.precomputeDetailCaches` still building two small |
|
| 431 |
+legacy SwiftData detail caches: |
|
| 432 |
+ |
|
| 433 |
+| Type | Current archive | Current count | Result | |
|
| 434 |
+|------|-----------------|---------------|--------| |
|
| 435 |
+| Stand Hours | 1.1 MB | 7,727 | built, 2 added | |
|
| 436 |
+| Environmental Sound Levels | 2 MB | 13,384 | built, 1 added | |
|
| 437 |
+ |
|
| 438 |
+Heart Rate and Active Energy were correctly skipped by the large-cache guard, |
|
| 439 |
+but Core Data still aborted immediately after `healthKit.precomputeDetailCaches.end` |
|
| 440 |
+with the same mutated-while-enumerated exception during change processing. |
|
| 441 |
+ |
|
| 442 |
+Conclusion: the issue is not only large archive size. Mutating legacy |
|
| 443 |
+`TypeCount.detailCacheData` during snapshot save is unsafe in this SwiftData / |
|
| 444 |
+Core Data context. Snapshot save must not precompute or clear this legacy cache. |
|
| 445 |
+The Snapshots list should also read observation timeline rows directly from |
|
| 446 |
+SQLite so it does not depend on a delayed Core Data cache rebuild to show the |
|
| 447 |
+freshly finished observation. |
|
| 448 |
+ |
|
| 426 | 449 |
## Optimization Iterations |
| 427 | 450 |
|
| 428 | 451 |
| Date | Commit | Change | Result / Status | |
@@ -449,6 +472,7 @@ result back to the main actor only when complete. |
||
| 449 | 472 |
| 2026-06-03 | `19ba656` | Copy previous type summaries and daily aggregates for unchanged metric observations instead of rebuilding from visible ranges. | First follow-up was not no-delta (+71 records), so it is inconclusive for the unchanged path. It did show `SummedInsertElapsed` 0.1s and a new changed-metric processing bottleneck: Heart Rate processing 13.9s, Active Energy processing 5.0s. | |
| 450 | 473 |
| 2026-06-03 | `e49a79d` | Skip legacy SwiftData detail-cache precompute for large type archives. | Triggered by a crash after building Heart Rate and Active Energy detail caches. Expected signal: no post-import `NSGenericException`, lower post-import wall-clock gap, and `healthKit.detailCache.skipLargeLegacyArchive` logs for high-volume types. | |
| 451 | 474 |
| 2026-06-03 | `7d52262` | Start Dashboard archive cache refresh without awaiting it after snapshot completion. | Triggered by continued app unresponsiveness after a successful 31.3s incremental snapshot. Expected signal: progress sheet/result UI remains responsive while cache rows refresh later. | |
| 475 |
+| 2026-06-03 | pending | Disable legacy SwiftData detail-cache precompute completely and load Snapshots timeline from SQLite. | Triggered by overnight crash after two small detail caches were built. Expected signal: no `healthKit.detailCache.buildBegin` logs during snapshot save, no Core Data mutated-while-enumerated abort, and the new SQLite observation appears in Snapshots without waiting for cache rebuild. | |
|
| 452 | 476 |
|
| 453 | 477 |
## Current Diagnosis |
| 454 | 478 |
|
@@ -467,8 +491,8 @@ The likely bottleneck is per-row SQLite work: |
||
| 467 | 491 |
- no-op visibility range maintenance for unchanged existing samples; |
| 468 | 492 |
- processing existing high-volume metrics when only a small number of records changed; |
| 469 | 493 |
- legacy SwiftData detail-cache precompute after SQLite completes. This became |
| 470 |
- the active crash/performance issue when Heart Rate and Active Energy detail |
|
| 471 |
- caches scanned large archives and Core Data aborted during change processing. |
|
| 494 |
+ the active crash/performance issue even for small caches; any snapshot-save |
|
| 495 |
+ mutation of `TypeCount.detailCacheData` is unsafe and should remain disabled. |
|
| 472 | 496 |
- Dashboard Core Data archive cache refresh after snapshot completion, when the |
| 473 | 497 |
UI path awaits the rebuild instead of publishing results asynchronously. |
| 474 | 498 |
|
@@ -481,6 +505,9 @@ The likely bottleneck is per-row SQLite work: |
||
| 481 | 505 |
post-import cache work. A 2026-06-03 console log showed Heart Rate and Active |
| 482 | 506 |
Energy `TypeCount.detailCacheData` precompute immediately before a Core Data |
| 483 | 507 |
mutated-while-enumerated abort. |
| 508 |
+- A later 2026-06-03 overnight log showed the same abort after only Stand Hours |
|
| 509 |
+ and Environmental Sound Levels detail caches were built. Size limits are not |
|
| 510 |
+ enough; the whole snapshot-save detail-cache precompute path is disabled. |
|
| 484 | 511 |
- Partial / old imported observations can pollute comparisons. Fresh first-snapshot performance comparisons should use a confirmed reset database. |
| 485 | 512 |
- Non-chain-start full scans can be slower than first imports if unchanged existing samples still write per-sample archive evidence. |
| 486 | 513 |
|
@@ -488,9 +515,9 @@ The likely bottleneck is per-row SQLite work: |
||
| 488 | 515 |
|
| 489 | 516 |
Prioritize experiments in this order: |
| 490 | 517 |
|
| 491 |
-1. Run an incremental snapshot after the fire-and-forget Dashboard cache refresh. |
|
| 492 |
- Confirm that the progress/result UI remains responsive immediately after |
|
| 493 |
- `complete_success`, even if dashboard archive rows update a little later. |
|
| 518 |
+1. Run an incremental snapshot after disabling legacy detail-cache precompute. |
|
| 519 |
+ Confirm there are no `healthKit.detailCache.buildBegin` logs, no Core Data |
|
| 520 |
+ mutated-while-enumerated abort, and the new observation appears in Snapshots. |
|
| 494 | 521 |
2. Run a repeated no-delta benchmark after copying unchanged metric summaries and daily aggregates. Compare `SummedFinalizeElapsed`, `Heart Rate finalizeElapsed`, `Active Energy finalizeElapsed`, and wall clock. |
| 495 | 522 |
3. Add or inspect timing around per-record processing for changed high-volume metrics, especially Heart Rate, to separate sample DTO/fingerprint work from SQLite idempotency checks. |
| 496 | 523 |
4. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`. |
+4
-103
HealthProbe/Services/HealthKitService.swift
@@ -5,8 +5,6 @@ import UIKit |
||
| 5 | 5 |
import os.log |
| 6 | 6 |
|
| 7 | 7 |
private let logger = Logger(subsystem: "ro.xdev.healthprobe", category: "HealthKitService") |
| 8 |
-private let legacyDetailCachePrecomputeRecordLimit = 250_000 |
|
| 9 |
-private let legacyDetailCachePrecomputeArchiveByteLimit = 32 * 1_024 * 1_024 |
|
| 10 | 8 |
|
| 11 | 9 |
enum TypeCategory: String, CaseIterable {
|
| 12 | 10 |
case activity = "Activity" |
@@ -644,80 +642,12 @@ final class HealthKitService {
|
||
| 644 | 642 |
private func precomputeTypeCountDetailCaches( |
| 645 | 643 |
snapshot: HealthSnapshot, |
| 646 | 644 |
typeCounts: [TypeCount], |
| 647 |
- context: ModelContext |
|
| 645 |
+ context _: ModelContext |
|
| 648 | 646 |
) {
|
| 649 |
- MemoryLog.log("healthKit.precomputeDetailCaches.begin", metadata: [
|
|
| 647 |
+ MemoryLog.log("healthKit.precomputeDetailCaches.disabled", metadata: [
|
|
| 650 | 648 |
"typeCountCount": "\(typeCounts.count)", |
| 651 |
- "hasPrevious": "\(snapshot.previousSnapshotID != nil)" |
|
| 652 |
- ]) |
|
| 653 |
- |
|
| 654 |
- guard let previousID = snapshot.previousSnapshotID, |
|
| 655 |
- let previous = fetchSnapshot(id: previousID, context: context) else {
|
|
| 656 |
- for typeCount in typeCounts {
|
|
| 657 |
- typeCount.setDetailCache(nil) |
|
| 658 |
- } |
|
| 659 |
- MemoryLog.log("healthKit.precomputeDetailCaches.noPrevious", metadata: [
|
|
| 660 |
- "typeCountCount": "\(typeCounts.count)" |
|
| 661 |
- ]) |
|
| 662 |
- return |
|
| 663 |
- } |
|
| 664 |
- |
|
| 665 |
- let previousByType = Dictionary( |
|
| 666 |
- uniqueKeysWithValues: (previous.typeCounts ?? []).map { ($0.typeIdentifier, $0) }
|
|
| 667 |
- ) |
|
| 668 |
- var builtCount = 0 |
|
| 669 |
- var skippedAliasCount = 0 |
|
| 670 |
- var skippedLargeCount = 0 |
|
| 671 |
- |
|
| 672 |
- for typeCount in typeCounts {
|
|
| 673 |
- if typeCount.isContentAlias {
|
|
| 674 |
- typeCount.setDetailCache(nil) |
|
| 675 |
- skippedAliasCount += 1 |
|
| 676 |
- continue |
|
| 677 |
- } |
|
| 678 |
- |
|
| 679 |
- let previousType = previousByType[typeCount.typeIdentifier] |
|
| 680 |
- if let skipReason = legacyDetailCachePrecomputeSkipReason( |
|
| 681 |
- current: typeCount, |
|
| 682 |
- previous: previousType |
|
| 683 |
- ) {
|
|
| 684 |
- typeCount.setDetailCache(nil) |
|
| 685 |
- skippedLargeCount += 1 |
|
| 686 |
- MemoryLog.log("healthKit.detailCache.skipLargeLegacyArchive", metadata: detailCacheMetadata(
|
|
| 687 |
- current: typeCount, |
|
| 688 |
- previous: previousType, |
|
| 689 |
- source: "snapshotSave" |
|
| 690 |
- ).merging([ |
|
| 691 |
- "reason": skipReason, |
|
| 692 |
- "recordLimit": "\(legacyDetailCachePrecomputeRecordLimit)", |
|
| 693 |
- "archiveByteLimit": MemoryLog.format(UInt64(legacyDetailCachePrecomputeArchiveByteLimit)) |
|
| 694 |
- ]) { _, new in new })
|
|
| 695 |
- continue |
|
| 696 |
- } |
|
| 697 |
- |
|
| 698 |
- MemoryLog.log("healthKit.detailCache.buildBegin", metadata: detailCacheMetadata(
|
|
| 699 |
- current: typeCount, |
|
| 700 |
- previous: previousType, |
|
| 701 |
- source: "snapshotSave" |
|
| 702 |
- )) |
|
| 703 |
- typeCount.setDetailCache( |
|
| 704 |
- TypeCountDetailCacheBuilder.build( |
|
| 705 |
- current: typeCount, |
|
| 706 |
- previous: previousType, |
|
| 707 |
- baselineSnapshotID: previous.id |
|
| 708 |
- ) |
|
| 709 |
- ) |
|
| 710 |
- builtCount += 1 |
|
| 711 |
- MemoryLog.log("healthKit.detailCache.buildEnd", metadata: detailCacheMetadata(
|
|
| 712 |
- current: typeCount, |
|
| 713 |
- previous: previousByType[typeCount.typeIdentifier], |
|
| 714 |
- source: "snapshotSave" |
|
| 715 |
- )) |
|
| 716 |
- } |
|
| 717 |
- MemoryLog.log("healthKit.precomputeDetailCaches.end", metadata: [
|
|
| 718 |
- "builtCount": "\(builtCount)", |
|
| 719 |
- "skippedAliasCount": "\(skippedAliasCount)", |
|
| 720 |
- "skippedLargeCount": "\(skippedLargeCount)" |
|
| 649 |
+ "hasPrevious": "\(snapshot.previousSnapshotID != nil)", |
|
| 650 |
+ "reason": "legacySwiftDataCacheCrash" |
|
| 721 | 651 |
]) |
| 722 | 652 |
} |
| 723 | 653 |
|
@@ -781,35 +711,6 @@ final class HealthKitService {
|
||
| 781 | 711 |
lhs.isUnsupported == rhs.isUnsupported |
| 782 | 712 |
} |
| 783 | 713 |
|
| 784 |
- private func detailCacheMetadata(current: TypeCount, previous: TypeCount?, source: String) -> [String: String] {
|
|
| 785 |
- [ |
|
| 786 |
- "source": source, |
|
| 787 |
- "type": current.typeIdentifier, |
|
| 788 |
- "currentCount": "\(current.count)", |
|
| 789 |
- "previousCount": "\(previous?.count ?? 0)", |
|
| 790 |
- "currentArchive": current.recordArchiveData.map { MemoryLog.format(UInt64($0.count)) } ?? "nil",
|
|
| 791 |
- "previousArchive": previous?.recordArchiveData.map { MemoryLog.format(UInt64($0.count)) } ?? "nil",
|
|
| 792 |
- "isAlias": "\(current.isContentAlias)" |
|
| 793 |
- ] |
|
| 794 |
- } |
|
| 795 |
- |
|
| 796 |
- private func legacyDetailCachePrecomputeSkipReason(current: TypeCount, previous: TypeCount?) -> String? {
|
|
| 797 |
- let largestRecordCount = max(max(current.count, 0), max(previous?.count ?? 0, 0)) |
|
| 798 |
- if largestRecordCount > legacyDetailCachePrecomputeRecordLimit {
|
|
| 799 |
- return "recordLimit" |
|
| 800 |
- } |
|
| 801 |
- |
|
| 802 |
- let largestArchiveByteCount = max( |
|
| 803 |
- current.recordArchiveData?.count ?? 0, |
|
| 804 |
- previous?.recordArchiveData?.count ?? 0 |
|
| 805 |
- ) |
|
| 806 |
- if largestArchiveByteCount > legacyDetailCachePrecomputeArchiveByteLimit {
|
|
| 807 |
- return "archiveByteLimit" |
|
| 808 |
- } |
|
| 809 |
- |
|
| 810 |
- return nil |
|
| 811 |
- } |
|
| 812 |
- |
|
| 813 | 714 |
private func hasAmbiguousCompleteDisappearance( |
| 814 | 715 |
snapshot: HealthSnapshot, |
| 815 | 716 |
typeCounts: [TypeCount], |
+61
-0
HealthProbe/Services/SQLiteHealthArchiveStore.swift
@@ -85,6 +85,14 @@ actor SQLiteHealthArchiveStore: HealthArchiveStore {
|
||
| 85 | 85 |
} |
| 86 | 86 |
} |
| 87 | 87 |
|
| 88 |
+ func observationRows(limit: Int = 200) async throws -> [CachedArchiveObservationRow] {
|
|
| 89 |
+ let db = try openDatabase() |
|
| 90 |
+ defer { sqlite3_close(db) }
|
|
| 91 |
+ try prepareSchemaIfNeeded(db) |
|
| 92 |
+ |
|
| 93 |
+ return try observationRows(limit: limit, db: db) |
|
| 94 |
+ } |
|
| 95 |
+ |
|
| 88 | 96 |
func upsertSamples(_ samples: [HKSample], observedAt: Date) async throws -> HealthArchiveWriteSummary {
|
| 89 | 97 |
guard !samples.isEmpty else {
|
| 90 | 98 |
return HealthArchiveWriteSummary(insertedCount: 0, updatedCount: 0, unchangedCount: 0) |
@@ -2273,6 +2281,59 @@ actor SQLiteHealthArchiveStore: HealthArchiveStore {
|
||
| 2273 | 2281 |
return sqlite3_last_insert_rowid(db) |
| 2274 | 2282 |
} |
| 2275 | 2283 |
|
| 2284 |
+ private func observationRows(limit: Int, db: OpaquePointer?) throws -> [CachedArchiveObservationRow] {
|
|
| 2285 |
+ let sql = """ |
|
| 2286 |
+ SELECT |
|
| 2287 |
+ o.id, |
|
| 2288 |
+ o.observed_at, |
|
| 2289 |
+ o.status, |
|
| 2290 |
+ o.trigger_reason, |
|
| 2291 |
+ o.time_zone_identifier, |
|
| 2292 |
+ COUNT(s.sample_type_id) AS tracked_type_count, |
|
| 2293 |
+ COALESCE(SUM(s.visible_record_count), 0) AS visible_record_count, |
|
| 2294 |
+ COALESCE(SUM(s.appeared_count), 0) AS appeared_count, |
|
| 2295 |
+ COALESCE(SUM(s.disappeared_count), 0) AS disappeared_count, |
|
| 2296 |
+ COALESCE(SUM(s.representation_changed_count), 0) AS representation_changed_count, |
|
| 2297 |
+ o.schema_version |
|
| 2298 |
+ FROM observations o |
|
| 2299 |
+ LEFT JOIN observation_type_summaries s ON s.observation_id = o.id |
|
| 2300 |
+ GROUP BY |
|
| 2301 |
+ o.id, o.observed_at, o.status, o.trigger_reason, o.time_zone_identifier, |
|
| 2302 |
+ o.schema_version |
|
| 2303 |
+ ORDER BY o.id DESC |
|
| 2304 |
+ LIMIT ? |
|
| 2305 |
+ """ |
|
| 2306 |
+ |
|
| 2307 |
+ var rows: [CachedArchiveObservationRow] = [] |
|
| 2308 |
+ try withStatement(sql, db: db) { statement in
|
|
| 2309 |
+ bindInt(max(limit, 0), to: 1, in: statement) |
|
| 2310 |
+ var stepResult = sqlite3_step(statement) |
|
| 2311 |
+ while stepResult == SQLITE_ROW {
|
|
| 2312 |
+ rows.append(CachedArchiveObservationRow( |
|
| 2313 |
+ observationID: sqlite3_column_int64(statement, 0), |
|
| 2314 |
+ observedAt: columnUnixDate(statement, 1) ?? Date(timeIntervalSince1970: 0), |
|
| 2315 |
+ status: columnText(statement, 2) ?? "", |
|
| 2316 |
+ triggerReason: columnText(statement, 3) ?? "", |
|
| 2317 |
+ timeZoneIdentifier: columnText(statement, 4), |
|
| 2318 |
+ trackedTypeCount: columnInt(statement, 5) ?? 0, |
|
| 2319 |
+ visibleRecordCount: columnInt(statement, 6) ?? 0, |
|
| 2320 |
+ appearedCount: columnInt(statement, 7) ?? 0, |
|
| 2321 |
+ disappearedCount: columnInt(statement, 8) ?? 0, |
|
| 2322 |
+ representationChangedCount: columnInt(statement, 9) ?? 0, |
|
| 2323 |
+ archiveSchemaVersion: columnInt(statement, 10) ?? Self.archiveSchemaVersion, |
|
| 2324 |
+ cacheSchemaVersion: CoreDataArchiveCacheStore.cacheSchemaVersion, |
|
| 2325 |
+ computedAt: Date() |
|
| 2326 |
+ )) |
|
| 2327 |
+ stepResult = sqlite3_step(statement) |
|
| 2328 |
+ } |
|
| 2329 |
+ |
|
| 2330 |
+ guard stepResult == SQLITE_DONE else {
|
|
| 2331 |
+ throw SQLiteHealthArchiveStoreError.stepFailed(lastErrorMessage(db)) |
|
| 2332 |
+ } |
|
| 2333 |
+ } |
|
| 2334 |
+ return rows |
|
| 2335 |
+ } |
|
| 2336 |
+ |
|
| 2276 | 2337 |
private func updateObservationStatus( |
| 2277 | 2338 |
observationID: Int64, |
| 2278 | 2339 |
status: String, |
+1
-2
HealthProbe/ViewModels/SnapshotsViewModel.swift
@@ -31,8 +31,7 @@ final class SnapshotsViewModel {
|
||
| 31 | 31 |
@MainActor |
| 32 | 32 |
func loadArchiveRows(limit: Int = 200) async {
|
| 33 | 33 |
do {
|
| 34 |
- let cache = try CoreDataArchiveCacheStore() |
|
| 35 |
- let rows = try cache.observationRows(limit: limit) |
|
| 34 |
+ let rows = try await SQLiteHealthArchiveStore.shared.observationRows(limit: limit) |
|
| 36 | 35 |
archiveRows = rows.isEmpty ? nil : rows |
| 37 | 36 |
archiveRowsError = nil |
| 38 | 37 |
} catch {
|