Showing 4 changed files with 29 additions and 4 deletions
+10 -1
HealthProbe/Doc/04-project/Import-Optimization-Log.md
@@ -582,6 +582,7 @@ rows exist".
582 582
 | 2026-06-03 | pending | Persist HealthKit capture anchors in SQLite. | A later no-delta attempt still reimported full high-volume types (`Records: 2,646,590`, `WallClock: 45.8s`, Heart Rate `922,521`, Active Energy `346,473`) because incremental anchors were still read from legacy SwiftData `TypeCount`, while the refactored app now chains observations from SQLite. SQLite now stores per-type capture state (`anchor_data`, count, content hash, yearly counts, range) and the import path uses it when legacy `TypeCount` state is missing. Expected signal: the first run after this change may seed anchors; the following no-delta run should report empty HealthKit delta pages instead of full `record_import` counts for large stable types. |
583 583
 | 2026-06-03 | pending | Clarify capture mode in record-import diagnostics. | Two full-profile repeated snapshots after SQLite capture-state persistence completed successfully with stable checksum and identical record count (`2,646,613`). The first ran in `6.3s` with `SummedProcessingElapsed: 0.0s`, `SummedInsertElapsed: 0.0s`, `SummedFinalizeElapsed: 2.6s`; the second ran in `6.4s` with `0.0s` processing/insert and `2.7s` finalize. Heart Rate `922,526` and Active Energy `346,478` each completed around `0.2s`, proving the heavy full reimport path was avoided. The report wording still said "`N` samples in 1 anchored segment", which is ambiguous for inherited unchanged summaries; diagnostics now label unchanged empty-delta, delta-apply, and full-import modes explicitly. |
584 584
 | 2026-06-03 | pending | Add capture-mode summary to diagnostics. | Repeated full-profile captures rarely produce a perfect no-delta report because at least one metric can change between manual runs. Diagnostic reports now include aggregate `CaptureModes` counts plus per-metric `captureMode`, so comparisons can separate unchanged empty-delta metrics from delta-applied metrics and full imports without manually reading every `record_import` line. Expected signal: stable checksum plus high `unchangedDelta` count and zero summed processing/insert confirms the fast path even when a few metrics changed. |
585
+| 2026-06-03 | pending | Add delta-event counts to diagnostics. | A full-profile follow-up completed in `47.4s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=115, delta=12, initialImport=0`, `SummedProcessingElapsed: 25.9s`, `SummedInsertElapsed: 0.2s`, and `SummedFinalizeElapsed: 16.0s`. This confirms anchors work and no full import ran. Remaining cost is delta application for large metrics: Heart Rate `23.5s` total (`14.1s` processing, `8.8s` finalize), Active Energy `7.1s`, and Basal Energy `6.0s`. Diagnostics now report aggregate/per-metric `DeltaEvents` so future logs can separate true HealthKit delta size from the final visible record count. |
585 586
 
586 587
 ## Current Diagnosis
587 588
 
@@ -633,6 +634,11 @@ The likely bottleneck is per-row SQLite work:
633 634
   identical record count, zero summed processing/insert, and sub-second
634 635
   high-volume metric timings are acceptable evidence that the expensive full
635 636
   import path was skipped.
637
+- Capture modes now show the heavy full import path is avoided, but a small
638
+  HealthKit delta on a high-volume type can still force expensive local
639
+  reconstruction of legacy compact record archives and type hashes. Future logs
640
+  should compare `DeltaEvents` against processing/finalize time before changing
641
+  page sizes or HealthKit query strategy.
636 642
 
637 643
 ## Open Issues / Observations
638 644
 
@@ -671,7 +677,10 @@ Prioritize experiments in this order:
671 677
    `CaptureModes`, and high-volume type timings. Treat stable checksum, a high
672 678
    `unchangedDelta` count, and zero processing/insert as the main unchanged-path
673 679
    signal.
674
-4. Add or inspect timing around per-record processing for changed high-volume metrics, especially Heart Rate, to separate sample DTO/fingerprint work from SQLite idempotency checks.
680
+4. Use `DeltaEvents` to quantify changed high-volume metrics, especially Heart
681
+   Rate, Active Energy, and Basal Energy. If delta events are small while
682
+   processing/finalize remain large, optimize legacy compact archive/hash
683
+   reconstruction rather than HealthKit fetch or SQLite insert.
675 684
 5. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
676 685
 6. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.
677 686
 7. Profile whether index maintenance dominates first-import insert cost.
+12 -2
HealthProbe/Services/HealthKitService.swift
@@ -930,7 +930,7 @@ final class HealthKitService {
930 930
             case .unchangedDelta:
931 931
                 return "\(distribution.totalCount) unchanged samples via empty HealthKit delta"
932 932
             case .delta:
933
-                return "\(distribution.totalCount) samples after applying HealthKit delta"
933
+                return "\(distribution.totalCount) samples after applying \(distribution.deltaEventCount) HealthKit delta events"
934 934
             case .initialImport:
935 935
                 return "\(distribution.totalCount) samples from full HealthKit import"
936 936
             }
@@ -1030,6 +1030,7 @@ final class HealthKitService {
1030 1030
             recordArchiveData: recordArchiveData
1031 1031
         )
1032 1032
         result.captureMode = distribution.captureMode.diagnosticValue
1033
+        result.deltaEventCount = distribution.deltaEventCount
1033 1034
         result.timingBreakdown = timingBreakdown
1034 1035
         await persistTypeCaptureState(from: result, observationID: archiveObservationID)
1035 1036
         return result
@@ -1177,6 +1178,7 @@ final class HealthKitService {
1177 1178
                     yearlyCounts: unchanged.yearlyCounts,
1178 1179
                     recordArchiveData: unchanged.recordArchiveData,
1179 1180
                     captureMode: .unchangedDelta,
1181
+                    deltaEventCount: 0,
1180 1182
                     timingBreakdown: captureTimings.importBreakdown
1181 1183
                 )
1182 1184
             }
@@ -1344,6 +1346,7 @@ final class HealthKitService {
1344 1346
                 yearlyCounts: nil,
1345 1347
                 recordArchiveData: nil,
1346 1348
                 captureMode: .initialImport,
1349
+                deltaEventCount: 0,
1347 1350
                 timingBreakdown: captureTimings.importBreakdown
1348 1351
             )
1349 1352
         }
@@ -1368,6 +1371,7 @@ final class HealthKitService {
1368 1371
             yearlyCounts: nil,
1369 1372
             recordArchiveData: nil,
1370 1373
             captureMode: .delta,
1374
+            deltaEventCount: processedEventCount,
1371 1375
             timingBreakdown: captureTimings.importBreakdown
1372 1376
         )
1373 1377
     }
@@ -1535,6 +1539,7 @@ final class HealthKitService {
1535 1539
                 yearlyCounts: nil,
1536 1540
                 recordArchiveData: nil,
1537 1541
                 captureMode: .initialImport,
1542
+                deltaEventCount: 0,
1538 1543
                 timingBreakdown: captureTimings.importBreakdown
1539 1544
             )
1540 1545
         }
@@ -1563,6 +1568,7 @@ final class HealthKitService {
1563 1568
             yearlyCounts: yearMap,
1564 1569
             recordArchiveData: recordArchiveData,
1565 1570
             captureMode: .initialImport,
1571
+            deltaEventCount: processedEventCount,
1566 1572
             timingBreakdown: captureTimings.importBreakdown
1567 1573
         )
1568 1574
     }
@@ -2479,6 +2485,7 @@ private struct SampleDistribution: Sendable {
2479 2485
     let yearlyCounts: [Int: Int]?
2480 2486
     let recordArchiveData: Data?
2481 2487
     let captureMode: CaptureMode
2488
+    let deltaEventCount: Int
2482 2489
     let timingBreakdown: ImportTimingBreakdown
2483 2490
 }
2484 2491
 
@@ -2687,6 +2694,7 @@ private struct PreviousDistributionState: Sendable {
2687 2694
             yearlyCounts: yearlyCounts,
2688 2695
             recordArchiveData: recordArchiveData,
2689 2696
             captureMode: .unchangedDelta,
2697
+            deltaEventCount: 0,
2690 2698
             timingBreakdown: .zero
2691 2699
         )
2692 2700
     }
@@ -2760,6 +2768,7 @@ private struct TypeCountFetchResult: Sendable {
2760 2768
     let records: [RecordData]
2761 2769
     let recordArchiveData: Data?
2762 2770
     var captureMode: String = "unavailable"
2771
+    var deltaEventCount: Int = 0
2763 2772
     var timeoutConfiguredSeconds: TimeInterval = 0
2764 2773
     var totalElapsedSeconds: TimeInterval = 0
2765 2774
     var timeoutMode: String = "default"
@@ -2858,7 +2867,8 @@ private extension SnapshotFetchProgress {
2858 2867
             timeoutCount: result.timeoutCount,
2859 2868
             successCount: result.successCount,
2860 2869
             timingBreakdown: result.timingBreakdown,
2861
-            captureMode: result.captureMode
2870
+            captureMode: result.captureMode,
2871
+            deltaEventCount: result.deltaEventCount
2862 2872
         )
2863 2873
     }
2864 2874
 
+4 -1
HealthProbe/Utilities/SnapshotFetchProgress.swift
@@ -114,6 +114,7 @@ final class SnapshotFetchProgress {
114 114
         var successCount: Int = 0
115 115
         var timingBreakdown: ImportTimingBreakdown = .zero
116 116
         var captureMode: String = "unavailable"
117
+        var deltaEventCount: Int = 0
117 118
         var blockProgress: String = ""
118 119
         var blockElapsedSeconds: TimeInterval = 0
119 120
         var blockSamplesPerSecond: Double = 0
@@ -215,7 +216,8 @@ final class SnapshotFetchProgress {
215 216
         timeoutCount: Int,
216 217
         successCount: Int,
217 218
         timingBreakdown: ImportTimingBreakdown,
218
-        captureMode: String = "unavailable"
219
+        captureMode: String = "unavailable",
220
+        deltaEventCount: Int = 0
219 221
     ) {
220 222
         let index = visibleTypeIndex(for: id)
221 223
         types[index].quality = quality
@@ -236,6 +238,7 @@ final class SnapshotFetchProgress {
236 238
         types[index].successCount = successCount
237 239
         types[index].timingBreakdown = timingBreakdown
238 240
         types[index].captureMode = captureMode
241
+        types[index].deltaEventCount = deltaEventCount
239 242
     }
240 243
 
241 244
     func updateTimeoutProfile(
+3 -0
HealthProbe/Views/Dashboard/DashboardView.swift
@@ -384,6 +384,7 @@ struct DashboardView: View {
384 384
         }
385 385
         let captureModeCounts = Dictionary(grouping: progress.types, by: \.captureMode)
386 386
             .mapValues(\.count)
387
+        let summedDeltaEventCount = progress.types.reduce(0) { $0 + max(0, $1.deltaEventCount) }
387 388
         lines.append("SummedMetricTotalElapsed: \(formatDuration(summedMetricTotalElapsed))")
388 389
         lines.append("SummedFetchElapsed: \(formatDuration(aggregateTiming.fetchElapsedSeconds))")
389 390
         lines.append("SummedProcessingElapsed: \(formatDuration(aggregateTiming.processingElapsedSeconds))")
@@ -391,6 +392,7 @@ struct DashboardView: View {
391 392
         lines.append("SummedFinalizeElapsed: \(formatDuration(aggregateTiming.finalizeElapsedSeconds))")
392 393
         lines.append("SummedResidualElapsed: \(formatDuration(summedMetricResidualElapsed))")
393 394
         lines.append("CaptureModes: unchangedDelta=\(captureModeCounts["unchangedDelta", default: 0]), delta=\(captureModeCounts["delta", default: 0]), initialImport=\(captureModeCounts["initialImport", default: 0]), unavailable=\(captureModeCounts["unavailable", default: 0])")
395
+        lines.append("DeltaEvents: \(summedDeltaEventCount)")
394 396
         lines.append("")
395 397
         lines.append(failedLines)
396 398
 
@@ -422,6 +424,7 @@ struct DashboardView: View {
422 424
             lines.append("  quality: \(type.quality)")
423 425
             lines.append("  count: \(type.recordCount)")
424 426
             lines.append("  captureMode: \(type.captureMode)")
427
+            lines.append("  deltaEvents: \(type.deltaEventCount)")
425 428
             lines.append("  timeoutMode: \(type.timeoutMode)")
426 429
             lines.append("  timeoutConfigured: \(formatDuration(type.timeoutConfiguredSeconds))")
427 430
             lines.append("  lastSuccessfulElapsed: \(type.lastSuccessfulElapsed > 0 ? formatDuration(type.lastSuccessfulElapsed) : "none")")