@@ -582,6 +582,7 @@ rows exist". |
||
| 582 | 582 |
| 2026-06-03 | pending | Persist HealthKit capture anchors in SQLite. | A later no-delta attempt still reimported full high-volume types (`Records: 2,646,590`, `WallClock: 45.8s`, Heart Rate `922,521`, Active Energy `346,473`) because incremental anchors were still read from legacy SwiftData `TypeCount`, while the refactored app now chains observations from SQLite. SQLite now stores per-type capture state (`anchor_data`, count, content hash, yearly counts, range) and the import path uses it when legacy `TypeCount` state is missing. Expected signal: the first run after this change may seed anchors; the following no-delta run should report empty HealthKit delta pages instead of full `record_import` counts for large stable types. | |
| 583 | 583 |
| 2026-06-03 | pending | Clarify capture mode in record-import diagnostics. | Two full-profile repeated snapshots after SQLite capture-state persistence completed successfully with stable checksum and identical record count (`2,646,613`). The first ran in `6.3s` with `SummedProcessingElapsed: 0.0s`, `SummedInsertElapsed: 0.0s`, `SummedFinalizeElapsed: 2.6s`; the second ran in `6.4s` with `0.0s` processing/insert and `2.7s` finalize. Heart Rate `922,526` and Active Energy `346,478` each completed around `0.2s`, proving the heavy full reimport path was avoided. The report wording still said "`N` samples in 1 anchored segment", which is ambiguous for inherited unchanged summaries; diagnostics now label unchanged empty-delta, delta-apply, and full-import modes explicitly. | |
| 584 | 584 |
| 2026-06-03 | pending | Add capture-mode summary to diagnostics. | Repeated full-profile captures rarely produce a perfect no-delta report because at least one metric can change between manual runs. Diagnostic reports now include aggregate `CaptureModes` counts plus per-metric `captureMode`, so comparisons can separate unchanged empty-delta metrics from delta-applied metrics and full imports without manually reading every `record_import` line. Expected signal: stable checksum plus high `unchangedDelta` count and zero summed processing/insert confirms the fast path even when a few metrics changed. | |
| 585 |
+| 2026-06-03 | pending | Add delta-event counts to diagnostics. | A full-profile follow-up completed in `47.4s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=115, delta=12, initialImport=0`, `SummedProcessingElapsed: 25.9s`, `SummedInsertElapsed: 0.2s`, and `SummedFinalizeElapsed: 16.0s`. This confirms anchors work and no full import ran. Remaining cost is delta application for large metrics: Heart Rate `23.5s` total (`14.1s` processing, `8.8s` finalize), Active Energy `7.1s`, and Basal Energy `6.0s`. Diagnostics now report aggregate/per-metric `DeltaEvents` so future logs can separate true HealthKit delta size from the final visible record count. | |
|
| 585 | 586 |
|
| 586 | 587 |
## Current Diagnosis |
| 587 | 588 |
|
@@ -633,6 +634,11 @@ The likely bottleneck is per-row SQLite work: |
||
| 633 | 634 |
identical record count, zero summed processing/insert, and sub-second |
| 634 | 635 |
high-volume metric timings are acceptable evidence that the expensive full |
| 635 | 636 |
import path was skipped. |
| 637 |
+- Capture modes now show the heavy full import path is avoided, but a small |
|
| 638 |
+ HealthKit delta on a high-volume type can still force expensive local |
|
| 639 |
+ reconstruction of legacy compact record archives and type hashes. Future logs |
|
| 640 |
+ should compare `DeltaEvents` against processing/finalize time before changing |
|
| 641 |
+ page sizes or HealthKit query strategy. |
|
| 636 | 642 |
|
| 637 | 643 |
## Open Issues / Observations |
| 638 | 644 |
|
@@ -671,7 +677,10 @@ Prioritize experiments in this order: |
||
| 671 | 677 |
`CaptureModes`, and high-volume type timings. Treat stable checksum, a high |
| 672 | 678 |
`unchangedDelta` count, and zero processing/insert as the main unchanged-path |
| 673 | 679 |
signal. |
| 674 |
-4. Add or inspect timing around per-record processing for changed high-volume metrics, especially Heart Rate, to separate sample DTO/fingerprint work from SQLite idempotency checks. |
|
| 680 |
+4. Use `DeltaEvents` to quantify changed high-volume metrics, especially Heart |
|
| 681 |
+ Rate, Active Energy, and Basal Energy. If delta events are small while |
|
| 682 |
+ processing/finalize remain large, optimize legacy compact archive/hash |
|
| 683 |
+ reconstruction rather than HealthKit fetch or SQLite insert. |
|
| 675 | 684 |
5. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`. |
| 676 | 685 |
6. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans. |
| 677 | 686 |
7. Profile whether index maintenance dominates first-import insert cost. |
@@ -930,7 +930,7 @@ final class HealthKitService {
|
||
| 930 | 930 |
case .unchangedDelta: |
| 931 | 931 |
return "\(distribution.totalCount) unchanged samples via empty HealthKit delta" |
| 932 | 932 |
case .delta: |
| 933 |
- return "\(distribution.totalCount) samples after applying HealthKit delta" |
|
| 933 |
+ return "\(distribution.totalCount) samples after applying \(distribution.deltaEventCount) HealthKit delta events" |
|
| 934 | 934 |
case .initialImport: |
| 935 | 935 |
return "\(distribution.totalCount) samples from full HealthKit import" |
| 936 | 936 |
} |
@@ -1030,6 +1030,7 @@ final class HealthKitService {
|
||
| 1030 | 1030 |
recordArchiveData: recordArchiveData |
| 1031 | 1031 |
) |
| 1032 | 1032 |
result.captureMode = distribution.captureMode.diagnosticValue |
| 1033 |
+ result.deltaEventCount = distribution.deltaEventCount |
|
| 1033 | 1034 |
result.timingBreakdown = timingBreakdown |
| 1034 | 1035 |
await persistTypeCaptureState(from: result, observationID: archiveObservationID) |
| 1035 | 1036 |
return result |
@@ -1177,6 +1178,7 @@ final class HealthKitService {
|
||
| 1177 | 1178 |
yearlyCounts: unchanged.yearlyCounts, |
| 1178 | 1179 |
recordArchiveData: unchanged.recordArchiveData, |
| 1179 | 1180 |
captureMode: .unchangedDelta, |
| 1181 |
+ deltaEventCount: 0, |
|
| 1180 | 1182 |
timingBreakdown: captureTimings.importBreakdown |
| 1181 | 1183 |
) |
| 1182 | 1184 |
} |
@@ -1344,6 +1346,7 @@ final class HealthKitService {
|
||
| 1344 | 1346 |
yearlyCounts: nil, |
| 1345 | 1347 |
recordArchiveData: nil, |
| 1346 | 1348 |
captureMode: .initialImport, |
| 1349 |
+ deltaEventCount: 0, |
|
| 1347 | 1350 |
timingBreakdown: captureTimings.importBreakdown |
| 1348 | 1351 |
) |
| 1349 | 1352 |
} |
@@ -1368,6 +1371,7 @@ final class HealthKitService {
|
||
| 1368 | 1371 |
yearlyCounts: nil, |
| 1369 | 1372 |
recordArchiveData: nil, |
| 1370 | 1373 |
captureMode: .delta, |
| 1374 |
+ deltaEventCount: processedEventCount, |
|
| 1371 | 1375 |
timingBreakdown: captureTimings.importBreakdown |
| 1372 | 1376 |
) |
| 1373 | 1377 |
} |
@@ -1535,6 +1539,7 @@ final class HealthKitService {
|
||
| 1535 | 1539 |
yearlyCounts: nil, |
| 1536 | 1540 |
recordArchiveData: nil, |
| 1537 | 1541 |
captureMode: .initialImport, |
| 1542 |
+ deltaEventCount: 0, |
|
| 1538 | 1543 |
timingBreakdown: captureTimings.importBreakdown |
| 1539 | 1544 |
) |
| 1540 | 1545 |
} |
@@ -1563,6 +1568,7 @@ final class HealthKitService {
|
||
| 1563 | 1568 |
yearlyCounts: yearMap, |
| 1564 | 1569 |
recordArchiveData: recordArchiveData, |
| 1565 | 1570 |
captureMode: .initialImport, |
| 1571 |
+ deltaEventCount: processedEventCount, |
|
| 1566 | 1572 |
timingBreakdown: captureTimings.importBreakdown |
| 1567 | 1573 |
) |
| 1568 | 1574 |
} |
@@ -2479,6 +2485,7 @@ private struct SampleDistribution: Sendable {
|
||
| 2479 | 2485 |
let yearlyCounts: [Int: Int]? |
| 2480 | 2486 |
let recordArchiveData: Data? |
| 2481 | 2487 |
let captureMode: CaptureMode |
| 2488 |
+ let deltaEventCount: Int |
|
| 2482 | 2489 |
let timingBreakdown: ImportTimingBreakdown |
| 2483 | 2490 |
} |
| 2484 | 2491 |
|
@@ -2687,6 +2694,7 @@ private struct PreviousDistributionState: Sendable {
|
||
| 2687 | 2694 |
yearlyCounts: yearlyCounts, |
| 2688 | 2695 |
recordArchiveData: recordArchiveData, |
| 2689 | 2696 |
captureMode: .unchangedDelta, |
| 2697 |
+ deltaEventCount: 0, |
|
| 2690 | 2698 |
timingBreakdown: .zero |
| 2691 | 2699 |
) |
| 2692 | 2700 |
} |
@@ -2760,6 +2768,7 @@ private struct TypeCountFetchResult: Sendable {
|
||
| 2760 | 2768 |
let records: [RecordData] |
| 2761 | 2769 |
let recordArchiveData: Data? |
| 2762 | 2770 |
var captureMode: String = "unavailable" |
| 2771 |
+ var deltaEventCount: Int = 0 |
|
| 2763 | 2772 |
var timeoutConfiguredSeconds: TimeInterval = 0 |
| 2764 | 2773 |
var totalElapsedSeconds: TimeInterval = 0 |
| 2765 | 2774 |
var timeoutMode: String = "default" |
@@ -2858,7 +2867,8 @@ private extension SnapshotFetchProgress {
|
||
| 2858 | 2867 |
timeoutCount: result.timeoutCount, |
| 2859 | 2868 |
successCount: result.successCount, |
| 2860 | 2869 |
timingBreakdown: result.timingBreakdown, |
| 2861 |
- captureMode: result.captureMode |
|
| 2870 |
+ captureMode: result.captureMode, |
|
| 2871 |
+ deltaEventCount: result.deltaEventCount |
|
| 2862 | 2872 |
) |
| 2863 | 2873 |
} |
| 2864 | 2874 |
|
@@ -114,6 +114,7 @@ final class SnapshotFetchProgress {
|
||
| 114 | 114 |
var successCount: Int = 0 |
| 115 | 115 |
var timingBreakdown: ImportTimingBreakdown = .zero |
| 116 | 116 |
var captureMode: String = "unavailable" |
| 117 |
+ var deltaEventCount: Int = 0 |
|
| 117 | 118 |
var blockProgress: String = "" |
| 118 | 119 |
var blockElapsedSeconds: TimeInterval = 0 |
| 119 | 120 |
var blockSamplesPerSecond: Double = 0 |
@@ -215,7 +216,8 @@ final class SnapshotFetchProgress {
|
||
| 215 | 216 |
timeoutCount: Int, |
| 216 | 217 |
successCount: Int, |
| 217 | 218 |
timingBreakdown: ImportTimingBreakdown, |
| 218 |
- captureMode: String = "unavailable" |
|
| 219 |
+ captureMode: String = "unavailable", |
|
| 220 |
+ deltaEventCount: Int = 0 |
|
| 219 | 221 |
) {
|
| 220 | 222 |
let index = visibleTypeIndex(for: id) |
| 221 | 223 |
types[index].quality = quality |
@@ -236,6 +238,7 @@ final class SnapshotFetchProgress {
|
||
| 236 | 238 |
types[index].successCount = successCount |
| 237 | 239 |
types[index].timingBreakdown = timingBreakdown |
| 238 | 240 |
types[index].captureMode = captureMode |
| 241 |
+ types[index].deltaEventCount = deltaEventCount |
|
| 239 | 242 |
} |
| 240 | 243 |
|
| 241 | 244 |
func updateTimeoutProfile( |
@@ -384,6 +384,7 @@ struct DashboardView: View {
|
||
| 384 | 384 |
} |
| 385 | 385 |
let captureModeCounts = Dictionary(grouping: progress.types, by: \.captureMode) |
| 386 | 386 |
.mapValues(\.count) |
| 387 |
+ let summedDeltaEventCount = progress.types.reduce(0) { $0 + max(0, $1.deltaEventCount) }
|
|
| 387 | 388 |
lines.append("SummedMetricTotalElapsed: \(formatDuration(summedMetricTotalElapsed))")
|
| 388 | 389 |
lines.append("SummedFetchElapsed: \(formatDuration(aggregateTiming.fetchElapsedSeconds))")
|
| 389 | 390 |
lines.append("SummedProcessingElapsed: \(formatDuration(aggregateTiming.processingElapsedSeconds))")
|
@@ -391,6 +392,7 @@ struct DashboardView: View {
|
||
| 391 | 392 |
lines.append("SummedFinalizeElapsed: \(formatDuration(aggregateTiming.finalizeElapsedSeconds))")
|
| 392 | 393 |
lines.append("SummedResidualElapsed: \(formatDuration(summedMetricResidualElapsed))")
|
| 393 | 394 |
lines.append("CaptureModes: unchangedDelta=\(captureModeCounts["unchangedDelta", default: 0]), delta=\(captureModeCounts["delta", default: 0]), initialImport=\(captureModeCounts["initialImport", default: 0]), unavailable=\(captureModeCounts["unavailable", default: 0])")
|
| 395 |
+ lines.append("DeltaEvents: \(summedDeltaEventCount)")
|
|
| 394 | 396 |
lines.append("")
|
| 395 | 397 |
lines.append(failedLines) |
| 396 | 398 |
|
@@ -422,6 +424,7 @@ struct DashboardView: View {
|
||
| 422 | 424 |
lines.append(" quality: \(type.quality)")
|
| 423 | 425 |
lines.append(" count: \(type.recordCount)")
|
| 424 | 426 |
lines.append(" captureMode: \(type.captureMode)")
|
| 427 |
+ lines.append(" deltaEvents: \(type.deltaEventCount)")
|
|
| 425 | 428 |
lines.append(" timeoutMode: \(type.timeoutMode)")
|
| 426 | 429 |
lines.append(" timeoutConfigured: \(formatDuration(type.timeoutConfiguredSeconds))")
|
| 427 | 430 |
lines.append(" lastSuccessfulElapsed: \(type.lastSuccessfulElapsed > 0 ? formatDuration(type.lastSuccessfulElapsed) : "none")")
|