Clarify import capture mode diagnostics · a607ed7

Clarify import capture mode diagnostics
Browse files

bogdan committed 4 days ago

main

1 parent bb61aa6

commit a607ed7

Showing 2 changed files with 30 additions and 9 deletions

+11 -5

HealthProbe/Doc/04-project/Import-Optimization-Log.md

@@ -580,6 +580,7 @@ rows exist".
 | 2026-06-03 | pending | Preserve completed import diagnostics and render large diagnostic reports lazily. | A copied full-profile reimport report completed successfully: `Types: 127/127 processed`, `Records: 2,646,527`, `WallClock: 40.3s`, `SummedProcessingElapsed: 21.7s`, `SummedFinalizeElapsed: 15.1s`, `SummedFetchElapsed: 1.7s`, `SummedInsertElapsed: 0.1s`, and `0` degraded metrics. The Diagnostics sheet copied the report text but displayed a blank body for the huge report; completed/partial/review reports are now persisted under Application Support and displayed in chunked lazy text blocks. |
 | 2026-06-03 | pending | Fast-path unchanged verification after an empty HealthKit delta page. | The successful full-profile reimport spent `15.1s` in finalize despite `0.1s` insert time; Heart Rate alone spent `9.0s` finalizing. The empty-delta path now calls an explicit unchanged-verification store method that copies the previous type summary and daily aggregates without first scanning sample events to prove zero changes. Expected signal: repeated no-delta/full-profile `SummedFinalizeElapsed` and Heart Rate finalize time drop sharply. |
 | 2026-06-03 | pending | Persist HealthKit capture anchors in SQLite. | A later no-delta attempt still reimported full high-volume types (`Records: 2,646,590`, `WallClock: 45.8s`, Heart Rate `922,521`, Active Energy `346,473`) because incremental anchors were still read from legacy SwiftData `TypeCount`, while the refactored app now chains observations from SQLite. SQLite now stores per-type capture state (`anchor_data`, count, content hash, yearly counts, range) and the import path uses it when legacy `TypeCount` state is missing. Expected signal: the first run after this change may seed anchors; the following no-delta run should report empty HealthKit delta pages instead of full `record_import` counts for large stable types. |
+| 2026-06-03 | pending | Clarify capture mode in record-import diagnostics. | Two full-profile repeated snapshots after SQLite capture-state persistence completed successfully with stable checksum and identical record count (`2,646,613`). The first ran in `6.3s` with `SummedProcessingElapsed: 0.0s`, `SummedInsertElapsed: 0.0s`, `SummedFinalizeElapsed: 2.6s`; the second ran in `6.4s` with `0.0s` processing/insert and `2.7s` finalize. Heart Rate `922,526` and Active Energy `346,478` each completed around `0.2s`, proving the heavy full reimport path was avoided. The report wording still said "`N` samples in 1 anchored segment", which is ambiguous for inherited unchanged summaries; diagnostics now label unchanged empty-delta, delta-apply, and full-import modes explicitly. |
 
 ## Current Diagnosis
 
@@ -626,6 +627,11 @@ The likely bottleneck is per-row SQLite work:
   `TypeCount.distributionBinsData`. Otherwise refactored SQLite-driven snapshots
   lose their HealthKit query anchor across launches/refactors and every
   high-volume metric becomes a full scan again.
+- No-delta reports are hard to obtain once the full HealthKit profile is enabled
+  because some metric often changes between manual captures. Identical checksum,
+  identical record count, zero summed processing/insert, and sub-second
+  high-volume metric timings are acceptable evidence that the expensive full
+  import path was skipped.
 
 ## Open Issues / Observations
 
@@ -658,11 +664,11 @@ Prioritize experiments in this order:
    diagnostic report does not freeze the app, and Dashboard/Snapshots show the
    latest observation from SQLite. Also verify Snapshot detail and Data Types
    show per-type summaries without a manual cache rebuild.
-3. Run two repeated no-delta benchmarks after SQLite capture-state persistence:
-   first to seed missing SQLite anchors if needed, second to confirm `record_import`
-   for large stable types uses empty delta pages. Compare `WallClockDuration`,
-   `SummedProcessingElapsed`, `SummedFinalizeElapsed`, Heart Rate count/timing,
-   and Active Energy count/timing.
+3. Continue repeated full-profile captures, but do not require perfect no-delta.
+   Compare `snapshotChecksum`, `Records`, `WallClockDuration`,
+   `SummedProcessingElapsed`, `SummedInsertElapsed`, `SummedFinalizeElapsed`,
+   and high-volume type timings. Treat stable checksum + zero processing/insert
+   as the main unchanged-path signal.
 4. Add or inspect timing around per-record processing for changed high-volume metrics, especially Heart Rate, to separate sample DTO/fingerprint work from SQLite idempotency checks.
 5. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
 6. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.


+19 -4

HealthProbe/Services/HealthKitService.swift

View

@@ -926,12 +926,14 @@ final class HealthKitService {
                 progress: progress
             )
         } resultDescription: { distribution in
-            if distribution.recordArchiveData == nil,
-               distribution.records.isEmpty,
-               distribution.totalCount > 0 {
+            switch distribution.captureMode {
+            case .unchangedDelta:
                 return "\(distribution.totalCount) unchanged samples via empty HealthKit delta"
+            case .delta:
+                return "\(distribution.totalCount) samples after applying HealthKit delta"
+            case .initialImport:
+                return "\(distribution.totalCount) samples from full HealthKit import"
             }
-            return "\(distribution.totalCount) samples in \(distribution.bins.count) anchored segment(s)"
         }
         apiCalls.insert(distributionResult.apiCall, at: 0)
 
@@ -1173,6 +1175,7 @@ final class HealthKitService {
                     contentHash: unchanged.contentHash,
                     yearlyCounts: unchanged.yearlyCounts,
                     recordArchiveData: unchanged.recordArchiveData,
+                    captureMode: .unchangedDelta,
                     timingBreakdown: captureTimings.importBreakdown
                 )
             }
@@ -1339,6 +1342,7 @@ final class HealthKitService {
                 contentHash: nil,
                 yearlyCounts: nil,
                 recordArchiveData: nil,
+                captureMode: .initialImport,
                 timingBreakdown: captureTimings.importBreakdown
             )
         }
@@ -1362,6 +1366,7 @@ final class HealthKitService {
             contentHash: contentHash,
             yearlyCounts: nil,
             recordArchiveData: nil,
+            captureMode: .delta,
             timingBreakdown: captureTimings.importBreakdown
         )
     }
@@ -1528,6 +1533,7 @@ final class HealthKitService {
                 contentHash: nil,
                 yearlyCounts: nil,
                 recordArchiveData: nil,
+                captureMode: .initialImport,
                 timingBreakdown: captureTimings.importBreakdown
             )
         }
@@ -1555,6 +1561,7 @@ final class HealthKitService {
             contentHash: contentHash,
             yearlyCounts: yearMap,
             recordArchiveData: recordArchiveData,
+            captureMode: .initialImport,
             timingBreakdown: captureTimings.importBreakdown
         )
     }
@@ -2440,6 +2447,12 @@ final class HealthKitService {
 }
 
 private struct SampleDistribution: Sendable {
+    enum CaptureMode: Sendable {
+        case initialImport
+        case delta
+        case unchangedDelta
+    }
+
     struct Bin: Sendable {
         let start: Date
         let end: Date
@@ -2453,6 +2466,7 @@ private struct SampleDistribution: Sendable {
     let contentHash: String?
     let yearlyCounts: [Int: Int]?
     let recordArchiveData: Data?
+    let captureMode: CaptureMode
     let timingBreakdown: ImportTimingBreakdown
 }
 
@@ -2660,6 +2674,7 @@ private struct PreviousDistributionState: Sendable {
             contentHash: effectiveContentHash,
             yearlyCounts: yearlyCounts,
             recordArchiveData: recordArchiveData,
+            captureMode: .unchangedDelta,
             timingBreakdown: .zero
         )
     }


	@@ -926,12 +926,14 @@ final class HealthKitService {
926	926	progress: progress
927	927	)
928	928	} resultDescription: { distribution in
929		- if distribution.recordArchiveData == nil,
930		- distribution.records.isEmpty,
931		- distribution.totalCount > 0 {
	929	+ switch distribution.captureMode {
	930	+ case .unchangedDelta:
932	931	return "\(distribution.totalCount) unchanged samples via empty HealthKit delta"
	932	+ case .delta:
	933	+ return "\(distribution.totalCount) samples after applying HealthKit delta"
	934	+ case .initialImport:
	935	+ return "\(distribution.totalCount) samples from full HealthKit import"
933	936	}
934		- return "\(distribution.totalCount) samples in \(distribution.bins.count) anchored segment(s)"
935	937	}
936	938	apiCalls.insert(distributionResult.apiCall, at: 0)
937	939
	@@ -1173,6 +1175,7 @@ final class HealthKitService {
1173	1175	contentHash: unchanged.contentHash,
1174	1176	yearlyCounts: unchanged.yearlyCounts,
1175	1177	recordArchiveData: unchanged.recordArchiveData,
	1178	+ captureMode: .unchangedDelta,
1176	1179	timingBreakdown: captureTimings.importBreakdown
1177	1180	)
1178	1181	}
	@@ -1339,6 +1342,7 @@ final class HealthKitService {
1339	1342	contentHash: nil,
1340	1343	yearlyCounts: nil,
1341	1344	recordArchiveData: nil,
	1345	+ captureMode: .initialImport,
1342	1346	timingBreakdown: captureTimings.importBreakdown
1343	1347	)
1344	1348	}
	@@ -1362,6 +1366,7 @@ final class HealthKitService {
1362	1366	contentHash: contentHash,
1363	1367	yearlyCounts: nil,
1364	1368	recordArchiveData: nil,
	1369	+ captureMode: .delta,
1365	1370	timingBreakdown: captureTimings.importBreakdown
1366	1371	)
1367	1372	}
	@@ -1528,6 +1533,7 @@ final class HealthKitService {
1528	1533	contentHash: nil,
1529	1534	yearlyCounts: nil,
1530	1535	recordArchiveData: nil,
	1536	+ captureMode: .initialImport,
1531	1537	timingBreakdown: captureTimings.importBreakdown
1532	1538	)
1533	1539	}
	@@ -1555,6 +1561,7 @@ final class HealthKitService {
1555	1561	contentHash: contentHash,
1556	1562	yearlyCounts: yearMap,
1557	1563	recordArchiveData: recordArchiveData,
	1564	+ captureMode: .initialImport,
1558	1565	timingBreakdown: captureTimings.importBreakdown
1559	1566	)
1560	1567	}
	@@ -2440,6 +2447,12 @@ final class HealthKitService {
2440	2447	}
2441	2448
2442	2449	private struct SampleDistribution: Sendable {
	2450	+ enum CaptureMode: Sendable {
	2451	+ case initialImport
	2452	+ case delta
	2453	+ case unchangedDelta
	2454	+ }
	2455	+
2443	2456	struct Bin: Sendable {
2444	2457	let start: Date
2445	2458	let end: Date
	@@ -2453,6 +2466,7 @@ private struct SampleDistribution: Sendable {
2453	2466	let contentHash: String?
2454	2467	let yearlyCounts: [Int: Int]?
2455	2468	let recordArchiveData: Data?
	2469	+ let captureMode: CaptureMode
2456	2470	let timingBreakdown: ImportTimingBreakdown
2457	2471	}
2458	2472
	@@ -2660,6 +2674,7 @@ private struct PreviousDistributionState: Sendable {
2660	2674	contentHash: effectiveContentHash,
2661	2675	yearlyCounts: yearlyCounts,
2662	2676	recordArchiveData: recordArchiveData,
	2677	+ captureMode: .unchangedDelta,
2663	2678	timingBreakdown: .zero
2664	2679	)
2665	2680	}

	@@ -580,6 +580,7 @@ rows exist".
580	580	\| 2026-06-03 \| pending \| Preserve completed import diagnostics and render large diagnostic reports lazily. \| A copied full-profile reimport report completed successfully: `Types: 127/127 processed`, `Records: 2,646,527`, `WallClock: 40.3s`, `SummedProcessingElapsed: 21.7s`, `SummedFinalizeElapsed: 15.1s`, `SummedFetchElapsed: 1.7s`, `SummedInsertElapsed: 0.1s`, and `0` degraded metrics. The Diagnostics sheet copied the report text but displayed a blank body for the huge report; completed/partial/review reports are now persisted under Application Support and displayed in chunked lazy text blocks. \|
581	581	\| 2026-06-03 \| pending \| Fast-path unchanged verification after an empty HealthKit delta page. \| The successful full-profile reimport spent `15.1s` in finalize despite `0.1s` insert time; Heart Rate alone spent `9.0s` finalizing. The empty-delta path now calls an explicit unchanged-verification store method that copies the previous type summary and daily aggregates without first scanning sample events to prove zero changes. Expected signal: repeated no-delta/full-profile `SummedFinalizeElapsed` and Heart Rate finalize time drop sharply. \|
582	582	\| 2026-06-03 \| pending \| Persist HealthKit capture anchors in SQLite. \| A later no-delta attempt still reimported full high-volume types (`Records: 2,646,590`, `WallClock: 45.8s`, Heart Rate `922,521`, Active Energy `346,473`) because incremental anchors were still read from legacy SwiftData `TypeCount`, while the refactored app now chains observations from SQLite. SQLite now stores per-type capture state (`anchor_data`, count, content hash, yearly counts, range) and the import path uses it when legacy `TypeCount` state is missing. Expected signal: the first run after this change may seed anchors; the following no-delta run should report empty HealthKit delta pages instead of full `record_import` counts for large stable types. \|
	583	+\| 2026-06-03 \| pending \| Clarify capture mode in record-import diagnostics. \| Two full-profile repeated snapshots after SQLite capture-state persistence completed successfully with stable checksum and identical record count (`2,646,613`). The first ran in `6.3s` with `SummedProcessingElapsed: 0.0s`, `SummedInsertElapsed: 0.0s`, `SummedFinalizeElapsed: 2.6s`; the second ran in `6.4s` with `0.0s` processing/insert and `2.7s` finalize. Heart Rate `922,526` and Active Energy `346,478` each completed around `0.2s`, proving the heavy full reimport path was avoided. The report wording still said "`N` samples in 1 anchored segment", which is ambiguous for inherited unchanged summaries; diagnostics now label unchanged empty-delta, delta-apply, and full-import modes explicitly. \|
583	584
584	585	## Current Diagnosis
585	586
	@@ -626,6 +627,11 @@ The likely bottleneck is per-row SQLite work:
626	627	`TypeCount.distributionBinsData`. Otherwise refactored SQLite-driven snapshots
627	628	lose their HealthKit query anchor across launches/refactors and every
628	629	high-volume metric becomes a full scan again.
	630	+- No-delta reports are hard to obtain once the full HealthKit profile is enabled
	631	+ because some metric often changes between manual captures. Identical checksum,
	632	+ identical record count, zero summed processing/insert, and sub-second
	633	+ high-volume metric timings are acceptable evidence that the expensive full
	634	+ import path was skipped.
629	635
630	636	## Open Issues / Observations
631	637
	@@ -658,11 +664,11 @@ Prioritize experiments in this order:
658	664	diagnostic report does not freeze the app, and Dashboard/Snapshots show the
659	665	latest observation from SQLite. Also verify Snapshot detail and Data Types
660	666	show per-type summaries without a manual cache rebuild.
661		-3. Run two repeated no-delta benchmarks after SQLite capture-state persistence:
662		- first to seed missing SQLite anchors if needed, second to confirm `record_import`
663		- for large stable types uses empty delta pages. Compare `WallClockDuration`,
664		- `SummedProcessingElapsed`, `SummedFinalizeElapsed`, Heart Rate count/timing,
665		- and Active Energy count/timing.
	667	+3. Continue repeated full-profile captures, but do not require perfect no-delta.
	668	+ Compare `snapshotChecksum`, `Records`, `WallClockDuration`,
	669	+ `SummedProcessingElapsed`, `SummedInsertElapsed`, `SummedFinalizeElapsed`,
	670	+ and high-volume type timings. Treat stable checksum + zero processing/insert
	671	+ as the main unchanged-path signal.
666	672	4. Add or inspect timing around per-record processing for changed high-volume metrics, especially Heart Rate, to separate sample DTO/fingerprint work from SQLite idempotency checks.
667	673	5. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
668	674	6. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.