Relax import paging for ordinary metrics · aa476be

Relax import paging for ordinary metrics
Browse files

bogdan committed a week ago

main

1 parent 708482d

commit aa476be

Showing 2 changed files with 31 additions and 15 deletions

+1 -1

HealthProbe/Doc/04-project/IMPLEMENTATION_STATUS.md

@@ -24,7 +24,7 @@ There are no real deployments, only test installations. Existing prototype datab
 | Area | Current Status | Target / Next Work |
 |------|----------------|--------------------|
 | Product docs | Updated | Keep `HealthProbe/Doc/README.md` as canonical index |
-| HealthKit capture | Capture now opens one archive observation per user-visible snapshot, attaches HealthKit pages, deleted-object evidence, and type verification to that observation id before finishing it, no longer aborts initial full-history imports after a fixed 30-minute wall-clock cap while page-level HealthKit timeouts remain in place, defers grouped observation summary/daily aggregate rebuilds until per-type verification instead of rebuilding after every imported page, and persists large HealthKit pages in smaller archive chunks while using smaller and more conservative initial-import page sizes for high-volume types, adaptive write chunk sizing, batched deleted-object persistence, explicit task yields, and lower-allocation streaming loops to avoid long monolithic SQLite stalls | Continue moving UI/cache reads to archive-backed observation ids and revisit full checkpoint/resume and background collection separately |
+| HealthKit capture | Capture now opens one archive observation per user-visible snapshot, attaches HealthKit pages, deleted-object evidence, and type verification to that observation id before finishing it, no longer aborts initial full-history imports after a fixed 30-minute wall-clock cap while page-level HealthKit timeouts remain in place, defers grouped observation summary/daily aggregate rebuilds until per-type verification instead of rebuilding after every imported page, and persists large HealthKit pages in smaller archive chunks while using type-specific import strategies: conservative paging for the heaviest metrics, more aggressive pages/chunks for ordinary metrics, adaptive write chunk sizing, batched deleted-object persistence, explicit task yields, and lower-allocation streaming loops to avoid long monolithic SQLite stalls | Continue moving UI/cache reads to archive-backed observation ids and revisit full checkpoint/resume and background collection separately |
 | SQLite archive | Archive v2 schema, snapshot-level observation grouping, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed, the hot write path now reuses prepared SQLite statements within grouped page writes instead of reparsing the same SQL for every sample, processes sample rows in a lower-allocation streaming loop, batches same-page deleted-object evidence in one transaction, adds composite indexes for visibility-range and sample-uuid hot lookups, and opens SQLite connections with import-friendly busy timeout / synchronous / temp-store pragmas | Continue moving capture/Dashboard actions to archive/cache DTOs |
 | Core Data cache | Initial programmatic Core Data model, full-cache rebuild service, read DTOs for observation/type/diff/health rows, and Dashboard archive-cache status wiring are in place | Move remaining export/report paths to cache DTOs and add targeted partial invalidation |
 | SwiftData cache | Exists; test builds now reset legacy prototype UI/archive/cache stores once for archive v2 so old SwiftData-only snapshots are not treated as backed-up observations. Metric timeout calibration, local device profile settings, operation logging, ContentView preview, Settings data maintenance, legacy detail/PDF views, unused legacy repair/observer services, Dashboard view/view-model access, and legacy anomaly/count-drop review have moved outside SwiftData or been removed. Remaining SwiftData imports are inventoried in [`SwiftData-Retirement-Inventory.md`](SwiftData-Retirement-Inventory.md) | Treat as disposable prototype data; stop returning/storing `HealthSnapshot` bridge handles before removing `ModelContainer` |


+30 -14

HealthProbe/Services/HealthKitService.swift

View

@@ -2551,38 +2551,54 @@ struct DistributionPageArchiveResult: Equatable {
 enum DistributionCaptureConfiguration {
     static let pageTimeoutSeconds: TimeInterval = 60
     static let incrementalStrategy = DistributionCaptureStrategy(
-        queryPageLimit: 5_000,
-        initialWriteChunkSize: 1_000,
-        minimumWriteChunkSize: 250,
-        slowBatchThresholdSeconds: 2,
+        queryPageLimit: 10_000,
+        initialWriteChunkSize: 2_000,
+        minimumWriteChunkSize: 500,
+        slowBatchThresholdSeconds: 2.5,
         severeBatchThresholdSeconds: 6
     )
 
-    private static let highVolumeTypeIdentifiers: Set<String> = [
+    private static let veryHighVolumeTypeIdentifiers: Set<String> = [
         HKQuantityTypeIdentifier.heartRate.rawValue,
+    ]
+
+    private static let highVolumeTypeIdentifiers: Set<String> = [
         HKQuantityTypeIdentifier.stepCount.rawValue,
         HKQuantityTypeIdentifier.distanceWalkingRunning.rawValue,
         HKQuantityTypeIdentifier.activeEnergyBurned.rawValue,
-        HKQuantityTypeIdentifier.appleExerciseTime.rawValue
+        HKQuantityTypeIdentifier.appleExerciseTime.rawValue,
+        HKCategoryTypeIdentifier.sleepAnalysis.rawValue,
+        HKQuantityTypeIdentifier.headphoneAudioExposure.rawValue,
+        HKQuantityTypeIdentifier.environmentalAudioExposure.rawValue
     ]
 
     static func initialImportStrategy(for typeIdentifier: String) -> DistributionCaptureStrategy {
-        if highVolumeTypeIdentifiers.contains(typeIdentifier) {
+        if veryHighVolumeTypeIdentifiers.contains(typeIdentifier) {
             return DistributionCaptureStrategy(
-                queryPageLimit: 1_000,
-                initialWriteChunkSize: 250,
+                queryPageLimit: 2_000,
+                initialWriteChunkSize: 500,
                 minimumWriteChunkSize: 100,
                 slowBatchThresholdSeconds: 1.25,
                 severeBatchThresholdSeconds: 4
             )
         }
 
+        if highVolumeTypeIdentifiers.contains(typeIdentifier) {
+            return DistributionCaptureStrategy(
+                queryPageLimit: 5_000,
+                initialWriteChunkSize: 1_000,
+                minimumWriteChunkSize: 250,
+                slowBatchThresholdSeconds: 1.75,
+                severeBatchThresholdSeconds: 4.5
+            )
+        }
+
         return DistributionCaptureStrategy(
-            queryPageLimit: 2_000,
-            initialWriteChunkSize: 500,
-            minimumWriteChunkSize: 100,
-            slowBatchThresholdSeconds: 1.5,
-            severeBatchThresholdSeconds: 4
+            queryPageLimit: 20_000,
+            initialWriteChunkSize: 5_000,
+            minimumWriteChunkSize: 500,
+            slowBatchThresholdSeconds: 2.5,
+            severeBatchThresholdSeconds: 6
         )
     }
 


	@@ -2551,38 +2551,54 @@ struct DistributionPageArchiveResult: Equatable {
2551	2551	enum DistributionCaptureConfiguration {
2552	2552	static let pageTimeoutSeconds: TimeInterval = 60
2553	2553	static let incrementalStrategy = DistributionCaptureStrategy(
2554		- queryPageLimit: 5_000,
2555		- initialWriteChunkSize: 1_000,
2556		- minimumWriteChunkSize: 250,
2557		- slowBatchThresholdSeconds: 2,
	2554	+ queryPageLimit: 10_000,
	2555	+ initialWriteChunkSize: 2_000,
	2556	+ minimumWriteChunkSize: 500,
	2557	+ slowBatchThresholdSeconds: 2.5,
2558	2558	severeBatchThresholdSeconds: 6
2559	2559	)
2560	2560
2561		- private static let highVolumeTypeIdentifiers: Set<String> = [
	2561	+ private static let veryHighVolumeTypeIdentifiers: Set<String> = [
2562	2562	HKQuantityTypeIdentifier.heartRate.rawValue,
	2563	+ ]
	2564	+
	2565	+ private static let highVolumeTypeIdentifiers: Set<String> = [
2563	2566	HKQuantityTypeIdentifier.stepCount.rawValue,
2564	2567	HKQuantityTypeIdentifier.distanceWalkingRunning.rawValue,
2565	2568	HKQuantityTypeIdentifier.activeEnergyBurned.rawValue,
2566		- HKQuantityTypeIdentifier.appleExerciseTime.rawValue
	2569	+ HKQuantityTypeIdentifier.appleExerciseTime.rawValue,
	2570	+ HKCategoryTypeIdentifier.sleepAnalysis.rawValue,
	2571	+ HKQuantityTypeIdentifier.headphoneAudioExposure.rawValue,
	2572	+ HKQuantityTypeIdentifier.environmentalAudioExposure.rawValue
2567	2573	]
2568	2574
2569	2575	static func initialImportStrategy(for typeIdentifier: String) -> DistributionCaptureStrategy {
2570		- if highVolumeTypeIdentifiers.contains(typeIdentifier) {
	2576	+ if veryHighVolumeTypeIdentifiers.contains(typeIdentifier) {
2571	2577	return DistributionCaptureStrategy(
2572		- queryPageLimit: 1_000,
2573		- initialWriteChunkSize: 250,
	2578	+ queryPageLimit: 2_000,
	2579	+ initialWriteChunkSize: 500,
2574	2580	minimumWriteChunkSize: 100,
2575	2581	slowBatchThresholdSeconds: 1.25,
2576	2582	severeBatchThresholdSeconds: 4
2577	2583	)
2578	2584	}
2579	2585
	2586	+ if highVolumeTypeIdentifiers.contains(typeIdentifier) {
	2587	+ return DistributionCaptureStrategy(
	2588	+ queryPageLimit: 5_000,
	2589	+ initialWriteChunkSize: 1_000,
	2590	+ minimumWriteChunkSize: 250,
	2591	+ slowBatchThresholdSeconds: 1.75,
	2592	+ severeBatchThresholdSeconds: 4.5
	2593	+ )
	2594	+ }
	2595	+
2580	2596	return DistributionCaptureStrategy(
2581		- queryPageLimit: 2_000,
2582		- initialWriteChunkSize: 500,
2583		- minimumWriteChunkSize: 100,
2584		- slowBatchThresholdSeconds: 1.5,
2585		- severeBatchThresholdSeconds: 4
	2597	+ queryPageLimit: 20_000,
	2598	+ initialWriteChunkSize: 5_000,
	2599	+ minimumWriteChunkSize: 500,
	2600	+ slowBatchThresholdSeconds: 2.5,
	2601	+ severeBatchThresholdSeconds: 6
2586	2602	)
2587	2603	}
2588	2604

	@@ -24,7 +24,7 @@ There are no real deployments, only test installations. Existing prototype datab
24	24	\| Area \| Current Status \| Target / Next Work \|
25	25	\|------\|----------------\|--------------------\|
26	26	\| Product docs \| Updated \| Keep `HealthProbe/Doc/README.md` as canonical index \|
27		-\| HealthKit capture \| Capture now opens one archive observation per user-visible snapshot, attaches HealthKit pages, deleted-object evidence, and type verification to that observation id before finishing it, no longer aborts initial full-history imports after a fixed 30-minute wall-clock cap while page-level HealthKit timeouts remain in place, defers grouped observation summary/daily aggregate rebuilds until per-type verification instead of rebuilding after every imported page, and persists large HealthKit pages in smaller archive chunks while using smaller and more conservative initial-import page sizes for high-volume types, adaptive write chunk sizing, batched deleted-object persistence, explicit task yields, and lower-allocation streaming loops to avoid long monolithic SQLite stalls \| Continue moving UI/cache reads to archive-backed observation ids and revisit full checkpoint/resume and background collection separately \|
	27	+\| HealthKit capture \| Capture now opens one archive observation per user-visible snapshot, attaches HealthKit pages, deleted-object evidence, and type verification to that observation id before finishing it, no longer aborts initial full-history imports after a fixed 30-minute wall-clock cap while page-level HealthKit timeouts remain in place, defers grouped observation summary/daily aggregate rebuilds until per-type verification instead of rebuilding after every imported page, and persists large HealthKit pages in smaller archive chunks while using type-specific import strategies: conservative paging for the heaviest metrics, more aggressive pages/chunks for ordinary metrics, adaptive write chunk sizing, batched deleted-object persistence, explicit task yields, and lower-allocation streaming loops to avoid long monolithic SQLite stalls \| Continue moving UI/cache reads to archive-backed observation ids and revisit full checkpoint/resume and background collection separately \|
28	28	\| SQLite archive \| Archive v2 schema, snapshot-level observation grouping, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed, the hot write path now reuses prepared SQLite statements within grouped page writes instead of reparsing the same SQL for every sample, processes sample rows in a lower-allocation streaming loop, batches same-page deleted-object evidence in one transaction, adds composite indexes for visibility-range and sample-uuid hot lookups, and opens SQLite connections with import-friendly busy timeout / synchronous / temp-store pragmas \| Continue moving capture/Dashboard actions to archive/cache DTOs \|
29	29	\| Core Data cache \| Initial programmatic Core Data model, full-cache rebuild service, read DTOs for observation/type/diff/health rows, and Dashboard archive-cache status wiring are in place \| Move remaining export/report paths to cache DTOs and add targeted partial invalidation \|
30	30	\| SwiftData cache \| Exists; test builds now reset legacy prototype UI/archive/cache stores once for archive v2 so old SwiftData-only snapshots are not treated as backed-up observations. Metric timeout calibration, local device profile settings, operation logging, ContentView preview, Settings data maintenance, legacy detail/PDF views, unused legacy repair/observer services, Dashboard view/view-model access, and legacy anomaly/count-drop review have moved outside SwiftData or been removed. Remaining SwiftData imports are inventoried in [`SwiftData-Retirement-Inventory.md`](SwiftData-Retirement-Inventory.md) \| Treat as disposable prototype data; stop returning/storing `HealthSnapshot` bridge handles before removing `ModelContainer` \|