Add composite indexes for import hot path · 8bdc72c

Add composite indexes for import hot path
Browse files

bogdan committed a week ago

main

1 parent 1b930c5

commit 8bdc72c

Showing 2 changed files with 4 additions and 1 deletions

+1 -1

HealthProbe/Doc/04-project/IMPLEMENTATION_STATUS.md

@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab
 |------|----------------|--------------------|
 | Product docs | Updated | Keep `HealthProbe/Doc/README.md` as canonical index |
 | HealthKit capture | Capture now opens one archive observation per user-visible snapshot, attaches HealthKit pages, deleted-object evidence, and type verification to that observation id before finishing it, no longer aborts initial full-history imports after a fixed 30-minute wall-clock cap while page-level HealthKit timeouts remain in place, defers grouped observation summary/daily aggregate rebuilds until per-type verification instead of rebuilding after every imported page, and persists large HealthKit pages in smaller archive chunks while using smaller and more conservative initial-import page sizes for high-volume types, adaptive write chunk sizing, batched deleted-object persistence, explicit task yields, and lower-allocation streaming loops to avoid long monolithic SQLite stalls | Continue moving UI/cache reads to archive-backed observation ids and revisit full checkpoint/resume and background collection separately |
-| SQLite archive | Archive v2 schema, snapshot-level observation grouping, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed, the hot write path now reuses prepared SQLite statements within grouped page writes instead of reparsing the same SQL for every sample, processes sample rows in a lower-allocation streaming loop, batches same-page deleted-object evidence in one transaction, and opens SQLite connections with import-friendly busy timeout / synchronous / temp-store pragmas | Continue moving capture/Dashboard actions to archive/cache DTOs |
+| SQLite archive | Archive v2 schema, snapshot-level observation grouping, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed, the hot write path now reuses prepared SQLite statements within grouped page writes instead of reparsing the same SQL for every sample, processes sample rows in a lower-allocation streaming loop, batches same-page deleted-object evidence in one transaction, adds composite indexes for visibility-range and sample-uuid hot lookups, and opens SQLite connections with import-friendly busy timeout / synchronous / temp-store pragmas | Continue moving capture/Dashboard actions to archive/cache DTOs |
 | Core Data cache | Initial programmatic Core Data model, full-cache rebuild service, read DTOs for observation/type/diff/health rows, and Dashboard archive-cache status wiring are in place | Move remaining export/report paths to cache DTOs and add targeted partial invalidation |
 | SwiftData cache | Exists; test builds now reset legacy prototype UI/archive/cache stores once for archive v2 so old SwiftData-only snapshots are not treated as backed-up observations. Metric timeout calibration, local device profile settings, operation logging, ContentView preview, Settings data maintenance, legacy detail/PDF views, unused legacy repair/observer services, Dashboard view/view-model access, and legacy anomaly/count-drop review have moved outside SwiftData or been removed. Remaining SwiftData imports are inventoried in [`SwiftData-Retirement-Inventory.md`](SwiftData-Retirement-Inventory.md) | Treat as disposable prototype data; stop returning/storing `HealthSnapshot` bridge handles before removing `ModelContainer` |
 | UI | Prototype exists; Dashboard status reads archive/cache observation rows and shows cache health, and Dashboard view/view-model code no longer imports SwiftData or reads `ModelContext`; capture/review actions now route through DTOs and snapshot ids, with the remaining legacy bridge isolated in `HealthKitService`. Snapshots and Data Types tab roots no longer import SwiftData, load Core Data cached observation rows, and open archive/cache-backed detail rows; `SnapshotArchiveDetailView` and `DataTypeArchiveDetailView` read Core Data type/diff summaries and page record drill-down through SQLite; unused legacy SwiftData snapshot/type detail and PDF views have been deleted; record-change evolution and temporal distribution screens now receive DTO rows/cache input instead of querying SwiftData directly; export preview reads the archive export API before showing/exporting JSON; simplified detail mode replaces heavy charts with summary rows on small/accessibility layouts or when enabled in Settings; visible change labels now use neutral new/missing/change-review language | Stop writing prototype `HealthSnapshot` bridge rows during capture/review |


+3 -0

HealthProbe/Services/SQLiteHealthArchiveStore.swift

View

@@ -1660,6 +1660,7 @@ actor SQLiteHealthArchiveStore: HealthArchiveStore {
         )
         """, db: db)
         try execute("CREATE INDEX IF NOT EXISTS idx_samples_uuid_hash ON samples(sample_uuid_hash)", db: db)
+        try execute("CREATE INDEX IF NOT EXISTS idx_samples_type_uuid_hash ON samples(sample_type_id, sample_uuid_hash)", db: db)
         try execute("CREATE INDEX IF NOT EXISTS idx_samples_type_semantic ON samples(sample_type_id, semantic_fingerprint)", db: db)
         try execute("""
         CREATE TABLE IF NOT EXISTS sample_versions (
@@ -1709,6 +1710,8 @@ actor SQLiteHealthArchiveStore: HealthArchiveStore {
         )
         """, db: db)
         try execute("CREATE INDEX IF NOT EXISTS idx_visibility_open_ranges ON sample_visibility_ranges(last_observation_id)", db: db)
+        try execute("CREATE INDEX IF NOT EXISTS idx_visibility_sample_open ON sample_visibility_ranges(sample_id, last_observation_id)", db: db)
+        try execute("CREATE INDEX IF NOT EXISTS idx_visibility_sample_version_open ON sample_visibility_ranges(sample_id, version_id, last_observation_id)", db: db)
         try execute("CREATE INDEX IF NOT EXISTS idx_visibility_point_lookup ON sample_visibility_ranges(first_observation_id, last_observation_id)", db: db)
         try execute("""
         CREATE TABLE IF NOT EXISTS sample_relationships (


	@@ -1660,6 +1660,7 @@ actor SQLiteHealthArchiveStore: HealthArchiveStore {
1660	1660	)
1661	1661	""", db: db)
1662	1662	try execute("CREATE INDEX IF NOT EXISTS idx_samples_uuid_hash ON samples(sample_uuid_hash)", db: db)
	1663	+ try execute("CREATE INDEX IF NOT EXISTS idx_samples_type_uuid_hash ON samples(sample_type_id, sample_uuid_hash)", db: db)
1663	1664	try execute("CREATE INDEX IF NOT EXISTS idx_samples_type_semantic ON samples(sample_type_id, semantic_fingerprint)", db: db)
1664	1665	try execute("""
1665	1666	CREATE TABLE IF NOT EXISTS sample_versions (
	@@ -1709,6 +1710,8 @@ actor SQLiteHealthArchiveStore: HealthArchiveStore {
1709	1710	)
1710	1711	""", db: db)
1711	1712	try execute("CREATE INDEX IF NOT EXISTS idx_visibility_open_ranges ON sample_visibility_ranges(last_observation_id)", db: db)
	1713	+ try execute("CREATE INDEX IF NOT EXISTS idx_visibility_sample_open ON sample_visibility_ranges(sample_id, last_observation_id)", db: db)
	1714	+ try execute("CREATE INDEX IF NOT EXISTS idx_visibility_sample_version_open ON sample_visibility_ranges(sample_id, version_id, last_observation_id)", db: db)
1712	1715	try execute("CREATE INDEX IF NOT EXISTS idx_visibility_point_lookup ON sample_visibility_ranges(first_observation_id, last_observation_id)", db: db)
1713	1716	try execute("""
1714	1717	CREATE TABLE IF NOT EXISTS sample_relationships (

	@@ -25,7 +25,7 @@ There are no real deployments, only test installations. Existing prototype datab
25	25	\|------\|----------------\|--------------------\|
26	26	\| Product docs \| Updated \| Keep `HealthProbe/Doc/README.md` as canonical index \|
27	27	\| HealthKit capture \| Capture now opens one archive observation per user-visible snapshot, attaches HealthKit pages, deleted-object evidence, and type verification to that observation id before finishing it, no longer aborts initial full-history imports after a fixed 30-minute wall-clock cap while page-level HealthKit timeouts remain in place, defers grouped observation summary/daily aggregate rebuilds until per-type verification instead of rebuilding after every imported page, and persists large HealthKit pages in smaller archive chunks while using smaller and more conservative initial-import page sizes for high-volume types, adaptive write chunk sizing, batched deleted-object persistence, explicit task yields, and lower-allocation streaming loops to avoid long monolithic SQLite stalls \| Continue moving UI/cache reads to archive-backed observation ids and revisit full checkpoint/resume and background collection separately \|
28		-\| SQLite archive \| Archive v2 schema, snapshot-level observation grouping, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed, the hot write path now reuses prepared SQLite statements within grouped page writes instead of reparsing the same SQL for every sample, processes sample rows in a lower-allocation streaming loop, batches same-page deleted-object evidence in one transaction, and opens SQLite connections with import-friendly busy timeout / synchronous / temp-store pragmas \| Continue moving capture/Dashboard actions to archive/cache DTOs \|
	28	+\| SQLite archive \| Archive v2 schema, snapshot-level observation grouping, differential write path, v2 verification/delete bookkeeping, daily aggregate rebuilds, integrity report, v2 record reads, SQL diff/count/aggregate/provenance/consolidation-evidence APIs, large synthetic diff pagination coverage, formal timing/memory metrics, and XCTest coverage are in place; the legacy `archive_samples` mirror has been removed, the hot write path now reuses prepared SQLite statements within grouped page writes instead of reparsing the same SQL for every sample, processes sample rows in a lower-allocation streaming loop, batches same-page deleted-object evidence in one transaction, adds composite indexes for visibility-range and sample-uuid hot lookups, and opens SQLite connections with import-friendly busy timeout / synchronous / temp-store pragmas \| Continue moving capture/Dashboard actions to archive/cache DTOs \|
29	29	\| Core Data cache \| Initial programmatic Core Data model, full-cache rebuild service, read DTOs for observation/type/diff/health rows, and Dashboard archive-cache status wiring are in place \| Move remaining export/report paths to cache DTOs and add targeted partial invalidation \|
30	30	\| SwiftData cache \| Exists; test builds now reset legacy prototype UI/archive/cache stores once for archive v2 so old SwiftData-only snapshots are not treated as backed-up observations. Metric timeout calibration, local device profile settings, operation logging, ContentView preview, Settings data maintenance, legacy detail/PDF views, unused legacy repair/observer services, Dashboard view/view-model access, and legacy anomaly/count-drop review have moved outside SwiftData or been removed. Remaining SwiftData imports are inventoried in [`SwiftData-Retirement-Inventory.md`](SwiftData-Retirement-Inventory.md) \| Treat as disposable prototype data; stop returning/storing `HealthSnapshot` bridge handles before removing `ModelContainer` \|
31	31	\| UI \| Prototype exists; Dashboard status reads archive/cache observation rows and shows cache health, and Dashboard view/view-model code no longer imports SwiftData or reads `ModelContext`; capture/review actions now route through DTOs and snapshot ids, with the remaining legacy bridge isolated in `HealthKitService`. Snapshots and Data Types tab roots no longer import SwiftData, load Core Data cached observation rows, and open archive/cache-backed detail rows; `SnapshotArchiveDetailView` and `DataTypeArchiveDetailView` read Core Data type/diff summaries and page record drill-down through SQLite; unused legacy SwiftData snapshot/type detail and PDF views have been deleted; record-change evolution and temporal distribution screens now receive DTO rows/cache input instead of querying SwiftData directly; export preview reads the archive export API before showing/exporting JSON; simplified detail mode replaces heavy charts with summary rows on small/accessibility layouts or when enabled in Settings; visible change labels now use neutral new/missing/change-review language \| Stop writing prototype `HealthSnapshot` bridge rows during capture/review \|