Copy aggregates for unchanged metric verification · 19ba656

Copy aggregates for unchanged metric verification
Browse files

bogdan committed 5 days ago

main

1 parent f60f09a

commit 19ba656

Showing 3 changed files with 176 additions and 9 deletions

+38 -9

HealthProbe/Doc/04-project/Import-Optimization-Log.md

	@@ -292,6 +292,32 @@ rebuild / dashboard refresh work. The dashboard cache refresh was moved to a
292	292	background task after this report; next reports should distinguish import time
293	293	from post-import UI recovery.
294	294
	295	+### 2026-06-03 Incremental No-Delta Snapshot After Visibility Fast-Path
	296	+
	297	+Commit context: after `f60f09a`. Source: user-provided diagnostic report with
	298	+`previousSnapshotID` present, `isChainStart: false`, and total record count
	299	+unchanged at 1,579,168.
	300	+
	301	+\| Metric \| Value \|
	302	+\|--------\|-------\|
	303	+\| Wall clock \| 13.6s \|
	304	+\| Summed metric total \| 13.2s \|
	305	+\| Summed fetch \| 0.2s \|
	306	+\| Summed processing \| 0.0s \|
	307	+\| Summed insert \| 0.0s \|
	308	+\| Summed finalize \| 12.9s \|
	309	+\| Total records \| 1,579,168 \|
	310	+\| Heart Rate finalize \| 9.1s \|
	311	+\| Active Energy finalize \| 1.8s \|
	312	+\| Steps finalize \| 0.5s \|
	313	+\| Walking + Running Distance finalize \| 0.4s \|
	314	+
	315	+Conclusion: the repeated no-delta write path is effectively eliminated. The
	316	+remaining measured import cost is finalization, especially Heart Rate daily
	317	+aggregate/materialized summary work. The next optimization should avoid
	318	+rescanning all visible records when a metric has no appeared, disappeared, or
	319	+representation-changed events in the current observation.
	320	+
295	321	## Optimization Iterations
296	322
297	323	\| Date \| Commit \| Change \| Result / Status \|
	@@ -314,7 +340,8 @@ from post-import UI recovery.
314	340	\| 2026-06-02 \| `3dd5f48` \| Fortified scheduled test database reset with a disk marker and extra SQLite sidecar cleanup. \| Real-device report confirmed reset produced `previousSnapshotID: none`, `isChainStart: true`, and a clean first-snapshot timeline. \|
315	341	\| 2026-06-02 \| `06ee6be` \| Removed unused `sample_versions(start_date, end_date)` and redundant `sample_visibility_ranges(sample_id, last_observation_id)` indexes. \| Comparable first-import report was flat: wall clock 12m43s -> 12m39s and summed insert 10m11s -> 10m07s. Treat as no material performance change. \|
316	342	\| 2026-06-02 \| pending \| Moved Dashboard archive cache refresh/rebuild off the UI path after snapshot completion. \| Awaiting real-device confirmation that the app no longer stays unresponsive for roughly one minute after a completed snapshot. \|
317		-\| 2026-06-03 \| pending \| Fast-path unchanged samples whose current version already has an open visibility range. \| Awaiting non-chain-start/full-scan report. Expected signal is lower insert elapsed on repeated captures with mostly unchanged rows, especially Heart Rate. \|
	343	+\| 2026-06-03 \| `f60f09a` \| Fast-path unchanged samples whose current version already has an open visibility range. \| Confirmed on repeated no-delta capture: `SummedInsertElapsed` remained 0.0s; remaining cost is `SummedFinalizeElapsed` 12.9s, with Heart Rate finalize 9.1s. \|
	344	+\| 2026-06-03 \| pending \| Copy previous type summaries and daily aggregates for unchanged metric observations instead of rebuilding from visible ranges. \| Awaiting repeated no-delta report. Expected signal is lower `SummedFinalizeElapsed`, especially Heart Rate finalize. \|
318	345
319	346	## Current Diagnosis
320	347
	@@ -331,6 +358,7 @@ The likely bottleneck is per-row SQLite work:
331	358	- multiple dependent writes per sample;
332	359	- commit / transaction shape;
333	360	- no-op visibility range maintenance for unchanged existing samples;
	361	+- full daily aggregate rebuilds for unchanged metric observations;
334	362	- Core Data or UI refresh work after SQLite completes. This became the active
335	363	bottleneck after a no-delta incremental snapshot completed in 9.2s but the UI
336	364	remained unresponsive for roughly one minute.
	@@ -349,17 +377,18 @@ The likely bottleneck is per-row SQLite work:
349	377	Prioritize experiments in this order:
350	378
351	379	1. Confirm whether the background dashboard cache refresh removes the post-import UI freeze. If not, add explicit timings around cache rebuild, dashboard refresh, diagnostic report generation, and sheet dismissal.
352		-2. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
353		-3. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.
354		-4. Profile whether index maintenance dominates first-import insert cost.
355		-5. Consider a guarded bulk-import mode for first observations:
	380	+2. Run a repeated no-delta benchmark after copying unchanged metric summaries and daily aggregates. Compare `SummedFinalizeElapsed`, `Heart Rate finalizeElapsed`, `Active Energy finalizeElapsed`, and wall clock.
	381	+3. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
	382	+4. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.
	383	+5. Profile whether index maintenance dominates first-import insert cost.
	384	+6. Consider a guarded bulk-import mode for first observations:
356	385	- keep archive semantics unchanged;
357	386	- only relax work that can be safely reconstructed or validated;
358	387	- re-enable normal idempotent paths for incremental observations.
359		-6. Run a fresh first-import benchmark after the unused-index removal and compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, and `Active Energy insertElapsed`.
360		-7. Investigate whether first-import-only deferred index creation or temporary staging tables can reduce `samples` / `sample_versions` / `sample_observation_events` write cost without weakening final archive integrity.
361		-8. Revisit adaptive page sizes only after SQLite write-path costs are reduced.
362		-9. Revisit background / scheduled collection once initial import can finish reliably and post-import UI recovery is bounded.
	388	+7. Run a fresh first-import benchmark after the unused-index removal and compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, and `Active Energy insertElapsed`.
	389	+8. Investigate whether first-import-only deferred index creation or temporary staging tables can reduce `samples` / `sample_versions` / `sample_observation_events` write cost without weakening final archive integrity.
	390	+9. Revisit adaptive page sizes only after SQLite write-path costs are reduced.
	391	+10. Revisit background / scheduled collection once initial import can finish reliably and post-import UI recovery is bounded.
363	392
364	393	## Verification Checklist For Each Optimization
365	394

+137 -0

HealthProbe/Services/SQLiteHealthArchiveStore.swift

View

+                                 lookupCache: lookupCache
+                             let counts = try eventCounts(observationID: observationID, sampleTypeID: sampleTypeID, db: db)
+                    +        if counts.appeared == 0,
+                    +           counts.disappeared == 0,
+                    +           counts.representationChanged == 0,
+                    +           let previousSummary = try previousTypeSummary(
+                    +            sampleTypeID: sampleTypeID,
+                    +            beforeObservationID: observationID,
+                    +            db: db,
+                    +            statementCache: statementCache
+                    +           ) {
+                    +            try insertObservationTypeRun(
+                    +                observationID: observationID,
+                    +                sampleTypeID: sampleTypeID,
+                    +                status: "completed",
+                    +                observedAt: verifiedAt,
+                    +                insertedEventCount: 0,
+                    +                deletedEventCount: 0,
+                    +                verifiedVisibleCount: previousSummary.aggregate.visibleRecordCount,
+                    +                db: db,
+                    +                statementCache: statementCache
+                    +            )
+                    +            try upsertTypeSummary(
+                    +                observationID: observationID,
+                    +                sampleTypeID: sampleTypeID,
+                    +                counts: counts,
+                    +                aggregate: previousSummary.aggregate,
+                    +                db: db
+                    +            )
+                    +            try copyDailyAggregates(
+                    +                fromObservationID: previousSummary.observationID,
+                    +                toObservationID: observationID,
+                    +                sampleTypeID: sampleTypeID,
+                    +                db: db
+                    +            )
+                    +            return
+                    +        }
+                             let aggregate = try visibleAggregate(sampleTypeID: sampleTypeID, db: db)
+                             let visibleCount = aggregate.visibleRecordCount
+                             try insertObservationTypeRun(
+                    +    private func previousTypeSummary(
+                    +        sampleTypeID: Int64,
+                    +        beforeObservationID observationID: Int64,
+                    +        db: OpaquePointer?,
+                    +        statementCache: SQLiteStatementCache? = nil
+                    +    ) throws -> (observationID: Int64, aggregate: ArchiveV2VisibleAggregate)? {
+                    +        let sql = """
+                    +        SELECT observation_id, visible_record_count, earliest_start_date, latest_end_date, value_sum, value_max
+                    +        FROM observation_type_summaries
+                    +        WHERE sample_type_id = ? AND observation_id < ?
+                    +        ORDER BY observation_id DESC
+                    +        LIMIT 1
+                    +        """
+                    +        return try withStatement(sql, db: db, statementCache: statementCache) { statement in
+                    +            bindInt64(sampleTypeID, to: 1, in: statement)
+                    +            bindInt64(observationID, to: 2, in: statement)
+                    +            guard sqlite3_step(statement) == SQLITE_ROW else {
+                    +                return nil
+                    +            }
+                    +            return (
+                    +                observationID: sqlite3_column_int64(statement, 0),
+                    +                aggregate: ArchiveV2VisibleAggregate(
+                    +                    visibleRecordCount: columnInt(statement, 1) ?? 0,
+                    +                    earliestStartDate: columnDouble(statement, 2),
+                    +                    latestEndDate: columnDouble(statement, 3),
+                    +                    valueSum: columnDouble(statement, 4),
+                    +                    valueMax: columnDouble(statement, 5)
+                    +                )
+                    +            )
+                    +        }
+                    +    }
+                         private func rebuildDailyAggregates(
+                             observationID: Int64,
+                             sampleTypeID: Int64,
+                             let rows = try dailyAggregateRows(sampleTypeID: sampleTypeID, secondsFromGMT: secondsFromGMT, db: db)
+                    +        try insertDailyAggregateRows(rows, observationID: observationID, sampleTypeID: sampleTypeID, db: db)
+                    +    }
+                    +    private func copyDailyAggregates(
+                    +        fromObservationID: Int64,
+                    +        toObservationID: Int64,
+                    +        sampleTypeID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws {
+                    +        try withStatement(
+                    +            "DELETE FROM daily_type_aggregates WHERE observation_id = ? AND sample_type_id = ?",
+                    +            db: db
+                    +        ) { statement in
+                    +            bindInt64(toObservationID, to: 1, in: statement)
+                    +            bindInt64(sampleTypeID, to: 2, in: statement)
+                    +            guard sqlite3_step(statement) == SQLITE_DONE else {
+                    +                throw SQLiteHealthArchiveStoreError.stepFailed(lastErrorMessage(db))
+                    +            }
+                    +        }
+                    +        let rows = try dailyAggregateRows(
+                    +            observationID: fromObservationID,
+                    +            sampleTypeID: sampleTypeID,
+                    +            db: db
+                    +        )
+                    +        try insertDailyAggregateRows(
+                    +            rows,
+                    +            observationID: toObservationID,
+                    +            sampleTypeID: sampleTypeID,
+                    +            db: db
+                    +        )
+                    +    }
+                    +    private func insertDailyAggregateRows(
+                    +        _ rows: [ArchiveV2DailyAggregateRow],
+                    +        observationID: Int64,
+                    +        sampleTypeID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws {
+                             try withStatement(
+                                 """
+                                 INSERT INTO daily_type_aggregates (
+                    +    private func dailyAggregateRows(
+                    +        observationID: Int64,
+                    +        sampleTypeID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws -> [ArchiveV2DailyAggregateRow] {
+                    +        let sql = """
+                    +        SELECT bucket_start, bucket_end, visible_record_count, value_sum, value_max, source_revision_id
+                    +        FROM daily_type_aggregates
+                    +        WHERE observation_id = ? AND sample_type_id = ?
+                    +        ORDER BY bucket_start ASC, source_revision_id ASC
+                    +        """
+                    +        return try withStatement(sql, db: db) { statement in
+                    +            bindInt64(observationID, to: 1, in: statement)
+                    +            bindInt64(sampleTypeID, to: 2, in: statement)
+                    +            var rows: [ArchiveV2DailyAggregateRow] = []
+                    +            while sqlite3_step(statement) == SQLITE_ROW {
+                    +                rows.append(ArchiveV2DailyAggregateRow(
+                    +                    bucketStart: sqlite3_column_double(statement, 0),
+                    +                    bucketEnd: sqlite3_column_double(statement, 1),
+                    +                    visibleRecordCount: columnInt(statement, 2) ?? 0,
+                    +                    valueSum: columnDouble(statement, 3),
+                    +                    valueMax: columnDouble(statement, 4),
+                    +                    sourceRevisionID: columnInt64(statement, 5)
+                    +                ))
+                    +            }
+                    +            return rows
+                    +        }
+                    +    }
+                         private func dailyAggregateRows(
+                             sampleTypeID: Int64,
+                             secondsFromGMT: Int,

+1 -0

HealthProbeTests/SQLiteHealthArchiveStoreTests.swift

View

@@ -99,6 +99,7 @@ final class SQLiteHealthArchiveStoreTests: XCTestCase {
         XCTAssertEqual(try countRows(in: "observation_type_runs", at: url), 2)
         XCTAssertEqual(try countRows(in: "observation_type_runs WHERE observation_id = \(observationIDs[0]) AND inserted_event_count = 1", at: url), 1)
         XCTAssertEqual(try countRows(in: "observation_type_summaries WHERE observation_id = \(observationIDs[1]) AND visible_record_count = 1", at: url), 1)
+        XCTAssertEqual(try countRows(in: "daily_type_aggregates WHERE observation_id = \(observationIDs[1]) AND visible_record_count = 1", at: url), 1)
         XCTAssertFalse(try tableExists("archive_samples", at: url))
     }
 


	@@ -99,6 +99,7 @@ final class SQLiteHealthArchiveStoreTests: XCTestCase {
99	99	XCTAssertEqual(try countRows(in: "observation_type_runs", at: url), 2)
100	100	XCTAssertEqual(try countRows(in: "observation_type_runs WHERE observation_id = \(observationIDs[0]) AND inserted_event_count = 1", at: url), 1)
101	101	XCTAssertEqual(try countRows(in: "observation_type_summaries WHERE observation_id = \(observationIDs[1]) AND visible_record_count = 1", at: url), 1)
	102	+ XCTAssertEqual(try countRows(in: "daily_type_aggregates WHERE observation_id = \(observationIDs[1]) AND visible_record_count = 1", at: url), 1)
102	103	XCTAssertFalse(try tableExists("archive_samples", at: url))
103	104	}
104	105