Incrementally update changed type summaries · 2ebfab3

Incrementally update changed type summaries
Browse files

bogdan committed 3 days ago

main

1 parent 457fd80

commit 2ebfab3

Showing 2 changed files with 192 additions and 12 deletions

+10 -11

HealthProbe/Doc/04-project/Import-Optimization-Log.md

@@ -584,7 +584,8 @@ rows exist".
 | 2026-06-03 | pending | Add capture-mode summary to diagnostics. | Repeated full-profile captures rarely produce a perfect no-delta report because at least one metric can change between manual runs. Diagnostic reports now include aggregate `CaptureModes` counts plus per-metric `captureMode`, so comparisons can separate unchanged empty-delta metrics from delta-applied metrics and full imports without manually reading every `record_import` line. Expected signal: stable checksum plus high `unchangedDelta` count and zero summed processing/insert confirms the fast path even when a few metrics changed. |
 | 2026-06-03 | pending | Add delta-event counts to diagnostics. | A full-profile follow-up completed in `47.4s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=115, delta=12, initialImport=0`, `SummedProcessingElapsed: 25.9s`, `SummedInsertElapsed: 0.2s`, and `SummedFinalizeElapsed: 16.0s`. This confirms anchors work and no full import ran. Remaining cost is delta application for large metrics: Heart Rate `23.5s` total (`14.1s` processing, `8.8s` finalize), Active Energy `7.1s`, and Basal Energy `6.0s`. Diagnostics now report aggregate/per-metric `DeltaEvents` so future logs can separate true HealthKit delta size from the final visible record count. |
 | 2026-06-04 | `a676df1` | Rebuild delta compact archives without large intermediate record arrays. | Follow-up full-profile delta report completed in `42.1s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=108, delta=19, initialImport=0`, and `DeltaEvents: 582`. Compared with the prior `47.4s` run, `SummedProcessingElapsed` dropped `25.9s -> 12.8s`; Heart Rate processing dropped `14.1s -> 6.4s` despite `187` delta events; Active Energy processing dropped `4.9s -> 2.5s` with `100` delta events; Basal Energy processing dropped `4.1s -> 2.0s` with `82` delta events. `SummedFinalizeElapsed` rose `16.0s -> 19.6s`, so the remaining bottleneck is now archive finalization / aggregate rebuild for changed high-volume types, not Swift archive reconstruction. |
-| 2026-06-04 | pending | Incrementally replace changed daily aggregate buckets. | The latest measured bottleneck is finalize work for changed high-volume delta types. Changed metrics now copy the previous observation's daily aggregates and recompute only the days touched by appeared, disappeared, or representation-changed events, with a bounded start-date window for the replacement query. Expected signal: lower `SummedFinalizeElapsed`, especially Heart Rate / Active Energy / Basal Energy finalize times, when `DeltaEvents` is small relative to the full visible type count. |
+| 2026-06-04 | `457fd80` | Incrementally replace changed daily aggregate buckets. | Follow-up full-profile delta report completed in `31.2s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=119, delta=8, initialImport=0`, and `DeltaEvents: 22`. Compared with the prior `42.1s` run, `SummedFinalizeElapsed` dropped `19.6s -> 15.5s`, Active Energy finalize dropped `4.8s -> 1.8s`, and total wall clock dropped `42.1s -> 31.2s`. Heart Rate finalize barely moved (`8.9s -> 8.7s`) despite only `5` delta events, proving that daily aggregate replacement helped some changed metrics but Heart Rate is still dominated by type summary `visibleAggregate` full scans. |
+| 2026-06-04 | pending | Incrementally update changed type summaries. | Heart Rate still spent `8.7s` in finalize with only `5` delta events because changed verification computed `visibleRecordCount`, range, value sum, and max by scanning all visible Heart Rate rows. Changed metrics now attempt to derive the new type summary from the previous materialized summary plus appeared/removed version deltas, falling back to the full visible scan if a removed row may have been the previous earliest, latest, or max value. Expected signal: Heart Rate finalize time drops sharply when a small recent delta does not remove historical extrema. |
 
 ## Current Diagnosis
 
@@ -650,11 +651,10 @@ The likely bottleneck is per-row SQLite work:
   high-value target is `markVerification` / daily aggregate rebuild for changed
   high-volume delta types, where Heart Rate still spent `8.9s` finalizing and
   Active Energy spent `4.8s`.
-- Changed high-volume metrics should not rebuild daily aggregates for the whole
-  type when the current observation touched only a few days. The active
-  experiment is to copy previous daily aggregates and replace only affected
-  buckets; compare `DeltaEvents` and per-type finalize time before doing deeper
-  schema/index work.
+- Incremental daily aggregate replacement helped Active Energy but did not move
+  Heart Rate enough. The next bottleneck is the changed-type `visibleAggregate`
+  summary query, which still scans all visible rows for the type before daily
+  aggregate replacement runs.
 
 ## Open Issues / Observations
 
@@ -693,11 +693,10 @@ Prioritize experiments in this order:
    `CaptureModes`, and high-volume type timings. Treat stable checksum, a high
    `unchangedDelta` count, and zero processing/insert as the main unchanged-path
    signal.
-4. Run a full-profile repeated capture after incremental changed-bucket
-   aggregate replacement. Compare `SummedFinalizeElapsed`, Heart Rate / Active
-   Energy / Basal Energy finalize times, and `DeltaEvents`. If finalize remains
-   high with small `DeltaEvents`, inspect the bounded replacement SQL plan before
-   adding new write-time indexes.
+4. Run a full-profile repeated capture after incremental type-summary updates.
+   Compare `SummedFinalizeElapsed`, Heart Rate finalize time, and `DeltaEvents`.
+   Expected success is Heart Rate finalize below the previous `8.7s` when its
+   delta does not remove the previous earliest/latest/max row.
 5. Keep using `DeltaEvents` to quantify changed high-volume metrics, especially
    Heart Rate, Active Energy, and Basal Energy. If delta events are small while
    finalize remains large, optimize aggregate rebuild/finalization rather than


+182 -1

HealthProbe/Services/SQLiteHealthArchiveStore.swift

View

+                                 return
+                    -        let aggregate = try visibleAggregate(sampleTypeID: sampleTypeID, db: db)
+                    +        let aggregate = try incrementalVisibleAggregate(
+                    +            previous: previousSummary?.aggregate,
+                    +            observationID: observationID,
+                    +            sampleTypeID: sampleTypeID,
+                    +            db: db
+                    +        ) ?? visibleAggregate(sampleTypeID: sampleTypeID, db: db)
+                             let visibleCount = aggregate.visibleRecordCount
+                             try insertObservationTypeRun(
+                                 observationID: observationID,
+                    +    private func incrementalVisibleAggregate(
+                    +        previous: ArchiveV2VisibleAggregate?,
+                    +        observationID: Int64,
+                    +        sampleTypeID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws -> ArchiveV2VisibleAggregate? {
+                    +        guard let previous else { return nil }
+                    +        let delta = try visibleAggregateDelta(
+                    +            observationID: observationID,
+                    +            sampleTypeID: sampleTypeID,
+                    +            db: db
+                    +        )
+                    +        guard delta.addedCount > 0 || delta.removedCount > 0 else {
+                    +            return previous
+                    +        }
+                    +        let visibleRecordCount = previous.visibleRecordCount + delta.addedCount - delta.removedCount
+                    +        guard visibleRecordCount >= 0 else { return nil }
+                    +        guard visibleRecordCount > 0 else {
+                    +            return ArchiveV2VisibleAggregate(
+                    +                visibleRecordCount: 0,
+                    +                earliestStartDate: nil,
+                    +                latestEndDate: nil,
+                    +                valueSum: nil,
+                    +                valueMax: nil
+                    +            )
+                    +        }
+                    +        if let previousEarliest = previous.earliestStartDate,
+                    +           let removedEarliest = delta.removedEarliestStartDate,
+                    +           sameSQLiteDouble(previousEarliest, removedEarliest) {
+                    +            return nil
+                    +        }
+                    +        if let previousLatest = previous.latestEndDate,
+                    +           let removedLatest = delta.removedLatestEndDate,
+                    +           sameSQLiteDouble(previousLatest, removedLatest) {
+                    +            return nil
+                    +        }
+                    +        if let previousMax = previous.valueMax,
+                    +           let removedMax = delta.removedValueMax,
+                    +           sameSQLiteDouble(previousMax, removedMax) {
+                    +            return nil
+                    +        }
+                    +        let earliestStartDate = minOptional(previous.earliestStartDate, delta.addedEarliestStartDate)
+                    +        let latestEndDate = maxOptional(previous.latestEndDate, delta.addedLatestEndDate)
+                    +        let hasValueSum = previous.valueSum != nil || delta.addedValueSum != nil || delta.removedValueSum != nil
+                    +        let valueSum = hasValueSum
+                    +            ? (previous.valueSum ?? 0) + (delta.addedValueSum ?? 0) - (delta.removedValueSum ?? 0)
+                    +            : nil
+                    +        let valueMax = maxOptional(previous.valueMax, delta.addedValueMax)
+                    +        return ArchiveV2VisibleAggregate(
+                    +            visibleRecordCount: visibleRecordCount,
+                    +            earliestStartDate: earliestStartDate,
+                    +            latestEndDate: latestEndDate,
+                    +            valueSum: valueSum,
+                    +            valueMax: valueMax
+                    +        )
+                    +    }
+                    +    private func visibleAggregateDelta(
+                    +        observationID: Int64,
+                    +        sampleTypeID: Int64,
+                    +        db: OpaquePointer?
+                    +    ) throws -> ArchiveV2VisibleAggregateDelta {
+                    +        let sql = """
+                    +        WITH added_versions AS (
+                    +            SELECT v.start_date, v.end_date, v.numeric_value
+                    +            FROM sample_observation_events e
+                    +            JOIN samples s ON s.id = e.sample_id
+                    +            JOIN sample_versions v ON v.id = e.version_id
+                    +            WHERE e.observation_id = ?
+                    +              AND s.sample_type_id = ?
+                    +              AND e.version_id IS NOT NULL
+                    +              AND e.event_kind IN ('appeared', 'representationChanged')
+                    +        ),
+                    +        removed_versions AS (
+                    +            SELECT v.start_date, v.end_date, v.numeric_value
+                    +            FROM sample_visibility_ranges r
+                    +            JOIN samples s ON s.id = r.sample_id
+                    +            JOIN sample_versions v ON v.id = r.version_id
+                    +            WHERE r.last_observation_id = ?
+                    +              AND s.sample_type_id = ?
+                    +        )
+                    +        SELECT
+                    +            (SELECT COUNT(*) FROM added_versions),
+                    +            (SELECT MIN(start_date) FROM added_versions),
+                    +            (SELECT MAX(end_date) FROM added_versions),
+                    +            (SELECT SUM(numeric_value) FROM added_versions),
+                    +            (SELECT MAX(numeric_value) FROM added_versions),
+                    +            (SELECT COUNT(*) FROM removed_versions),
+                    +            (SELECT MIN(start_date) FROM removed_versions),
+                    +            (SELECT MAX(end_date) FROM removed_versions),
+                    +            (SELECT SUM(numeric_value) FROM removed_versions),
+                    +            (SELECT MAX(numeric_value) FROM removed_versions)
+                    +        """
+                    +        return try withStatement(sql, db: db) { statement in
+                    +            bindInt64(observationID, to: 1, in: statement)
+                    +            bindInt64(sampleTypeID, to: 2, in: statement)
+                    +            bindInt64(observationID, to: 3, in: statement)
+                    +            bindInt64(sampleTypeID, to: 4, in: statement)
+                    +            guard sqlite3_step(statement) == SQLITE_ROW else {
+                    +                return ArchiveV2VisibleAggregateDelta.empty
+                    +            }
+                    +            return ArchiveV2VisibleAggregateDelta(
+                    +                addedCount: columnInt(statement, 0) ?? 0,
+                    +                addedEarliestStartDate: columnDouble(statement, 1),
+                    +                addedLatestEndDate: columnDouble(statement, 2),
+                    +                addedValueSum: columnDouble(statement, 3),
+                    +                addedValueMax: columnDouble(statement, 4),
+                    +                removedCount: columnInt(statement, 5) ?? 0,
+                    +                removedEarliestStartDate: columnDouble(statement, 6),
+                    +                removedLatestEndDate: columnDouble(statement, 7),
+                    +                removedValueSum: columnDouble(statement, 8),
+                    +                removedValueMax: columnDouble(statement, 9)
+                    +            )
+                    +        }
+                    +    }
+                    +    private func minOptional(_ lhs: Double?, _ rhs: Double?) -> Double? {
+                    +        switch (lhs, rhs) {
+                    +        case let (lhs?, rhs?):
+                    +            return min(lhs, rhs)
+                    +        case let (lhs?, nil):
+                    +            return lhs
+                    +        case let (nil, rhs?):
+                    +            return rhs
+                    +        case (nil, nil):
+                    +            return nil
+                    +        }
+                    +    }
+                    +    private func maxOptional(_ lhs: Double?, _ rhs: Double?) -> Double? {
+                    +        switch (lhs, rhs) {
+                    +        case let (lhs?, rhs?):
+                    +            return max(lhs, rhs)
+                    +        case let (lhs?, nil):
+                    +            return lhs
+                    +        case let (nil, rhs?):
+                    +            return rhs
+                    +        case (nil, nil):
+                    +            return nil
+                    +        }
+                    +    }
+                    +    private func sameSQLiteDouble(_ lhs: Double, _ rhs: Double) -> Bool {
+                    +        abs(lhs - rhs) < 0.000_001
+                    +    }
+                         private func bindDiffObservationIDs(
+                             _ fromObservationID: Int64,
+                             _ toObservationID: Int64,
+                         let valueMax: Double?
+                    +private struct ArchiveV2VisibleAggregateDelta {
+                    +    let addedCount: Int
+                    +    let addedEarliestStartDate: Double?
+                    +    let addedLatestEndDate: Double?
+                    +    let addedValueSum: Double?
+                    +    let addedValueMax: Double?
+                    +    let removedCount: Int
+                    +    let removedEarliestStartDate: Double?
+                    +    let removedLatestEndDate: Double?
+                    +    let removedValueSum: Double?
+                    +    let removedValueMax: Double?
+                    +    static let empty = ArchiveV2VisibleAggregateDelta(
+                    +        addedCount: 0,
+                    +        addedEarliestStartDate: nil,
+                    +        addedLatestEndDate: nil,
+                    +        addedValueSum: nil,
+                    +        addedValueMax: nil,
+                    +        removedCount: 0,
+                    +        removedEarliestStartDate: nil,
+                    +        removedLatestEndDate: nil,
+                    +        removedValueSum: nil,
+                    +        removedValueMax: nil
+                    +    )
+                    +}
+                     private struct ArchiveV2DailyAggregateRow {
+                         let bucketStart: Double
+                         let bucketEnd: Double

	@@ -584,7 +584,8 @@ rows exist".
584	584	\| 2026-06-03 \| pending \| Add capture-mode summary to diagnostics. \| Repeated full-profile captures rarely produce a perfect no-delta report because at least one metric can change between manual runs. Diagnostic reports now include aggregate `CaptureModes` counts plus per-metric `captureMode`, so comparisons can separate unchanged empty-delta metrics from delta-applied metrics and full imports without manually reading every `record_import` line. Expected signal: stable checksum plus high `unchangedDelta` count and zero summed processing/insert confirms the fast path even when a few metrics changed. \|
585	585	\| 2026-06-03 \| pending \| Add delta-event counts to diagnostics. \| A full-profile follow-up completed in `47.4s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=115, delta=12, initialImport=0`, `SummedProcessingElapsed: 25.9s`, `SummedInsertElapsed: 0.2s`, and `SummedFinalizeElapsed: 16.0s`. This confirms anchors work and no full import ran. Remaining cost is delta application for large metrics: Heart Rate `23.5s` total (`14.1s` processing, `8.8s` finalize), Active Energy `7.1s`, and Basal Energy `6.0s`. Diagnostics now report aggregate/per-metric `DeltaEvents` so future logs can separate true HealthKit delta size from the final visible record count. \|
586	586	\| 2026-06-04 \| `a676df1` \| Rebuild delta compact archives without large intermediate record arrays. \| Follow-up full-profile delta report completed in `42.1s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=108, delta=19, initialImport=0`, and `DeltaEvents: 582`. Compared with the prior `47.4s` run, `SummedProcessingElapsed` dropped `25.9s -> 12.8s`; Heart Rate processing dropped `14.1s -> 6.4s` despite `187` delta events; Active Energy processing dropped `4.9s -> 2.5s` with `100` delta events; Basal Energy processing dropped `4.1s -> 2.0s` with `82` delta events. `SummedFinalizeElapsed` rose `16.0s -> 19.6s`, so the remaining bottleneck is now archive finalization / aggregate rebuild for changed high-volume types, not Swift archive reconstruction. \|
587		-\| 2026-06-04 \| pending \| Incrementally replace changed daily aggregate buckets. \| The latest measured bottleneck is finalize work for changed high-volume delta types. Changed metrics now copy the previous observation's daily aggregates and recompute only the days touched by appeared, disappeared, or representation-changed events, with a bounded start-date window for the replacement query. Expected signal: lower `SummedFinalizeElapsed`, especially Heart Rate / Active Energy / Basal Energy finalize times, when `DeltaEvents` is small relative to the full visible type count. \|
	587	+\| 2026-06-04 \| `457fd80` \| Incrementally replace changed daily aggregate buckets. \| Follow-up full-profile delta report completed in `31.2s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=119, delta=8, initialImport=0`, and `DeltaEvents: 22`. Compared with the prior `42.1s` run, `SummedFinalizeElapsed` dropped `19.6s -> 15.5s`, Active Energy finalize dropped `4.8s -> 1.8s`, and total wall clock dropped `42.1s -> 31.2s`. Heart Rate finalize barely moved (`8.9s -> 8.7s`) despite only `5` delta events, proving that daily aggregate replacement helped some changed metrics but Heart Rate is still dominated by type summary `visibleAggregate` full scans. \|
	588	+\| 2026-06-04 \| pending \| Incrementally update changed type summaries. \| Heart Rate still spent `8.7s` in finalize with only `5` delta events because changed verification computed `visibleRecordCount`, range, value sum, and max by scanning all visible Heart Rate rows. Changed metrics now attempt to derive the new type summary from the previous materialized summary plus appeared/removed version deltas, falling back to the full visible scan if a removed row may have been the previous earliest, latest, or max value. Expected signal: Heart Rate finalize time drops sharply when a small recent delta does not remove historical extrema. \|
588	589
589	590	## Current Diagnosis
590	591
	@@ -650,11 +651,10 @@ The likely bottleneck is per-row SQLite work:
650	651	high-value target is `markVerification` / daily aggregate rebuild for changed
651	652	high-volume delta types, where Heart Rate still spent `8.9s` finalizing and
652	653	Active Energy spent `4.8s`.
653		-- Changed high-volume metrics should not rebuild daily aggregates for the whole
654		- type when the current observation touched only a few days. The active
655		- experiment is to copy previous daily aggregates and replace only affected
656		- buckets; compare `DeltaEvents` and per-type finalize time before doing deeper
657		- schema/index work.
	654	+- Incremental daily aggregate replacement helped Active Energy but did not move
	655	+ Heart Rate enough. The next bottleneck is the changed-type `visibleAggregate`
	656	+ summary query, which still scans all visible rows for the type before daily
	657	+ aggregate replacement runs.
658	658
659	659	## Open Issues / Observations
660	660
	@@ -693,11 +693,10 @@ Prioritize experiments in this order:
693	693	`CaptureModes`, and high-volume type timings. Treat stable checksum, a high
694	694	`unchangedDelta` count, and zero processing/insert as the main unchanged-path
695	695	signal.
696		-4. Run a full-profile repeated capture after incremental changed-bucket
697		- aggregate replacement. Compare `SummedFinalizeElapsed`, Heart Rate / Active
698		- Energy / Basal Energy finalize times, and `DeltaEvents`. If finalize remains
699		- high with small `DeltaEvents`, inspect the bounded replacement SQL plan before
700		- adding new write-time indexes.
	696	+4. Run a full-profile repeated capture after incremental type-summary updates.
	697	+ Compare `SummedFinalizeElapsed`, Heart Rate finalize time, and `DeltaEvents`.
	698	+ Expected success is Heart Rate finalize below the previous `8.7s` when its
	699	+ delta does not remove the previous earliest/latest/max row.
701	700	5. Keep using `DeltaEvents` to quantify changed high-volume metrics, especially
702	701	Heart Rate, Active Energy, and Basal Energy. If delta events are small while
703	702	finalize remains large, optimize aggregate rebuild/finalization rather than