Narrow incremental summary fallback · 7e3b997

Narrow incremental summary fallback
Browse files

bogdan committed 3 days ago

main

1 parent c9091de

commit 7e3b997

Showing 3 changed files with 213 additions and 25 deletions

+23 -10

HealthProbe/Doc/04-project/Import-Optimization-Log.md

@@ -589,7 +589,8 @@ rows exist".
 | 2026-06-04 | `4894b77` | Patch compact archives from delta without full record maps. | Follow-up full-profile delta report completed in `52.1s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=106, delta=21, initialImport=0`, and `DeltaEvents: 11,093`. This is not comparable to the previous `46`-event baseline: Active Energy had `2,377` events, Basal Energy `2,347`, Cycling Distance `6,052`, and Heart Rate `231`. Processing remained bounded relative to delta size (`SummedProcessingElapsed: 16.0s`; Heart Rate `5.9s`, Active Energy `2.2s`, Basal Energy `1.9s`, Cycling Distance `4.2s`), but wall clock rose because fetch `16.1s`, insert `2.3s`, and finalize `14.8s` all had real work. Conclusion: compact dictionary removal did not regress and looks healthy for large deltas, but a small-delta repeat is still needed to validate the original `6.1s` Heart Rate target. |
 | 2026-06-04 | `1ba6c38` | Hash delta compact archives in recorded order. | Small-delta follow-up completed in `24.0s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=124, delta=3, initialImport=0`, and `DeltaEvents: 13`. This finally validated the remaining bottleneck: Heart Rate still spent `5.7s` processing only `6` events, Active Energy spent `2.1s` for `5` events, and Basal Energy spent `1.8s` for `2` events. The delta rebuild no longer builds a full UUID record map, but it still collected every fingerprint into a large array and sorted it for the per-type hash. Delta rebuild now uses the same recorded-order `TypeHashBuilder` strategy as initial import, avoiding the all-record fingerprint array and sort. |
 | 2026-06-04 | `d4de48c` | Copy unchanged daily aggregates inside SQLite. | First small-delta run before this commit completed in `23.6s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=120, delta=7`, and `DeltaEvents: 11`. The hash change helped processing (`9.6s -> 8.1s`; Heart Rate `5.7s -> 4.2s`), but finalization stayed high (`11.4s`, Heart Rate `4.8s`). Changed daily aggregates were copying all previous daily rows through Swift before replacing affected buckets. This commit moved unchanged daily aggregate copying to SQLite `INSERT ... SELECT`, while affected buckets remain recalculated. Follow-up report completed in `21.1s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=123, delta=4`, and `DeltaEvents: 50`. `SummedFinalizeElapsed` improved `11.4s -> 9.3s` and wall clock improved `23.6s -> 21.1s`; however Heart Rate finalize was still `5.0s` with `4` events, so this helped overall finalize cost but did not remove the high-volume changed-type floor. |
-| 2026-06-04 | pending | Include build identity in diagnostic reports. | The latest diagnostic report only included `App Version: 1.0(1)` near the end and did not include a commit/build source identifier. This makes it too easy to compare reports from the wrong installed binary. Diagnostics now emit `appVersion`, `buildFingerprint`, `sourceCommit`, and `sourceDirty` in `OPERATION METADATA`. `buildFingerprint` is derived from the installed executable and should change when a different binary is installed; `sourceCommit/sourceDirty` remain available for builds that inject those Info.plist keys. Expected signal: future pasted reports have enough build identity to detect wrong-version reports immediately. |
+| 2026-06-04 | `c9091de` | Include build identity in diagnostic reports. | The latest diagnostic report only included `App Version: 1.0(1)` near the end and did not include a commit/build source identifier. This makes it too easy to compare reports from the wrong installed binary. Diagnostics now emit `appVersion`, `buildFingerprint`, `sourceCommit`, and `sourceDirty` in `OPERATION METADATA`. `buildFingerprint` is derived from the installed executable and should change when a different binary is installed; `sourceCommit/sourceDirty` remain available for builds that inject those Info.plist keys. Expected signal: future pasted reports have enough build identity to detect wrong-version reports immediately. |
+| 2026-06-04 | pending | Avoid summary full scans when delta safely replaces extremes. | Follow-up after `d4de48c` still showed `SummedFinalizeElapsed: 9.3s`, with Heart Rate finalize at `5.0s` despite only `4` Heart Rate delta events. `markVerification` already tries to update type summaries incrementally, but it fell back to a full visible-row aggregate scan whenever a removed row matched the previous earliest/latest/max, even if the same delta added a row that safely preserved or extended that extreme. The fallback is now narrower: full scan is still used when an extreme becomes unknown, but not when added rows replace the removed earliest/latest/max with an equivalent or stronger value. Expected signal: on repeated full-profile captures with small recent deltas, `SummedFinalizeElapsed` and high-volume type finalize times, especially Heart Rate, should drop when deltas replace recent/latest rows. Correctness is covered by synthetic SQLite archive tests for replaced latest/max and removed-latest-without-replacement cases. |
 
 ## Current Diagnosis
 
@@ -688,6 +689,11 @@ The likely bottleneck is per-row SQLite work:
   from conversation/order unless backed by an external build note. New reports
   should include `buildFingerprint`; `sourceCommit` and `sourceDirty` may still
   be `unknown` unless the build pipeline injects those Info.plist keys.
+- Incremental type-summary finalization was still too conservative around
+  removed extremes. A delta that removes the previous latest/max but adds a
+  newer/larger replacement no longer needs a full visible aggregate scan. The
+  archive still falls back to the full scan when a removed earliest/latest/max
+  leaves the new extreme genuinely unknown.
 
 ## Open Issues / Observations
 
@@ -746,21 +752,28 @@ Prioritize experiments in this order:
    identity unless the build provenance is otherwise certain. `sourceCommit`
    and `sourceDirty` are useful when present, but may be `unknown` for normal
    Xcode test installs.
-8. Investigate full-profile empty anchored-query cost for zero-count types.
+8. Run a repeated full-profile capture after narrowing incremental summary
+   fallback. Compare `SummedFinalizeElapsed` and Heart Rate finalize against the
+   `9.3s` total finalize / `5.0s` Heart Rate finalize report after `d4de48c`.
+   This optimization should help only when changed high-volume metrics replace
+   extremes safely; if Heart Rate finalize remains around `5s`, the remaining
+   cost is likely daily-bucket replacement or compact archive processing rather
+   than type summary full-scan fallback.
+9. Investigate full-profile empty anchored-query cost for zero-count types.
    Compare slow empty types across reports before changing behavior; any skip or
    lower-frequency strategy must preserve the promise that full authorized
    backup can notice newly appearing data.
-9. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
-10. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.
-11. Profile whether index maintenance dominates first-import insert cost.
-12. Consider a guarded bulk-import mode for first observations:
+10. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
+11. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.
+12. Profile whether index maintenance dominates first-import insert cost.
+13. Consider a guarded bulk-import mode for first observations:
    - keep archive semantics unchanged;
    - only relax work that can be safely reconstructed or validated;
    - re-enable normal idempotent paths for incremental observations.
-13. Run a fresh first-import benchmark after the unused-index removal and compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, and `Active Energy insertElapsed`.
-14. Investigate whether first-import-only deferred index creation or temporary staging tables can reduce `samples` / `sample_versions` / `sample_observation_events` write cost without weakening final archive integrity.
-15. Revisit adaptive page sizes only after SQLite write-path costs are reduced.
-16. Revisit background / scheduled collection once initial import can finish reliably and post-import UI recovery is bounded.
+14. Run a fresh first-import benchmark after the unused-index removal and compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, and `Active Energy insertElapsed`.
+15. Investigate whether first-import-only deferred index creation or temporary staging tables can reduce `samples` / `sample_versions` / `sample_observation_events` write cost without weakening final archive integrity.
+16. Revisit adaptive page sizes only after SQLite write-path costs are reduced.
+17. Revisit background / scheduled collection once initial import can finish reliably and post-import UI recovery is bounded.
 
 ## Verification Checklist For Each Optimization
 


+47 -15

HealthProbe/Services/SQLiteHealthArchiveStore.swift

View

+                    -        if let previousEarliest = previous.earliestStartDate,
+                    -           let removedEarliest = delta.removedEarliestStartDate,
+                    -           sameSQLiteDouble(previousEarliest, removedEarliest) {
+                    +        let earliestUpdate = incrementallyUpdatedMinimum(
+                    +            previous: previous.earliestStartDate,
+                    +            removed: delta.removedEarliestStartDate,
+                    +            added: delta.addedEarliestStartDate
+                    +        )
+                    +        guard earliestUpdate.isKnown else {
+                                 return nil
+                    -        if let previousLatest = previous.latestEndDate,
+                    -           let removedLatest = delta.removedLatestEndDate,
+                    -           sameSQLiteDouble(previousLatest, removedLatest) {
+                    +        let latestUpdate = incrementallyUpdatedMaximum(
+                    +            previous: previous.latestEndDate,
+                    +            removed: delta.removedLatestEndDate,
+                    +            added: delta.addedLatestEndDate
+                    +        )
+                    +        guard latestUpdate.isKnown else {
+                                 return nil
+                    -        if let previousMax = previous.valueMax,
+                    -           let removedMax = delta.removedValueMax,
+                    -           sameSQLiteDouble(previousMax, removedMax) {
+                    +        let maxUpdate = incrementallyUpdatedMaximum(
+                    +            previous: previous.valueMax,
+                    +            removed: delta.removedValueMax,
+                    +            added: delta.addedValueMax
+                    +        )
+                    +        guard maxUpdate.isKnown else {
+                                 return nil
+                    -        let earliestStartDate = minOptional(previous.earliestStartDate, delta.addedEarliestStartDate)
+                    -        let latestEndDate = maxOptional(previous.latestEndDate, delta.addedLatestEndDate)
+                             let hasValueSum = previous.valueSum != nil || delta.addedValueSum != nil || delta.removedValueSum != nil
+                             let valueSum = hasValueSum
+                                 ? (previous.valueSum ?? 0) + (delta.addedValueSum ?? 0) - (delta.removedValueSum ?? 0)
+                                 : nil
+                    -        let valueMax = maxOptional(previous.valueMax, delta.addedValueMax)
+                             return ArchiveV2VisibleAggregate(
+                                 visibleRecordCount: visibleRecordCount,
+                    -            earliestStartDate: earliestStartDate,
+                    -            latestEndDate: latestEndDate,
+                    +            earliestStartDate: earliestUpdate.value,
+                    +            latestEndDate: latestUpdate.value,
+                                 valueSum: valueSum,
+                    -            valueMax: valueMax
+                    +            valueMax: maxUpdate.value
+                    +    private func incrementallyUpdatedMinimum(previous: Double?, removed: Double?, added: Double?) -> (isKnown: Bool, value: Double?) {
+                    +        guard let previous else {
+                    +            return (true, added)
+                    +        }
+                    +        guard let removed, sameSQLiteDouble(previous, removed) else {
+                    +            return (true, minOptional(previous, added))
+                    +        }
+                    +        guard let added, added <= previous || sameSQLiteDouble(added, previous) else {
+                    +            return (false, nil)
+                    +        }
+                    +        return (true, added)
+                    +    }
+                    +    private func incrementallyUpdatedMaximum(previous: Double?, removed: Double?, added: Double?) -> (isKnown: Bool, value: Double?) {
+                    +        guard let previous else {
+                    +            return (true, added)
+                    +        }
+                    +        guard let removed, sameSQLiteDouble(previous, removed) else {
+                    +            return (true, maxOptional(previous, added))
+                    +        }
+                    +        guard let added, added >= previous || sameSQLiteDouble(added, previous) else {
+                    +            return (false, nil)
+                    +        }
+                    +        return (true, added)
+                    +    }
+                         private func minOptional(_ lhs: Double?, _ rhs: Double?) -> Double? {
+                             switch (lhs, rhs) {
+                             case let (lhs?, rhs?):

+143 -0

HealthProbeTests/SQLiteHealthArchiveStoreTests.swift

View

+                    +    func testChangedVerificationKeepsSummaryCorrectWhenLatestAndMaxAreReplaced() async throws {
+                    +        let url = databaseURL()
+                    +        let store = SQLiteHealthArchiveStore(databaseURL: url)
+                    +        let oldSample = makeStepCountSample(value: 10, start: 1_000, end: 1_300)
+                    +        let oldLatestMaxSample = makeStepCountSample(value: 20, start: 4_000, end: 4_300)
+                    +        let newLatestMaxSample = makeStepCountSample(value: 25, start: 5_000, end: 5_300)
+                    +        let typeIdentifier = HKQuantityTypeIdentifier.stepCount.rawValue
+                    +        _ = try await store.upsertSamples(
+                    +            [oldSample, oldLatestMaxSample],
+                    +            observedAt: Date(timeIntervalSince1970: 6_000)
+                    +        )
+                    +        try await store.markVerification(
+                    +            sampleType: oldSample.sampleType,
+                    +            verifiedAt: Date(timeIntervalSince1970: 6_060)
+                    +        )
+                    +        let changedObservationID = try await store.beginObservation(
+                    +            observedAt: Date(timeIntervalSince1970: 6_120),
+                    +            triggerReason: "manual",
+                    +            selectedTypeSetHash: "selected-types"
+                    +        )
+                    +        _ = try await store.upsertSamples(
+                    +            [newLatestMaxSample],
+                    +            observedAt: Date(timeIntervalSince1970: 6_120),
+                    +            observationID: changedObservationID
+                    +        )
+                    +        try await store.recordDisappearance(
+                    +            sampleUUIDHash: HashService.sampleUUIDHash(oldLatestMaxSample.uuid.uuidString),
+                    +            sampleTypeIdentifier: typeIdentifier,
+                    +            observedMissingAt: Date(timeIntervalSince1970: 6_120),
+                    +            observationID: changedObservationID
+                    +        )
+                    +        try await store.markVerification(
+                    +            sampleType: oldSample.sampleType,
+                    +            verifiedAt: Date(timeIntervalSince1970: 6_120),
+                    +            observationID: changedObservationID
+                    +        )
+                    +        let summary = try typeSummary(observationID: changedObservationID, at: url)
+                    +        XCTAssertEqual(summary.visibleRecordCount, 2)
+                    +        XCTAssertEqual(summary.appearedCount, 1)
+                    +        XCTAssertEqual(summary.disappearedCount, 1)
+                    +        XCTAssertEqual(summary.earliestStartDate, oldSample.startDate.timeIntervalSince1970, accuracy: 0.000_001)
+                    +        XCTAssertEqual(summary.latestEndDate, newLatestMaxSample.endDate.timeIntervalSince1970, accuracy: 0.000_001)
+                    +        XCTAssertEqual(summary.valueSum, 35, accuracy: 0.000_001)
+                    +        XCTAssertEqual(summary.valueMax, 25, accuracy: 0.000_001)
+                    +    }
+                    +    func testChangedVerificationKeepsSummaryCorrectWhenLatestIsRemovedWithoutReplacement() async throws {
+                    +        let url = databaseURL()
+                    +        let store = SQLiteHealthArchiveStore(databaseURL: url)
+                    +        let remainingSample = makeStepCountSample(value: 10, start: 1_000, end: 1_300)
+                    +        let removedLatestSample = makeStepCountSample(value: 20, start: 4_000, end: 4_300)
+                    +        let typeIdentifier = HKQuantityTypeIdentifier.stepCount.rawValue
+                    +        _ = try await store.upsertSamples(
+                    +            [remainingSample, removedLatestSample],
+                    +            observedAt: Date(timeIntervalSince1970: 6_000)
+                    +        )
+                    +        try await store.markVerification(
+                    +            sampleType: remainingSample.sampleType,
+                    +            verifiedAt: Date(timeIntervalSince1970: 6_060)
+                    +        )
+                    +        let changedObservationID = try await store.beginObservation(
+                    +            observedAt: Date(timeIntervalSince1970: 6_120),
+                    +            triggerReason: "manual",
+                    +            selectedTypeSetHash: "selected-types"
+                    +        )
+                    +        try await store.recordDisappearance(
+                    +            sampleUUIDHash: HashService.sampleUUIDHash(removedLatestSample.uuid.uuidString),
+                    +            sampleTypeIdentifier: typeIdentifier,
+                    +            observedMissingAt: Date(timeIntervalSince1970: 6_120),
+                    +            observationID: changedObservationID
+                    +        )
+                    +        try await store.markVerification(
+                    +            sampleType: remainingSample.sampleType,
+                    +            verifiedAt: Date(timeIntervalSince1970: 6_120),
+                    +            observationID: changedObservationID
+                    +        )
+                    +        let summary = try typeSummary(observationID: changedObservationID, at: url)
+                    +        XCTAssertEqual(summary.visibleRecordCount, 1)
+                    +        XCTAssertEqual(summary.appearedCount, 0)
+                    +        XCTAssertEqual(summary.disappearedCount, 1)
+                    +        XCTAssertEqual(summary.earliestStartDate, remainingSample.startDate.timeIntervalSince1970, accuracy: 0.000_001)
+                    +        XCTAssertEqual(summary.latestEndDate, remainingSample.endDate.timeIntervalSince1970, accuracy: 0.000_001)
+                    +        XCTAssertEqual(summary.valueSum, 10, accuracy: 0.000_001)
+                    +        XCTAssertEqual(summary.valueMax, 10, accuracy: 0.000_001)
+                    +    }
+                         func testDiffSummaryAndRecordsBetweenObservationsUseSQLVisibility() async throws {
+                             let url = databaseURL()
+                             let store = SQLiteHealthArchiveStore(databaseURL: url)
+                             return ids
+                    +    private func typeSummary(observationID: Int64, at url: URL) throws -> (
+                    +        visibleRecordCount: Int,
+                    +        appearedCount: Int,
+                    +        disappearedCount: Int,
+                    +        earliestStartDate: Double,
+                    +        latestEndDate: Double,
+                    +        valueSum: Double,
+                    +        valueMax: Double
+                    +    ) {
+                    +        var db: OpaquePointer?
+                    +        guard sqlite3_open_v2(url.path, &db, SQLITE_OPEN_READONLY | SQLITE_OPEN_FULLMUTEX, nil) == SQLITE_OK else {
+                    +            sqlite3_close(db)
+                    +            XCTFail("Could not open test database")
+                    +            return (0, 0, 0, 0, 0, 0, 0)
+                    +        }
+                    +        defer { sqlite3_close(db) }
+                    +        let sql = """
+                    +        SELECT visible_record_count, appeared_count, disappeared_count,
+                    +               earliest_start_date, latest_end_date, value_sum, value_max
+                    +        FROM observation_type_summaries
+                    +        WHERE observation_id = ?
+                    +        LIMIT 1
+                    +        """
+                    +        var statement: OpaquePointer?
+                    +        guard sqlite3_prepare_v2(db, sql, -1, &statement, nil) == SQLITE_OK else {
+                    +            sqlite3_finalize(statement)
+                    +            XCTFail("Could not prepare type summary query")
+                    +            return (0, 0, 0, 0, 0, 0, 0)
+                    +        }
+                    +        defer { sqlite3_finalize(statement) }
+                    +        sqlite3_bind_int64(statement, 1, observationID)
+                    +        guard sqlite3_step(statement) == SQLITE_ROW else {
+                    +            XCTFail("Missing type summary row")
+                    +            return (0, 0, 0, 0, 0, 0, 0)
+                    +        }
+                    +        return (
+                    +            visibleRecordCount: Int(sqlite3_column_int(statement, 0)),
+                    +            appearedCount: Int(sqlite3_column_int(statement, 1)),
+                    +            disappearedCount: Int(sqlite3_column_int(statement, 2)),
+                    +            earliestStartDate: sqlite3_column_double(statement, 3),
+                    +            latestEndDate: sqlite3_column_double(statement, 4),
+                    +            valueSum: sqlite3_column_double(statement, 5),
+                    +            valueMax: sqlite3_column_double(statement, 6)
+                    +        )
+                    +    }
+                         private func sampleVersionDebugRows(at url: URL) throws -> String {
+                             var db: OpaquePointer?
+                             guard sqlite3_open_v2(url.path, &db, SQLITE_OPEN_READONLY | SQLITE_OPEN_FULLMUTEX, nil) == SQLITE_OK else {

	@@ -589,7 +589,8 @@ rows exist".
589	589	\| 2026-06-04 \| `4894b77` \| Patch compact archives from delta without full record maps. \| Follow-up full-profile delta report completed in `52.1s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=106, delta=21, initialImport=0`, and `DeltaEvents: 11,093`. This is not comparable to the previous `46`-event baseline: Active Energy had `2,377` events, Basal Energy `2,347`, Cycling Distance `6,052`, and Heart Rate `231`. Processing remained bounded relative to delta size (`SummedProcessingElapsed: 16.0s`; Heart Rate `5.9s`, Active Energy `2.2s`, Basal Energy `1.9s`, Cycling Distance `4.2s`), but wall clock rose because fetch `16.1s`, insert `2.3s`, and finalize `14.8s` all had real work. Conclusion: compact dictionary removal did not regress and looks healthy for large deltas, but a small-delta repeat is still needed to validate the original `6.1s` Heart Rate target. \|
590	590	\| 2026-06-04 \| `1ba6c38` \| Hash delta compact archives in recorded order. \| Small-delta follow-up completed in `24.0s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=124, delta=3, initialImport=0`, and `DeltaEvents: 13`. This finally validated the remaining bottleneck: Heart Rate still spent `5.7s` processing only `6` events, Active Energy spent `2.1s` for `5` events, and Basal Energy spent `1.8s` for `2` events. The delta rebuild no longer builds a full UUID record map, but it still collected every fingerprint into a large array and sorted it for the per-type hash. Delta rebuild now uses the same recorded-order `TypeHashBuilder` strategy as initial import, avoiding the all-record fingerprint array and sort. \|
591	591	\| 2026-06-04 \| `d4de48c` \| Copy unchanged daily aggregates inside SQLite. \| First small-delta run before this commit completed in `23.6s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=120, delta=7`, and `DeltaEvents: 11`. The hash change helped processing (`9.6s -> 8.1s`; Heart Rate `5.7s -> 4.2s`), but finalization stayed high (`11.4s`, Heart Rate `4.8s`). Changed daily aggregates were copying all previous daily rows through Swift before replacing affected buckets. This commit moved unchanged daily aggregate copying to SQLite `INSERT ... SELECT`, while affected buckets remain recalculated. Follow-up report completed in `21.1s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=123, delta=4`, and `DeltaEvents: 50`. `SummedFinalizeElapsed` improved `11.4s -> 9.3s` and wall clock improved `23.6s -> 21.1s`; however Heart Rate finalize was still `5.0s` with `4` events, so this helped overall finalize cost but did not remove the high-volume changed-type floor. \|
592		-\| 2026-06-04 \| pending \| Include build identity in diagnostic reports. \| The latest diagnostic report only included `App Version: 1.0(1)` near the end and did not include a commit/build source identifier. This makes it too easy to compare reports from the wrong installed binary. Diagnostics now emit `appVersion`, `buildFingerprint`, `sourceCommit`, and `sourceDirty` in `OPERATION METADATA`. `buildFingerprint` is derived from the installed executable and should change when a different binary is installed; `sourceCommit/sourceDirty` remain available for builds that inject those Info.plist keys. Expected signal: future pasted reports have enough build identity to detect wrong-version reports immediately. \|
	592	+\| 2026-06-04 \| `c9091de` \| Include build identity in diagnostic reports. \| The latest diagnostic report only included `App Version: 1.0(1)` near the end and did not include a commit/build source identifier. This makes it too easy to compare reports from the wrong installed binary. Diagnostics now emit `appVersion`, `buildFingerprint`, `sourceCommit`, and `sourceDirty` in `OPERATION METADATA`. `buildFingerprint` is derived from the installed executable and should change when a different binary is installed; `sourceCommit/sourceDirty` remain available for builds that inject those Info.plist keys. Expected signal: future pasted reports have enough build identity to detect wrong-version reports immediately. \|
	593	+\| 2026-06-04 \| pending \| Avoid summary full scans when delta safely replaces extremes. \| Follow-up after `d4de48c` still showed `SummedFinalizeElapsed: 9.3s`, with Heart Rate finalize at `5.0s` despite only `4` Heart Rate delta events. `markVerification` already tries to update type summaries incrementally, but it fell back to a full visible-row aggregate scan whenever a removed row matched the previous earliest/latest/max, even if the same delta added a row that safely preserved or extended that extreme. The fallback is now narrower: full scan is still used when an extreme becomes unknown, but not when added rows replace the removed earliest/latest/max with an equivalent or stronger value. Expected signal: on repeated full-profile captures with small recent deltas, `SummedFinalizeElapsed` and high-volume type finalize times, especially Heart Rate, should drop when deltas replace recent/latest rows. Correctness is covered by synthetic SQLite archive tests for replaced latest/max and removed-latest-without-replacement cases. \|
593	594
594	595	## Current Diagnosis
595	596
	@@ -688,6 +689,11 @@ The likely bottleneck is per-row SQLite work:
688	689	from conversation/order unless backed by an external build note. New reports
689	690	should include `buildFingerprint`; `sourceCommit` and `sourceDirty` may still
690	691	be `unknown` unless the build pipeline injects those Info.plist keys.
	692	+- Incremental type-summary finalization was still too conservative around
	693	+ removed extremes. A delta that removes the previous latest/max but adds a
	694	+ newer/larger replacement no longer needs a full visible aggregate scan. The
	695	+ archive still falls back to the full scan when a removed earliest/latest/max
	696	+ leaves the new extreme genuinely unknown.
691	697
692	698	## Open Issues / Observations
693	699
	@@ -746,21 +752,28 @@ Prioritize experiments in this order:
746	752	identity unless the build provenance is otherwise certain. `sourceCommit`
747	753	and `sourceDirty` are useful when present, but may be `unknown` for normal
748	754	Xcode test installs.
749		-8. Investigate full-profile empty anchored-query cost for zero-count types.
	755	+8. Run a repeated full-profile capture after narrowing incremental summary
	756	+ fallback. Compare `SummedFinalizeElapsed` and Heart Rate finalize against the
	757	+ `9.3s` total finalize / `5.0s` Heart Rate finalize report after `d4de48c`.
	758	+ This optimization should help only when changed high-volume metrics replace
	759	+ extremes safely; if Heart Rate finalize remains around `5s`, the remaining
	760	+ cost is likely daily-bucket replacement or compact archive processing rather
	761	+ than type summary full-scan fallback.
	762	+9. Investigate full-profile empty anchored-query cost for zero-count types.
750	763	Compare slow empty types across reports before changing behavior; any skip or
751	764	lower-frequency strategy must preserve the promise that full authorized
752	765	backup can notice newly appearing data.
753		-9. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
754		-10. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.
755		-11. Profile whether index maintenance dominates first-import insert cost.
756		-12. Consider a guarded bulk-import mode for first observations:
	766	+10. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
	767	+11. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.
	768	+12. Profile whether index maintenance dominates first-import insert cost.
	769	+13. Consider a guarded bulk-import mode for first observations:
757	770	- keep archive semantics unchanged;
758	771	- only relax work that can be safely reconstructed or validated;
759	772	- re-enable normal idempotent paths for incremental observations.
760		-13. Run a fresh first-import benchmark after the unused-index removal and compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, and `Active Energy insertElapsed`.
761		-14. Investigate whether first-import-only deferred index creation or temporary staging tables can reduce `samples` / `sample_versions` / `sample_observation_events` write cost without weakening final archive integrity.
762		-15. Revisit adaptive page sizes only after SQLite write-path costs are reduced.
763		-16. Revisit background / scheduled collection once initial import can finish reliably and post-import UI recovery is bounded.
	773	+14. Run a fresh first-import benchmark after the unused-index removal and compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, and `Active Energy insertElapsed`.
	774	+15. Investigate whether first-import-only deferred index creation or temporary staging tables can reduce `samples` / `sample_versions` / `sample_observation_events` write cost without weakening final archive integrity.
	775	+16. Revisit adaptive page sizes only after SQLite write-path costs are reduced.
	776	+17. Revisit background / scheduled collection once initial import can finish reliably and post-import UI recovery is bounded.
764	777
765	778	## Verification Checklist For Each Optimization
766	779