Showing 1 changed files with 8 additions and 6 deletions
+8 -6
HealthProbe/Doc/04-project/Import-Optimization-Log.md
@@ -598,7 +598,7 @@ rows exist".
598 598
 | 2026-06-05 | `bf5a861` | Rebuild changed daily aggregate buckets from time-ranged versions. | Confirmed on two full-profile repeated captures with `buildFingerprint: 1.0(1)-1780640325-92064`. The overnight-data run completed in `21.8s` with `127/127` complete, `CaptureModes: unchangedDelta=111, delta=16`, and `DeltaEvents: 322`; daily aggregate rebuild dropped from `6.9s` to `0.0s`, with daily aggregate work now only `0.6s` copy. The low-delta run completed in `6.6s` with `CaptureModes: unchangedDelta=125, delta=2`; daily aggregate rebuild again stayed `0.0s`. Conclusion: the time-ranged `sample_versions(start_date, sample_id)` query solved the affected-bucket rebuild bottleneck. Continue monitoring first-import insert timing because the new index is a write-path tradeoff. |
599 599
 | 2026-06-05 | `e7d45a2` | Count observation events from the event table first. | Confirmed on a full-profile repeated capture with `buildFingerprint: 1.0(1)-1780646299-92064`: wall clock `15.6s`, `127/127` complete, `CaptureModes: unchangedDelta=114, delta=13`, and `DeltaEvents: 99`. `SummedFinalizeEventCountElapsed` dropped from the post-`bf5a861` overnight baseline `2.5s` to `0.0s`; total finalize dropped to `1.7s`, while daily aggregate rebuild stayed `0.0s`. Remaining cost is processing: `9.1s` total, led by Heart Rate `4.6s` for `18` events, Active Energy `1.8s` for `11` events, and Basal Energy `1.5s` for `5` events. |
600 600
 | 2026-06-05 | `cfd9de8` | Split processing timing diagnostics. | Confirmed on a full-profile repeated capture with `buildFingerprint: 1.0(1)-1780683224-92064`: wall clock `14.7s`, `127/127` complete, `CaptureModes: unchangedDelta=120, delta=7`, and `DeltaEvents: 11`. `SummedProcessingElapsed` was `8.5s` and `SummedProcessingRecordArchiveRebuildElapsed` was also `8.5s`; delta apply, initial record processing, record archive finalization, and processing other all rounded to `0.0s`. Per-type rebuild cost dominated changed high-volume metrics: Heart Rate `4.4s`, Active Energy `1.7s`, Basal Energy `1.5s`, Steps `0.4s`, and Walking + Running Distance `0.4s`. Conclusion: the repeated-capture bottleneck is no longer SQLite finalization; it is the legacy compact `recordArchiveData` rebuild for changed types. |
601
-| 2026-06-05 | `0db2f5e` | Skip legacy compact archive rebuild for SQLite-backed deltas. | Confirmed on a full-profile repeated capture with `buildFingerprint: 1.0(1)-1780689289-92064`: wall clock `6.2s`, `127/127` complete, `CaptureModes: unchangedDelta=118, delta=9`, and `DeltaEvents: 116`. `SummedProcessingElapsed` dropped from `8.5s` to `0.0s`; `SummedProcessingRecordArchiveRebuildElapsed` also dropped from `8.5s` to `0.0s`. Heart Rate had `15` events and completed in `0.2s` with `0.0s` rebuild; Active Energy had `76` events and completed in `0.2s` with `0.0s` rebuild; Basal Energy had `7` events and completed in `0.1s`. Insert stayed low at `0.2s`, finalization stayed bounded at `1.8s`, and no `initialImport` occurred. Conclusion: the legacy compact archive rebuild was the repeated-capture bottleneck and is now removed from the normal SQLite-backed delta path. |
601
+| 2026-06-05 | `0db2f5e` | Skip legacy compact archive rebuild for SQLite-backed deltas. | Confirmed on two full-profile repeated captures. First report, `buildFingerprint: 1.0(1)-1780689289-92064`, completed in `6.2s` with `127/127` complete, `CaptureModes: unchangedDelta=118, delta=9`, `DeltaEvents: 116`, `SummedProcessingElapsed: 0.0s`, and `SummedProcessingRecordArchiveRebuildElapsed: 0.0s`; Heart Rate and Active Energy each completed in `0.2s` with no rebuild. Second report, `buildFingerprint: 1.0(1)-1780689890-92064`, completed in `5.8s` with `CaptureModes: unchangedDelta=121, delta=6`, `DeltaEvents: 7`, fetch `2.0s`, insert `0.1s`, finalize `1.7s`, residual `1.1s`, and processing/rebuild still `0.0s`. Conclusion: the legacy compact archive rebuild was the repeated-capture bottleneck and is now removed from the normal SQLite-backed delta path; the repeated full-profile floor is now roughly `6s` on this device/database. |
602 602
 
603 603
 ## Current Diagnosis
604 604
 
@@ -744,6 +744,9 @@ The likely bottleneck is per-row SQLite work:
744 744
   rebuild for SQLite-backed deltas: repeated full-profile wall clock dropped to
745 745
   `6.2s`, `SummedProcessingElapsed` dropped to `0.0s`, and changed high-volume
746 746
   metrics such as Heart Rate and Active Energy completed in about `0.2s`.
747
+- A second `0db2f5e` report confirmed the new floor is stable: wall clock
748
+  `5.8s`, fetch `2.0s`, insert `0.1s`, finalize `1.7s`, residual `1.1s`,
749
+  `initialImport=0`, and processing/rebuild still `0.0s`.
747 750
 - A large older-build first import on an `8.4M`-record database completed but
748 751
   took `166m10s`, with `137m31s` summed insert time. This confirms that full
749 752
   authorized backup volume can be much larger than the original 15-type test
@@ -807,11 +810,10 @@ Prioritize experiments in this order:
807 810
    identity unless the build provenance is otherwise certain. `sourceCommit`
808 811
    and `sourceDirty` are useful when present, but may be `unknown` for normal
809 812
    Xcode test installs.
810
-8. Run one more repeated full-profile capture after `0db2f5e` to confirm the
811
-   new floor is stable across a second delta shape. Compare wall clock,
812
-   `SummedFetchElapsed`, `SummedInsertElapsed`, `SummedFinalizeElapsed`, and
813
-   `SummedResidualElapsed` against the first validated run: wall `6.2s`, fetch
814
-   `2.1s`, insert `0.2s`, finalize `1.8s`, residual `1.2s`.
813
+8. Treat repeated full-profile captures around `6s` as the current validated
814
+   floor on this device/database. The next repeated-capture optimization should
815
+   target only a measured subphase above about `1s`: currently fetch is around
816
+   `2.0s`, finalize around `1.7s`, and residual around `1.1s`.
815 817
 9. Keep watching first-import insert timing on the next clean large-database
816 818
    import because the new `sample_versions(start_date, sample_id)` index from
817 819
    `bf5a861` is a write-path tradeoff.