Showing 1 changed files with 13 additions and 8 deletions
+13 -8
HealthProbe/Doc/04-project/Import-Optimization-Log.md
@@ -598,7 +598,7 @@ rows exist".
598 598
 | 2026-06-05 | `bf5a861` | Rebuild changed daily aggregate buckets from time-ranged versions. | Confirmed on two full-profile repeated captures with `buildFingerprint: 1.0(1)-1780640325-92064`. The overnight-data run completed in `21.8s` with `127/127` complete, `CaptureModes: unchangedDelta=111, delta=16`, and `DeltaEvents: 322`; daily aggregate rebuild dropped from `6.9s` to `0.0s`, with daily aggregate work now only `0.6s` copy. The low-delta run completed in `6.6s` with `CaptureModes: unchangedDelta=125, delta=2`; daily aggregate rebuild again stayed `0.0s`. Conclusion: the time-ranged `sample_versions(start_date, sample_id)` query solved the affected-bucket rebuild bottleneck. Continue monitoring first-import insert timing because the new index is a write-path tradeoff. |
599 599
 | 2026-06-05 | `e7d45a2` | Count observation events from the event table first. | Confirmed on a full-profile repeated capture with `buildFingerprint: 1.0(1)-1780646299-92064`: wall clock `15.6s`, `127/127` complete, `CaptureModes: unchangedDelta=114, delta=13`, and `DeltaEvents: 99`. `SummedFinalizeEventCountElapsed` dropped from the post-`bf5a861` overnight baseline `2.5s` to `0.0s`; total finalize dropped to `1.7s`, while daily aggregate rebuild stayed `0.0s`. Remaining cost is processing: `9.1s` total, led by Heart Rate `4.6s` for `18` events, Active Energy `1.8s` for `11` events, and Basal Energy `1.5s` for `5` events. |
600 600
 | 2026-06-05 | `cfd9de8` | Split processing timing diagnostics. | Confirmed on a full-profile repeated capture with `buildFingerprint: 1.0(1)-1780683224-92064`: wall clock `14.7s`, `127/127` complete, `CaptureModes: unchangedDelta=120, delta=7`, and `DeltaEvents: 11`. `SummedProcessingElapsed` was `8.5s` and `SummedProcessingRecordArchiveRebuildElapsed` was also `8.5s`; delta apply, initial record processing, record archive finalization, and processing other all rounded to `0.0s`. Per-type rebuild cost dominated changed high-volume metrics: Heart Rate `4.4s`, Active Energy `1.7s`, Basal Energy `1.5s`, Steps `0.4s`, and Walking + Running Distance `0.4s`. Conclusion: the repeated-capture bottleneck is no longer SQLite finalization; it is the legacy compact `recordArchiveData` rebuild for changed types. |
601
-| 2026-06-05 | pending | Skip legacy compact archive rebuild for SQLite-backed deltas. | Changed delta captures now derive the saved count/date range/hash state from the previous SQLite-backed capture state plus the current HealthKit delta events, without decoding and rewriting the legacy compact record archive. TypeCount no longer writes an empty compact archive when no legacy records are present. Expected signal: `SummedProcessingRecordArchiveRebuildElapsed` should drop from the `cfd9de8` baseline `8.5s` toward `0.0s`, with Heart Rate no longer spending about `4.4s` to process tiny deltas. |
601
+| 2026-06-05 | `0db2f5e` | Skip legacy compact archive rebuild for SQLite-backed deltas. | Confirmed on a full-profile repeated capture with `buildFingerprint: 1.0(1)-1780689289-92064`: wall clock `6.2s`, `127/127` complete, `CaptureModes: unchangedDelta=118, delta=9`, and `DeltaEvents: 116`. `SummedProcessingElapsed` dropped from `8.5s` to `0.0s`; `SummedProcessingRecordArchiveRebuildElapsed` also dropped from `8.5s` to `0.0s`. Heart Rate had `15` events and completed in `0.2s` with `0.0s` rebuild; Active Energy had `76` events and completed in `0.2s` with `0.0s` rebuild; Basal Energy had `7` events and completed in `0.1s`. Insert stayed low at `0.2s`, finalization stayed bounded at `1.8s`, and no `initialImport` occurred. Conclusion: the legacy compact archive rebuild was the repeated-capture bottleneck and is now removed from the normal SQLite-backed delta path. |
602 602
 
603 603
 ## Current Diagnosis
604 604
 
@@ -740,6 +740,10 @@ The likely bottleneck is per-row SQLite work:
740 740
   high-volume metrics. The latest report spent `8.5s` in processing and all of
741 741
   it was `processingRecordArchiveRebuildElapsed`, including Heart Rate `4.4s`
742 742
   for a tiny delta.
743
+- The `0db2f5e` follow-up report validated skipping legacy compact archive
744
+  rebuild for SQLite-backed deltas: repeated full-profile wall clock dropped to
745
+  `6.2s`, `SummedProcessingElapsed` dropped to `0.0s`, and changed high-volume
746
+  metrics such as Heart Rate and Active Energy completed in about `0.2s`.
743 747
 - A large older-build first import on an `8.4M`-record database completed but
744 748
   took `166m10s`, with `137m31s` summed insert time. This confirms that full
745 749
   authorized backup volume can be much larger than the original 15-type test
@@ -803,16 +807,17 @@ Prioritize experiments in this order:
803 807
    identity unless the build provenance is otherwise certain. `sourceCommit`
804 808
    and `sourceDirty` are useful when present, but may be `unknown` for normal
805 809
    Xcode test installs.
806
-8. Run a repeated full-profile capture after skipping legacy compact archive
807
-   rebuild for SQLite-backed deltas. Compare
808
-   `SummedProcessingRecordArchiveRebuildElapsed` against the `cfd9de8` baseline:
809
-   `8.5s` total, Heart Rate `4.4s`, Active Energy `1.7s`, Basal Energy `1.5s`.
810
+8. Run one more repeated full-profile capture after `0db2f5e` to confirm the
811
+   new floor is stable across a second delta shape. Compare wall clock,
812
+   `SummedFetchElapsed`, `SummedInsertElapsed`, `SummedFinalizeElapsed`, and
813
+   `SummedResidualElapsed` against the first validated run: wall `6.2s`, fetch
814
+   `2.1s`, insert `0.2s`, finalize `1.8s`, residual `1.2s`.
810 815
 9. Keep watching first-import insert timing on the next clean large-database
811 816
    import because the new `sample_versions(start_date, sample_id)` index from
812 817
    `bf5a861` is a write-path tradeoff.
813
-10. After the compact archive rebuild skip is validated, decide whether the
814
-   legacy `recordArchiveData` field can be retired for new SwiftData TypeCount
815
-   rows or must remain populated only for first-import compatibility.
818
+10. Decide whether the legacy `recordArchiveData` field can be retired for new
819
+   SwiftData TypeCount rows or must remain populated only for first-import
820
+   compatibility. The validated repeated-capture path no longer needs it.
816 821
 11. Investigate full-profile empty anchored-query cost for zero-count types.
817 822
    Compare slow empty types across reports before changing behavior; any skip or
818 823
    lower-frequency strategy must preserve the promise that full authorized