+**Last Updated:** 2026-06-02

+Import performance iterations and measured reports live in [`Import-Optimization-Log.md`](Import-Optimization-Log.md).

+# HealthProbe Import Optimization Log

+

+**Canonical path:** `HealthProbe/Doc/04-project/Import-Optimization-Log.md`  

+**Created:** 2026-06-02  

+**Purpose:** Track import performance work, measured results, regressions, and next experiments.

+

+This is a living project log. Update it after each import optimization commit and after each real-device import report.

+

+## Scope

+

+The current optimization target is the initial / full-history HealthKit import into the SQLite archive.

+

+Primary goals:

+- complete large first-run imports without app freeze or watchdog-like stalls;

+- keep memory bounded for low-end devices;

+- reduce wall-clock duration enough to make future background / scheduled collection realistic;

+- keep archive writes idempotent and differential;

+- preserve SQLite as the source of truth.

+

+Non-goals for this log:

+- redesigning HealthKit background collection strategy;

+- changing archive semantics;

+- optimizing UI rendering after import, except when post-import work blocks the app.

+

+## Measurement Fields

+

+Use the HealthProbe diagnostic report fields below for comparisons:

+

+| Field | Meaning |

+|-------|---------|

+| `WallClockDuration` / `Duration` | User-visible total operation time. |

+| `SummedFetchElapsed` | Time spent fetching HealthKit samples. Per-metric sums may overlap. |

+| `SummedProcessingElapsed` | Time spent converting HealthKit samples into archive rows. |

+| `SummedInsertElapsed` | Time spent writing rows to SQLite. Current main bottleneck. |

+| `SummedFinalizeElapsed` | Type verification, aggregate rebuild, and finalization cost. |

+| Per-type `insertElapsed` | Most useful field for high-volume types such as Heart Rate and Active Energy. |

+

+Important interpretation:

+- per-metric timing sums can exceed wall-clock time because type fetches overlap;

+- progress rates shown during import may overestimate throughput if overhead is not included;

+- compare first snapshots only against first snapshots after a database reset.

+

+## Real-Device Results

+

+### 2026-06-02 Baseline Before Latest Batch/Chunk Work

+

+Source: user-provided diagnostic report.

+

+| Metric | Value |

+|--------|-------|

+| Wall clock | 20m 25s |

+| Summed fetch | 1m 03s |

+| Summed processing | 1m 31s |

+| Summed insert | 17m 02s |

+| Summed finalize | 9.2s |

+| Heart Rate count | 923,466 |

+| Heart Rate insert | 10m 45s |

+| Active Energy insert | 4m 29s |

+| Steps insert | 27.1s |

+| Walking + Running Distance insert | 21.8s |

+

+Conclusion: SQLite insert dominated the run. HealthKit fetch was not the limiting factor.

+

+### 2026-06-02 After Batched Initial Archive Writes

+

+Source: user-provided diagnostic report after commit `a026566`.

+

+| Metric | Value |

+|--------|-------|

+| Wall clock | 18m 21s |

+| Summed insert | 15m 44s |

+| Heart Rate insert | 10m 03s |

+| Active Energy insert | 3m 51s |

+| Steps insert | 28.1s |

+| Walking + Running Distance insert | 25.4s |

+

+Conclusion: batching produced a useful improvement, but insert remained dominant.

+

+### 2026-06-02 After Larger Initial Write Chunks

+

+Source: user-provided diagnostic report after commit `c138b7b`.

+

+| Metric | Value |

+|--------|-------|

+| Wall clock | 18m 30s |

+| Summed metric total | 18m 02s |

+| Summed fetch | 46.8s |

+| Summed processing | 1m 37s |

+| Summed insert | 15m 24s |

+| Summed finalize | 10.5s |

+| Heart Rate count | 922,404 |

+| Heart Rate total | 11m 23s |

+| Heart Rate fetch | 21.2s |

+| Heart Rate processing | 56.1s |

+| Heart Rate insert | 9m 58s |

+| Active Energy count | 348,635 |

+| Active Energy insert | 3m 48s |

+| Steps insert | 24.2s |

+| Walking + Running Distance insert | 20.0s |

+

+Conclusion: larger chunks gave only marginal gains. Further optimization should reduce per-sample SQLite work rather than only increasing page/chunk size.

+

+### 2026-06-02 After Direct Inserts For New Archive Samples

+

+Commit: `44d9ebd` (`Use direct inserts for new archive samples`)  

+Source: no real-device import report yet.

+

+Expected signal:

+- `Heart Rate insertElapsed` should drop first;

+- `SummedInsertElapsed` should drop if most first-import rows are new;

+- no semantic change should appear in diff counts or repeated-page idempotency.

+

+## Optimization Iterations

+

+| Date | Commit | Change | Result / Status |

+|------|--------|--------|-----------------|

+| 2026-06-02 | `fd08ded` | Added explicit fetch / processing / insert / finalize timings to reports. | Made phase comparisons possible without inferring from UI progress. |

+| 2026-06-02 | `87f1a85` | Cached repeated SQLite write-path lookups within grouped imports. | Reduced repeated id lookup pressure in hot path. |

+| 2026-06-02 | `7294a01` | Fast-pathed visibility writes for new archive samples. | Removed redundant visibility close/existence work for brand-new samples. |

+| 2026-06-02 | `585d77f` | Tightened archive verification aggregate queries. | Reduced finalization / verification rescans. |

+| 2026-06-02 | `2dd279c` | Used rowid fast path for new archive sample rows. | Avoided follow-up id lookup queries when SQLite confirmed new inserts. |

+| 2026-06-02 | `f569b6c` | Fixed scheduled test database reset. | Restored ability to compare fresh first-snapshot imports. |

+| 2026-06-02 | `986f343` | Increased Heart Rate import write chunks. | Early attempt to reduce paging/write overhead for the largest metric. |

+| 2026-06-02 | `c1ebd37` | Sped up archive verification finalization. | Reduced finalize pressure; insert remained dominant. |

+| 2026-06-02 | `bcbf9a5` | Cleaned up import diagnostic timings. | Corrected date-fetch wall-clock measurement and report text. |

+| 2026-06-02 | `a026566` | Batched initial import archive writes across several fetched pages. | Wall clock improved from about 20m25s to 18m21s on the measured first import. |

+| 2026-06-02 | `c138b7b` | Increased initial import write chunk sizes. | Marginal improvement: summed insert from 15m44s to 15m24s on the next comparable run. |

+| 2026-06-02 | `44d9ebd` | Used direct inserts for dependent rows when `samples` creates a new sample. | Awaiting real-device report. Tests passed. |

+

+## Current Diagnosis

+

+The import is no longer primarily a HealthKit fetch problem. On the latest measured run:

+

+- total wall clock was 18m30s;

+- summed fetch was only 46.8s;

+- summed insert was 15m24s;

+- Heart Rate alone spent 9m58s inserting.

+

+The likely bottleneck is per-row SQLite work:

+- uniqueness checks on hot tables;

+- index maintenance while importing high-volume rows;

+- multiple dependent writes per sample;

+- commit / transaction shape;

+- Core Data or UI refresh work after SQLite completes, if the app remains unresponsive after import.

+

+## Open Issues / Observations

+

+- Very small pages reduced freeze risk but introduced visible overhead.

+- Some progress timing displayed in the UI did not include overhead, so elapsed time and rates looked better than the real operation.

+- A previous Heart Rate import appeared to stall for long periods around roughly 900k records, but later progress resumed; avoid classifying this as a hard timeout without report evidence.

+- After a completed import, the app may remain unresponsive for more than one minute. This needs separate timing around post-import cache rebuild, UI refresh, report generation, and main-thread work.

+- Partial / old imported observations can pollute comparisons. Fresh first-snapshot performance comparisons should use a confirmed reset database.

+

+## Next Experiments

+

+Prioritize experiments in this order:

+

+1. Run a fresh first-snapshot import after `44d9ebd` and compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, and `Active Energy insertElapsed`.

+2. Add explicit post-import timings if the app is still unresponsive after the operation reports success.

+3. Profile whether index maintenance dominates first-import insert cost.

+4. Consider a guarded bulk-import mode for first observations:

+   - keep archive semantics unchanged;

+   - only relax work that can be safely reconstructed or validated;

+   - re-enable normal idempotent paths for incremental observations.

+5. Revisit adaptive page sizes only after SQLite write-path costs are reduced.

+6. Revisit background / scheduled collection once initial import can finish reliably and post-import UI recovery is bounded.

+

+## Verification Checklist For Each Optimization

+

+- [ ] `git diff --check` passes.

+- [ ] SQLite archive store tests pass.

+- [ ] Import configuration tests pass if capture strategy changed.

+- [ ] Repeated-page/idempotency behavior remains covered.

+- [ ] A real-device report is attached or summarized in this log.

+- [ ] The next experiment is recorded before moving on.

+| Import performance iterations and real-device timing comparisons | [`04-project/Import-Optimization-Log.md`](04-project/Import-Optimization-Log.md) |

+- [`04-project/Import-Optimization-Log.md`](04-project/Import-Optimization-Log.md)

+  Living log for first-import optimization work, measured real-device reports, bottleneck diagnosis, and next experiments.

+