# HealthProbe Import Optimization Log

**Canonical path:** `HealthProbe/Doc/04-project/Import-Optimization-Log.md`  
**Created:** 2026-06-02  
**Purpose:** Track import performance work, measured results, regressions, and next experiments.

This is a living project log. Update it after each import optimization commit and after each real-device import report.

## Scope

The current optimization target is the initial / full-history HealthKit import into the SQLite archive.

Primary goals:
- complete large first-run imports without app freeze or watchdog-like stalls;
- keep memory bounded for low-end devices;
- reduce wall-clock duration enough to make future background / scheduled collection realistic;
- keep archive writes idempotent and differential;
- preserve SQLite as the source of truth.

Non-goals for this log:
- redesigning HealthKit background collection strategy;
- changing archive semantics;
- optimizing UI rendering after import, except when post-import work blocks the app.

## Measurement Fields

Use the HealthProbe diagnostic report fields below for comparisons:

| Field | Meaning |
|-------|---------|
| `WallClockDuration` / `Duration` | User-visible total operation time. |
| `SummedFetchElapsed` | Time spent fetching HealthKit samples. Per-metric sums may overlap. |
| `SummedProcessingElapsed` | Time spent converting HealthKit samples into archive rows. |
| `SummedProcessingRecordArchiveRebuildElapsed` | Time spent rebuilding legacy compact record archives during delta captures. |
| `SummedInsertElapsed` | Time spent writing rows to SQLite. Current main bottleneck. |
| `SummedFinalizeElapsed` | Type verification, aggregate rebuild, and finalization cost. |
| Per-type `insertElapsed` | Most useful field for high-volume types such as Heart Rate and Active Energy. |

Important interpretation:
- per-metric timing sums can exceed wall-clock time because type fetches overlap;
- progress rates shown during import may overestimate throughput if overhead is not included;
- compare first snapshots only against first snapshots after a database reset.

## Benchmark Harness

The repo now keeps one opt-in XCTest benchmark for synthetic large full imports:
- test: `SQLiteLargeImportPerformanceTests.testLargeSyntheticFullImportBenchmark`
- enable with:
  `HP_ENABLE_LARGE_IMPORT_BENCHMARK=1` or
  `-HP_ENABLE_LARGE_IMPORT_BENCHMARK YES`
- optional tuning:
  `HP_LARGE_IMPORT_SAMPLE_COUNT=<count>`,
  `HP_LARGE_IMPORT_ITERATIONS=<count>`, or matching launch arguments
  (`-HP_LARGE_IMPORT_SAMPLE_COUNT <count>`,
  `-HP_LARGE_IMPORT_ITERATIONS <count>`)
- metrics: `XCTClockMetric` and `XCTMemoryMetric`

To keep the path from drifting, the same test file also contains a small
always-on smoke case:
- test: `SQLiteLargeImportPerformanceTests.testLargeSyntheticFullImportSmoke`
- purpose: prove the archive can still complete a synthetic first-import flow in
  normal CI/test runs even when the heavier benchmark stays disabled

Interpretation rules:
- this benchmark measures SQLite archive write/finalize cost, not real HealthKit
  fetch latency or device authorization behavior;
- it is the repeatable regression gate for "can the archive still ingest a very
  large first import with bounded time/memory?";
- a background-import decision still needs one clean real-device full-import
  report from the large database device in addition to this synthetic benchmark.

## Real-Device Results

### 2026-06-07 Overnight Initial Imports On Both Large Devices

Source: two user-provided overnight first-import reports from large devices,
both on build fingerprint `1.0(1)-1780767682-92064`.

Observed behavior:
- report A: `264,844` records, `80 complete / 47 degraded`, wall clock
  `405m 54s`, summed fetch `157m 3s`, summed insert `56.6s`;
- report B: `783,190` records, `51 complete / 76 degraded`, wall clock
  `250m 47s`, summed fetch `237m 43s`, summed insert `9m 5s`;
- the dominant regression was not SQLite write time; it was fetch time spent on
  failed or stalled type-boundary queries;
- one report showed explicit `HKErrorDatabaseInaccessible` (`Protected health
  data is inaccessible`) for many failed types, while the other showed the same
  failure mode amplified by adaptive timeout-based retries.

Interpretation:
- this is primarily a HealthKit availability / device-lock regression path, not
  an archive insert regression;
- debug binaries may add overhead, but they do not explain multi-hour fetch-only
  inflation while insert remains in seconds or single-digit minutes;
- once protected health data becomes inaccessible mid-run, continuing to probe
  all remaining types wastes wall-clock time and pollutes timeout learning.

Action taken:
- detect `HKErrorDatabaseInaccessible` explicitly;
- stop inferring that protected-data failures should keep probing the remaining
  types;
- fast-fail all remaining types in the same snapshot once protected data becomes
  unavailable, instead of paying one timeout/error cycle per metric.

### 2026-06-02 Baseline Before Latest Batch/Chunk Work

Source: user-provided diagnostic report.

| Metric | Value |
|--------|-------|
| Wall clock | 20m 25s |
| Summed fetch | 1m 03s |
| Summed processing | 1m 31s |
| Summed insert | 17m 02s |
| Summed finalize | 9.2s |
| Heart Rate count | 923,466 |
| Heart Rate insert | 10m 45s |
| Active Energy insert | 4m 29s |
| Steps insert | 27.1s |
| Walking + Running Distance insert | 21.8s |

Conclusion: SQLite insert dominated the run. HealthKit fetch was not the limiting factor.

### 2026-06-02 After Batched Initial Archive Writes

Source: user-provided diagnostic report after commit `a026566`.

| Metric | Value |
|--------|-------|
| Wall clock | 18m 21s |
| Summed insert | 15m 44s |
| Heart Rate insert | 10m 03s |
| Active Energy insert | 3m 51s |
| Steps insert | 28.1s |
| Walking + Running Distance insert | 25.4s |

Conclusion: batching produced a useful improvement, but insert remained dominant.

### 2026-06-02 After Larger Initial Write Chunks

Source: user-provided diagnostic report after commit `c138b7b`.

| Metric | Value |
|--------|-------|
| Wall clock | 18m 30s |
| Summed metric total | 18m 02s |
| Summed fetch | 46.8s |
| Summed processing | 1m 37s |
| Summed insert | 15m 24s |
| Summed finalize | 10.5s |
| Heart Rate count | 922,404 |
| Heart Rate total | 11m 23s |
| Heart Rate fetch | 21.2s |
| Heart Rate processing | 56.1s |
| Heart Rate insert | 9m 58s |
| Active Energy count | 348,635 |
| Active Energy insert | 3m 48s |
| Steps insert | 24.2s |
| Walking + Running Distance insert | 20.0s |

Conclusion: larger chunks gave only marginal gains. Further optimization should reduce per-sample SQLite work rather than only increasing page/chunk size.

### 2026-06-02 After Direct Inserts For New Archive Samples

Commit: `44d9ebd` (`Use direct inserts for new archive samples`)  
Source: user-provided first-import diagnostic report after database reset.

Note: an earlier user-provided report looked significantly faster, but it was a
reimport rather than a fresh first snapshot and is not used as a direct
comparison against first-import runs.

| Metric | Value |
|--------|-------|
| Wall clock | 17m 13s |
| Summed metric total | 17m 13s |
| Summed fetch | 43.0s |
| Summed processing | 1m 40s |
| Summed insert | 14m 38s |
| Summed finalize | 9.5s |
| Heart Rate count | 922,431 |
| Heart Rate total | 10m 25s |
| Heart Rate fetch | 20.8s |
| Heart Rate processing | 57.0s |
| Heart Rate insert | 8m 59s |
| Active Energy count | 348,669 |
| Active Energy insert | 3m 54s |
| Steps insert | 24.2s |
| Walking + Running Distance insert | 20.7s |

Comparison against the previous comparable first-import run (`c138b7b`):

| Metric | Previous | Current | Change |
|--------|----------|---------|--------|
| Wall clock | 18m 30s | 17m 13s | -1m 17s / -7% |
| Summed insert | 15m 24s | 14m 38s | -46s / -5% |
| Heart Rate insert | 9m 58s | 8m 59s | -59s / -10% |
| Active Energy insert | 3m 48s | 3m 54s | +6s / +3% |
| Steps insert | 24.2s | 24.2s | flat |
| Walking + Running Distance insert | 20.0s | 20.7s | +0.7s / +4% |

Conclusion: direct inserts for brand-new dependent rows produced a valid but
modest first-import gain. The large reimport improvement was not representative
of a clean first snapshot. SQLite insert remains the dominant bottleneck.

### 2026-06-02 Non-Chain-Start Full Scan After Index Removal

Commit context: after `ff59257` (`Drop unused sample import indexes`)
Source: user-provided diagnostic report with `previousSnapshotID` present and
`isChainStart: false`.

This is not a comparable first-import benchmark for the unused-index removal,
but it is important because it shows that non-initial captures can be slower
than first imports when the app performs a full-history scan.

| Metric | Value |
|--------|-------|
| Wall clock | 22m 33s |
| Summed metric total | 22m 14s |
| Summed fetch | 52.0s |
| Summed processing | 2m 23s |
| Summed insert | 18m 44s |
| Summed finalize | 11.5s |
| Heart Rate count | 922,440 |
| Heart Rate total | 13m 30s |
| Heart Rate fetch | 24.3s |
| Heart Rate processing | 1m 29s |
| Heart Rate insert | 11m 25s |
| Active Energy count | 348,698 |
| Active Energy insert | 4m 44s |
| Steps insert | 40.4s |
| Walking + Running Distance insert | 36.0s |

Conclusion: this run should not be used to judge first-import index removal.
However, it indicates a separate bottleneck: subsequent full scans still spend
most of their time in SQLite writes, likely because unchanged samples are still
touching the archive write path. The next implementation target should reduce
per-sample work for unchanged existing samples during verification/full-scan
captures.

### 2026-06-02 First Import After Index Removal And Reset Fortification

Commit context: after `3dd5f48` (`Fortify scheduled test database reset`), with
the unused index removal from `ff59257` included. Source: user-provided
diagnostic report with `previousSnapshotID: none` and `isChainStart: true`.

This is a comparable first-import benchmark. The `a281c51` verified-event change
is included in the build, but it should not materially affect this run because a
clean first import creates brand-new samples rather than unchanged existing
samples.

| Metric | Value |
|--------|-------|
| Wall clock | 12m 43s |
| Summed metric total | 12m 42s |
| Summed fetch | 40.4s |
| Summed processing | 1m 37s |
| Summed insert | 10m 11s |
| Summed finalize | 10.8s |
| Total records | 1,579,168 |
| Heart Rate count | 922,450 |
| Heart Rate total | 8m 06s |
| Heart Rate fetch | 18.6s |
| Heart Rate processing | 56.1s |
| Heart Rate insert | 6m 41s |
| Active Energy count | 348,701 |
| Active Energy insert | 2m 09s |
| Steps insert | 21.6s |
| Walking + Running Distance insert | 19.2s |

Comparison against the previous comparable first-import run (`44d9ebd`):

| Metric | Previous | Current | Change |
|--------|----------|---------|--------|
| Wall clock | 17m 13s | 12m 43s | -4m 30s / -26% |
| Summed insert | 14m 38s | 10m 11s | -4m 27s / -30% |
| Heart Rate insert | 8m 59s | 6m 41s | -2m 18s / -26% |
| Active Energy insert | 3m 54s | 2m 09s | -1m 45s / -45% |
| Steps insert | 24.2s | 21.6s | -2.6s / -11% |
| Walking + Running Distance insert | 20.7s | 19.2s | -1.5s / -7% |

Conclusion: first-import reset is now clean and the unused-index removal produced
a large measurable gain. SQLite insert remains dominant, but the main bottleneck
has moved from about 14m38s to 10m11s.

### 2026-06-02 First Import After Additional Index Removal

Commit context: after `06ee6be` (`Drop additional import write indexes`). Source:
user-provided diagnostic report with `previousSnapshotID: none`,
`isChainStart: true`, and the same total record count as the previous clean run.

This is a comparable first-import benchmark for removing
`idx_sample_versions_time` and `idx_visibility_sample_open`.

| Metric | Value |
|--------|-------|
| Wall clock | 12m 39s |
| Summed metric total | 12m 39s |
| Summed fetch | 43.8s |
| Summed processing | 1m 35s |
| Summed insert | 10m 07s |
| Summed finalize | 10.1s |
| Total records | 1,579,168 |
| Heart Rate count | 922,450 |
| Heart Rate total | 8m 08s |
| Heart Rate fetch | 19.2s |
| Heart Rate processing | 55.0s |
| Heart Rate insert | 6m 44s |
| Active Energy count | 348,701 |
| Active Energy insert | 2m 07s |
| Steps insert | 20.7s |
| Walking + Running Distance insert | 18.2s |

Comparison against the previous comparable first-import run (`3dd5f48` context):

| Metric | Previous | Current | Change |
|--------|----------|---------|--------|
| Wall clock | 12m 43s | 12m 39s | -4s / -1% |
| Summed insert | 10m 11s | 10m 07s | -4s / -1% |
| Heart Rate insert | 6m 41s | 6m 44s | +3s / +1% |
| Active Energy insert | 2m 09s | 2m 07s | -2s / -2% |
| Steps insert | 21.6s | 20.7s | -0.9s / -4% |
| Walking + Running Distance insert | 19.2s | 18.2s | -1.0s / -5% |

Conclusion: removing these two extra indexes did not materially change first
import performance. The small differences are within expected run-to-run noise.
The larger first-import gain remains attributable to the earlier hot `samples`
index removal plus clean reset conditions.

### 2026-06-02 Incremental No-Delta Snapshot With Post-Import UI Freeze

Commit context: after `06ee6be`. Source: user-provided diagnostic report with
`previousSnapshotID` present, `isChainStart: false`, and the same total record
count as the preceding first snapshot.

| Metric | Value |
|--------|-------|
| Wall clock | 9.2s |
| Summed metric total | 8.9s |
| Summed fetch | 0.2s |
| Summed processing | 0.0s |
| Summed insert | 0.0s |
| Summed finalize | 8.5s |
| Total records | 1,579,168 |
| Heart Rate finalize | 4.8s |
| Active Energy finalize | 1.8s |

Conclusion: repeated no-delta capture is now fast and does not write unchanged
records. The user still observed roughly one minute of app unresponsiveness
after finalization, so the remaining issue is outside the measured import
phases. The most likely culprit is synchronous post-import Core Data cache
rebuild / dashboard refresh work. The dashboard cache refresh was moved to a
background task after this report; next reports should distinguish import time
from post-import UI recovery.

### 2026-06-03 Incremental No-Delta Snapshot After Visibility Fast-Path

Commit context: after `f60f09a`. Source: user-provided diagnostic report with
`previousSnapshotID` present, `isChainStart: false`, and total record count
unchanged at 1,579,168.

| Metric | Value |
|--------|-------|
| Wall clock | 13.6s |
| Summed metric total | 13.2s |
| Summed fetch | 0.2s |
| Summed processing | 0.0s |
| Summed insert | 0.0s |
| Summed finalize | 12.9s |
| Total records | 1,579,168 |
| Heart Rate finalize | 9.1s |
| Active Energy finalize | 1.8s |
| Steps finalize | 0.5s |
| Walking + Running Distance finalize | 0.4s |

Conclusion: the repeated no-delta write path is effectively eliminated. The
remaining measured import cost is finalization, especially Heart Rate daily
aggregate/materialized summary work. The next optimization should avoid
rescanning all visible records when a metric has no appeared, disappeared, or
representation-changed events in the current observation.

### 2026-06-03 Incremental Snapshot After Aggregate Copy Path

Commit context: after `19ba656`. Source: user-provided diagnostic report with
`previousSnapshotID` present and `isChainStart: false`.

This was not a pure no-delta run: total record count increased from 1,579,168 to
1,579,239 (+71 records). That means high-volume changed metrics still needed
normal processing/finalization paths, so the aggregate-copy optimization was not
fully exercised for Heart Rate or Active Energy.

| Metric | Value |
|--------|-------|
| Wall clock | 51.3s |
| Summed metric total | 34.3s |
| Summed fetch | 0.2s |
| Summed processing | 21.1s |
| Summed insert | 0.1s |
| Summed finalize | 12.1s |
| Total records | 1,579,239 |
| Heart Rate processing | 13.9s |
| Heart Rate finalize | 8.7s |
| Active Energy processing | 5.0s |
| Active Energy finalize | 1.9s |
| Steps processing | 1.1s |
| Walking + Running Distance processing | 0.9s |

Comparison against the previous repeated snapshot report:

| Metric | Previous | Current | Change |
|--------|----------|---------|--------|
| Total records | 1,579,168 | 1,579,239 | +71 |
| Wall clock | 13.6s | 51.3s | +37.7s |
| Summed processing | 0.0s | 21.1s | +21.1s |
| Summed insert | 0.0s | 0.1s | +0.1s |
| Summed finalize | 12.9s | 12.1s | -0.8s |
| Heart Rate finalize | 9.1s | 8.7s | -0.4s |

Conclusion: writes remain bounded (`SummedInsertElapsed` 0.1s), but this report
shows a new bottleneck for changed high-volume metrics: processing existing
records when a small number of records changed. A pure no-delta report is still
needed to validate the aggregate-copy path. The unexplained global gap between
wall clock and summed metric total also needs timing if it persists.

### 2026-06-03 Incremental Snapshot With Large Legacy Detail Cache Crash

Commit context: after `19ba656`. Source: user-provided diagnostic report plus
console crash log.

This was still not a pure no-delta run: total record count increased from
1,579,239 to 1,579,243 (+4 records). The report shows that archive inserts
remained effectively eliminated, while aggregate finalization improved versus
the previous changed incremental run.

| Metric | Previous | Current | Change |
|--------|----------|---------|--------|
| Total records | 1,579,239 | 1,579,243 | +4 |
| Wall clock | 51.3s | 44.7s | -6.6s |
| Summed metric total | 34.3s | 26.7s | -7.6s |
| Summed fetch | 0.2s | 0.2s | flat |
| Summed processing | 21.1s | 18.4s | -2.7s |
| Summed insert | 0.1s | 0.0s | -0.1s |
| Summed finalize | 12.1s | 7.4s | -4.7s |
| Heart Rate processing | 13.9s | 13.5s | -0.4s |
| Heart Rate finalize | 8.7s | 4.8s | -3.9s |
| Active Energy processing | 5.0s | 4.9s | -0.1s |
| Active Energy finalize | 1.9s | 1.8s | -0.1s |

The follow-up console log identifies the remaining post-import issue:
`healthKit.precomputeDetailCaches` built legacy SwiftData detail caches for
Heart Rate and Active Energy, scanning archives of about 137 MB / 922k records
and 52 MB / 349k records. Immediately after that work completed, Core Data
aborted on the main thread with a mutated-while-enumerated exception during
change processing.

Conclusion: the SQLite archive path is not the crash source. The expensive and
crash-prone work is the legacy `TypeCount.detailCacheData` precompute after the
snapshot save. Large type detail caches should be skipped and served from the
SQLite/Core Data archive/cache path instead.

### 2026-06-03 Incremental Snapshot After Large Detail Cache Skip

Commit context: after `e49a79d`. Source: user-provided diagnostic report. The
user still observed app unresponsiveness after the operation completed.

| Metric | Previous | Current | Change |
|--------|----------|---------|--------|
| Total records | 1,579,243 | 1,579,253 | +10 |
| Wall clock | 44.7s | 31.3s | -13.4s |
| Summed metric total | 26.7s | 30.6s | +3.9s |
| Summed fetch | 0.2s | 0.2s | flat |
| Summed processing | 18.4s | 18.1s | -0.3s |
| Summed insert | 0.0s | 0.0s | flat |
| Summed finalize | 7.4s | 11.5s | +4.1s |
| Heart Rate processing | 13.5s | 13.1s | -0.4s |
| Heart Rate finalize | 4.8s | 8.8s | +4.0s |
| Active Energy processing | 4.9s | 4.7s | -0.2s |
| Active Energy finalize | 1.8s | 1.9s | +0.1s |

Conclusion: the large legacy detail-cache skip removed the earlier crash signal
and reduced wall clock, but the app can still become unresponsive after the
snapshot reports success. The remaining suspect is the Dashboard Core Data cache
refresh because the view model still awaited cache rebuild before releasing the
post-snapshot UI path. That refresh should run fire-and-forget and publish its
result back to the main actor only when complete.

### 2026-06-03 Final Freeze Log: Small Detail Cache Still Crashes

Source: user-provided overnight freeze/crash log after the large-cache skip.

The log shows `healthKit.precomputeDetailCaches` still building two small
legacy SwiftData detail caches:

| Type | Current archive | Current count | Result |
|------|-----------------|---------------|--------|
| Stand Hours | 1.1 MB | 7,727 | built, 2 added |
| Environmental Sound Levels | 2 MB | 13,384 | built, 1 added |

Heart Rate and Active Energy were correctly skipped by the large-cache guard,
but Core Data still aborted immediately after `healthKit.precomputeDetailCaches.end`
with the same mutated-while-enumerated exception during change processing.

Conclusion: the issue is not only large archive size. Mutating legacy
`TypeCount.detailCacheData` during snapshot save is unsafe in this SwiftData /
Core Data context. Snapshot save must not precompute or clear this legacy cache.
The Snapshots list should also read observation timeline rows directly from
SQLite so it does not depend on a delayed Core Data cache rebuild to show the
freshly finished observation.

### 2026-06-03 Successful Snapshot Still Freezes After Copying Report

Commit context: after `1229f19`. Source: user-provided diagnostic report copied
from the app before a new freeze. Unlike the previous overnight log, this report
contains no `healthKit.detailCache.buildBegin` evidence and the snapshot itself
completed successfully.

| Metric | Previous | Current | Change |
|--------|----------|---------|--------|
| Total records | 1,579,253 | 1,579,445 | +192 |
| Wall clock | 31.3s | 35.5s | +4.2s |
| Summed metric total | 30.6s | 35.0s | +4.4s |
| Summed fetch | 0.2s | 0.2s | flat |
| Summed processing | 18.1s | 21.0s | +2.9s |
| Summed insert | 0.0s | 0.2s | +0.2s |
| Summed finalize | 11.5s | 12.6s | +1.1s |

Conclusion: the legacy detail-cache crash path was removed, but the app can
still freeze after the user copies the diagnostic report. The remaining
post-snapshot culprit is the automatic Dashboard Core Data cache rebuild, which
can consume enough device I/O/CPU to make the app appear frozen even when run
from a detached task. Automatic post-snapshot refresh should read the small
timeline/status data directly from SQLite; full Core Data cache rebuild should
remain explicit/manual until partial invalidation exists.

### 2026-06-03 Core Data Cache Rebuild Crash Stack

Source: user-provided LLDB backtrace after a fast crash. The stack still pointed
to an app binary that called `CoreDataArchiveCacheStore.rebuild` from
`DashboardViewModel.startArchiveCacheRefresh`, which was removed in `199d2ef`.
However, the stack also exposed a real cache-store bug: rebuild inserted
`NSManagedObject`s through `container.viewContext` while running on a Swift
utility task.

Crash location:

- `CoreDataArchiveCacheStore.insertDailyAggregateRows`
- `NSEntityDescription.insertNewObject`
- `NSManagedObjectContext insertObject`
- Core Data `__CFBasicHashAddValue` / `EXC_BAD_ACCESS`

Conclusion: even though automatic post-snapshot rebuild has been removed, manual
cache rebuild must also be safe. `CoreDataArchiveCacheStore.rebuild` and
`deleteCache` should use a dedicated background context and perform all Core
Data mutations on that context's queue.

### 2026-06-03 No Crash, But Snapshot Detail Lost Data Types

Commit context: after disabling automatic/legacy Core Data cache work. Source:
three user-provided reports and a screenshot of `SnapshotArchiveDetailView`.

Observed sequence:

| Time | Total records | Duration | Notes |
|------|---------------|----------|-------|
| 13:36 | 1,579,596 | 36.2s | Successful incremental snapshot, no crash. |
| 13:45 | 1,579,601 | 31.3s | Not a true no-delta run: +5 records versus the prior snapshot. |
| 13:47 | 1,579,601 | 2.5s | True no-delta run; processing and insert were 0.0s, finalize about 2.0s. |

The detail screen showed `Metrics: 15`, `Records: 1579596`, and
`Record Changes: 175`, but `Data Range` and `Data Types` were empty.

Conclusion: the archive/import path is working, and no-delta is now fast. The
UI regression came from remaining reads of `CachedTypeSummary` and
`CachedDiffSummary` through `CoreDataArchiveCacheStore`. Since automatic cache
rebuild is intentionally disabled to prevent freezes/crashes, snapshot detail,
Data Types, drilldown screens, and charts must read materialized SQLite archive
summaries directly. Do not re-enable automatic full Core Data rebuild as a fix
for missing UI details.

### 2026-06-03 Data Types Restored, Startup Shows False Empty State

Commit context: after `2a82f67`. Source: user confirmation, screenshots, and
two reports.

The user confirmed that Snapshot detail data types reappeared. A follow-up
snapshot at 14:10 was a true no-delta run:

| Metric | Value |
|--------|-------|
| Total records | 1,579,611 |
| Wall clock | 3.0s |
| Summed fetch | 0.2s |
| Summed processing | 0.0s |
| Summed insert | 0.0s |
| Summed finalize | 2.2s |

Screenshots also showed a UI startup issue: the app briefly opened Dashboard /
Snapshots as if there were no archive observations, then populated the real
SQLite rows about three seconds later.

Conclusion: this is a loading-state bug, not an archive loss. Main tabs must
distinguish "SQLite rows are still loading" from "SQLite query completed and no
rows exist".

## Optimization Iterations

| Date | Commit | Change | Result / Status |
|------|--------|--------|-----------------|
| 2026-06-02 | `fd08ded` | Added explicit fetch / processing / insert / finalize timings to reports. | Made phase comparisons possible without inferring from UI progress. |
| 2026-06-02 | `87f1a85` | Cached repeated SQLite write-path lookups within grouped imports. | Reduced repeated id lookup pressure in hot path. |
| 2026-06-02 | `7294a01` | Fast-pathed visibility writes for new archive samples. | Removed redundant visibility close/existence work for brand-new samples. |
| 2026-06-02 | `585d77f` | Tightened archive verification aggregate queries. | Reduced finalization / verification rescans. |
| 2026-06-02 | `2dd279c` | Used rowid fast path for new archive sample rows. | Avoided follow-up id lookup queries when SQLite confirmed new inserts. |
| 2026-06-02 | `f569b6c` | Fixed scheduled test database reset. | Restored ability to compare fresh first-snapshot imports. |
| 2026-06-02 | `986f343` | Increased Heart Rate import write chunks. | Early attempt to reduce paging/write overhead for the largest metric. |
| 2026-06-02 | `c1ebd37` | Sped up archive verification finalization. | Reduced finalize pressure; insert remained dominant. |
| 2026-06-02 | `bcbf9a5` | Cleaned up import diagnostic timings. | Corrected date-fetch wall-clock measurement and report text. |
| 2026-06-02 | `a026566` | Batched initial import archive writes across several fetched pages. | Wall clock improved from about 20m25s to 18m21s on the measured first import. |
| 2026-06-02 | `c138b7b` | Increased initial import write chunk sizes. | Marginal improvement: summed insert from 15m44s to 15m24s on the next comparable run. |
| 2026-06-02 | `44d9ebd` | Used direct inserts for dependent rows when `samples` creates a new sample. | Confirmed modest first-import gain: wall clock 18m30s -> 17m13s, summed insert 15m24s -> 14m38s, Heart Rate insert 9m58s -> 8m59s. |
| 2026-06-02 | `ff59257` | Removed unused `samples` indexes on global UUID hash and semantic fingerprint. | Confirmed large first-import gain after clean reset: wall clock 17m13s -> 12m43s, summed insert 14m38s -> 10m11s, Heart Rate insert 8m59s -> 6m41s. Deleted-object lookup remains covered by `(sample_type_id, sample_uuid_hash)`. |
| 2026-06-02 | pending | Captured non-chain-start full-scan report after index removal. | Not comparable for first-import performance; reveals a separate full-scan/unchanged-sample write bottleneck. |
| 2026-06-02 | `a281c51` | Stopped writing `verified` observation events for unchanged existing samples. | Awaiting comparable non-chain-start/full-scan report. Expected signal is lower `SummedInsertElapsed` and especially lower Heart Rate insert time when most rows are unchanged. |
| 2026-06-02 | `3dd5f48` | Fortified scheduled test database reset with a disk marker and extra SQLite sidecar cleanup. | Real-device report confirmed reset produced `previousSnapshotID: none`, `isChainStart: true`, and a clean first-snapshot timeline. |
| 2026-06-02 | `06ee6be` | Removed unused `sample_versions(start_date, end_date)` and redundant `sample_visibility_ranges(sample_id, last_observation_id)` indexes. | Comparable first-import report was flat: wall clock 12m43s -> 12m39s and summed insert 10m11s -> 10m07s. Treat as no material performance change. |
| 2026-06-02 | pending | Moved Dashboard archive cache refresh/rebuild off the UI path after snapshot completion. | Awaiting real-device confirmation that the app no longer stays unresponsive for roughly one minute after a completed snapshot. |
| 2026-06-03 | `f60f09a` | Fast-path unchanged samples whose current version already has an open visibility range. | Confirmed on repeated no-delta capture: `SummedInsertElapsed` remained 0.0s; remaining cost is `SummedFinalizeElapsed` 12.9s, with Heart Rate finalize 9.1s. |
| 2026-06-03 | `19ba656` | Copy previous type summaries and daily aggregates for unchanged metric observations instead of rebuilding from visible ranges. | First follow-up was not no-delta (+71 records), so it is inconclusive for the unchanged path. It did show `SummedInsertElapsed` 0.1s and a new changed-metric processing bottleneck: Heart Rate processing 13.9s, Active Energy processing 5.0s. |
| 2026-06-03 | `e49a79d` | Skip legacy SwiftData detail-cache precompute for large type archives. | Triggered by a crash after building Heart Rate and Active Energy detail caches. Expected signal: no post-import `NSGenericException`, lower post-import wall-clock gap, and `healthKit.detailCache.skipLargeLegacyArchive` logs for high-volume types. |
| 2026-06-03 | `7d52262` | Start Dashboard archive cache refresh without awaiting it after snapshot completion. | Triggered by continued app unresponsiveness after a successful 31.3s incremental snapshot. Expected signal: progress sheet/result UI remains responsive while cache rows refresh later. |
| 2026-06-03 | `1229f19` | Disable legacy SwiftData detail-cache precompute completely and load Snapshots timeline from SQLite. | Triggered by overnight crash after two small detail caches were built. Expected signal: no `healthKit.detailCache.buildBegin` logs during snapshot save, no Core Data mutated-while-enumerated abort, and the new SQLite observation appears in Snapshots without waiting for cache rebuild. |
| 2026-06-03 | `199d2ef` | Stop automatic Dashboard Core Data cache rebuild after snapshot; refresh latest rows from SQLite only. | Triggered by freeze after copying a successful diagnostic report. Expected signal: copying diagnostics and returning to Dashboard/Snapshots remains responsive; Core Data cache rebuild is no longer started automatically after snapshot completion. |
| 2026-06-03 | `3abf63d` | Run Core Data cache rebuild/delete on a dedicated background context. | Triggered by `EXC_BAD_ACCESS` inside Core Data object insertion during cache rebuild. Expected signal: manual Settings cache rebuild no longer crashes due to `NSManagedObjectContext` queue misuse. |
| 2026-06-03 | `2a82f67` | Load snapshot/type detail UI from SQLite materialized summaries instead of Core Data cache. | Triggered by successful snapshots whose detail screens showed no data types after automatic cache rebuild was disabled. Expected signal: Snapshot detail, Data Types, per-type drilldown, and evolution chart show current archive details without rebuilding Core Data cache. |
| 2026-06-03 | `ec7ee29` | Add explicit loading states for Dashboard, Snapshots, and Data Types archive rows. | Triggered by false "no observations/no snapshots/not enough data" states during the first few seconds after app launch. Expected signal: startup shows loading state until SQLite rows are available, then shows real archive data without flicker. |
| 2026-06-03 | `e231eaf` | Use the HealthKit registry as SQLite sample type display-name fallback. | Triggered by Snapshot detail showing raw identifiers such as `HKCategoryTypeIdentifierAppleStandHour` after UI moved from Core Data cache to SQLite summaries. Expected signal: existing and new archive rows show human-readable names such as `Stand Hours` without requiring reset/reimport. |
| 2026-06-03 | `5fafcdd` | Expand the HealthKit type registry for full-dataset discovery while keeping the original 15-type profile as the tested default. | Triggered by the decision that import/storage cannot be considered complete based only on the restricted v1 dataset. Expected signal: Settings/authorization can expose a much broader quantity/category/workout catalog, unsupported types are explicit, and real-device coverage reports can measure full authorized backup volume. |
| 2026-06-03 | committed | Add explicit capture profile controls for full-dataset discovery. | A real-device report after registry expansion still showed `Types: 15/15 processed` and the old monitored type-set hash because `selectedTypeIDs` persisted the v1 core profile in `UserDefaults`. Settings now exposes `Select All Available Types`, `Select Core Profile`, and selected/available counts so the next real-device run can deliberately switch from the v1 sample set to the expanded supported registry. |
| 2026-06-03 | pending | Migrate legacy core-profile selections to full available capture by default. | A follow-up real-device report still showed the old `4907...` monitored type-set hash and `Types: 15/15 processed`, proving the running app still used the old persisted selected type set. New installs and pre-profile settings that exactly match the old core profile now migrate to `All available`; only an explicit `Select Core Profile` action persists the core subset. Settings also shows the active profile label (`All available`, `Core`, or `Custom`) for quick verification before capture. |
| 2026-06-03 | pending | Harden quantity unit conversion for full-profile imports. | The first 127-metric run crashed while archiving `HKQuantityTypeIdentifierDietaryWater`: the previous fallback converted unknown quantities with `.count()`, and HealthKit raised `NSInvalidArgumentException` for incompatible `mL` to `count`. The archive now maps known extended units, stores quantity rows with nil numeric/unit when a future type is unmapped, and removes the unsafe display fallback. Next run should pass Water and reveal any remaining type-specific unit gaps. |
| 2026-06-03 | pending | Preserve completed import diagnostics and render large diagnostic reports lazily. | A copied full-profile reimport report completed successfully: `Types: 127/127 processed`, `Records: 2,646,527`, `WallClock: 40.3s`, `SummedProcessingElapsed: 21.7s`, `SummedFinalizeElapsed: 15.1s`, `SummedFetchElapsed: 1.7s`, `SummedInsertElapsed: 0.1s`, and `0` degraded metrics. The Diagnostics sheet copied the report text but displayed a blank body for the huge report; completed/partial/review reports are now persisted under Application Support and displayed in chunked lazy text blocks. |
| 2026-06-03 | pending | Fast-path unchanged verification after an empty HealthKit delta page. | The successful full-profile reimport spent `15.1s` in finalize despite `0.1s` insert time; Heart Rate alone spent `9.0s` finalizing. The empty-delta path now calls an explicit unchanged-verification store method that copies the previous type summary and daily aggregates without first scanning sample events to prove zero changes. Expected signal: repeated no-delta/full-profile `SummedFinalizeElapsed` and Heart Rate finalize time drop sharply. |
| 2026-06-03 | pending | Persist HealthKit capture anchors in SQLite. | A later no-delta attempt still reimported full high-volume types (`Records: 2,646,590`, `WallClock: 45.8s`, Heart Rate `922,521`, Active Energy `346,473`) because incremental anchors were still read from legacy SwiftData `TypeCount`, while the refactored app now chains observations from SQLite. SQLite now stores per-type capture state (`anchor_data`, count, content hash, yearly counts, range) and the import path uses it when legacy `TypeCount` state is missing. Expected signal: the first run after this change may seed anchors; the following no-delta run should report empty HealthKit delta pages instead of full `record_import` counts for large stable types. |
| 2026-06-03 | pending | Clarify capture mode in record-import diagnostics. | Two full-profile repeated snapshots after SQLite capture-state persistence completed successfully with stable checksum and identical record count (`2,646,613`). The first ran in `6.3s` with `SummedProcessingElapsed: 0.0s`, `SummedInsertElapsed: 0.0s`, `SummedFinalizeElapsed: 2.6s`; the second ran in `6.4s` with `0.0s` processing/insert and `2.7s` finalize. Heart Rate `922,526` and Active Energy `346,478` each completed around `0.2s`, proving the heavy full reimport path was avoided. The report wording still said "`N` samples in 1 anchored segment", which is ambiguous for inherited unchanged summaries; diagnostics now label unchanged empty-delta, delta-apply, and full-import modes explicitly. |
| 2026-06-03 | pending | Add capture-mode summary to diagnostics. | Repeated full-profile captures rarely produce a perfect no-delta report because at least one metric can change between manual runs. Diagnostic reports now include aggregate `CaptureModes` counts plus per-metric `captureMode`, so comparisons can separate unchanged empty-delta metrics from delta-applied metrics and full imports without manually reading every `record_import` line. Expected signal: stable checksum plus high `unchangedDelta` count and zero summed processing/insert confirms the fast path even when a few metrics changed. |
| 2026-06-03 | pending | Add delta-event counts to diagnostics. | A full-profile follow-up completed in `47.4s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=115, delta=12, initialImport=0`, `SummedProcessingElapsed: 25.9s`, `SummedInsertElapsed: 0.2s`, and `SummedFinalizeElapsed: 16.0s`. This confirms anchors work and no full import ran. Remaining cost is delta application for large metrics: Heart Rate `23.5s` total (`14.1s` processing, `8.8s` finalize), Active Energy `7.1s`, and Basal Energy `6.0s`. Diagnostics now report aggregate/per-metric `DeltaEvents` so future logs can separate true HealthKit delta size from the final visible record count. |
| 2026-06-04 | `a676df1` | Rebuild delta compact archives without large intermediate record arrays. | Follow-up full-profile delta report completed in `42.1s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=108, delta=19, initialImport=0`, and `DeltaEvents: 582`. Compared with the prior `47.4s` run, `SummedProcessingElapsed` dropped `25.9s -> 12.8s`; Heart Rate processing dropped `14.1s -> 6.4s` despite `187` delta events; Active Energy processing dropped `4.9s -> 2.5s` with `100` delta events; Basal Energy processing dropped `4.1s -> 2.0s` with `82` delta events. `SummedFinalizeElapsed` rose `16.0s -> 19.6s`, so the remaining bottleneck is now archive finalization / aggregate rebuild for changed high-volume types, not Swift archive reconstruction. |
| 2026-06-04 | `457fd80` | Incrementally replace changed daily aggregate buckets. | Follow-up full-profile delta report completed in `31.2s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=119, delta=8, initialImport=0`, and `DeltaEvents: 22`. Compared with the prior `42.1s` run, `SummedFinalizeElapsed` dropped `19.6s -> 15.5s`, Active Energy finalize dropped `4.8s -> 1.8s`, and total wall clock dropped `42.1s -> 31.2s`. Heart Rate finalize barely moved (`8.9s -> 8.7s`) despite only `5` delta events, proving that daily aggregate replacement helped some changed metrics but Heart Rate is still dominated by type summary `visibleAggregate` full scans. |
| 2026-06-04 | `2ebfab3` | Incrementally update changed type summaries. | Follow-up full-profile delta report completed in `27.5s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=118, delta=9, initialImport=0`, and `DeltaEvents: 46`. Compared with the prior `31.2s` run, `SummedFinalizeElapsed` dropped `15.5s -> 11.7s`; Heart Rate finalize dropped `8.7s -> 4.8s`; Active Energy finalize stayed bounded at `1.7s`. Remaining cost moved back to delta archive processing: `SummedProcessingElapsed` was `11.6s`, with Heart Rate processing `6.1s`, Active Energy `2.3s`, and Basal Energy `1.9s` for small deltas. |
| 2026-06-04 | `4894b77` | Patch compact archives from delta without full record maps. | Follow-up full-profile delta report completed in `52.1s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=106, delta=21, initialImport=0`, and `DeltaEvents: 11,093`. This is not comparable to the previous `46`-event baseline: Active Energy had `2,377` events, Basal Energy `2,347`, Cycling Distance `6,052`, and Heart Rate `231`. Processing remained bounded relative to delta size (`SummedProcessingElapsed: 16.0s`; Heart Rate `5.9s`, Active Energy `2.2s`, Basal Energy `1.9s`, Cycling Distance `4.2s`), but wall clock rose because fetch `16.1s`, insert `2.3s`, and finalize `14.8s` all had real work. Conclusion: compact dictionary removal did not regress and looks healthy for large deltas, but a small-delta repeat is still needed to validate the original `6.1s` Heart Rate target. |
| 2026-06-04 | `1ba6c38` | Hash delta compact archives in recorded order. | Small-delta follow-up completed in `24.0s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=124, delta=3, initialImport=0`, and `DeltaEvents: 13`. This finally validated the remaining bottleneck: Heart Rate still spent `5.7s` processing only `6` events, Active Energy spent `2.1s` for `5` events, and Basal Energy spent `1.8s` for `2` events. The delta rebuild no longer builds a full UUID record map, but it still collected every fingerprint into a large array and sorted it for the per-type hash. Delta rebuild now uses the same recorded-order `TypeHashBuilder` strategy as initial import, avoiding the all-record fingerprint array and sort. |
| 2026-06-04 | `d4de48c` | Copy unchanged daily aggregates inside SQLite. | First small-delta run before this commit completed in `23.6s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=120, delta=7`, and `DeltaEvents: 11`. The hash change helped processing (`9.6s -> 8.1s`; Heart Rate `5.7s -> 4.2s`), but finalization stayed high (`11.4s`, Heart Rate `4.8s`). Changed daily aggregates were copying all previous daily rows through Swift before replacing affected buckets. This commit moved unchanged daily aggregate copying to SQLite `INSERT ... SELECT`, while affected buckets remain recalculated. Follow-up report completed in `21.1s` with `127/127` complete, `0` degraded, `CaptureModes: unchangedDelta=123, delta=4`, and `DeltaEvents: 50`. `SummedFinalizeElapsed` improved `11.4s -> 9.3s` and wall clock improved `23.6s -> 21.1s`; however Heart Rate finalize was still `5.0s` with `4` events, so this helped overall finalize cost but did not remove the high-volume changed-type floor. |
| 2026-06-04 | `c9091de` | Include build identity in diagnostic reports. | The latest diagnostic report only included `App Version: 1.0(1)` near the end and did not include a commit/build source identifier. This makes it too easy to compare reports from the wrong installed binary. Diagnostics now emit `appVersion`, `buildFingerprint`, `sourceCommit`, and `sourceDirty` in `OPERATION METADATA`. `buildFingerprint` is derived from the installed executable and should change when a different binary is installed; `sourceCommit/sourceDirty` remain available for builds that inject those Info.plist keys. Expected signal: future pasted reports have enough build identity to detect wrong-version reports immediately. |
| 2026-06-04 | `7e3b997` | Avoid summary full scans when delta safely replaces extremes. | Follow-up after `d4de48c` still showed `SummedFinalizeElapsed: 9.3s`, with Heart Rate finalize at `5.0s` despite only `4` Heart Rate delta events. `markVerification` already tries to update type summaries incrementally, but it fell back to a full visible-row aggregate scan whenever a removed row matched the previous earliest/latest/max, even if the same delta added a row that safely preserved or extended that extreme. The fallback is now narrower: full scan is still used when an extreme becomes unknown, but not when added rows replace the removed earliest/latest/max with an equivalent or stronger value. Follow-up report with `reportSchemaVersion: 3` and `buildFingerprint: 1.0(1)-1780603665-92064` completed in `21.0s`, with `127/127` complete, `CaptureModes: unchangedDelta=121, delta=6`, and `DeltaEvents: 20`. This was effectively flat versus the `d4de48c` report: wall `21.1s -> 21.0s`, processing `7.8s -> 7.4s`, finalize `9.3s -> 9.4s`, and Heart Rate finalize `5.0s -> 4.8s` while Heart Rate had `7` delta events. Conclusion: this optimization did not materially move the bottleneck; either the safe-extreme case did not trigger or type-summary fallback is no longer the dominant finalize cost. |
| 2026-06-04 | `f73f076` | Add archive finalization phase timings to diagnostics. | The post-`7e3b997` report proved the top-level `finalizeElapsed` bucket was too opaque for the next optimization. Diagnostics now split finalization into event-count/previous-summary lookup, type-summary work, daily-aggregate work, observation-type-run update, and residual other time. Follow-up report with `reportSchemaVersion: 3` and `buildFingerprint: 1.0(1)-1780606903-92064` completed in `22.6s`, with `127/127` complete, `CaptureModes: unchangedDelta=119, delta=8`, and `DeltaEvents: 27`. Finalization was `10.3s`: event-count/previous-summary lookup `1.8s`, type-summary `0.0s`, daily aggregates `7.3s`, run update `0.0s`, and other `1.2s`. Heart Rate had `9` delta events and spent `4.8s` finalizing, of which `3.8s` was daily aggregate work and `0.9s` was event-count/previous-summary lookup. Conclusion: the remaining finalize bottleneck is not type-summary fallback; it is changed-type daily aggregate maintenance, especially Heart Rate. |
| 2026-06-04 | older build / schema v2 | Captured large first-import baseline on a bigger device database. | Initial full-profile snapshot on an older build completed with `127/127` metrics and `8,421,978` records, but it used `reportSchemaVersion: 2` and has no build fingerprint. Treat it as a volume/shape baseline, not a precise current-build comparison. Wall clock was `166m10s`; summed fetch `5m19s`, processing `20m29s`, insert `137m31s`, finalize `1m53s`. The high-volume types dominated: Heart Rate `2,225,738` records and `46m57s` total (`39m16s` insert), Active Energy `1,914,449` records and `41m35s` total (`35m21s` insert), another high-volume type around `2,007,920` records and `41m20s` total (`34m29s` insert), and Basal Energy `1,116,074` records and `21m37s` total (`17m48s` insert). Conclusion: for clean first imports on very large databases, SQLite insert/index/write-path cost remains the central risk; incremental daily-aggregate optimization should not add first-import indexes without measurement. |
| 2026-06-05 | `6041bac` | Split daily aggregate finalization timings. | The first finalization phase report identified daily aggregate work as the remaining changed-type bottleneck, but `finalizeDailyAggregateElapsed` still mixed affected-bucket lookup, previous aggregate copy, destination delete, affected-bucket rebuild, replacement insert, and residual SQL/transaction overhead. Diagnostics now emit aggregate and per-type daily subphase fields: bucket lookup, copy, delete, rebuild, insert, and other. Follow-up report with `buildFingerprint: 1.0(1)-1780618540-92064` completed in `23.5s`, with `127/127` complete, `CaptureModes: unchangedDelta=118, delta=9`, and `DeltaEvents: 97`. Finalization was `10.5s`, daily aggregate work was `7.4s`, and daily rebuild alone was `6.9s`; daily copy was only `0.5s`. Heart Rate had `40` delta events and spent `4.8s` finalizing, of which `3.8s` was daily aggregate rebuild. Conclusion: copying previous materialized daily rows is not the bottleneck; affected-bucket rebuild scans are. |
| 2026-06-05 | `bf5a861` | Rebuild changed daily aggregate buckets from time-ranged versions. | Confirmed on two full-profile repeated captures with `buildFingerprint: 1.0(1)-1780640325-92064`. The overnight-data run completed in `21.8s` with `127/127` complete, `CaptureModes: unchangedDelta=111, delta=16`, and `DeltaEvents: 322`; daily aggregate rebuild dropped from `6.9s` to `0.0s`, with daily aggregate work now only `0.6s` copy. The low-delta run completed in `6.6s` with `CaptureModes: unchangedDelta=125, delta=2`; daily aggregate rebuild again stayed `0.0s`. Conclusion: the time-ranged `sample_versions(start_date, sample_id)` query solved the affected-bucket rebuild bottleneck. Continue monitoring first-import insert timing because the new index is a write-path tradeoff. |
| 2026-06-05 | `e7d45a2` | Count observation events from the event table first. | Confirmed on a full-profile repeated capture with `buildFingerprint: 1.0(1)-1780646299-92064`: wall clock `15.6s`, `127/127` complete, `CaptureModes: unchangedDelta=114, delta=13`, and `DeltaEvents: 99`. `SummedFinalizeEventCountElapsed` dropped from the post-`bf5a861` overnight baseline `2.5s` to `0.0s`; total finalize dropped to `1.7s`, while daily aggregate rebuild stayed `0.0s`. Remaining cost is processing: `9.1s` total, led by Heart Rate `4.6s` for `18` events, Active Energy `1.8s` for `11` events, and Basal Energy `1.5s` for `5` events. |
| 2026-06-05 | `cfd9de8` | Split processing timing diagnostics. | Confirmed on a full-profile repeated capture with `buildFingerprint: 1.0(1)-1780683224-92064`: wall clock `14.7s`, `127/127` complete, `CaptureModes: unchangedDelta=120, delta=7`, and `DeltaEvents: 11`. `SummedProcessingElapsed` was `8.5s` and `SummedProcessingRecordArchiveRebuildElapsed` was also `8.5s`; delta apply, initial record processing, record archive finalization, and processing other all rounded to `0.0s`. Per-type rebuild cost dominated changed high-volume metrics: Heart Rate `4.4s`, Active Energy `1.7s`, Basal Energy `1.5s`, Steps `0.4s`, and Walking + Running Distance `0.4s`. Conclusion: the repeated-capture bottleneck is no longer SQLite finalization; it is the legacy compact `recordArchiveData` rebuild for changed types. |
| 2026-06-05 | `0db2f5e` | Skip legacy compact archive rebuild for SQLite-backed deltas. | Confirmed on two full-profile repeated captures. First report, `buildFingerprint: 1.0(1)-1780689289-92064`, completed in `6.2s` with `127/127` complete, `CaptureModes: unchangedDelta=118, delta=9`, `DeltaEvents: 116`, `SummedProcessingElapsed: 0.0s`, and `SummedProcessingRecordArchiveRebuildElapsed: 0.0s`; Heart Rate and Active Energy each completed in `0.2s` with no rebuild. Second report, `buildFingerprint: 1.0(1)-1780689890-92064`, completed in `5.8s` with `CaptureModes: unchangedDelta=121, delta=6`, `DeltaEvents: 7`, fetch `2.0s`, insert `0.1s`, finalize `1.7s`, residual `1.1s`, and processing/rebuild still `0.0s`. Conclusion: the legacy compact archive rebuild was the repeated-capture bottleneck and is now removed from the normal SQLite-backed delta path; the repeated full-profile floor is now roughly `6s` on this device/database. |
| 2026-06-06 | `fbbca81` | Expose saved diagnostic reports in Settings. | Diagnostic reports were already persisted under Application Support at snapshot completion, but they were not discoverable from the app after dismissing the result sheet. Settings now lists the latest saved reports, opens them with the same chunked lazy diagnostics viewer, supports copy, and allows per-report deletion. Expected signal: future import analysis can recover the exact report text after the fact instead of depending on screenshots or manual copy at completion time. |
| 2026-06-06 | `92592cd` | Stop HealthKit date-boundary queries on timeout cancellation. | A restored-device report with `buildFingerprint: 1.0(1)-1780695759-92064` recorded `263m 4s` total and `262m 50s` fetch for `Zinc`, with both `earliest_sample` and `latest_sample` reported as timeout after `15770.00s` despite `timeoutConfigured: 15.0s`. This timing is not a clean import-duration measurement: the phone had already been unresponsive for roughly `4h22m`, while the app/snapshot attempt was started shortly before the report. The report still exposed a defensive gap: date-boundary queries used plain continuations, so cancellation could mark timeout without explicitly stopping the underlying `HKSampleQuery`. Date-boundary queries now use the cancellable HealthKit continuation box and call `HKHealthStore.stop(_:)` when the timeout cancels the task. Expected signal: if a future boundary query wedges, it should fail near its configured timeout and leave the rest of the capture able to continue; do not use the `263m` value as a throughput datapoint. |
| 2026-06-06 | pending | Use monotonic import timers and non-blocking per-query timeout race. | Follow-up review clarified that the app and phone had been restarted before the suspicious report, so a persisted armed timer should not be assumed. Code inspection found no persisted deadline/start timer, only adaptive timeout profiles. However, import durations and API-call diagnostics used `Date()` wall-clock deltas, which can jump after system time or device-state changes, and the per-query timeout helper used a structured task group that could wait for a stuck operation child after timeout. User-visible snapshot duration, per-type total elapsed, boundary fetch elapsed, and API-call elapsed now use a `DispatchTime` monotonic timer. Per-query timeout now races unstructured operation/timeout tasks so the timeout path can return after cancelling the operation while the HealthKit cancellation handler stops the underlying query. Expected signal: after reboot or Health UI instability, diagnostics should not inherit impossible multi-hour elapsed values from wall-clock jumps, and a wedged boundary query should report near the configured timeout. |

## Current Diagnosis

The import is no longer primarily a HealthKit fetch problem. On the latest comparable first-import measured run:

- total wall clock was 12m43s after the unused-index removal and clean reset;
- summed fetch was only 40.4s;
- summed insert was 10m11s;
- Heart Rate alone spent 6m41s inserting.

The likely bottleneck is per-row SQLite work:
- uniqueness checks on hot tables;
- index maintenance while importing high-volume rows;
- multiple dependent writes per sample;
- commit / transaction shape;
- no-op visibility range maintenance for unchanged existing samples;
- processing existing high-volume metrics when only a small number of records changed;
- legacy SwiftData detail-cache precompute after SQLite completes. This became
  the active crash/performance issue even for small caches; any snapshot-save
  mutation of `TypeCount.detailCacheData` is unsafe and should remain disabled.
- Dashboard Core Data archive cache refresh after snapshot completion, when the
  app starts a full rebuild immediately after the import. Even detached rebuilds
  can overwhelm real-device I/O/CPU, so automatic post-snapshot UI refresh should
  use SQLite summary rows only.
- Core Data cache rebuild must not mutate `viewContext` from background Swift
  tasks. Rebuild/delete should use a private background context.
- Snapshot and Data Types UI must not rely on Core Data cache rows being present.
  SQLite observation rows and type summaries are already materialized during
  archive finalization and should be the primary UI source for fresh snapshots.
- UI state should distinguish loading from empty results. A nil or empty in-memory
  row list during app launch is not evidence that the archive is empty.
- The validated import metrics are based on the original 15-type profile. The
  next correctness/performance question is full-dataset coverage and volume, not
  further confidence from the restricted sample alone.
- Full-profile reimport after unit hardening completed successfully across 127
  metrics and 2.6M archived rows. This is not comparable to clean first-import
  timing, but it confirms that the expanded registry can pass the previous Water
  unit crash point.
- On unchanged full-profile reimports, `markVerification` can still be expensive
  if it counts nonexistent events for high-volume types. The capture layer knows
  the first anchored delta page is empty, so unchanged verification should skip
  event-count scans and copy previous materialized summaries directly.
- Incremental anchors must be persisted in SQLite, not only in legacy SwiftData
  `TypeCount.distributionBinsData`. Otherwise refactored SQLite-driven snapshots
  lose their HealthKit query anchor across launches/refactors and every
  high-volume metric becomes a full scan again.
- No-delta reports are hard to obtain once the full HealthKit profile is enabled
  because some metric often changes between manual captures. Identical checksum,
  identical record count, zero summed processing/insert, and sub-second
  high-volume metric timings are acceptable evidence that the expensive full
  import path was skipped.
- Capture modes now show the heavy full import path is avoided, but a small
  HealthKit delta on a high-volume type can still force expensive local
  reconstruction of legacy compact record archives and type hashes. Future logs
  should compare `DeltaEvents` against processing/finalize time before changing
  page sizes or HealthKit query strategy.
- The current delta optimization keeps checksum semantics unchanged and only
  removes avoidable Swift allocations. A larger future optimization would need a
  deliberate replacement for the legacy per-type fingerprint hash, not an ad hoc
  switch to SQLite aggregate hashes.
- Incremental daily aggregate replacement and type-summary updates achieved the
  intended finalization reduction. Do not repeat those experiments unless a
  regression appears. The latest measured bottleneck is again delta archive
  processing: Heart Rate spent `6.1s` processing only `9` delta events because
  the compact archive was still materialized into a large UUID dictionary before
  being rebuilt.
- The current compact-archive patch keeps checksum/hash semantics unchanged and
  only removes avoidable Swift dictionary materialization. If processing remains
  high after this patch, the next larger step would be a deliberate redesign of
  the legacy per-type compact archive/hash maintenance, not more HealthKit page
  tuning.
- The latest post-`4894b77` report was a large-delta capture (`11,093` events),
  not a small-delta repeat. It showed no full import fallback and no degraded
  metrics, but it cannot prove the Heart Rate small-delta target because Heart
  Rate had `231` events and other high-volume metrics had thousands. It does
  reveal a second full-profile cost: some zero-count types can spend several
  seconds in an empty anchored HealthKit query (`Wheelchair Pushes`, `Wheezing`,
  `Zinc`). Treat this as a HealthKit fetch / profile scheduling issue, separate
  from compact archive rebuild.
- The next small-delta report after `4894b77` confirmed that compact archive
  rebuild itself is still expensive even with tiny deltas. Heart Rate had only
  `6` delta events but still spent `5.7s` in processing. The current hypothesis
  is that per-type hash sorting over all record fingerprints is a large part of
  that residual cost. Delta hashing now uses recorded order, matching the
  initial-import hash builder path and removing the full fingerprint array/sort.
- The first report after recorded-order delta hashing confirmed the direction:
  processing improved, but finalization became the dominant cost. `Heart Rate`
  still spent `4.8s` in finalize with only `3` delta events. The current
  hypothesis is that changed daily aggregate replacement spends avoidable Swift
  time copying unchanged materialized daily rows before replacing affected
  buckets. Those copied rows now use SQLite `INSERT ... SELECT`; changed buckets
  are still rebuilt normally.
- Reports before `reportSchemaVersion: 3` do not identify the build beyond
  `App Version: 1.0(1)`, which is not enough for performance attribution during
  rapid test-install iteration. Treat older report-to-commit mapping as inferred
  from conversation/order unless backed by an external build note. New reports
  should include `buildFingerprint`; `sourceCommit` and `sourceDirty` may still
  be `unknown` unless the build pipeline injects those Info.plist keys.
- Incremental type-summary finalization was still too conservative around
  removed extremes. A delta that removes the previous latest/max but adds a
  newer/larger replacement no longer needs a full visible aggregate scan. The
  archive still falls back to the full scan when a removed earliest/latest/max
  leaves the new extreme genuinely unknown.
- The first post-extreme-fallback report was flat: total finalize stayed around
  `9.4s` and Heart Rate finalize stayed around `4.8s`. Do not continue guessing
  inside finalization without phase timings; the diagnostic report now needs to
  show whether the remaining time is type-summary, daily-aggregate,
  event-count, run-update, or other overhead.
- Finalization phase timings identified the current small-delta bottleneck:
  type-summary work is effectively gone, while daily aggregate maintenance is
  `7.3s` of `10.3s` finalize on the latest full-profile repeated capture. Heart
  Rate spent `3.8s` in daily aggregates for only `9` delta events. The changed
  daily aggregate path still copies previous materialized rows for the type and
  then rebuilds affected buckets; distinguish copy cost from affected-bucket
  rebuild cost before adding new SQLite indexes, because first-import reports
  show insert/index overhead is already the dominant large-database risk.
- Daily aggregate subphase timings proved the problem is affected-bucket rebuild,
  not daily aggregate copy. On the `6041bac` report, daily rebuild was `6.9s`
  of `7.4s` daily aggregate work, while copy was `0.5s`. Heart Rate alone spent
  `3.8s` rebuilding affected daily buckets. The next experiment changes the
  affected-bucket rebuild query shape so it starts from time-ranged
  `sample_versions` instead of all samples of the type.
- The `bf5a861` follow-up reports validated that time-ranged daily aggregate
  rebuilds worked: `SummedFinalizeDailyAggregateRebuildElapsed` was `0.0s` on
  both an overnight-data run (`322` delta events, `21.8s` wall clock) and a
  low-delta run (`2` delta events, `6.6s` wall clock). Daily aggregate cost is
  no longer the active repeated-capture bottleneck.
- After daily rebuild was removed, the next measured finalization floor is
  event-count / previous-summary lookup. The old event-count query started from
  `samples(sample_type_id)` and could traverse a high-volume type to count a
  handful of events. The next query shape starts from
  `sample_observation_events(observation_id, event_kind)` and joins to samples
  for type filtering.
- The `e7d45a2` follow-up report validated the event-count query shape:
  `SummedFinalizeEventCountElapsed` dropped to `0.0s` and finalization is no
  longer the active repeated-capture bottleneck.
- The `cfd9de8` diagnostic split confirmed the active repeated-capture
  bottleneck is the legacy compact record archive/hash path for changed
  high-volume metrics. The latest report spent `8.5s` in processing and all of
  it was `processingRecordArchiveRebuildElapsed`, including Heart Rate `4.4s`
  for a tiny delta.
- The `0db2f5e` follow-up report validated skipping legacy compact archive
  rebuild for SQLite-backed deltas: repeated full-profile wall clock dropped to
  `6.2s`, `SummedProcessingElapsed` dropped to `0.0s`, and changed high-volume
  metrics such as Heart Rate and Active Energy completed in about `0.2s`.
- A second `0db2f5e` report confirmed the new floor is stable: wall clock
  `5.8s`, fetch `2.0s`, insert `0.1s`, finalize `1.7s`, residual `1.1s`,
  `initialImport=0`, and processing/rebuild still `0.0s`.
- A large older-build first import on an `8.4M`-record database completed but
  took `166m10s`, with `137m31s` summed insert time. This confirms that full
  authorized backup volume can be much larger than the original 15-type test
  subset and that first-import write-path performance must remain a separate
  acceptance track from repeated incremental capture performance.
- The next first-import optimization removes compact `recordArchiveData`
  generation for new SwiftData TypeCount rows. SQLite is now the source of
  truth for record details, repeated captures no longer need compact archives,
  and legacy SwiftData detail-cache precompute is disabled. Keeping the compact
  blob on fresh first imports only adds CPU, memory, and SwiftData payload size.
  Existing optional fields remain for old rows and compatibility, but new
  SQLite-backed captures should report no compact archive finalize cost.
- A 2026-06-06 screenshot from the large-database device showed a fast partial
  retry after first import: `127` metrics, `8,411,713` records fetched, `11.2s`
  duration, and `2` failed metrics (`Blood Pressure Diastolic` and
  `Blood Pressure Systolic`). The device's Health database is known to be a
  restored pre-loss reference state with unstable Health/authorization behavior.
  Entries dated 2026-05-24 were later clarified as fresh data from an Apple
  Watch paired on that day in an attempt to repair synchronization, not evidence
  that HealthProbe caused old data to be pulled from or pushed to iCloud. Treat
  the two failed blood-pressure metrics as device/data-state evidence, not as an
  import-performance regression, unless a matching diagnostic report proves a
  HealthProbe error. The attached copied diagnostic text did not match that
  screenshot; it was the older `1.0(1)-1780689890-92064` repeated-capture report
  with `2,625,217` records and `5.8s` duration. Do not use that pasted text as
  evidence for the large-device partial run or for commit `0d61338`. A matching
  Diagnostics text copy is still needed before drawing performance conclusions
  from the large-device partial run.
- The matching Diagnostics report for that retry was captured later. Build
  fingerprint was `1.0(1)-1780695759-92064`, operation result was
  `partial_success`, `previousSnapshotID` was `none`, `isChainStart` was `true`,
  but capture modes were `unchangedDelta=121`, `delta=4`, `initialImport=0`,
  `unavailable=2`. This confirms the run was a retry using previously persisted
  per-type capture state, not a clean first import rewrite of `8.4M` rows. The
  performance result is still important for large-database retry/incremental
  behavior: wall clock `11.2s`, fetch `6.0s`, processing `0.0s`, insert `1.0s`,
  finalize `1.9s`, residual `1.5s`, with `793` delta events. The two failed
  Blood Pressure metrics failed before record import because earliest/latest
  queries returned HealthKit `com.apple.healthkit` error code `5`:
  `Authorization not determined`. Treat this as a permission/device-state issue,
  not an import engine regression.
- An earlier matching report from the same large-database device
  (`Device: 4A8B3EE5-BA8B-44F2-A3D9-44105FFEA033`) captured a complete first
  import on 2026-06-04 at 20:42:44: `8,421,978` records, `127/127` complete,
  `initialImport=127`, wall clock `166m10s`, fetch `5m19s`, processing
  `20m29s`, insert `137m31s`, finalize `1m53s`. In that report Blood Pressure
  Diastolic and Systolic were both `authorizationStatus: granted`, both had
  `5,120` samples, and both ranged from `2014-12-18T11:24:11Z` to
  `2026-05-29T03:06:36Z`. This is a strong before/after reference: the same
  device later reported `Authorization not determined` for those same types
  after authorization/query interaction. Preserve this report as forensic
  evidence that HealthKit access/authorization state changed, not merely that
  the Health UI displayed different data.
- This large-database device is not the untouched original source. It is a
  restored copy of the pre-data-loss database from a backup that still exists,
  and it has already been used for multiple attempts to restart or repair Health
  synchronization, including pairing an Apple Watch that later produced fresh
  entries dated 2026-05-24. It can therefore be used for aggressive experiments
  when needed, but results must be labelled as coming from a restored
  pre-loss-reference copy rather than from the original live device state.

## Open Issues / Observations

- Very small pages reduced freeze risk but introduced visible overhead.
- Some progress timing displayed in the UI did not include overhead, so elapsed time and rates looked better than the real operation.
- A previous Heart Rate import appeared to stall for long periods around roughly 900k records, but later progress resumed; avoid classifying this as a hard timeout without report evidence.
- Diagnostic reports must remain available after each import. Copying from the
  Diagnostics sheet is useful, but future comparisons should not depend on the
  operator remembering to copy the visible report. Saved reports are now exposed
  in Settings so the exact text can be recovered later.
- A timeout report with elapsed time far above `timeoutConfigured` can indicate
  either a wall-clock timing artifact after device unresponsiveness/system time
  changes or an underlying HealthKit query that was not stopped on task
  cancellation. Operational import timings now use monotonic uptime, and
  date-boundary queries explicitly stop their `HKSampleQuery` on cancellation.
  Future reports should be judged by whether the elapsed time is near the
  configured timeout.
- After a completed import, the app may remain unresponsive or crash in legacy
  post-import cache work. A 2026-06-03 console log showed Heart Rate and Active
  Energy `TypeCount.detailCacheData` precompute immediately before a Core Data
  mutated-while-enumerated abort.
- A later 2026-06-03 overnight log showed the same abort after only Stand Hours
  and Environmental Sound Levels detail caches were built. Size limits are not
  enough; the whole snapshot-save detail-cache precompute path is disabled.
- Partial / old imported observations can pollute comparisons. Fresh first-snapshot performance comparisons should use a confirmed reset database.
- Non-chain-start full scans can be slower than first imports if unchanged existing samples still write per-sample archive evidence.

## Next Experiments

Prioritize experiments in this order:

1. Run full-dataset discovery with the expanded registry:
   request/refresh permissions, inspect supported vs unsupported types, and run
   a capture with all desired supported types enabled on a real device. Record
   type count, total records, failed/unsupported/empty types, and phase timings.
2. Run an incremental snapshot after removing automatic Core Data cache rebuild.
   Confirm there are no `healthKit.detailCache.buildBegin` logs, copying the
   diagnostic report does not freeze the app, and Dashboard/Snapshots show the
   latest observation from SQLite. Also verify Snapshot detail and Data Types
   show per-type summaries without a manual cache rebuild.
3. Continue repeated full-profile captures, but do not require perfect no-delta.
   Compare `snapshotChecksum`, `Records`, `WallClockDuration`,
   `SummedProcessingElapsed`, `SummedInsertElapsed`, `SummedFinalizeElapsed`,
   `CaptureModes`, and high-volume type timings. Treat stable checksum, a high
   `unchangedDelta` count, and zero processing/insert as the main unchanged-path
   signal.
4. Run a full-profile repeated capture after compact-delta archive patching.
   Compare `SummedProcessingElapsed`, Heart Rate processing time, and
   `DeltaEvents`. Expected success is Heart Rate processing below the previous
   `5.7s` baseline when its delta remains small. This target was met in the
   `23.6s` / `11`-event report, where Heart Rate processing dropped to `4.2s`.
5. Keep using `DeltaEvents` to quantify changed high-volume metrics, especially
   Heart Rate, Active Energy, and Basal Energy. If delta events are small while
   finalize remains large, optimize aggregate rebuild/finalization rather than
   HealthKit fetch, SQLite insert, or legacy compact archive reconstruction.
6. Run a full-profile repeated capture after SQL-side daily aggregate copying.
   Compare `SummedFinalizeElapsed` and Heart Rate finalize against the current
   `11.4s` total finalize / `4.8s` Heart Rate finalize baseline. The first
   follow-up improved total finalize to `9.3s`, but Heart Rate stayed around
   `5.0s`; keep this as evidence that there is still a high-volume
   changed-type floor.
7. Verify the next diagnostic report contains `reportSchemaVersion: 3` and
   `buildFingerprint`. Do not compare performance reports without a build
   identity unless the build provenance is otherwise certain. `sourceCommit`
   and `sourceDirty` are useful when present, but may be `unknown` for normal
   Xcode test installs.
8. Treat repeated full-profile captures around `6s` as the current validated
   floor on this device/database. The next repeated-capture optimization should
   target only a measured subphase above about `1s`: currently fetch is around
   `2.0s`, finalize around `1.7s`, and residual around `1.1s`.
9. Keep watching first-import insert timing on the next clean large-database
   import because the new `sample_versions(start_date, sample_id)` index from
   `bf5a861` is a write-path tradeoff.
10. Run a clean full first import after retiring compact `recordArchiveData` for
   new SwiftData TypeCount rows. Compare wall clock, `SummedProcessingElapsed`,
   `SummedProcessingInitialRecordElapsed`,
   `SummedProcessingRecordArchiveFinalizeElapsed`, `SummedInsertElapsed`, and
   post-import responsiveness. Expected success: record archive finalize is
   zero/negligible, details still load from SQLite, and first-import memory /
   post-save behavior does not regress.
11. On the large-database device, resolve or deliberately document the two
   Blood Pressure permissions. The matching report shows HealthKit
   `Authorization not determined` for systolic/diastolic earliest/latest
   queries, so the next app-side check is whether `Request Health Access`
   re-prompts for those types or whether iOS Health settings expose them for
   HealthProbe.
   Compare any future attempt against the 2026-06-04 first-import reference,
   where the same device granted HealthProbe access and imported `5,120`
   systolic + `5,120` diastolic records through 2026-05-29.
12. Investigate full-profile empty anchored-query cost for zero-count types.
   Compare slow empty types across reports before changing behavior; any skip or
   lower-frequency strategy must preserve the promise that full authorized
   backup can notice newly appearing data.
13. Run a non-chain-start/full-scan benchmark after skipping unchanged `verified` events and fast-pathing already-open visibility ranges. Compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, `Steps insertElapsed`, and `Walking + Running Distance insertElapsed`.
14. Reduce any remaining per-sample SQLite writes for unchanged existing samples during non-chain-start full scans.
15. Profile whether index maintenance dominates first-import insert cost.
16. Consider a guarded bulk-import mode for first observations:
   - keep archive semantics unchanged;
   - only relax work that can be safely reconstructed or validated;
   - re-enable normal idempotent paths for incremental observations.
17. Run a fresh first-import benchmark after the unused-index removal and compare `SummedInsertElapsed`, `Heart Rate insertElapsed`, and `Active Energy insertElapsed`.
18. Investigate whether first-import-only deferred index creation or temporary staging tables can reduce `samples` / `sample_versions` / `sample_observation_events` write cost without weakening final archive integrity.
19. Revisit adaptive page sizes only after SQLite write-path costs are reduced.
20. Revisit background / scheduled collection once initial import can finish reliably and post-import UI recovery is bounded.

## Verification Checklist For Each Optimization

- [ ] `git diff --check` passes.
- [ ] SQLite archive store tests pass.
- [ ] Import configuration tests pass if capture strategy changed.
- [ ] Repeated-page/idempotency behavior remains covered.
- [ ] A real-device report is attached or summarized in this log.
- [ ] The next experiment is recorded before moving on.
