# HealthProbe – Complete Specification & Motivations

**Version:** 1.5
**Status:** MVP (iOS local Health DB Time Machine)
**Last Updated:** 2026-05-23

---

## 1. Executive Summary

HealthProbe is a **local time machine for Apple HealthKit-accessible data**. It captures selected local HealthKit observations over time so the user can inspect how their Health database looked at a chosen date, understand what changed, and export details that may later be unavailable after aggregation, consolidation, pruning, or API limitations.

**Core Problem:** Apple Health is not a stable forensic record store. The same user-visible health history may be represented differently over time: high-frequency samples can be aggregated, old records can become intervalized, exports can lose detail, and different Apple devices may expose different local HealthKit database states. A count drop is therefore not reliable proof of loss.

**Solution:** HealthProbe incrementally captures selected HealthKit data into a robust local SQLite archive and presents a single-device observation timeline. The archive is differential and analysis-capable: it stores observations, sample identities, payload versions, visibility ranges/events, and materialized aggregates rather than recurring complete snapshot copies. Core Data is the target UI/reporting cache for expensive counts, summaries, report metadata, and display state; it is rebuildable and not the source of truth.

---

## 2. Motivation & Product History

### 2.1 Original Trigger

HealthProbe started after a user-observed mass disappearance of Apple Health detail. The first idea was simple: count records, compare snapshots, and warn the user when Apple Health appeared to "lose" data.

That framing was useful for discovery but incomplete.

### 2.2 What Changed

Further observation showed that older HealthKit data can change representation without necessarily representing user-meaningful loss:
- high-frequency samples may be thinned
- old point samples may become interval samples
- per-record values may change while daily/monthly aggregates remain explainable
- HealthKit exports taken months apart may not contain the same record-level detail
- different devices can expose different local HealthKit database states

Because of this, record-by-record cross-device comparison is out of scope and count-only alerts create false alarms.

### 2.3 Current Objective

HealthProbe is now a single-device local Health DB Time Machine:
- capture selected HealthKit-accessible data as it exists at observation time
- reconstruct how the local Health database looked at a chosen date
- show additions, removals, representation changes, and aggregate changes between observations
- preserve local evidence that HealthKit may later aggregate or no longer export
- export scoped historical views for personal backup, support, research, or external analysis

### 2.4 Interpretation Model

HealthProbe describes changes neutrally:
- **Appeared:** a record/fingerprint is newly visible
- **Disappeared:** a record/fingerprint is no longer visible
- **Representation changed:** timestamps, intervals, values, or granularity changed
- **Consolidation likely:** record detail decreased while aggregates remain explainable
- **Uncertain:** available evidence cannot distinguish user action, HealthKit behavior, app permissions, background timing, or system state

These classifications are evidence labels, not claims about Apple's intent or definitive proof of corruption.

### 2.5 Why This Matters

| Concern | Impact | HealthProbe Role |
|---------|--------|-----------------|
| **Historical detail is unstable** | Older HealthKit detail may later be aggregated or unavailable | Preserve selected observations locally |
| **Counts mislead** | A sample-count drop may be consolidation, not loss | Explain changes with record and aggregate evidence |
| **No built-in time travel** | Health.app shows current state, not prior database states | Provide local point-in-time views |
| **Exports are time-sensitive** | Future exports may not contain old detail | Export selected evidence while it exists locally |
| **Privacy of monitoring** | Health data is sensitive | Local-only archive, explicit user exports |

---

## 3. Core Architecture

### 3.1 Design Principles

1. **Read-only operations** (never modify HealthKit data)
2. **Local-first** (full functionality without network)
3. **Incremental queries** (efficient, avoid repeating work)
4. **Single archive store** (do not split the forensic store per data type; cross-type relationships and shared metadata matter)
5. **Auditability** (every observation logged, timestamped, reproducible)
6. **Privacy by default** (no HealthProbe cloud sync; local storage remains under user control)
7. **Time-machine capture** (selected data types are archived locally so prior HealthKit-accessible states can be revisited)
8. **Single-device timeline** (snapshot comparisons stay within the local device chain; cross-device record comparison is not a product goal)
9. **Consolidation-aware explanations** (record-count changes are described with uncertainty and aggregate context)
10. **Legacy device support** (target iOS 15-era Health collection devices; avoid SwiftData as a required foundation)
11. **SQL-first analysis** (large diffs and reports run inside SQLite using indexes, temporary tables, joins, and paged results)

### 3.2 Threading Model

```
┌─────────────────────────────────────────┐
│  Main Thread (UI)                       │
│  - Display archive and capture status   │
│  - Show timeline, diffs, exports        │
│  - User interaction                     │
└──────────────┬──────────────────────────┘
               │
               ├─ Delegate query results
               │
┌──────────────▼──────────────────────────┐
│  Background Queue (HealthKit Queries)   │
│  - HKAnchoredObjectQuery (efficient)    │
│  - HKObserverQuery (reactive)           │
│  - Observation comparisons              │
│  - Change explanation logic             │
└──────────────┬──────────────────────────┘
               │
               ├─ Write observations and change summaries
               │
┌──────────────▼──────────────────────────┐
│  Local Archive Store                    │
│  - Canonical HealthKit samples          │
│  - Sources, devices, metadata           │
│  - Cross-type relationships             │
│  - Fingerprints and verification hashes │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│  Core Data UI/Report Cache              │
│  - Precomputed counts/statistics        │
│  - Visualization state and settings     │
│  - Logs, history, report indexes        │
└─────────────────────────────────────────┘
```

### 3.3 Storage Model

**SQLite Archive Store (source of truth):**
- one robust local database for all archived samples and analysis, not one archive per data type
- differential observation storage, not recurring complete snapshot copies
- normalized entities for samples, sample payload versions, workouts, sources, source revisions, devices, metadata, relationships, and observations
- multiple fingerprints per sample: HealthKit UUID hash, strict fingerprint, semantic fingerprint, and fuzzy matching keys for export/backup reconciliation
- append-only observation history (`firstSeen`, `lastSeen`, `lastVerified`, disappearance evidence)
- visibility ranges/events so point-in-time reconstruction can be queried without duplicating every record per observation
- materialized aggregates for expensive counts and report inputs
- snapshot/observation-level and table-level hashes for integrity checks
- SQL-first analysis using indexes, temporary tables, joins, CTEs, and streaming/paged result sets

**Core Data UI/Report Cache (derived/cache layer):**
- settings and selected data types
- import job state and progress
- precomputed counts, temporal bins, display ranges, and summary statistics
- audit log entries and report indexes
- change summaries and links into the archive store

Core Data cache rows must be rebuildable from the local archive store. If the two disagree, the SQLite archive wins. Current SwiftData models are legacy/prototype implementation details until this cache layer replaces them.

There are no real deployments, only test installations. Existing prototype stores are disposable during the archive v2 refactor and do not require backward-compatible migration.

### 3.4 Storage Architecture Decision

HealthProbe keeps the database on-device. The current objectives do not require a server or external analytical engine for the iOS app. SQLite is the durable archive and analysis engine because it supports the operations needed for large local Health datasets: indexes, temporary tables, joins, CTEs, transactions, and paged/streaming reads.

Core Data is the right destination for cached counts and UI/reporting state because:
- it supports legacy iOS versions that SwiftData does not;
- it is well-suited for bounded object graphs and presentation-ready summaries;
- cached rows can be deleted and rebuilt from SQLite;
- expensive counts can be persisted without forcing every screen/report to recalculate them.

Core Data is not the right place for heavy archive analysis. Diffs across large observations, export selection, consolidation heuristics over high-frequency records, and record table pagination should run against SQLite and return bounded result pages or materialized summary rows.

The target storage split is therefore:
- `HealthProbeArchive.sqlite`: source of truth, differential observation storage, SQL analysis, export source;
- Core Data cache store: rebuildable UI/report summaries, expensive counts, timeline rows, progress, settings, and export metadata.

This resolves the product tension: the app remains usable on older Health collection devices while still allowing large-dataset analysis without loading entire snapshots into RAM.

---

## 4. Time Machine Features (MVP)

### 4.1 Incremental Capture

**Using `HKAnchoredObjectQuery`:**
```
Query pattern:
├─ Initial query: anchor = 0 → captures all existing data
├─ Store anchor locally
├─ Periodic queries: anchor = stored → captures only new/modified samples
└─ Update anchor → efficient incremental updates
```

**What triggers capture:**
- App launch
- Background refresh (iOS allows periodic background queries)
- User manually triggers capture
- Every 12-24 hours (configurable)

### 4.2 Tracked Sample Types (Extensible)

| Type | Why Captured | Change Signal |
|------|---------------|----------------|
| **Workouts** | High-value user records | Appeared/disappeared records, metadata changes |
| **Heart Rate** | High-frequency detail likely to be consolidated | Granularity changes, intervalization, aggregate drift |
| **Activity Summary** | Auto-computed, depends on other types | Recalculation between observations |
| **Steps** | Cumulative and often consolidated | Aggregate preservation vs record thinning |
| **Sleep** | Frequently edited and reclassified | Stage/category representation changes |
| **Blood Pressure** | Manual/clinical-style records | Point-in-time history and export preservation |
| **Audio Exposure** | Often high-frequency/device-specific | Detail retention and later aggregation |

### 4.3 Change Explanation Logic

#### A. Point-In-Time Reconstruction
```
Input: observation timestamp T and selected sample type
Use: archived records whose observation history makes them visible at T
Output: table, aggregates, source breakdown, and manifest hash
```

#### B. Adjacent Observation Diff
```
Previous observation: S_prev
Current observation:  S_now

Appeared     = S_now - S_prev
Disappeared = S_prev - S_now
Retained    = S_now ∩ S_prev

For retained semantic groups:
  compare record count, interval length, value sum, value max, and source metadata
```

#### C. Consolidation Heuristic
```
IF old high-frequency records disappear
AND newer interval records cover similar time ranges
AND aggregate sums remain within tolerance
THEN classify as "consolidation likely"
ELSE classify as "changed/uncertain" with evidence
```

#### D. Export Preservation
```
For selected records or diffs:
  export archived details, observation metadata, hashes, and explanatory labels
  never require a cloud round trip
```

---

## 5. Context Logging

HealthProbe does **not** sync its own archive through iCloud or CloudKit. Observed HealthKit databases can diverge between devices, and HealthProbe no longer attempts to compare snapshots from different devices. The product scope is one local observation timeline on the current device.

Health/iCloud state is still useful as **context** for interpreting local changes, but it is not treated as proof of cause.

### 5.1 Context Tracking

**Observe HealthKit permission & coarse system context:**
```swift
HKHealthStore().requestAuthorization(...)
// → Detect when user grants/revokes permissions

// Monitor iCloud sign-in state as context only
FileManager.default.ubiquityIdentityToken
// → Detects iCloud sign-in/sign-out
// → Logs context for later correlation
```

**Capture lifecycle events:**
- iCloud sign-in detected → log context and schedule a local archive verification pass
- iCloud sign-out detected → note local-only mode
- Device backup initiated → pre-backup snapshot
- App backgrounded/foregrounded → capture if needed

### 5.2 Context Documentation

**Audit trail entries:**
```
[2026-05-01 14:23:15] SYNC_STATE_CHANGE: iCloud enabled
  - Previous: local-only
  - Action: archive verification scheduled
  - Result: no HealthProbe cloud sync performed

[2026-05-01 14:24:02] CAPTURE_COMPLETED: local HealthKit observation
  - Samples observed: 87 new, 3 no longer visible
  - Representation changes: 2 groups
  - Cause: not inferred

[2026-05-01 16:15:00] CHANGE_SUMMARY: Historical record appeared
  - Type: Workout
  - Record date: 2024-03-15
  - First observed by HealthProbe: 2026-05-01
  - Label: appeared; cause unknown
```

### 5.3 Background Monitoring

**iOS Background Modes enabled:**
- `background-fetch` — periodic archive and context checks
- `remote-notification` → not required for HealthProbe archive sync

**Check frequency:**
- Min: 2 hours
- Max: 24 hours
- Adapts based on archive cost and user preference

---

## 6. Local Archive, Reports & Forensics

### 6.1 Local Archive Store

The main backup artifact is the on-device archive store. It is populated incrementally from HealthKit and is not dependent on Apple Health ZIP exports or full encrypted iPhone backups.

The archive must preserve as much HealthKit information as the API exposes:
- sample UUID, type, start/end date, value, unit, and metadata
- source, source revision, bundle identifier, product type, version/build if available
- device fields exposed by `HKDevice`
- relationships between workouts, samples, events, and other linked records where available
- first-seen / last-seen / last-verified observations
- fingerprints suitable for matching against Apple Health XML exports and extracted backup databases

The archive is selected by data type for performance and privacy, but it is stored in **one schema** so later analysis can follow relationships between types.

### 6.2 Reports and Point Exports

HealthProbe does not need to optimize for routine complete exports. The local archive is the backup; point exports are the user-facing way to preserve or share a historical view.

Export is scoped to what the user is inspecting:
- point-in-time record tables
- diff reports between two observations
- point-in-time manifests and hashes
- selected record sets needed for external analysis

### 6.3 Forensic Query Examples

**"What did my step data look like on March 1?"**
```
1. Select the March 1 observation
2. Load archived visible records and aggregates for Steps
3. Show counts, time range, daily totals, and source breakdown
4. Offer JSON/CSV export for the selected view
```

**"What changed since the previous observation?"**
```
1. Compare adjacent local observations
2. Group appeared, disappeared, retained, and representation-changed records
3. Explain likely consolidation when aggregates remain stable
```

**"Can I still export detail that HealthKit no longer shows?"**
```
1. Search the local archive for the earlier observation
2. Select the preserved records or diff set
3. Export archived details with manifest hashes and observation metadata
```

---

## 7. User-Facing Features

### 7.1 Dashboard (iOS App)

**Home Screen:**
- **Latest Observation** — timestamp and capture quality
- **Archive Coverage** — selected data types, date range, storage use
- **Recent Changes** — neutral summary of appeared/disappeared/changed records
- **Export Shortcuts** — selected observation or diff report

**Detail Views:**
- **Timeline** — local observations over time
- **Observation Detail** — point-in-time tables and aggregates
- **Diff Detail** — changes between two observations
- **Audit Trail** — complete immutable log
- **Archive Status** — current local archive health, last verification, selected data types

**Settings:**
- Capture frequency
- Sample types to track
- Change-label thresholds/tolerances
- Local archive retention and report export options

### 7.2 Notifications

Notification-led alerting is not a current product objective. The app may later add reminders for scheduled capture or completed exports, but alerts about presumed data loss are explicitly out of scope.

---

## 8. Future Work Parking Lot

Items in this section are not active product objectives. They require a separate scope decision before implementation.

### 8.1 Better Reconstruction
- richer point-in-time query language
- improved consolidation heuristics
- archive compaction without losing observation history

### 8.2 External Analysis Tools
- Analyze explicit HealthProbe exports outside the iOS app if needed for the DearApple investigation/article
- Do not treat a macOS companion, community sharing, or open-source publication as committed product scope

### 8.3 Recovery-Compatible Archives

HealthProbe will not perform recovery workflows. It will not patch iOS backups, transplant Health database files, or re-publish archived values into HealthKit.

However, HealthProbe archives and exports should be suitable input for external recovery/salvage procedures, including:

1. **Backup transplant restoration outside the app**
   - External tooling may use HealthProbe evidence alongside the reverse of the DearApple scratchpad HealthDB extraction/reinsertion workflow.
   - Archive/export requirements: preserve source database identity where available, record identity, payload versions, dates, values, units, metadata, relationships, observation timestamps, and manifest hashes.

2. **HealthKit re-publication outside the app**
   - External tooling may choose to write missing values back through HealthKit as new app-owned samples.
   - Archive/export requirements: preserve enough value/date/unit/type detail to recreate user-visible values, plus explicit provenance warnings because original source/device/sync metadata may be lost.

Recovery compatibility is therefore an archive/export design requirement, not an in-app restore feature. The app remains read-only.

---

## 9. Technical Specifications

### 9.1 Platform
- **iOS 15.0+** (HealthKit framework support; keeps iPhone 6s-era Health collection devices in scope)
Legacy Apple Watch devices remain relevant as Health data sources paired to the target iPhone, but HealthProbe itself is scoped as an iOS app.

### 9.2 Permissions Required
- `HealthKit` — read-only access to specified types
- `Background Modes` — "Background Fetch"

### 9.3 Data Storage
- **SQLite Archive Store:** canonical differential HealthKit observation archive and analysis engine (source of truth)
- **Core Data:** derived UI/report/cache/settings/log/history store, rebuildable from SQLite
- **No CloudKit sync:** HealthProbe data remains local unless the user exports a report or selected record table

### 9.4 Performance
- Query time: < 5 seconds (anchored queries)
- UI/report cache size: bounded, rebuildable, and safe to purge
- Archive storage: differential; depends on selected high-frequency data types and number of representation changes, not number of full periodic snapshots
- Large analysis: runs in SQLite with paged results; Swift must not load full high-frequency datasets into RAM

---

## 10. Privacy & Security

### 10.1 What HealthProbe Never Does
- ❌ Exports raw health samples to cloud
- ❌ Identifies users by name/account
- ❌ Shares device location or personal context
- ❌ Modifies any HealthKit data
- ❌ Patches, transplants, or rewrites iOS backup databases
- ❌ Sells or shares data with third parties

### 10.2 What HealthProbe Collects (Local Only)
- ✅ Aggregated counts and per-sample archive data for user-selected types
- ✅ Observation and change timestamps
- ✅ Device model & iOS version (for context)
- ✅ Change labels and evidence summaries

**Local archive:**
- ✅ Per-sample archive for user-selected types, stored on-device and exportable by user
- ✅ Metadata needed for recognition in Apple Health XML exports, backup database extracts, and future datasets

### 10.3 Cloud Policy
- No HealthProbe CloudKit/iCloud sync
- No cross-device HealthProbe snapshot comparison
- No automatic upload of raw samples, digests, reports, or device fingerprints
- User-triggered exports are explicit, scoped, and local-file based

---

## 11. Success Criteria

| Objective | Metric | Target |
|-----------|--------|--------|
| **Time-machine inspection** | User can inspect a selected observation | All captured types |
| **Change explanation** | Diffs include neutral labels and evidence | > 95% of visible changes classified or marked uncertain |
| **Export preservation** | Selected historical records can be exported | JSON/CSV with manifest hashes |
| **False alarms** | Count-only drops framed as critical loss | 0 by design |
| **Privacy** | % of users comfortable with data practices | > 90% |
| **Performance** | Background capture battery impact | < 2% drain/day |
| **Reproducibility** | Users can preserve scoped evidence | High relevance for personal analysis and the DearApple investigation context |

---

## 12. References & Related Work

- [DearApple Issue #001](https://github.com/overbog/dear-apple/issues/0001-apple-health-mass-data-loss.md) — historical context for reported Apple Health data anomalies
- [Apple HealthKit Documentation](https://developer.apple.com/documentation/healthkit/)
- [HKAnchoredObjectQuery](https://developer.apple.com/documentation/healthkit/hkanchoredrobjectquery) — Efficient incremental queries

---

## Appendix A: Example Diff Export

```json
{
  "report_id": "DIFF_20260501_001",
  "type": "observation_diff",
  "exported_at": "2026-05-01T14:35:22Z",
  "from_observation": "2026-04-01T08:00:00Z",
  "to_observation": "2026-05-01T08:00:00Z",
  "evidence": {
    "sample_type": "HKQuantityTypeIdentifierStepCount",
    "appeared": 12,
    "disappeared": 84,
    "representation_changed": 6,
    "aggregate_delta": {
      "value_sum_percent": 0.1,
      "covered_days": 31
    },
    "label": "consolidation_likely",
    "cause": "not inferred"
  },
  "manifest_hash": "synthetic-example-hash"
}
```

---

*HealthProbe — A local time machine for your Health database.*
