# HealthProbe – Complete Specification & Motivations

**Version:** 1.2  
**Status:** MVP (iOS monitoring agent)  
**Last Updated:** 2026-05-18  

---

## 1. Executive Summary

HealthProbe is an **audit and integrity monitoring tool for Apple HealthKit**, designed to detect and document anomalies in health data that would otherwise go unnoticed. It serves as a **local sentinel** — read-only observation with forensic-grade logging for later analysis.

**Core Problem:** Apple Health data loss events (confirmed Sept 2025 incident, ongoing sporadic reports) lack detective mechanisms. Users do not know when data has been lost, corrupted, or silently modified.

**Solution:** HealthProbe incrementally captures HealthKit data into a robust local archive and maintains an audit trail for post-incident forensic analysis. SwiftData is used for UI assistance, settings, logs, history, and precomputed values; it is not the source of truth. The local archive exists because Apple Health/iCloud can **rewrite / downsample / consolidate** historical samples in-place (not only add/delete), which cannot be proven by counts alone.

---

## 2. Motivations & Concrete Observed Cases

### 2.1 The September 2025 Mass Data Loss Event

**What happened:**
- Large-scale loss of Apple Health records reported across multiple devices and iOS versions
- Timeframe: September 2025 (correlated with iOS 26 release)
- Suspected triggers:
  - Device migration (iCloud sync state transitions)
  - OS upgrade/downgrade cycles
  - Backup restore operations
  - HealthKit database re-indexing
  - iCloud sync divergence

**Why undetected:**
- No notification from Apple Health
- Users discover loss retrospectively (weeks/months later)
- No audit trail to identify exactly *when* or *what* was lost
- No differentiation between: user deletion, sync loss, corruption, or app bug

**HealthProbe's answer:**
- Continuous monitoring detects *first occurrence* of loss
- Timestamped snapshots enable forensic reconstruction
- Pattern detection identifies trends (gradual loss vs. sudden wipe)
- Allows users to file reproducible bug reports with evidence

### 2.2 Observed Anomalies

#### A. Historical Insertions (Backdated Data)
**Pattern:** HealthKit receives samples with `startDate` far in the past, but `firstSeen` ≈ now
- **Examples:**
  - Workout from Jan 2023 suddenly appears in Feb 2026
  - Step count "corrected" retroactively without user action
  - Heart rate baseline recalibration affecting past months
  
**Root cause theories:**
  - iCloud sync restoring from outdated backup
  - Third-party fitness app injecting historical reconstructions
  - HealthKit recovery logic applying retroactive corrections
  - Cross-device sync desynchronization

**Detection method:** Anchored queries + timestamp comparison

---

#### B. Silent Deletions
**Pattern:** Samples present in previous snapshot, absent in current, no `HKDeletedObject` notification
- **Examples:**
  - 2-week gap of step data (no deletion events logged)
  - Entire workout history from old iPhone missing post-restore
  - Selective loss (e.g., only workouts, heart rate preserved)

**Root cause theories:**
  - Incomplete restore from backup
  - Selective iCloud sync pruning based on storage limits
  - Corrupted local database during indexing
  - Race condition during multi-device sync

**Detection method:** Snapshot comparison + gap detection

---

#### C. Duplicate Records
**Pattern:** Identical samples (same type, time, value) appearing multiple times
- **Examples:**
  - Duplicate step counts from watch syncing (15 min apart)
  - Duplicate workouts after iCloud re-sync
  - Conflicting HR readings within 30-second window

**Root cause theories:**
  - Multi-device sync collision (watch + phone + iPad)
  - Retry logic without deduplication
  - Backup restore merging with live data

**Detection method:** Fingerprinting (type + date + value + source)

---

#### D. Divergence Trends
**Pattern:** Measurable drift in aggregated metrics over time
- **Examples:**
  - Active energy expenditure trending down 30% without behavior change
  - Sleep records shifting systematically earlier/later
  - Heart rate variability calculation method changing unexpectedly

**Root cause theories:**
  - Algorithm updates not backfilled uniformly
  - Device calibration drift
  - Source priority shifting (watch → phone)
  - Health app recalculation without user visibility

**Detection method:** Time-series aggregation + statistical outlier detection

---

#### E. Consolidation / Downsampling (Historical Rewrites)
**Pattern:** For high-frequency data types, Apple Health/iCloud may rewrite historical samples such that:
- total sample `count` decreases for older months
- timestamps in a later snapshot/export become a **subset** of an earlier snapshot/export (thinning)
- some samples become **interval-based** (`endDate > startDate`) and/or values become fractional
- for cumulative quantities, `value_sum` can remain stable while per-sample `value_max` increases (consolidation)

**Why it matters:** A “record counter” cannot distinguish discard vs consolidation. HealthProbe must support value-level forensics and (optionally) preserve complete evidence locally.

**Detection method:** Snapshot comparison of fingerprints *plus* optional per-sample archives for selected data types.

---

### 2.3 Why This Matters

| Concern | Impact | HealthProbe Role |
|---------|--------|-----------------|
| **Data loss undetected** | Users lose personal health history with no notification | Immediate detection & alert |
| **No forensic trail** | Impossible to reproduce for bug reports | Audit trail enables Apple debugging |
| **Blame uncertainty** | "Is it sync? Backup? A bug?" | Precise classification of anomaly type |
| **Third-party apps** | Apps assume data is trustworthy, may make wrong decisions | Detect corruption before downstream use |
| **Privacy of monitoring** | Users fear data exfiltration by health apps | Local-only observation, no cloud upload |

---

## 3. Core Architecture

### 3.1 Design Principles

1. **Read-only operations** (never modify HealthKit data)
2. **Local-first** (full functionality without network)
3. **Incremental queries** (efficient, avoid repeating work)
4. **Single archive store** (do not split the forensic store per data type; cross-type relationships and shared metadata matter)
5. **Auditability** (every observation logged, timestamped, reproducible)
6. **Privacy by default** (no HealthProbe cloud sync; local storage remains under user control)
7. **Forensic capture** (selected data types are archived locally as complete per-sample records with metadata to preserve evidence against silent rewrites)

### 3.2 Threading Model

```
┌─────────────────────────────────────────┐
│  Main Thread (UI)                       │
│  - Display current health status        │
│  - Show alerts & anomalies              │
│  - User interaction                     │
└──────────────┬──────────────────────────┘
               │
               ├─ Delegate query results
               │
┌──────────────▼──────────────────────────┐
│  Background Queue (HealthKit Queries)   │
│  - HKAnchoredObjectQuery (efficient)    │
│  - HKObserverQuery (reactive)           │
│  - Snapshot comparisons                 │
│  - Anomaly detection logic              │
└──────────────┬──────────────────────────┘
               │
               ├─ Write detected anomalies
               │
┌──────────────▼──────────────────────────┐
│  Local Archive Store                    │
│  - Canonical HealthKit samples          │
│  - Sources, devices, metadata           │
│  - Cross-type relationships             │
│  - Fingerprints and verification hashes │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│  SwiftData UI Store                     │
│  - Precomputed counts/statistics        │
│  - Visualization state and settings     │
│  - Logs, history, report indexes        │
└─────────────────────────────────────────┘
```

### 3.3 Storage Model

**Local Archive Store (source of truth):**
- one robust local database for all archived samples, not one archive per data type
- normalized entities for samples, workouts, sources, source revisions, devices, metadata, relationships, and observations
- multiple fingerprints per sample: HealthKit UUID, strict fingerprint, semantic fingerprint, and fuzzy matching keys for export/backup reconciliation
- append-only observation history (`firstSeen`, `lastSeen`, `lastVerified`, disappearance evidence)
- snapshot-level and table-level hashes for integrity checks

**SwiftData UI Store (derived/cache layer):**
- settings and selected data types
- import job state and progress
- precomputed counts, temporal bins, display ranges, and summary statistics
- audit log entries and report indexes
- anomaly summaries and links into the archive store

SwiftData rows must be rebuildable from the local archive store. If the two disagree, the archive store wins.

---

## 4. Monitoring Features (MVP)

### 4.1 Incremental Change Detection

**Using `HKAnchoredObjectQuery`:**
```
Query pattern:
├─ Initial query: anchor = 0 → captures all existing data
├─ Store anchor locally
├─ Periodic queries: anchor = stored → captures only new/modified samples
└─ Update anchor → efficient incremental updates
```

**What triggers a query:**
- App launch
- Background refresh (iOS allows periodic background queries)
- User manually triggers "Check Now"
- Every 12-24 hours (configurable)

### 4.2 Tracked Sample Types (Extensible)

| Type | Why Monitored | Anomaly Signal |
|------|---------------|----------------|
| **Workouts** | High-value data, often synced from watch | Historical insertions, duplicates |
| **Heart Rate** | Continuous stream, high modification risk | Gaps, divergence |
| **Activity Summary** | Auto-computed, depends on other types | Recalculation without notice |
| **Steps** | Cross-device (watch/phone), sync-heavy | Duplicate from retries |
| **Sleep** | Frequently "corrected" post-recording | Backdated entries, loss |
| **Blood Pressure** | Manual entry, sync state-dependent | Divergence trends |
| **Audio Exposure** | Often device-specific | Selective loss |

### 4.3 Anomaly Detection Logic

#### A. Historical Insertion Detection
```
For each sample:
  Δt = now - startDate  (age of sample)
  Δt_observed = now - firstSeen  (how long we've known about it)
  
  IF Δt >> Δt_observed (e.g., Δt ≈ 6 months, Δt_observed ≈ 5 minutes):
    → Flag as "historical insertion"
    → Severity: MEDIUM (might be legitimate correction)
```

#### B. Deletion Detection
```
Current snapshot: S_now
Previous snapshot: S_prev

Missing = S_prev - S_now
  (samples present before, absent now)

IF |Missing| > 0 AND no HKDeletedObject notification:
  → Flag as "silent deletion"
  → Severity: CRITICAL
  → Record gap_duration = time between last observation and absence
```

#### C. Duplicate Detection
```
Fingerprint = (sampleType, startDate, value, unit, source)

IF count(fingerprint) > 1:
  → Flag as "duplicate record"
  → Severity: LOW (data integrity risk)
  → Calculate time between duplicates
```

#### D. Divergence Detection
```
Track aggregated metrics:
  total_steps_per_day[date]
  active_energy_per_day[date]
  hr_average_per_day[date]

For each metric over time:
  σ_expected = standard deviation (normal range)
  σ_observed = recent variance
  
  IF σ_observed > 2 * σ_expected:
    → Flag as "divergence trend"
    → Severity: MEDIUM
    → Record trend direction & magnitude
```

---

## 5. Sync Context Logging

HealthProbe does **not** sync its own archive through iCloud or CloudKit. Observed HealthKit databases can diverge between devices, so cross-device HealthProbe sync would increase complexity without providing a reliable forensic source of truth.

Health/iCloud state is still useful as **context** for anomalies.

### 5.1 Context Tracking

**Observe HealthKit permission & sync state:**
```swift
HKHealthStore().requestAuthorization(...)
// → Detect when user grants/revokes permissions

// Monitor iCloud state
FileManager.default.ubiquityIdentityToken
// → Detects iCloud sign-in/sign-out
// → Logs context for later correlation
```

**Capture lifecycle events:**
- iCloud sign-in detected → log context and schedule a local archive verification pass
- iCloud sign-out detected → note local-only mode
- Device backup initiated → pre-backup snapshot
- App backgrounded/foregrounded → check for sync activity

### 5.2 Context Documentation

**Audit trail entries:**
```
[2026-05-01 14:23:15] SYNC_STATE_CHANGE: iCloud enabled
  - Previous: local-only
  - Action: archive verification scheduled
  - Result: no HealthProbe cloud sync performed

[2026-05-01 14:24:02] SYNC_COMPLETE: iCloud data merged
  - Samples added: 87
  - Samples deleted: 3
  - Duplicates found: 2
  - Divergence detected: NO

[2026-05-01 16:15:00] ANOMALY_DETECTED: Historical insertion
  - Sample: Workout "Running" 
  - Original date: 2024-03-15
  - First observed: 2026-05-01
  - Age: 778 days
  - Severity: MEDIUM
```

### 5.3 Background Monitoring

**iOS Background Modes enabled:**
- `background-fetch` — periodic archive and context checks
- `remote-notification` → not required for HealthProbe archive sync

**Check frequency:**
- Min: 2 hours
- Max: 24 hours
- Adapts based on anomaly detection frequency

---

## 6. Local Archive, Reports & Forensics

### 6.1 Local Archive Store

The main backup artifact is the on-device archive store. It is populated incrementally from HealthKit and is not dependent on Apple Health ZIP exports or full encrypted iPhone backups.

The archive must preserve as much HealthKit information as the API exposes:
- sample UUID, type, start/end date, value, unit, and metadata
- source, source revision, bundle identifier, product type, version/build if available
- device fields exposed by `HKDevice`
- relationships between workouts, samples, events, and other linked records where available
- first-seen / last-seen / last-verified observations
- fingerprints suitable for matching against Apple Health XML exports and extracted backup databases

The archive is selected by data type for performance and privacy, but it is stored in **one schema** so later analysis can follow relationships between types.

### 6.2 Reports and Point Exports

HealthProbe does not need to optimize for routine complete exports. The local archive is the backup.

Export is scoped to what the user is inspecting:
- anomaly reports
- tables of records shown in UI (e.g., “these 1,000 HK records disappeared”)
- point-in-time manifests and hashes
- selected record sets needed for external analysis

### 6.3 Forensic Query Examples

**"Has my step data been compromised?"**
```
1. Load all snapshots for "Steps" type
2. Plot sample count over time
3. Identify gaps > 6 hours
4. Report: when, how many missing, context
```

**"Did iCloud sync break my data?"**
```
1. Correlate anomalies with observed Health/iCloud state changes
2. Show timeline: before state change, during reconciliation, after
3. Calculate: samples lost, duplicates introduced
```

**"Is my health data drifting?"**
```
1. Compute daily aggregates (steps, energy, HR)
2. Fit trend line over 30-90 days
3. Report: slope (drift direction), R² (confidence)
4. Compare to device baseline
```

---

## 7. User-Facing Features

### 7.1 Dashboard (iOS App)

**Home Screen:**
- **Health Status** — "✅ Healthy" / "⚠️ Check" / "🚨 Critical"
- **Last Check** — timestamp of last monitoring run
- **Quick Stats** — samples tracked, anomalies found (all-time)
- **Active Alerts** — up to 3 most recent anomalies

**Detail Views:**
- **Anomalies** — sortable list by date/severity
- **Snapshots** — historical timeline of known-good snapshots
- **Audit Trail** — complete immutable log
- **Archive Status** — current local archive health, last verification, selected data types

**Settings:**
- Check frequency
- Sample types to track
- Alert thresholds
- Local archive retention and report export options

### 7.2 Alerts

**Push Notifications (opt-in):**
- 🚨 "Critical data loss detected" (> 10% samples missing)
- ⚠️ "Unexpected historical data inserted" (> 100 samples)
- ℹ️ "Archive check completed, 2 duplicates found"

---

## 8. Future Enhancements (Beyond MVP)

### 8.1 macOS Companion (Visualization Layer)
- Open and analyze exported HealthProbe reports or archive copies
- Long-term trend visualization (6-12 month history)
- Cross-device anomaly correlation
- Export to reproducible bug reports

### 8.2 Machine Learning
- Personalized baseline generation
- Anomaly confidence scoring
- Predictive detection (flag drift before threshold hit)

### 8.3 Community Patterns
- Anonymized digest sharing → identify systemic issues
- Detect if data loss correlates with: iOS version, device model, iCloud region, etc.
- Contribute to DearApple bug reports with statistical evidence

---

## 9. Technical Specifications

### 9.1 Platform
- **iOS 15.0+** (HealthKit framework support)
- **watchOS 8.0+** (future sync awareness)
- **macOS 12.0+** (visualization, analysis)

### 9.2 Permissions Required
- `HealthKit` — read-only access to specified types
- `Background Modes` — "Background Fetch"

### 9.3 Data Storage
- **Local Archive Store:** canonical HealthKit sample archive (source of truth)
- **SwiftData:** derived UI/cache/settings/log/history store
- **No CloudKit sync:** HealthProbe data remains local unless the user exports a report or selected record table

### 9.4 Performance
- Query time: < 5 seconds (anchored queries)
- Snapshot/index size: ≈ 5-10 KB per type per snapshot in SwiftData
- Archive storage: depends on selected high-frequency data types; report per-type storage costs in settings

---

## 10. Privacy & Security

### 10.1 What HealthProbe Never Does
- ❌ Exports raw health samples to cloud
- ❌ Identifies users by name/account
- ❌ Shares device location or personal context
- ❌ Modifies any HealthKit data
- ❌ Sells or shares data with third parties

### 10.2 What HealthProbe Collects (Local Only)
- ✅ Aggregated counts (not samples)
- ✅ Timestamps of anomalies
- ✅ Device model & iOS version (for context)
- ✅ Anomaly types & severity

**Local archive:**
- ✅ Per-sample archive for user-selected types, stored on-device and exportable by user
- ✅ Metadata needed for recognition in Apple Health XML exports, backup database extracts, and future datasets

### 10.3 Cloud Policy
- No HealthProbe CloudKit/iCloud sync
- No automatic upload of raw samples, digests, reports, or device fingerprints
- User-triggered exports are explicit, scoped, and local-file based

---

## 11. Success Criteria

| Objective | Metric | Target |
|-----------|--------|--------|
| **Detect loss** | Time to detection after loss occurs | < 24 hours |
| **Forensic completeness** | % of anomalies with sufficient evidence | > 95% |
| **False positives** | Alerts user shouldn't worry about | < 5% of total |
| **Privacy** | % of users comfortable with data practices | > 90% |
| **Performance** | Background capture battery impact | < 2% drain/day |
| **Adoption** | Users can reproduce bugs with HealthProbe data | High relevance in Apple feedback |

---

## 12. References & Related Work

- [DearApple Issue #001](https://github.com/overbog/dear-apple/issues/0001-apple-health-mass-data-loss.md) — Sept 2025 mass data loss
- [Apple HealthKit Documentation](https://developer.apple.com/documentation/healthkit/)
- [HKAnchoredObjectQuery](https://developer.apple.com/documentation/healthkit/hkanchoredrobjectquery) — Efficient incremental queries

---

## Appendix A: Example Anomaly Report

```json
{
  "anomaly_id": "ANML_20260501_001",
  "type": "historical_insertion",
  "timestamp_detected": "2026-05-01T14:35:22Z",
  "severity": "MEDIUM",
  "evidence": {
    "sample_type": "HKWorkout",
    "workout_type": "Running",
    "start_date": "2025-01-15T07:30:00Z",
    "end_date": "2025-01-15T08:15:00Z",
    "duration_minutes": 45,
    "calories": 420,
    "first_observed": "2026-05-01T14:35:00Z",
    "age_days": 106,
    "source": "Health.app",
    "context": "iCloud sync completed 2 hours prior"
  },
  "classification": "Likely data recovery from cloud",
  "recommended_action": "Monitor for similar patterns"
}
```

---

*HealthProbe — Guarding the integrity of your health data.*
