# autoSMART Installation and Setup Guide

## 🚀 Quick Start

### Prerequisites Checklist

#### System Requirements
- ✅ **Operating System**: Linux (Ubuntu 20.04+, CentOS 8+, Proxmox VE 7+)
- ✅ **Perl**: Version 5.20+ with CPAN access
- ✅ **PostgreSQL**: Version 13+ with JSONB support
- ✅ **Hardware Access**: sudo/root access for SMART data collection
- ✅ **Network**: Access to OpenAI API (optional, for AI predictions)

#### Test Database Available
```
Host: 192.168.2.102
Database: autosmart
User: postgres
Password: (no password)
Port: 5432
```

## 🔧 Installation Steps

### 1. System Dependencies

#### Ubuntu/Debian
```bash
# Update system packages
sudo apt update && sudo apt upgrade -y

# Install system dependencies
sudo apt install -y perl postgresql-client smartmontools git curl

# Install PostgreSQL server (if not using remote database)
sudo apt install -y postgresql postgresql-contrib

# Install Perl development tools
sudo apt install -y build-essential cpanminus libdbi-perl
```

#### CentOS/RHEL/Rocky Linux
```bash
# Update system packages
sudo dnf update -y

# Install system dependencies
sudo dnf install -y perl postgresql smartmontools git curl

# Install development tools
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y perl-App-cpanminus perl-DBI
```

#### Proxmox VE
```bash
# Proxmox already includes most dependencies
apt update
apt install -y cpanminus libdbi-perl libdbd-pg-perl libjson-xs-perl
```

### 2. Perl Modules Installation

#### Required Modules
```bash
# Core database connectivity
sudo cpanm DBI DBD::Pg

# JSON processing
sudo cpanm JSON::XS

# Configuration and utilities
sudo cpanm Config::Simple File::Slurp Time::HiRes Digest::SHA

# HTTP clients for API integration
sudo cpanm LWP::UserAgent HTTP::Request::Common

# Optional: Testing modules
sudo cpanm Test::More Test::Exception Data::Dumper
```

#### Verify Perl Module Installation
```bash
perl -e "
use DBI;
use JSON::XS; 
use Config::Simple;
use Digest::SHA;
use LWP::UserAgent;
print \"All required Perl modules installed successfully!\n\";
"
```

### 3. Database Setup

#### Option A: Use Test Database (Recommended for Development)
```bash
# Test connection to existing database
psql -h 192.168.2.102 -U postgres -d autosmart -c "SELECT version();"

# If successful, skip to step 4 - Project Installation
```

#### Option B: Local PostgreSQL Installation
```bash
# Install PostgreSQL
sudo apt install -y postgresql postgresql-contrib

# Start and enable PostgreSQL
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Create database and user
sudo -u postgres psql << EOF
CREATE DATABASE autosmart;
CREATE USER autosmart WITH PASSWORD 'smartpassword';
GRANT ALL PRIVILEGES ON DATABASE autosmart TO autosmart;
ALTER USER autosmart CREATEDB;
\q
EOF
```

#### Option C: Remote PostgreSQL Setup
```bash
# Connect to your PostgreSQL server
psql -h your-db-host -U postgres

# Create database and configure
CREATE DATABASE autosmart;
CREATE USER autosmart WITH PASSWORD 'your-secure-password';
GRANT ALL PRIVILEGES ON DATABASE autosmart TO autosmart;
```

### 4. Project Installation

#### Download and Setup
```bash
# Create installation directory
sudo mkdir -p /etc/pve/autoSMART
cd /etc/pve/autoSMART

# Clone or copy project files (adjust as needed)
# git clone https://github.com/your-repo/autoSMART.git .
# OR copy from development workspace:
cp -r /Users/bogdan/Documents/workspace/autoSMART/* .

# Set proper ownership and permissions
sudo chown -R root:root .
chmod +x scripts/*.pl
chmod 600 config/cluster.conf
```

#### Directory Structure Verification
```bash
tree /etc/pve/autoSMART
# Should show:
# ├── config/
# ├── docs/
# ├── lib/
# ├── scripts/
# ├── sql/
# └── README.md
```

### 5. Database Deployment

autoSMART uses PostgreSQL for storing SMART data, configurations, and analysis results. You can deploy the database schema from your development machine using the included deployment scripts.

#### Prerequisites for Database Deployment
- ✅ **psql** client installed on development machine (macOS/Linux)
- ✅ **Network access** to target PostgreSQL server
- ✅ **Database credentials** with schema creation privileges
- ✅ **Target database** already created and accessible

#### Database Deployment with deploy.sh

The `deploy.sh` script can install the database schema remotely using psql from your development machine:

```bash
# Show help and available options
./deploy.sh

# Deploy database schema to remote PostgreSQL server
./deploy.sh install database --db-host 192.168.2.102 --db-user postgres --db-name autosmart

# Deploy with custom credentials
./deploy.sh install database \
  --db-host your-postgres-server.local \
  --db-user autosmart \
  --db-pass your-password \
  --db-name autosmart_prod
```

#### Manual Database Installation from Development Machine

If you prefer manual control over the database installation:

```bash
# 1. Ensure psql is available on your development machine
# macOS:
brew install postgresql

# Ubuntu/Debian:
sudo apt install postgresql-client

# 2. Test connection to target database
psql -h 192.168.2.102 -U postgres -d autosmart -c "SELECT version();"

# 3. Install the complete schema
psql -h 192.168.2.102 -U postgres -d autosmart -f sql/schema.sql

# 4. Verify schema installation
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT 
    schemaname,
    tablename,
    tableowner 
FROM pg_tables 
WHERE schemaname = 'public' 
ORDER BY tablename;
"

# 5. Check database functions and triggers
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT 
    proname as function_name,
    pg_get_function_result(oid) as return_type
FROM pg_proc 
WHERE pronamespace = (SELECT oid FROM pg_namespace WHERE nspname = 'public')
ORDER BY proname;
"
```

#### Database Schema Components

The autoSMART database schema includes:

**Core Tables:**
- `hdd_inventory` - Physical drive tracking and migration history
- `smart_readings` - Raw SMART data collection (differential storage)
- `smart_thresholds` - Drive-specific alert thresholds
- `predictions` - AI-generated failure predictions
- `alert_history` - System alerts and notifications
- `system_config` - Cluster-wide configuration settings

**Analytical Views:**
- `smart_readings_reconstructed` - Full SMART data reconstruction from differential storage
- `latest_smart_readings` - Most recent SMART values per drive
- `drive_health_summary` - Drive health status and trend analysis

**Functions and Triggers:**
- `differential_storage_trigger()` - Automatic differential storage on SMART updates
- `update_drive_health()` - Health score calculation
- `cleanup_old_readings()` - Data retention management

#### Database Verification Commands

```bash
# Verify all components are installed
psql -h 192.168.2.102 -U postgres -d autosmart << EOF

-- Check table count and sizes
SELECT 
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
FROM pg_tables 
WHERE schemaname = 'public';

-- Check views
SELECT 
    schemaname,
    viewname,
    definition
FROM pg_views 
WHERE schemaname = 'public';

-- Test differential storage function
SELECT differential_storage_trigger() as function_test;

-- Verify database is ready
SELECT 'autoSMART database ready!' as status;

EOF
```

#### Troubleshooting Database Installation

**Connection Issues:**
```bash
# Test basic connectivity
ping 192.168.2.102

# Test PostgreSQL port
telnet 192.168.2.102 5432

# Test authentication
psql -h 192.168.2.102 -U postgres -d postgres -c "SELECT current_user;"
```

**Schema Installation Issues:**
```bash
# Check for existing schema conflicts
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT table_name FROM information_schema.tables 
WHERE table_schema = 'public' AND table_name LIKE '%smart%';
"

# Force clean installation (⚠️ DESTRUCTIVE)
psql -h 192.168.2.102 -U postgres -d autosmart -c "
DROP SCHEMA public CASCADE;
CREATE SCHEMA public;
GRANT ALL ON SCHEMA public TO postgres;
GRANT ALL ON SCHEMA public TO public;
"

# Reinstall schema
psql -h 192.168.2.102 -U postgres -d autosmart -f sql/schema.sql
```

### 6. Database Schema Installation (Legacy Method)

#### Using Test Database (192.168.2.102)
```bash
# Install the complete schema
cd /etc/pve/autoSMART
psql -h 192.168.2.102 -U postgres -d autosmart -f sql/schema.sql

# Verify installation
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT table_name FROM information_schema.tables 
WHERE table_schema = 'public' 
ORDER BY table_name;
"
```

#### Expected Tables
- ✅ hdd_inventory
- ✅ hdd_migrations  
- ✅ smart_readings
- ✅ predictions
- ✅ smart_thresholds
- ✅ alert_history
- ✅ system_config

#### Expected Views
- ✅ smart_readings_reconstructed
- ✅ latest_smart_readings
- ✅ drive_health_summary

### 7. Configuration

#### Cluster Configuration
```bash
# Edit cluster-wide settings
nano /etc/pve/autoSMART/config/cluster.conf
```

```ini
[database]
host = 192.168.2.102
port = 5432
name = autosmart
user = postgres
password = 

[collection]
interval = 1800
timeout = 60
madagascar_inventory_path = /opt/madagascar/inventory.json

[ai_predictions]
enabled = true
openai_api_key = your-openai-api-key-here
openai_model = gpt-4
prediction_interval = 86400

[alerts]
enabled = true
email_notifications = true
slack_webhook = https://hooks.slack.com/your-webhook
```

#### Local Node Configuration
```bash
# Copy default configuration
cp config/defaults/autosmart /etc/default/autosmart

# Edit local settings
nano /etc/default/autosmart
```

```bash
# autoSMART local configuration
AUTOSMART_DEBUG=2
AUTOSMART_NODE_ID=$(hostname)
AUTOSMART_CLUSTER_CONFIG="/etc/pve/autoSMART/config/cluster.conf"

# Local database override (if needed)
# AUTOSMART_DB_HOST=192.168.2.102
# AUTOSMART_DB_USER=postgres
# AUTOSMART_DB_PASS=

# OpenAI API configuration
OPENAI_API_KEY=your-api-key-here

# Collection settings
SMART_COLLECTION_ENABLED=true
MIGRATION_DETECTION_ENABLED=true
DIFFERENTIAL_STORAGE_ENABLED=true
```

### 8. Testing Installation

#### Database Connectivity Test
```bash
cd /etc/pve/autoSMART
perl -e "
use lib 'lib';
use DBI;

my \$dsn = 'DBI:Pg:dbname=autosmart;host=192.168.2.102;port=5432';
my \$dbh = DBI->connect(\$dsn, 'postgres', '', {RaiseError => 1});

print \"✅ Database connection successful!\n\";

# Test schema
my \$sth = \$dbh->prepare('SELECT COUNT(*) FROM hdd_inventory');
\$sth->execute();
my (\$count) = \$sth->fetchrow_array();

print \"✅ Schema installed - hdd_inventory table accessible\n\";
\$dbh->disconnect();
"
```

#### SMART Data Collection Test  
```bash
# Test SMART data access
sudo smartctl -a /dev/sda | head -20

# Test collection script (dry-run mode)
cd /etc/pve/autoSMART/scripts
sudo perl collect-smart-data.pl --test --dry-run
```

#### Differential Storage Test
```bash
# Run comprehensive storage test
cd /etc/pve/autoSMART/scripts
perl test-differential-storage.pl
```

Expected output:
```
=== autoSMART Differential Storage Test ===
✓ Connected to database
✓ Created test HDD (ID: 1)
✓ Inserted baseline reading (ID: 1)
✓ Identical reading test - Should store: NO (Type: baseline)
✓ Temperature change reading - Should store: YES (Type: differential, ID: 2)
✓ Critical change reading - Should store: YES (Type: full, ID: 3)
--- Storage Statistics ---
baseline     : 1 readings, avg size: 245 bytes
differential : 1 readings, avg size: 89 bytes  
full         : 1 readings, avg size: 245 bytes
Total: 3 readings, estimated size: 579 bytes
=== Test Complete ===
```

### 9. Service Configuration (Optional)

#### SystemD Service Files

Create `/etc/systemd/system/autosmart-collector.service`:
```ini
[Unit]
Description=autoSMART Data Collector
After=network.target postgresql.service

[Service]
Type=simple
User=root
WorkingDirectory=/etc/pve/autoSMART
ExecStart=/usr/bin/perl scripts/collect-smart-data.pl --daemon
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target
```

Create `/etc/systemd/system/autosmart-analyzer.service`:
```ini
[Unit]
Description=autoSMART AI Analyzer
After=network.target postgresql.service autosmart-collector.service

[Service]
Type=simple
User=root
WorkingDirectory=/etc/pve/autoSMART
ExecStart=/usr/bin/perl scripts/analyze-smart-data.pl --daemon
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target
```

#### Enable Services
```bash
# Reload systemd configuration
sudo systemctl daemon-reload

# Enable and start services
sudo systemctl enable autosmart-collector
sudo systemctl start autosmart-collector

sudo systemctl enable autosmart-analyzer  
sudo systemctl start autosmart-analyzer

# Check service status
sudo systemctl status autosmart-collector
sudo systemctl status autosmart-analyzer
```

### 10. Verification and Monitoring

#### Log Files
```bash
# View collection logs
sudo journalctl -u autosmart-collector -f

# View analysis logs  
sudo journalctl -u autosmart-analyzer -f

# Check syslog for SMART events
sudo tail -f /var/log/syslog | grep -i smart
```

#### Database Monitoring
```bash
# Monitor data collection
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT 
    COUNT(*) as total_readings,
    MAX(timestamp) as latest_reading,
    COUNT(DISTINCT hdd_id) as active_drives
FROM smart_readings 
WHERE timestamp > NOW() - INTERVAL '24 hours';
"

# Monitor storage efficiency
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT 
    reading_type,
    COUNT(*) as count,
    COUNT(*) * 100.0 / SUM(COUNT(*)) OVER() as percentage
FROM smart_readings 
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY reading_type;
"
```

## 🎯 Post-Installation Steps

### 1. Madagascar Integration
```bash
# Ensure Madagascar inventory is accessible
ls -la /opt/madagascar/inventory.json

# Test Madagascar data parsing
cd /etc/pve/autoSMART/scripts
perl -e "
use JSON::XS;
my \$data = decode_json(qx(cat /opt/madagascar/inventory.json));
print 'Madagascar drives found: ' . scalar(\@{\$data->{drives}}) . \"\n\";
"
```

### 2. OpenAI API Setup (Optional)
```bash
# Test OpenAI API access
curl -H "Authorization: Bearer your-api-key" \
     -H "Content-Type: application/json" \
     -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello"}]}' \
     https://api.openai.com/v1/chat/completions
```

### 3. Alert Configuration
```bash
# Test email notifications (if configured)
echo "Test autoSMART alert" | mail -s "autoSMART Test" admin@yourcompany.com

# Test Slack webhook (if configured)
curl -X POST -H 'Content-type: application/json' \
     --data '{"text":"autoSMART installation test"}' \
     YOUR_SLACK_WEBHOOK_URL
```

## 🔍 Troubleshooting

### Common Issues

#### Database Connection Failed
```bash
# Check PostgreSQL service
sudo systemctl status postgresql

# Test network connectivity
telnet 192.168.2.102 5432

# Check firewall
sudo ufw status
sudo iptables -L | grep 5432
```

#### Permission Denied for SMART Data
```bash
# Check smartctl permissions
ls -la /usr/sbin/smartctl

# Test with sudo
sudo smartctl -a /dev/sda

# Add user to disk group (if running as non-root)
sudo usermod -a -G disk $USER
```

#### Perl Module Issues
```bash
# Check module installation
perl -MDBI -e 'print "DBI version: $DBI::VERSION\n"'
perl -MJSON::XS -e 'print "JSON::XS installed OK\n"'

# Reinstall problematic modules
sudo cpanm --force --reinstall DBD::Pg
```

### Log Analysis
```bash
# Enable debug logging
export AUTOSMART_DEBUG=3

# Run collection manually for debugging
cd /etc/pve/autoSMART/scripts
sudo perl collect-smart-data.pl --verbose --test
```

## 📚 Next Steps

1. **Customize Configuration**: Adjust thresholds and intervals for your environment
2. **Set Up Monitoring**: Configure alerts and dashboards  
3. **Schedule Regular Backups**: Database and configuration files
4. **Plan for Scaling**: Consider performance optimization for large deployments
5. **Implement AI Predictions**: Configure OpenAI integration for failure prediction

For more detailed information, see:
- [DEVELOPMENT.md](DEVELOPMENT.md) - Development and customization guide
- [API.md](API.md) - OpenAI API integration details
- [DIFFERENTIAL_STORAGE.md](DIFFERENTIAL_STORAGE.md) - Storage optimization details
- [MIGRATION_DETECTION.md](MIGRATION_DETECTION.md) - HDD tracking system details

## 🆘 Getting Help

If you encounter issues during installation:

1. **Check logs**: `journalctl -u autosmart-collector -n 50`
2. **Verify database**: Test database connectivity and schema
3. **Test components**: Run individual scripts manually with debug output
4. **Review configuration**: Ensure all paths and credentials are correct
5. **Check dependencies**: Verify all system and Perl dependencies are installed

The autoSMART system is designed to be robust and self-healing, but proper installation and configuration are essential for optimal performance.