autoSMART Installation and Setup Guide

🚀 Quick Start

Prerequisites Checklist

System Requirements

✅ Operating System: Linux (Ubuntu 20.04+, CentOS 8+, Proxmox VE 7+)
✅ Perl: Version 5.20+ with CPAN access
✅ PostgreSQL: Version 13+ with JSONB support
✅ Hardware Access: sudo/root access for SMART data collection
✅ Network: Access to OpenAI API (optional, for AI predictions)

Test Database Available

Host: 192.168.2.102
Database: autosmart
User: postgres
Password: (no password)
Port: 5432

🔧 Installation Steps

1. System Dependencies

Ubuntu/Debian

# Update system packages
sudo apt update && sudo apt upgrade -y

# Install system dependencies
sudo apt install -y perl postgresql-client smartmontools git curl

# Install PostgreSQL server (if not using remote database)
sudo apt install -y postgresql postgresql-contrib

# Install Perl development tools
sudo apt install -y build-essential cpanminus libdbi-perl

CentOS/RHEL/Rocky Linux

# Update system packages
sudo dnf update -y

# Install system dependencies
sudo dnf install -y perl postgresql smartmontools git curl

# Install development tools
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y perl-App-cpanminus perl-DBI

Proxmox VE

# Proxmox already includes most dependencies
apt update
apt install -y cpanminus libdbi-perl libdbd-pg-perl libjson-xs-perl

2. Perl Modules Installation

Required Modules

# Core database connectivity
sudo cpanm DBI DBD::Pg

# JSON processing
sudo cpanm JSON::XS

# Configuration and utilities
sudo cpanm Config::Simple File::Slurp Time::HiRes Digest::SHA

# HTTP clients for API integration
sudo cpanm LWP::UserAgent HTTP::Request::Common

# Optional: Testing modules
sudo cpanm Test::More Test::Exception Data::Dumper

Verify Perl Module Installation

perl -e "
use DBI;
use JSON::XS; 
use Config::Simple;
use Digest::SHA;
use LWP::UserAgent;
print \"All required Perl modules installed successfully!\n\";
"

3. Database Setup

Option A: Use Test Database (Recommended for Development)

# Test connection to existing database
psql -h 192.168.2.102 -U postgres -d autosmart -c "SELECT version();"

# If successful, skip to step 4 - Project Installation

Option B: Local PostgreSQL Installation

# Install PostgreSQL
sudo apt install -y postgresql postgresql-contrib

# Start and enable PostgreSQL
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Create database and user
sudo -u postgres psql << EOF
CREATE DATABASE autosmart;
CREATE USER autosmart WITH PASSWORD 'smartpassword';
GRANT ALL PRIVILEGES ON DATABASE autosmart TO autosmart;
ALTER USER autosmart CREATEDB;
\q
EOF

Option C: Remote PostgreSQL Setup

# Connect to your PostgreSQL server
psql -h your-db-host -U postgres

# Create database and configure
CREATE DATABASE autosmart;
CREATE USER autosmart WITH PASSWORD 'your-secure-password';
GRANT ALL PRIVILEGES ON DATABASE autosmart TO autosmart;

4. Project Installation

Download and Setup

# Create installation directory
sudo mkdir -p /etc/pve/autoSMART
cd /etc/pve/autoSMART

# Clone or copy project files (adjust as needed)
# git clone https://github.com/your-repo/autoSMART.git .
# OR copy from development workspace:
cp -r /Users/bogdan/Documents/workspace/autoSMART/* .

# Set proper ownership and permissions
sudo chown -R root:root .
chmod +x scripts/*.pl
chmod 600 config/cluster.conf

Directory Structure Verification

tree /etc/pve/autoSMART
# Should show:
# ├── config/
# ├── docs/
# ├── lib/
# ├── scripts/
# ├── sql/
# └── README.md

5. Database Deployment

autoSMART uses PostgreSQL for storing SMART data, configurations, and analysis results. You can deploy the database schema from your development machine using the included deployment scripts.

Prerequisites for Database Deployment

✅ psql client installed on development machine (macOS/Linux)
✅ Network access to target PostgreSQL server
✅ Database credentials with schema creation privileges
✅ Target database already created and accessible

Database Deployment with deploy.sh

The deploy.sh script can install the database schema remotely using psql from your development machine:

# Show help and available options
./deploy.sh

# Deploy database schema to remote PostgreSQL server
./deploy.sh install database --db-host 192.168.2.102 --db-user postgres --db-name autosmart

# Deploy with custom credentials
./deploy.sh install database \
  --db-host your-postgres-server.local \
  --db-user autosmart \
  --db-pass your-password \
  --db-name autosmart_prod

Manual Database Installation from Development Machine

If you prefer manual control over the database installation:

# 1. Ensure psql is available on your development machine
# macOS:
brew install postgresql

# Ubuntu/Debian:
sudo apt install postgresql-client

# 2. Test connection to target database
psql -h 192.168.2.102 -U postgres -d autosmart -c "SELECT version();"

# 3. Install the complete schema
psql -h 192.168.2.102 -U postgres -d autosmart -f sql/schema.sql

# 4. Verify schema installation
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT 
    schemaname,
    tablename,
    tableowner 
FROM pg_tables 
WHERE schemaname = 'public' 
ORDER BY tablename;
"

# 5. Check database functions and triggers
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT 
    proname as function_name,
    pg_get_function_result(oid) as return_type
FROM pg_proc 
WHERE pronamespace = (SELECT oid FROM pg_namespace WHERE nspname = 'public')
ORDER BY proname;
"

Database Schema Components

The autoSMART database schema includes:

Core Tables: - hdd_inventory - Physical drive tracking and migration history - smart_readings - Raw SMART data collection (differential storage) - smart_thresholds - Drive-specific alert thresholds - predictions - AI-generated failure predictions - alert_history - System alerts and notifications - system_config - Cluster-wide configuration settings

Analytical Views: - smart_readings_reconstructed - Full SMART data reconstruction from differential storage - latest_smart_readings - Most recent SMART values per drive - drive_health_summary - Drive health status and trend analysis

Functions and Triggers: - differential_storage_trigger() - Automatic differential storage on SMART updates - update_drive_health() - Health score calculation - cleanup_old_readings() - Data retention management

Database Verification Commands

# Verify all components are installed
psql -h 192.168.2.102 -U postgres -d autosmart << EOF

-- Check table count and sizes
SELECT 
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
FROM pg_tables 
WHERE schemaname = 'public';

-- Check views
SELECT 
    schemaname,
    viewname,
    definition
FROM pg_views 
WHERE schemaname = 'public';

-- Test differential storage function
SELECT differential_storage_trigger() as function_test;

-- Verify database is ready
SELECT 'autoSMART database ready!' as status;

EOF

Troubleshooting Database Installation

Connection Issues: ```bash

Test basic connectivity

ping 192.168.2.102

Test PostgreSQL port

telnet 192.168.2.102 5432

Test authentication

psql -h 192.168.2.102 -U postgres -d postgres -c "SELECT current_user;" ```

Schema Installation Issues: ```bash

Check for existing schema conflicts

psql -h 192.168.2.102 -U postgres -d autosmart -c " SELECT table_name FROM information_schema.tables WHERE table_schema = 'public' AND table_name LIKE '%smart%'; "

Force clean installation (⚠️ DESTRUCTIVE)

psql -h 192.168.2.102 -U postgres -d autosmart -c " DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO postgres; GRANT ALL ON SCHEMA public TO public; "

Reinstall schema

psql -h 192.168.2.102 -U postgres -d autosmart -f sql/schema.sql ```

6. Database Schema Installation (Legacy Method)

Using Test Database (192.168.2.102)

# Install the complete schema
cd /etc/pve/autoSMART
psql -h 192.168.2.102 -U postgres -d autosmart -f sql/schema.sql

# Verify installation
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT table_name FROM information_schema.tables 
WHERE table_schema = 'public' 
ORDER BY table_name;
"

Expected Tables

✅ hdd_inventory
✅ hdd_migrations
✅ smart_readings
✅ predictions
✅ smart_thresholds
✅ alert_history
✅ system_config

Expected Views

✅ smart_readings_reconstructed
✅ latest_smart_readings
✅ drive_health_summary

7. Configuration

Cluster Configuration

# Edit cluster-wide settings
nano /etc/pve/autoSMART/config/cluster.conf

[database]
host = 192.168.2.102
port = 5432
name = autosmart
user = postgres
password = 

[collection]
interval = 1800
timeout = 60
madagascar_inventory_path = /opt/madagascar/inventory.json

[ai_predictions]
enabled = true
openai_api_key = your-openai-api-key-here
openai_model = gpt-4
prediction_interval = 86400

[alerts]
enabled = true
email_notifications = true
slack_webhook = https://hooks.slack.com/your-webhook

Local Node Configuration

# Copy default configuration
cp config/defaults/autosmart /etc/default/autosmart

# Edit local settings
nano /etc/default/autosmart

# autoSMART local configuration
AUTOSMART_DEBUG=2
AUTOSMART_NODE_ID=$(hostname)
AUTOSMART_CLUSTER_CONFIG="/etc/pve/autoSMART/config/cluster.conf"

# Local database override (if needed)
# AUTOSMART_DB_HOST=192.168.2.102
# AUTOSMART_DB_USER=postgres
# AUTOSMART_DB_PASS=

# OpenAI API configuration
OPENAI_API_KEY=your-api-key-here

# Collection settings
SMART_COLLECTION_ENABLED=true
MIGRATION_DETECTION_ENABLED=true
DIFFERENTIAL_STORAGE_ENABLED=true

8. Testing Installation

Database Connectivity Test

cd /etc/pve/autoSMART
perl -e "
use lib 'lib';
use DBI;

my \$dsn = 'DBI:Pg:dbname=autosmart;host=192.168.2.102;port=5432';
my \$dbh = DBI->connect(\$dsn, 'postgres', '', {RaiseError => 1});

print \"✅ Database connection successful!\n\";

# Test schema
my \$sth = \$dbh->prepare('SELECT COUNT(*) FROM hdd_inventory');
\$sth->execute();
my (\$count) = \$sth->fetchrow_array();

print \"✅ Schema installed - hdd_inventory table accessible\n\";
\$dbh->disconnect();
"

SMART Data Collection Test

# Test SMART data access
sudo smartctl -a /dev/sda | head -20

# Test collection script (dry-run mode)
cd /etc/pve/autoSMART/scripts
sudo perl collect-smart-data.pl --test --dry-run

Differential Storage Test

# Run comprehensive storage test
cd /etc/pve/autoSMART/scripts
perl test-differential-storage.pl

Expected output: === autoSMART Differential Storage Test === ✓ Connected to database ✓ Created test HDD (ID: 1) ✓ Inserted baseline reading (ID: 1) ✓ Identical reading test - Should store: NO (Type: baseline) ✓ Temperature change reading - Should store: YES (Type: differential, ID: 2) ✓ Critical change reading - Should store: YES (Type: full, ID: 3) --- Storage Statistics --- baseline : 1 readings, avg size: 245 bytes differential : 1 readings, avg size: 89 bytes full : 1 readings, avg size: 245 bytes Total: 3 readings, estimated size: 579 bytes === Test Complete ===

9. Service Configuration (Optional)

SystemD Service Files

Create /etc/systemd/system/autosmart-collector.service: ```ini [Unit] Description=autoSMART Data Collector After=network.target postgresql.service

[Service] Type=simple User=root WorkingDirectory=/etc/pve/autoSMART ExecStart=/usr/bin/perl scripts/collect-smart-data.pl --daemon Restart=always RestartSec=30

[Install] WantedBy=multi-user.target ```

Create /etc/systemd/system/autosmart-analyzer.service: ```ini [Unit] Description=autoSMART AI Analyzer After=network.target postgresql.service autosmart-collector.service

[Service] Type=simple User=root WorkingDirectory=/etc/pve/autoSMART ExecStart=/usr/bin/perl scripts/analyze-smart-data.pl --daemon Restart=always RestartSec=60

[Install] WantedBy=multi-user.target ```

Enable Services

# Reload systemd configuration
sudo systemctl daemon-reload

# Enable and start services
sudo systemctl enable autosmart-collector
sudo systemctl start autosmart-collector

sudo systemctl enable autosmart-analyzer  
sudo systemctl start autosmart-analyzer

# Check service status
sudo systemctl status autosmart-collector
sudo systemctl status autosmart-analyzer

10. Verification and Monitoring

Log Files

# View collection logs
sudo journalctl -u autosmart-collector -f

# View analysis logs  
sudo journalctl -u autosmart-analyzer -f

# Check syslog for SMART events
sudo tail -f /var/log/syslog | grep -i smart

Database Monitoring

# Monitor data collection
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT 
    COUNT(*) as total_readings,
    MAX(timestamp) as latest_reading,
    COUNT(DISTINCT hdd_id) as active_drives
FROM smart_readings 
WHERE timestamp > NOW() - INTERVAL '24 hours';
"

# Monitor storage efficiency
psql -h 192.168.2.102 -U postgres -d autosmart -c "
SELECT 
    reading_type,
    COUNT(*) as count,
    COUNT(*) * 100.0 / SUM(COUNT(*)) OVER() as percentage
FROM smart_readings 
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY reading_type;
"

🎯 Post-Installation Steps

1. Madagascar Integration

# Ensure Madagascar inventory is accessible
ls -la /opt/madagascar/inventory.json

# Test Madagascar data parsing
cd /etc/pve/autoSMART/scripts
perl -e "
use JSON::XS;
my \$data = decode_json(qx(cat /opt/madagascar/inventory.json));
print 'Madagascar drives found: ' . scalar(\@{\$data->{drives}}) . \"\n\";
"

2. OpenAI API Setup (Optional)

# Test OpenAI API access
curl -H "Authorization: Bearer your-api-key" \
     -H "Content-Type: application/json" \
     -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello"}]}' \
     https://api.openai.com/v1/chat/completions

3. Alert Configuration

# Test email notifications (if configured)
echo "Test autoSMART alert" | mail -s "autoSMART Test" admin@yourcompany.com

# Test Slack webhook (if configured)
curl -X POST -H 'Content-type: application/json' \
     --data '{"text":"autoSMART installation test"}' \
     YOUR_SLACK_WEBHOOK_URL

🔍 Troubleshooting

Common Issues

Database Connection Failed

# Check PostgreSQL service
sudo systemctl status postgresql

# Test network connectivity
telnet 192.168.2.102 5432

# Check firewall
sudo ufw status
sudo iptables -L | grep 5432

Permission Denied for SMART Data

# Check smartctl permissions
ls -la /usr/sbin/smartctl

# Test with sudo
sudo smartctl -a /dev/sda

# Add user to disk group (if running as non-root)
sudo usermod -a -G disk $USER

Perl Module Issues

# Check module installation
perl -MDBI -e 'print "DBI version: $DBI::VERSION\n"'
perl -MJSON::XS -e 'print "JSON::XS installed OK\n"'

# Reinstall problematic modules
sudo cpanm --force --reinstall DBD::Pg

Log Analysis

# Enable debug logging
export AUTOSMART_DEBUG=3

# Run collection manually for debugging
cd /etc/pve/autoSMART/scripts
sudo perl collect-smart-data.pl --verbose --test

📚 Next Steps

Customize Configuration: Adjust thresholds and intervals for your environment
Set Up Monitoring: Configure alerts and dashboards
Schedule Regular Backups: Database and configuration files
Plan for Scaling: Consider performance optimization for large deployments
Implement AI Predictions: Configure OpenAI integration for failure prediction

For more detailed information, see: - DEVELOPMENT.md - Development and customization guide - API.md - OpenAI API integration details - DIFFERENTIAL_STORAGE.md - Storage optimization details - MIGRATION_DETECTION.md - HDD tracking system details

🆘 Getting Help

If you encounter issues during installation:

Check logs: journalctl -u autosmart-collector -n 50
Verify database: Test database connectivity and schema
Test components: Run individual scripts manually with debug output
Review configuration: Ensure all paths and credentials are correct
Check dependencies: Verify all system and Perl dependencies are installed

The autoSMART system is designed to be robust and self-healing, but proper installation and configuration are essential for optimal performance.