POTE/docs/12_automation_and_reporting.md

# Automation and Reporting Guide

This guide covers automated data updates, email reporting, and CI/CD pipelines for POTE.

## Table of Contents

1. [Email Reporting Setup](#email-reporting-setup)
2. [Automated Daily/Weekly Runs](#automated-dailyweekly-runs)
3. [Cron Setup](#cron-setup)
4. [Health Checks](#health-checks)
5. [CI/CD Pipeline](#cicd-pipeline)

---

## Email Reporting Setup

### Configure SMTP Settings

POTE can send automated email reports. You need to configure SMTP settings in your `.env` file.

#### Option 1: Gmail

```bash
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your-email@gmail.com
SMTP_PASSWORD=your-app-password  # NOT your regular password!
FROM_EMAIL=pote-reports@gmail.com
REPORT_RECIPIENTS=user1@example.com,user2@example.com
```

**Important:** For Gmail, you must use an [App Password](https://support.google.com/accounts/answer/185833), not your regular password:
1. Enable 2-factor authentication on your Google account
2. Go to https://myaccount.google.com/apppasswords
3. Generate an app password for "Mail"
4. Use that 16-character password in `.env`

#### Option 2: SendGrid

```bash
SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USER=apikey
SMTP_PASSWORD=your-sendgrid-api-key
FROM_EMAIL=noreply@yourdomain.com
REPORT_RECIPIENTS=user1@example.com,user2@example.com
```

#### Option 3: Custom SMTP Server

```bash
SMTP_HOST=mail.yourdomain.com
SMTP_PORT=587  # or 465 for SSL
SMTP_USER=your-username
SMTP_PASSWORD=your-password
FROM_EMAIL=pote@yourdomain.com
REPORT_RECIPIENTS=admin@yourdomain.com
```

### Test SMTP Connection

Before setting up automation, test your SMTP settings:

```bash
python scripts/send_daily_report.py --to your-email@example.com --test-smtp
```

### Manual Report Generation

#### Send Daily Report

```bash
python scripts/send_daily_report.py --to user@example.com
```

Options:
- `--to EMAIL` - Recipient(s), comma-separated
- `--date YYYY-MM-DD` - Report date (default: today)
- `--test-smtp` - Test SMTP connection first
- `--save-to-file PATH` - Also save report to file

#### Send Weekly Report

```bash
python scripts/send_weekly_report.py --to user@example.com
```

---

## Automated Daily/Weekly Runs

POTE includes shell scripts for automated execution:

### Daily Run Script

`scripts/automated_daily_run.sh` performs:

1. Fetch congressional trades
2. Enrich securities (company names, sectors)
3. Fetch latest price data
4. Run market monitoring
5. Analyze disclosure timing
6. Send daily report via email

### Weekly Run Script

`scripts/automated_weekly_run.sh` performs:

1. Generate pattern detection report
2. Send weekly summary via email

### Manual Execution

Test the scripts manually before setting up cron:

```bash
# Daily run
./scripts/automated_daily_run.sh

# Weekly run
./scripts/automated_weekly_run.sh
```

Check logs:
```bash
tail -f ~/logs/daily_run.log
tail -f ~/logs/weekly_run.log
```

---

## Cron Setup

### Automated Setup (Recommended)

Use the interactive setup script:

```bash
cd /path/to/pote
./scripts/setup_cron.sh
```

This will:
1. Prompt for your email address
2. Ask for preferred schedule
3. Configure `.env` with email settings
4. Add cron jobs
5. Backup existing crontab

### Manual Setup

If you prefer manual configuration:

1. Make scripts executable:
```bash
chmod +x scripts/automated_daily_run.sh
chmod +x scripts/automated_weekly_run.sh
```

2. Create logs directory:
```bash
mkdir -p ~/logs
```

3. Edit crontab:
```bash
crontab -e
```

4. Add these lines:
```cron
# POTE Automated Daily Run (6 AM daily)
0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1

# POTE Automated Weekly Run (Sunday 8 AM)
0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1
```

### Verify Cron Jobs

```bash
crontab -l
```

### Remove Cron Jobs

```bash
crontab -e
# Delete the POTE lines and save
```

---

## Health Checks

### Run Health Check

```bash
python scripts/health_check.py
```

Output:
```
============================================================
POTE HEALTH CHECK
============================================================
Timestamp: 2025-12-15T10:30:00
Overall Status: ✓ OK

✓ Database Connection: Database connection successful
✓ Data Freshness: Data is fresh (2 days old)
    latest_trade_date: 2025-12-13
✓ Data Counts: Database has 1,234 trades
    officials: 45
    securities: 123
    trades: 1,234
    prices: 12,345
    market_alerts: 567
✓ Recent Alerts: 23 alerts in last 24 hours
============================================================
```

### JSON Output

For programmatic use:

```bash
python scripts/health_check.py --json
```

### Integrate with Monitoring

Add health check to cron for monitoring:

```cron
# POTE Health Check (every 6 hours)
0 */6 * * * /home/poteapp/pote/venv/bin/python /home/poteapp/pote/scripts/health_check.py >> /home/poteapp/logs/health.log 2>&1
```

---

## CI/CD Pipeline

POTE includes a GitHub Actions / Gitea Actions compatible CI/CD pipeline.

### What the Pipeline Does

On every push or pull request:

1. **Lint & Test**
   - Runs ruff, black, mypy
   - Executes full test suite with coverage
   - Tests against PostgreSQL

2. **Security Scan**
   - Checks dependencies with `safety`
   - Scans code with `bandit`

3. **Dependency Scan**
   - Scans for vulnerabilities with Trivy

4. **Docker Build Test**
   - Builds Docker image
   - Verifies POTE imports correctly

### GitHub Actions Setup

The pipeline is at `.github/workflows/ci.yml` and will run automatically when you push to GitHub.

### Gitea Actions Setup

Gitea Actions is compatible with GitHub Actions syntax. To enable:

1. Ensure Gitea Actions runner is installed on your server
2. Push the repository to Gitea
3. The workflow will run automatically

### Local Testing

Test the pipeline locally with Docker:

```bash
# Build image
docker build -t pote:test .

# Run tests in container
docker run --rm pote:test pytest tests/
```

### Secrets Configuration

For the CI pipeline to work fully, configure these secrets in your repository settings:

- `SONAR_HOST_URL` (optional, for SonarQube)
- `SONAR_TOKEN` (optional, for SonarQube)

---

## Deployment Workflow

### Development → Production

1. **Develop locally**
   ```bash
   git checkout -b feature/my-feature
   # Make changes
   pytest tests/
   git commit -m "Add feature"
   ```

2. **Push and test**
   ```bash
   git push origin feature/my-feature
   # CI pipeline runs automatically
   ```

3. **Merge to main**
   ```bash
   # After PR approval
   git checkout main
   git pull
   ```

4. **Deploy to Proxmox**
   ```bash
   ssh poteapp@10.0.10.95
   cd ~/pote
   git pull
   source venv/bin/activate
   pip install -e .
   alembic upgrade head
   ```

5. **Restart services**
   ```bash
   # If using systemd
   sudo systemctl restart pote
   
   # Or just restart cron jobs (they'll pick up changes)
   # No action needed
   ```

---

## Troubleshooting

### Email Not Sending

1. Check SMTP settings in `.env`
2. Test connection:
   ```bash
   python scripts/send_daily_report.py --to your-email@example.com --test-smtp
   ```
3. For Gmail: Ensure you're using an App Password, not regular password
4. Check firewall: Ensure port 587 (or 465) is open

### Cron Jobs Not Running

1. Check cron service is running:
   ```bash
   systemctl status cron
   ```

2. Verify cron jobs are installed:
   ```bash
   crontab -l
   ```

3. Check logs for errors:
   ```bash
   tail -f ~/logs/daily_run.log
   ```

4. Ensure scripts are executable:
   ```bash
   chmod +x scripts/automated_*.sh
   ```

5. Check paths in crontab (use absolute paths)

### No Data in Reports

1. Run health check:
   ```bash
   python scripts/health_check.py
   ```

2. Manually fetch data:
   ```bash
   python scripts/fetch_congressional_trades.py
   python scripts/enrich_securities.py
   ```

3. Check database connection in `.env`

---

## Summary

**After deployment, you have three ways to use POTE:**

1. **Manual**: SSH to server, run scripts manually
2. **Automated (Recommended)**: Set up cron jobs, receive daily/weekly email reports
3. **Programmatic**: Use the Python API directly in your scripts

**For fully automated operation:**

```bash
# One-time setup
cd /path/to/pote
./scripts/setup_cron.sh

# That's it! You'll now receive:
# - Daily reports at 6 AM (or your chosen time)
# - Weekly reports on Sundays at 8 AM
# - Reports will be sent to your configured email
```

**To access reports without email:**
- Reports are saved to `~/logs/daily_report_YYYYMMDD.txt`
- Reports are saved to `~/logs/weekly_report_YYYYMMDD.txt`
- You can SSH to the server and read them directly