POTE/AUTOMATION_QUICKSTART.md
ilia 0d8d85adc1 Add complete automation, reporting, and CI/CD system
Features Added:
==============

📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py

🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup

🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration

🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI

📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide

🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron

WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
   - New trades, market alerts, suspicious timing
   - Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)

USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh

# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com

# Check system health
python scripts/health_check.py

See AUTOMATION_QUICKSTART.md for full instructions.

93 tests passing | Full CI/CD | Email reports ready
2025-12-15 15:34:31 -05:00

4.3 KiB

POTE Automation Quickstart

Get automated daily/weekly reports in 5 minutes.

Prerequisites

  • POTE deployed and working on Proxmox (or any server)
  • SSH access to the server
  • Email account for sending reports (Gmail recommended)

Quick Setup

Step 1: Configure Email

SSH to your POTE server:

ssh poteapp@your-proxmox-ip
cd ~/pote

Edit .env and add SMTP settings:

nano .env

Add these lines (for Gmail):

SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your-email@gmail.com
SMTP_PASSWORD=your-app-password
FROM_EMAIL=pote-reports@gmail.com
REPORT_RECIPIENTS=your-email@example.com

Gmail users: Get an App Password at https://myaccount.google.com/apppasswords

Step 2: Test Email

source venv/bin/activate
python scripts/send_daily_report.py --to your-email@example.com --test-smtp

If successful, you should receive a test email!

Step 3: Set Up Automation

Run the interactive setup:

./scripts/setup_cron.sh

Follow the prompts:

  1. Enter your email address
  2. Choose daily report time (recommend 6 AM)
  3. Confirm

That's it! 🎉


What You'll Get

Daily Reports (6 AM)

Includes:

  • New congressional trades filed yesterday
  • Market alerts (unusual volume, price spikes)
  • Suspicious timing detections
  • Critical alerts

Weekly Reports (Sunday 8 AM)

Includes:

  • Most active officials
  • Most traded securities
  • Repeat offenders (officials with consistent suspicious timing)
  • Pattern analysis

Verify Setup

Check cron jobs are installed:

crontab -l

You should see:

# POTE Automated Daily Run
0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
# POTE Automated Weekly Run
0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1

Test Now (Don't Wait)

Run the daily script manually to test:

./scripts/automated_daily_run.sh

Check if email arrived! 📧


View Logs

# Daily run log
tail -f ~/logs/daily_run.log

# Weekly run log
tail -f ~/logs/weekly_run.log

# Saved reports
ls -lh ~/logs/*.txt
cat ~/logs/daily_report_$(date +%Y%m%d).txt

Troubleshooting

No Email Received

  1. Check spam folder
  2. Verify SMTP settings in .env
  3. Test connection:
    python scripts/send_daily_report.py --to your-email@example.com --test-smtp
    

Cron Not Running

  1. Check logs:

    tail -50 ~/logs/daily_run.log
    
  2. Ensure scripts are executable:

    chmod +x scripts/automated_*.sh
    
  3. Test manually:

    ./scripts/automated_daily_run.sh
    

Empty Reports

System needs data first. Manually fetch:

source venv/bin/activate
python scripts/fetch_congressional_trades.py
python scripts/enrich_securities.py
python scripts/monitor_market.py --scan

Then try sending a report again.


Advanced Configuration

Change Report Schedule

Edit crontab:

crontab -e

Cron syntax: minute hour day month weekday command

Examples:

# 9 AM daily
0 9 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1

# Twice daily: 6 AM and 6 PM
0 6,18 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1

# Weekdays only at 6 AM
0 6 * * 1-5 /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1

Multiple Recipients

In .env:

REPORT_RECIPIENTS=user1@example.com,user2@example.com,user3@example.com

Disable Email, Keep Logs

Comment out the email step in scripts/automated_daily_run.sh:

# python scripts/send_daily_report.py --to "$REPORT_RECIPIENTS" ...

Reports will still be saved to ~/logs/


System Health

Check system health anytime:

python scripts/health_check.py

Add to cron for regular health checks:

# Health check every 6 hours
0 */6 * * * /home/poteapp/pote/venv/bin/python /home/poteapp/pote/scripts/health_check.py >> /home/poteapp/logs/health.log 2>&1

Next Steps

  • See full documentation: docs/12_automation_and_reporting.md
  • Explore CI/CD pipeline: .github/workflows/ci.yml
  • Customize reports: src/pote/reporting/report_generator.py

You're all set! POTE will now run automatically and send you daily/weekly reports. 🚀