Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready
4.3 KiB
POTE Automation Quickstart
Get automated daily/weekly reports in 5 minutes.
Prerequisites
- POTE deployed and working on Proxmox (or any server)
- SSH access to the server
- Email account for sending reports (Gmail recommended)
Quick Setup
Step 1: Configure Email
SSH to your POTE server:
ssh poteapp@your-proxmox-ip
cd ~/pote
Edit .env and add SMTP settings:
nano .env
Add these lines (for Gmail):
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your-email@gmail.com
SMTP_PASSWORD=your-app-password
FROM_EMAIL=pote-reports@gmail.com
REPORT_RECIPIENTS=your-email@example.com
Gmail users: Get an App Password at https://myaccount.google.com/apppasswords
Step 2: Test Email
source venv/bin/activate
python scripts/send_daily_report.py --to your-email@example.com --test-smtp
If successful, you should receive a test email!
Step 3: Set Up Automation
Run the interactive setup:
./scripts/setup_cron.sh
Follow the prompts:
- Enter your email address
- Choose daily report time (recommend 6 AM)
- Confirm
That's it! 🎉
What You'll Get
Daily Reports (6 AM)
Includes:
- New congressional trades filed yesterday
- Market alerts (unusual volume, price spikes)
- Suspicious timing detections
- Critical alerts
Weekly Reports (Sunday 8 AM)
Includes:
- Most active officials
- Most traded securities
- Repeat offenders (officials with consistent suspicious timing)
- Pattern analysis
Verify Setup
Check cron jobs are installed:
crontab -l
You should see:
# POTE Automated Daily Run
0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
# POTE Automated Weekly Run
0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1
Test Now (Don't Wait)
Run the daily script manually to test:
./scripts/automated_daily_run.sh
Check if email arrived! 📧
View Logs
# Daily run log
tail -f ~/logs/daily_run.log
# Weekly run log
tail -f ~/logs/weekly_run.log
# Saved reports
ls -lh ~/logs/*.txt
cat ~/logs/daily_report_$(date +%Y%m%d).txt
Troubleshooting
No Email Received
- Check spam folder
- Verify SMTP settings in
.env - Test connection:
python scripts/send_daily_report.py --to your-email@example.com --test-smtp
Cron Not Running
-
Check logs:
tail -50 ~/logs/daily_run.log -
Ensure scripts are executable:
chmod +x scripts/automated_*.sh -
Test manually:
./scripts/automated_daily_run.sh
Empty Reports
System needs data first. Manually fetch:
source venv/bin/activate
python scripts/fetch_congressional_trades.py
python scripts/enrich_securities.py
python scripts/monitor_market.py --scan
Then try sending a report again.
Advanced Configuration
Change Report Schedule
Edit crontab:
crontab -e
Cron syntax: minute hour day month weekday command
Examples:
# 9 AM daily
0 9 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
# Twice daily: 6 AM and 6 PM
0 6,18 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
# Weekdays only at 6 AM
0 6 * * 1-5 /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
Multiple Recipients
In .env:
REPORT_RECIPIENTS=user1@example.com,user2@example.com,user3@example.com
Disable Email, Keep Logs
Comment out the email step in scripts/automated_daily_run.sh:
# python scripts/send_daily_report.py --to "$REPORT_RECIPIENTS" ...
Reports will still be saved to ~/logs/
System Health
Check system health anytime:
python scripts/health_check.py
Add to cron for regular health checks:
# Health check every 6 hours
0 */6 * * * /home/poteapp/pote/venv/bin/python /home/poteapp/pote/scripts/health_check.py >> /home/poteapp/logs/health.log 2>&1
Next Steps
- See full documentation:
docs/12_automation_and_reporting.md - Explore CI/CD pipeline:
.github/workflows/ci.yml - Customize reports:
src/pote/reporting/report_generator.py
You're all set! POTE will now run automatically and send you daily/weekly reports. 🚀