Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready
249 lines
4.3 KiB
Markdown
249 lines
4.3 KiB
Markdown
# POTE Automation Quickstart
|
|
|
|
Get automated daily/weekly reports in 5 minutes.
|
|
|
|
## Prerequisites
|
|
|
|
- POTE deployed and working on Proxmox (or any server)
|
|
- SSH access to the server
|
|
- Email account for sending reports (Gmail recommended)
|
|
|
|
---
|
|
|
|
## Quick Setup
|
|
|
|
### Step 1: Configure Email
|
|
|
|
SSH to your POTE server:
|
|
|
|
```bash
|
|
ssh poteapp@your-proxmox-ip
|
|
cd ~/pote
|
|
```
|
|
|
|
Edit `.env` and add SMTP settings:
|
|
|
|
```bash
|
|
nano .env
|
|
```
|
|
|
|
Add these lines (for Gmail):
|
|
|
|
```env
|
|
SMTP_HOST=smtp.gmail.com
|
|
SMTP_PORT=587
|
|
SMTP_USER=your-email@gmail.com
|
|
SMTP_PASSWORD=your-app-password
|
|
FROM_EMAIL=pote-reports@gmail.com
|
|
REPORT_RECIPIENTS=your-email@example.com
|
|
```
|
|
|
|
**Gmail users:** Get an App Password at https://myaccount.google.com/apppasswords
|
|
|
|
### Step 2: Test Email
|
|
|
|
```bash
|
|
source venv/bin/activate
|
|
python scripts/send_daily_report.py --to your-email@example.com --test-smtp
|
|
```
|
|
|
|
If successful, you should receive a test email!
|
|
|
|
### Step 3: Set Up Automation
|
|
|
|
Run the interactive setup:
|
|
|
|
```bash
|
|
./scripts/setup_cron.sh
|
|
```
|
|
|
|
Follow the prompts:
|
|
1. Enter your email address
|
|
2. Choose daily report time (recommend 6 AM)
|
|
3. Confirm
|
|
|
|
That's it! 🎉
|
|
|
|
---
|
|
|
|
## What You'll Get
|
|
|
|
### Daily Reports (6 AM)
|
|
|
|
Includes:
|
|
- New congressional trades filed yesterday
|
|
- Market alerts (unusual volume, price spikes)
|
|
- Suspicious timing detections
|
|
- Critical alerts
|
|
|
|
### Weekly Reports (Sunday 8 AM)
|
|
|
|
Includes:
|
|
- Most active officials
|
|
- Most traded securities
|
|
- Repeat offenders (officials with consistent suspicious timing)
|
|
- Pattern analysis
|
|
|
|
---
|
|
|
|
## Verify Setup
|
|
|
|
Check cron jobs are installed:
|
|
|
|
```bash
|
|
crontab -l
|
|
```
|
|
|
|
You should see:
|
|
```
|
|
# POTE Automated Daily Run
|
|
0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
|
|
# POTE Automated Weekly Run
|
|
0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1
|
|
```
|
|
|
|
---
|
|
|
|
## Test Now (Don't Wait)
|
|
|
|
Run the daily script manually to test:
|
|
|
|
```bash
|
|
./scripts/automated_daily_run.sh
|
|
```
|
|
|
|
Check if email arrived! 📧
|
|
|
|
---
|
|
|
|
## View Logs
|
|
|
|
```bash
|
|
# Daily run log
|
|
tail -f ~/logs/daily_run.log
|
|
|
|
# Weekly run log
|
|
tail -f ~/logs/weekly_run.log
|
|
|
|
# Saved reports
|
|
ls -lh ~/logs/*.txt
|
|
cat ~/logs/daily_report_$(date +%Y%m%d).txt
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### No Email Received
|
|
|
|
1. Check spam folder
|
|
2. Verify SMTP settings in `.env`
|
|
3. Test connection:
|
|
```bash
|
|
python scripts/send_daily_report.py --to your-email@example.com --test-smtp
|
|
```
|
|
|
|
### Cron Not Running
|
|
|
|
1. Check logs:
|
|
```bash
|
|
tail -50 ~/logs/daily_run.log
|
|
```
|
|
|
|
2. Ensure scripts are executable:
|
|
```bash
|
|
chmod +x scripts/automated_*.sh
|
|
```
|
|
|
|
3. Test manually:
|
|
```bash
|
|
./scripts/automated_daily_run.sh
|
|
```
|
|
|
|
### Empty Reports
|
|
|
|
System needs data first. Manually fetch:
|
|
|
|
```bash
|
|
source venv/bin/activate
|
|
python scripts/fetch_congressional_trades.py
|
|
python scripts/enrich_securities.py
|
|
python scripts/monitor_market.py --scan
|
|
```
|
|
|
|
Then try sending a report again.
|
|
|
|
---
|
|
|
|
## Advanced Configuration
|
|
|
|
### Change Report Schedule
|
|
|
|
Edit crontab:
|
|
|
|
```bash
|
|
crontab -e
|
|
```
|
|
|
|
Cron syntax: `minute hour day month weekday command`
|
|
|
|
Examples:
|
|
```cron
|
|
# 9 AM daily
|
|
0 9 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
|
|
|
|
# Twice daily: 6 AM and 6 PM
|
|
0 6,18 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
|
|
|
|
# Weekdays only at 6 AM
|
|
0 6 * * 1-5 /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
|
|
```
|
|
|
|
### Multiple Recipients
|
|
|
|
In `.env`:
|
|
|
|
```env
|
|
REPORT_RECIPIENTS=user1@example.com,user2@example.com,user3@example.com
|
|
```
|
|
|
|
### Disable Email, Keep Logs
|
|
|
|
Comment out the email step in `scripts/automated_daily_run.sh`:
|
|
|
|
```bash
|
|
# python scripts/send_daily_report.py --to "$REPORT_RECIPIENTS" ...
|
|
```
|
|
|
|
Reports will still be saved to `~/logs/`
|
|
|
|
---
|
|
|
|
## System Health
|
|
|
|
Check system health anytime:
|
|
|
|
```bash
|
|
python scripts/health_check.py
|
|
```
|
|
|
|
Add to cron for regular health checks:
|
|
|
|
```cron
|
|
# Health check every 6 hours
|
|
0 */6 * * * /home/poteapp/pote/venv/bin/python /home/poteapp/pote/scripts/health_check.py >> /home/poteapp/logs/health.log 2>&1
|
|
```
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
- See full documentation: `docs/12_automation_and_reporting.md`
|
|
- Explore CI/CD pipeline: `.github/workflows/ci.yml`
|
|
- Customize reports: `src/pote/reporting/report_generator.py`
|
|
|
|
---
|
|
|
|
**You're all set! POTE will now run automatically and send you daily/weekly reports. 🚀**
|
|
|