POTE/docs/12_automation_and_reporting.md

Automation and Reporting Guide

This guide covers automated data updates, email reporting, and CI/CD pipelines for POTE.

Table of Contents

1. Email Reporting Setup
2. Automated Daily/Weekly Runs
3. Cron Setup
4. Health Checks
5. CI/CD Pipeline

---

Email Reporting Setup

Configure SMTP Settings

POTE can send automated email reports. You need to configure SMTP settings in your .env file.

Option 1: Gmail

SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your-email@gmail.com
SMTP_PASSWORD=your-app-password  # NOT your regular password!
FROM_EMAIL=pote-reports@gmail.com
REPORT_RECIPIENTS=user1@example.com,user2@example.com

Important: For Gmail, you must use an App Password, not your regular password:

1. Enable 2-factor authentication on your Google account
2. Go to https://myaccount.google.com/apppasswords
3. Generate an app password for "Mail"
4. Use that 16-character password in .env

Option 2: SendGrid

SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USER=apikey
SMTP_PASSWORD=your-sendgrid-api-key
FROM_EMAIL=noreply@yourdomain.com
REPORT_RECIPIENTS=user1@example.com,user2@example.com

Option 3: Custom SMTP Server

SMTP_HOST=mail.yourdomain.com
SMTP_PORT=587  # or 465 for SSL
SMTP_USER=your-username
SMTP_PASSWORD=your-password
FROM_EMAIL=pote@yourdomain.com
REPORT_RECIPIENTS=admin@yourdomain.com

Test SMTP Connection

Before setting up automation, test your SMTP settings:

python scripts/send_daily_report.py --to your-email@example.com --test-smtp

Manual Report Generation

Send Daily Report

python scripts/send_daily_report.py --to user@example.com

Options:

• --to EMAIL - Recipient(s), comma-separated
• --date YYYY-MM-DD - Report date (default: today)
• --test-smtp - Test SMTP connection first
• --save-to-file PATH - Also save report to file

Send Weekly Report

python scripts/send_weekly_report.py --to user@example.com

---

Automated Daily/Weekly Runs

POTE includes shell scripts for automated execution:

Daily Run Script

scripts/automated_daily_run.sh performs:

1. Fetch congressional trades
2. Enrich securities (company names, sectors)
3. Fetch latest price data
4. Run market monitoring
5. Analyze disclosure timing
6. Send daily report via email

Weekly Run Script

scripts/automated_weekly_run.sh performs:

1. Generate pattern detection report
2. Send weekly summary via email

Manual Execution

Test the scripts manually before setting up cron:

# Daily run
./scripts/automated_daily_run.sh

# Weekly run
./scripts/automated_weekly_run.sh

Check logs:

tail -f ~/logs/daily_run.log
tail -f ~/logs/weekly_run.log

---

Cron Setup

Automated Setup (Recommended)

Use the interactive setup script:

cd /path/to/pote
./scripts/setup_cron.sh

This will:

1. Prompt for your email address
2. Ask for preferred schedule
3. Configure .env with email settings
4. Add cron jobs
5. Backup existing crontab

Manual Setup

If you prefer manual configuration:

1. Make scripts executable:

chmod +x scripts/automated_daily_run.sh
chmod +x scripts/automated_weekly_run.sh

2. Create logs directory:

mkdir -p ~/logs

3. Edit crontab:

crontab -e

4. Add these lines:

# POTE Automated Daily Run (6 AM daily)
0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1

# POTE Automated Weekly Run (Sunday 8 AM)
0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1

Verify Cron Jobs

crontab -l

Remove Cron Jobs

crontab -e
# Delete the POTE lines and save

---

Health Checks

Run Health Check

python scripts/health_check.py

Output:

============================================================
POTE HEALTH CHECK
============================================================
Timestamp: 2025-12-15T10:30:00
Overall Status: ✓ OK

✓ Database Connection: Database connection successful
✓ Data Freshness: Data is fresh (2 days old)
    latest_trade_date: 2025-12-13
✓ Data Counts: Database has 1,234 trades
    officials: 45
    securities: 123
    trades: 1,234
    prices: 12,345
    market_alerts: 567
✓ Recent Alerts: 23 alerts in last 24 hours
============================================================

JSON Output

For programmatic use:

python scripts/health_check.py --json

Integrate with Monitoring

Add health check to cron for monitoring:

# POTE Health Check (every 6 hours)
0 */6 * * * /home/poteapp/pote/venv/bin/python /home/poteapp/pote/scripts/health_check.py >> /home/poteapp/logs/health.log 2>&1

---

CI/CD Pipeline

POTE includes a GitHub Actions / Gitea Actions compatible CI/CD pipeline.

What the Pipeline Does

On every push or pull request:

1. Lint & Test

   • Runs ruff, black, mypy
   • Executes full test suite with coverage
   • Tests against PostgreSQL

2. Security Scan

   • Checks dependencies with safety
   • Scans code with bandit

3. Dependency Scan

   • Scans for vulnerabilities with Trivy

4. Docker Build Test

   • Builds Docker image
   • Verifies POTE imports correctly

GitHub Actions Setup

The pipeline is at .github/workflows/ci.yml and will run automatically when you push to GitHub.

Gitea Actions Setup

Gitea Actions is compatible with GitHub Actions syntax. To enable:

1. Ensure Gitea Actions runner is installed on your server
2. Push the repository to Gitea
3. The workflow will run automatically

Local Testing

Test the pipeline locally with Docker:

# Build image
docker build -t pote:test .

# Run tests in container
docker run --rm pote:test pytest tests/

Secrets Configuration

For the CI pipeline to work fully, configure these secrets in your repository settings:

• SONAR_HOST_URL (optional, for SonarQube)
• SONAR_TOKEN (optional, for SonarQube)

---

Deployment Workflow

Development → Production

1. Develop locally

   git checkout -b feature/my-feature
   # Make changes
   pytest tests/
   git commit -m "Add feature"
   

2. Push and test

   git push origin feature/my-feature
   # CI pipeline runs automatically
   

3. Merge to main

   # After PR approval
   git checkout main
   git pull
   

4. Deploy to Proxmox

   ssh poteapp@10.0.10.95
   cd ~/pote
   git pull
   source venv/bin/activate
   pip install -e .
   alembic upgrade head
   

5. Restart services

   # If using systemd
   sudo systemctl restart pote
   
   # Or just restart cron jobs (they'll pick up changes)
   # No action needed
   

---

Troubleshooting

Email Not Sending

1. Check SMTP settings in .env
2. Test connection:

   python scripts/send_daily_report.py --to your-email@example.com --test-smtp
   

3. For Gmail: Ensure you're using an App Password, not regular password
4. Check firewall: Ensure port 587 (or 465) is open

Cron Jobs Not Running

1. Check cron service is running:

   systemctl status cron
   

2. Verify cron jobs are installed:

   crontab -l
   

3. Check logs for errors:

   tail -f ~/logs/daily_run.log
   

4. Ensure scripts are executable:

   chmod +x scripts/automated_*.sh
   

5. Check paths in crontab (use absolute paths)

No Data in Reports

1. Run health check:

   python scripts/health_check.py
   

2. Manually fetch data:

   python scripts/fetch_congressional_trades.py
   python scripts/enrich_securities.py
   

3. Check database connection in .env

---

Summary

After deployment, you have three ways to use POTE:

1. Manual: SSH to server, run scripts manually
2. Automated (Recommended): Set up cron jobs, receive daily/weekly email reports
3. Programmatic: Use the Python API directly in your scripts

For fully automated operation:

# One-time setup
cd /path/to/pote
./scripts/setup_cron.sh

# That's it! You'll now receive:
# - Daily reports at 6 AM (or your chosen time)
# - Weekly reports on Sundays at 8 AM
# - Reports will be sent to your configured email

To access reports without email:

• Reports are saved to ~/logs/daily_report_YYYYMMDD.txt
• Reports are saved to ~/logs/weekly_report_YYYYMMDD.txt
• You can SSH to the server and read them directly