0d8d85adc1
Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready
78 lines
2.4 KiB
Bash
Executable File
78 lines
2.4 KiB
Bash
Executable File
#!/bin/bash
|
|
# Daily update script for POTE
|
|
# Run this via cron to automatically fetch new data
|
|
|
|
set -e
|
|
|
|
# Configuration
|
|
POTE_DIR="/home/poteapp/pote"
|
|
LOG_DIR="/home/poteapp/logs"
|
|
LOG_FILE="$LOG_DIR/daily_update_$(date +%Y%m%d).log"
|
|
|
|
# Ensure log directory exists
|
|
mkdir -p "$LOG_DIR"
|
|
|
|
echo "=== POTE Daily Update: $(date) ===" | tee -a "$LOG_FILE"
|
|
|
|
cd "$POTE_DIR"
|
|
source venv/bin/activate
|
|
|
|
# 1. Fetch new congressional trades (if House Stock Watcher is back up)
|
|
echo "[1/4] Fetching congressional trades..." | tee -a "$LOG_FILE"
|
|
if python scripts/fetch_congressional_trades.py --days 7 >> "$LOG_FILE" 2>&1; then
|
|
echo "✓ Trades fetched successfully" | tee -a "$LOG_FILE"
|
|
else
|
|
echo "✗ Trade fetch failed (API might be down)" | tee -a "$LOG_FILE"
|
|
fi
|
|
|
|
# 2. Enrich any new securities
|
|
echo "[2/4] Enriching securities..." | tee -a "$LOG_FILE"
|
|
if python scripts/enrich_securities.py >> "$LOG_FILE" 2>&1; then
|
|
echo "✓ Securities enriched" | tee -a "$LOG_FILE"
|
|
else
|
|
echo "✗ Security enrichment failed" | tee -a "$LOG_FILE"
|
|
fi
|
|
|
|
# 3. Update prices for all securities
|
|
echo "[3/4] Fetching price data..." | tee -a "$LOG_FILE"
|
|
if python scripts/fetch_sample_prices.py >> "$LOG_FILE" 2>&1; then
|
|
echo "✓ Prices updated" | tee -a "$LOG_FILE"
|
|
else
|
|
echo "✗ Price fetch failed" | tee -a "$LOG_FILE"
|
|
fi
|
|
|
|
# 4. Generate summary
|
|
echo "[4/4] Generating summary..." | tee -a "$LOG_FILE"
|
|
python << 'EOF' | tee -a "$LOG_FILE"
|
|
from sqlalchemy import text
|
|
from pote.db import engine
|
|
from datetime import datetime, timedelta
|
|
|
|
with engine.connect() as conn:
|
|
# Get counts
|
|
officials = conn.execute(text("SELECT COUNT(*) FROM officials")).scalar()
|
|
trades = conn.execute(text("SELECT COUNT(*) FROM trades")).scalar()
|
|
securities = conn.execute(text("SELECT COUNT(*) FROM securities")).scalar()
|
|
|
|
# Get new trades in last 7 days
|
|
week_ago = (datetime.now() - timedelta(days=7)).strftime('%Y-%m-%d')
|
|
new_trades = conn.execute(
|
|
text(f"SELECT COUNT(*) FROM trades WHERE created_at >= '{week_ago}'")
|
|
).scalar()
|
|
|
|
print(f"\n📊 Database Summary:")
|
|
print(f" Officials: {officials:,}")
|
|
print(f" Securities: {securities:,}")
|
|
print(f" Trades: {trades:,}")
|
|
print(f" New (7d): {new_trades:,}")
|
|
EOF
|
|
|
|
echo "" | tee -a "$LOG_FILE"
|
|
echo "=== Update Complete: $(date) ===" | tee -a "$LOG_FILE"
|
|
echo "" | tee -a "$LOG_FILE"
|
|
|
|
# Keep only last 30 days of logs
|
|
find "$LOG_DIR" -name "daily_update_*.log" -mtime +30 -delete
|
|
|
|
|