Add complete automation, reporting, and CI/CD system

Features Added:
==============

📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py

🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup

🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration

🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI

📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide

🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron

WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
   - New trades, market alerts, suspicious timing
   - Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)

USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh

# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com

# Check system health
python scripts/health_check.py

See AUTOMATION_QUICKSTART.md for full instructions.

93 tests passing | Full CI/CD | Email reports ready
This commit is contained in:
ilia
2025-12-15 15:34:31 -05:00
parent 53d631a903
commit 0d8d85adc1
44 changed files with 2206 additions and 61 deletions
+116
View File
@@ -0,0 +1,116 @@
"""
Email Reporter for POTE
Sends formatted reports via SMTP email.
"""
import logging
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from typing import List, Optional
from pote.config import settings
logger = logging.getLogger(__name__)
class EmailReporter:
"""Sends email reports via SMTP."""
def __init__(
self,
smtp_host: Optional[str] = None,
smtp_port: Optional[int] = None,
smtp_user: Optional[str] = None,
smtp_password: Optional[str] = None,
from_email: Optional[str] = None,
):
"""
Initialize email reporter.
If parameters are not provided, will attempt to use settings from config.
"""
self.smtp_host = smtp_host or getattr(settings, "smtp_host", "localhost")
self.smtp_port = smtp_port or getattr(settings, "smtp_port", 587)
self.smtp_user = smtp_user or getattr(settings, "smtp_user", None)
self.smtp_password = smtp_password or getattr(settings, "smtp_password", None)
self.from_email = from_email or getattr(
settings, "from_email", "pote@localhost"
)
def send_report(
self,
to_emails: List[str],
subject: str,
body_text: str,
body_html: Optional[str] = None,
) -> bool:
"""
Send an email report.
Args:
to_emails: List of recipient email addresses
subject: Email subject line
body_text: Plain text email body
body_html: Optional HTML email body
Returns:
True if email sent successfully, False otherwise
"""
try:
msg = MIMEMultipart("alternative")
msg["Subject"] = subject
msg["From"] = self.from_email
msg["To"] = ", ".join(to_emails)
# Attach plain text part
msg.attach(MIMEText(body_text, "plain"))
# Attach HTML part if provided
if body_html:
msg.attach(MIMEText(body_html, "html"))
# Connect to SMTP server and send
with smtplib.SMTP(self.smtp_host, self.smtp_port) as server:
server.ehlo()
if self.smtp_port == 587: # TLS
server.starttls()
server.ehlo()
if self.smtp_user and self.smtp_password:
server.login(self.smtp_user, self.smtp_password)
server.send_message(msg)
logger.info(f"Email sent successfully to {', '.join(to_emails)}")
return True
except Exception as e:
logger.error(f"Failed to send email: {e}")
return False
def test_connection(self) -> bool:
"""
Test SMTP connection.
Returns:
True if connection successful, False otherwise
"""
try:
with smtplib.SMTP(self.smtp_host, self.smtp_port, timeout=10) as server:
server.ehlo()
if self.smtp_port == 587:
server.starttls()
server.ehlo()
if self.smtp_user and self.smtp_password:
server.login(self.smtp_user, self.smtp_password)
logger.info("SMTP connection test successful")
return True
except Exception as e:
logger.error(f"SMTP connection test failed: {e}")
return False