POTE/docs/PR4_PLAN.md
ilia 0d8d85adc1 Add complete automation, reporting, and CI/CD system
Features Added:
==============

📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py

🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup

🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration

🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI

📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide

🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron

WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
   - New trades, market alerts, suspicious timing
   - Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)

USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh

# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com

# Check system health
python scripts/health_check.py

See AUTOMATION_QUICKSTART.md for full instructions.

93 tests passing | Full CI/CD | Email reports ready
2025-12-15 15:34:31 -05:00

6.4 KiB

PR4: Phase 2 - Analytics Foundation

Goal

Calculate abnormal returns and performance metrics for congressional trades.

What We'll Build

1. Return Calculator (src/pote/analytics/returns.py)

class ReturnCalculator:
    """Calculate returns for trades over various windows."""
    
    def calculate_trade_return(
        self,
        trade: Trade,
        window_days: int = 90
    ) -> dict:
        """
        Calculate return for a single trade.
        
        Returns:
            {
                'ticker': 'NVDA',
                'transaction_date': '2024-01-15',
                'window_days': 90,
                'entry_price': 495.00,
                'exit_price': 650.00,
                'return_pct': 31.3,
                'return_abs': 155.00
            }
        """
        pass
    
    def calculate_benchmark_return(
        self,
        start_date: date,
        end_date: date,
        benchmark: str = "SPY"  # S&P 500
    ) -> float:
        """Calculate benchmark return over period."""
        pass
    
    def calculate_abnormal_return(
        self,
        trade_return: float,
        benchmark_return: float
    ) -> float:
        """Return - Benchmark = Abnormal Return (alpha)."""
        return trade_return - benchmark_return

2. Performance Metrics (src/pote/analytics/metrics.py)

class PerformanceMetrics:
    """Aggregate performance metrics by official, sector, etc."""
    
    def official_performance(
        self,
        official_id: int,
        window_days: int = 90
    ) -> dict:
        """
        Aggregate stats for an official.
        
        Returns:
            {
                'name': 'Nancy Pelosi',
                'total_trades': 50,
                'buy_trades': 35,
                'sell_trades': 15,
                'avg_return': 12.5,
                'avg_abnormal_return': 5.2,
                'win_rate': 0.68,
                'total_value': 2500000,
                'best_trade': {'ticker': 'NVDA', 'return': 85.3},
                'worst_trade': {'ticker': 'META', 'return': -15.2}
            }
        """
        pass
    
    def sector_analysis(self, window_days: int = 90) -> list:
        """Performance by sector (Tech, Healthcare, etc.)."""
        pass
    
    def timing_analysis(self) -> dict:
        """Analyze disclosure lag vs performance."""
        pass

3. Benchmark Comparisons (src/pote/analytics/benchmarks.py)

class BenchmarkComparison:
    """Compare official performance vs market indices."""
    
    BENCHMARKS = {
        'SPY': 'S&P 500',
        'QQQ': 'NASDAQ-100',
        'DIA': 'Dow Jones',
        'IWM': 'Russell 2000'
    }
    
    def compare_to_market(
        self,
        official_id: int,
        benchmark: str = 'SPY',
        period_start: date = None
    ) -> dict:
        """
        Compare official's returns to market.
        
        Returns:
            {
                'official_return': 15.2,
                'benchmark_return': 8.5,
                'alpha': 6.7,
                'sharpe_ratio': 1.35,
                'win_rate_vs_market': 0.72
            }
        """
        pass

4. Database Schema Updates

Add metrics_performance table:

CREATE TABLE metrics_performance (
    id SERIAL PRIMARY KEY,
    official_id INTEGER REFERENCES officials(id),
    security_id INTEGER REFERENCES securities(id),
    trade_id INTEGER REFERENCES trades(id),
    
    -- Return metrics
    window_days INTEGER NOT NULL,
    entry_price DECIMAL(15, 2),
    exit_price DECIMAL(15, 2),
    return_pct DECIMAL(10, 4),
    return_abs DECIMAL(15, 2),
    
    -- Benchmark comparison
    benchmark_ticker VARCHAR(10),
    benchmark_return_pct DECIMAL(10, 4),
    abnormal_return_pct DECIMAL(10, 4),  -- alpha
    
    -- Calculated at
    calculated_at TIMESTAMP,
    
    INDEX(official_id, window_days),
    INDEX(security_id, window_days),
    INDEX(trade_id)
);

Implementation Steps

  1. Create analytics module structure

    src/pote/analytics/
    ├── __init__.py
    ├── returns.py       # Return calculations
    ├── metrics.py       # Aggregate metrics
    ├── benchmarks.py    # Benchmark comparisons
    └── utils.py         # Helper functions
    
  2. Add database migration

    alembic revision -m "add_performance_metrics_table"
    
  3. Implement return calculator

    • Fetch prices from database
    • Calculate returns for various windows (30, 60, 90, 180 days)
    • Handle edge cases (IPOs, delisting, missing data)
  4. Implement benchmark comparisons

    • Fetch benchmark data (SPY, QQQ, etc.)
    • Calculate abnormal returns
    • Statistical significance tests
  5. Create calculation scripts

    scripts/calculate_returns.py      # Calculate all returns
    scripts/update_metrics.py         # Update performance table
    scripts/analyze_official.py       # Analyze specific official
    
  6. Add tests

    • Unit tests for calculators
    • Integration tests with sample data
    • Edge case handling

Example Usage

# Calculate returns for all trades
from pote.analytics.returns import ReturnCalculator
from pote.db import get_session

calculator = ReturnCalculator()

with next(get_session()) as session:
    trades = session.query(Trade).all()
    
    for trade in trades:
        for window in [30, 60, 90]:
            result = calculator.calculate_trade_return(trade, window)
            print(f"{trade.official.name} {trade.security.ticker}: "
                  f"{result['return_pct']:.1f}% ({window}d)")
# Get official performance summary
from pote.analytics.metrics import PerformanceMetrics

metrics = PerformanceMetrics()
pelosi_stats = metrics.official_performance(official_id=1, window_days=90)

print(f"Average Return: {pelosi_stats['avg_return']:.1f}%")
print(f"Alpha: {pelosi_stats['avg_abnormal_return']:.1f}%")
print(f"Win Rate: {pelosi_stats['win_rate']:.1%}")

Success Criteria

  • Can calculate returns for any trade + window
  • Can compare to S&P 500 benchmark
  • Can generate official performance summaries
  • All calculations tested and accurate
  • Performance data stored efficiently
  • Documentation complete

Timeline

  • Implementation: 2-3 hours
  • Testing: 1 hour
  • Documentation: 30 minutes
  • Total: ~4 hours

Next Steps After PR4

PR5: Clustering & Behavioral Analysis PR6: Research Signals (follow_research, avoid_risk, watch) PR7: API & Dashboard