POTE/docs/PR4_PLAN.md
ilia 0d8d85adc1 Add complete automation, reporting, and CI/CD system
Features Added:
==============

📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py

🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup

🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration

🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI

📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide

🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron

WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
   - New trades, market alerts, suspicious timing
   - Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)

USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh

# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com

# Check system health
python scripts/health_check.py

See AUTOMATION_QUICKSTART.md for full instructions.

93 tests passing | Full CI/CD | Email reports ready
2025-12-15 15:34:31 -05:00

247 lines
6.4 KiB
Markdown

# PR4: Phase 2 - Analytics Foundation
## Goal
Calculate abnormal returns and performance metrics for congressional trades.
## What We'll Build
### 1. Return Calculator (`src/pote/analytics/returns.py`)
```python
class ReturnCalculator:
"""Calculate returns for trades over various windows."""
def calculate_trade_return(
self,
trade: Trade,
window_days: int = 90
) -> dict:
"""
Calculate return for a single trade.
Returns:
{
'ticker': 'NVDA',
'transaction_date': '2024-01-15',
'window_days': 90,
'entry_price': 495.00,
'exit_price': 650.00,
'return_pct': 31.3,
'return_abs': 155.00
}
"""
pass
def calculate_benchmark_return(
self,
start_date: date,
end_date: date,
benchmark: str = "SPY" # S&P 500
) -> float:
"""Calculate benchmark return over period."""
pass
def calculate_abnormal_return(
self,
trade_return: float,
benchmark_return: float
) -> float:
"""Return - Benchmark = Abnormal Return (alpha)."""
return trade_return - benchmark_return
```
### 2. Performance Metrics (`src/pote/analytics/metrics.py`)
```python
class PerformanceMetrics:
"""Aggregate performance metrics by official, sector, etc."""
def official_performance(
self,
official_id: int,
window_days: int = 90
) -> dict:
"""
Aggregate stats for an official.
Returns:
{
'name': 'Nancy Pelosi',
'total_trades': 50,
'buy_trades': 35,
'sell_trades': 15,
'avg_return': 12.5,
'avg_abnormal_return': 5.2,
'win_rate': 0.68,
'total_value': 2500000,
'best_trade': {'ticker': 'NVDA', 'return': 85.3},
'worst_trade': {'ticker': 'META', 'return': -15.2}
}
"""
pass
def sector_analysis(self, window_days: int = 90) -> list:
"""Performance by sector (Tech, Healthcare, etc.)."""
pass
def timing_analysis(self) -> dict:
"""Analyze disclosure lag vs performance."""
pass
```
### 3. Benchmark Comparisons (`src/pote/analytics/benchmarks.py`)
```python
class BenchmarkComparison:
"""Compare official performance vs market indices."""
BENCHMARKS = {
'SPY': 'S&P 500',
'QQQ': 'NASDAQ-100',
'DIA': 'Dow Jones',
'IWM': 'Russell 2000'
}
def compare_to_market(
self,
official_id: int,
benchmark: str = 'SPY',
period_start: date = None
) -> dict:
"""
Compare official's returns to market.
Returns:
{
'official_return': 15.2,
'benchmark_return': 8.5,
'alpha': 6.7,
'sharpe_ratio': 1.35,
'win_rate_vs_market': 0.72
}
"""
pass
```
### 4. Database Schema Updates
Add `metrics_performance` table:
```sql
CREATE TABLE metrics_performance (
id SERIAL PRIMARY KEY,
official_id INTEGER REFERENCES officials(id),
security_id INTEGER REFERENCES securities(id),
trade_id INTEGER REFERENCES trades(id),
-- Return metrics
window_days INTEGER NOT NULL,
entry_price DECIMAL(15, 2),
exit_price DECIMAL(15, 2),
return_pct DECIMAL(10, 4),
return_abs DECIMAL(15, 2),
-- Benchmark comparison
benchmark_ticker VARCHAR(10),
benchmark_return_pct DECIMAL(10, 4),
abnormal_return_pct DECIMAL(10, 4), -- alpha
-- Calculated at
calculated_at TIMESTAMP,
INDEX(official_id, window_days),
INDEX(security_id, window_days),
INDEX(trade_id)
);
```
## Implementation Steps
1. **Create analytics module structure**
```
src/pote/analytics/
├── __init__.py
├── returns.py # Return calculations
├── metrics.py # Aggregate metrics
├── benchmarks.py # Benchmark comparisons
└── utils.py # Helper functions
```
2. **Add database migration**
```bash
alembic revision -m "add_performance_metrics_table"
```
3. **Implement return calculator**
- Fetch prices from database
- Calculate returns for various windows (30, 60, 90, 180 days)
- Handle edge cases (IPOs, delisting, missing data)
4. **Implement benchmark comparisons**
- Fetch benchmark data (SPY, QQQ, etc.)
- Calculate abnormal returns
- Statistical significance tests
5. **Create calculation scripts**
```bash
scripts/calculate_returns.py # Calculate all returns
scripts/update_metrics.py # Update performance table
scripts/analyze_official.py # Analyze specific official
```
6. **Add tests**
- Unit tests for calculators
- Integration tests with sample data
- Edge case handling
## Example Usage
```python
# Calculate returns for all trades
from pote.analytics.returns import ReturnCalculator
from pote.db import get_session
calculator = ReturnCalculator()
with next(get_session()) as session:
trades = session.query(Trade).all()
for trade in trades:
for window in [30, 60, 90]:
result = calculator.calculate_trade_return(trade, window)
print(f"{trade.official.name} {trade.security.ticker}: "
f"{result['return_pct']:.1f}% ({window}d)")
```
```python
# Get official performance summary
from pote.analytics.metrics import PerformanceMetrics
metrics = PerformanceMetrics()
pelosi_stats = metrics.official_performance(official_id=1, window_days=90)
print(f"Average Return: {pelosi_stats['avg_return']:.1f}%")
print(f"Alpha: {pelosi_stats['avg_abnormal_return']:.1f}%")
print(f"Win Rate: {pelosi_stats['win_rate']:.1%}")
```
## Success Criteria
- ✅ Can calculate returns for any trade + window
- ✅ Can compare to S&P 500 benchmark
- ✅ Can generate official performance summaries
- ✅ All calculations tested and accurate
- ✅ Performance data stored efficiently
- ✅ Documentation complete
## Timeline
- Implementation: 2-3 hours
- Testing: 1 hour
- Documentation: 30 minutes
- **Total: ~4 hours**
## Next Steps After PR4
**PR5**: Clustering & Behavioral Analysis
**PR6**: Research Signals (follow_research, avoid_risk, watch)
**PR7**: API & Dashboard