Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready
247 lines
6.4 KiB
Markdown
247 lines
6.4 KiB
Markdown
# PR4: Phase 2 - Analytics Foundation
|
|
|
|
## Goal
|
|
Calculate abnormal returns and performance metrics for congressional trades.
|
|
|
|
## What We'll Build
|
|
|
|
### 1. Return Calculator (`src/pote/analytics/returns.py`)
|
|
```python
|
|
class ReturnCalculator:
|
|
"""Calculate returns for trades over various windows."""
|
|
|
|
def calculate_trade_return(
|
|
self,
|
|
trade: Trade,
|
|
window_days: int = 90
|
|
) -> dict:
|
|
"""
|
|
Calculate return for a single trade.
|
|
|
|
Returns:
|
|
{
|
|
'ticker': 'NVDA',
|
|
'transaction_date': '2024-01-15',
|
|
'window_days': 90,
|
|
'entry_price': 495.00,
|
|
'exit_price': 650.00,
|
|
'return_pct': 31.3,
|
|
'return_abs': 155.00
|
|
}
|
|
"""
|
|
pass
|
|
|
|
def calculate_benchmark_return(
|
|
self,
|
|
start_date: date,
|
|
end_date: date,
|
|
benchmark: str = "SPY" # S&P 500
|
|
) -> float:
|
|
"""Calculate benchmark return over period."""
|
|
pass
|
|
|
|
def calculate_abnormal_return(
|
|
self,
|
|
trade_return: float,
|
|
benchmark_return: float
|
|
) -> float:
|
|
"""Return - Benchmark = Abnormal Return (alpha)."""
|
|
return trade_return - benchmark_return
|
|
```
|
|
|
|
### 2. Performance Metrics (`src/pote/analytics/metrics.py`)
|
|
```python
|
|
class PerformanceMetrics:
|
|
"""Aggregate performance metrics by official, sector, etc."""
|
|
|
|
def official_performance(
|
|
self,
|
|
official_id: int,
|
|
window_days: int = 90
|
|
) -> dict:
|
|
"""
|
|
Aggregate stats for an official.
|
|
|
|
Returns:
|
|
{
|
|
'name': 'Nancy Pelosi',
|
|
'total_trades': 50,
|
|
'buy_trades': 35,
|
|
'sell_trades': 15,
|
|
'avg_return': 12.5,
|
|
'avg_abnormal_return': 5.2,
|
|
'win_rate': 0.68,
|
|
'total_value': 2500000,
|
|
'best_trade': {'ticker': 'NVDA', 'return': 85.3},
|
|
'worst_trade': {'ticker': 'META', 'return': -15.2}
|
|
}
|
|
"""
|
|
pass
|
|
|
|
def sector_analysis(self, window_days: int = 90) -> list:
|
|
"""Performance by sector (Tech, Healthcare, etc.)."""
|
|
pass
|
|
|
|
def timing_analysis(self) -> dict:
|
|
"""Analyze disclosure lag vs performance."""
|
|
pass
|
|
```
|
|
|
|
### 3. Benchmark Comparisons (`src/pote/analytics/benchmarks.py`)
|
|
```python
|
|
class BenchmarkComparison:
|
|
"""Compare official performance vs market indices."""
|
|
|
|
BENCHMARKS = {
|
|
'SPY': 'S&P 500',
|
|
'QQQ': 'NASDAQ-100',
|
|
'DIA': 'Dow Jones',
|
|
'IWM': 'Russell 2000'
|
|
}
|
|
|
|
def compare_to_market(
|
|
self,
|
|
official_id: int,
|
|
benchmark: str = 'SPY',
|
|
period_start: date = None
|
|
) -> dict:
|
|
"""
|
|
Compare official's returns to market.
|
|
|
|
Returns:
|
|
{
|
|
'official_return': 15.2,
|
|
'benchmark_return': 8.5,
|
|
'alpha': 6.7,
|
|
'sharpe_ratio': 1.35,
|
|
'win_rate_vs_market': 0.72
|
|
}
|
|
"""
|
|
pass
|
|
```
|
|
|
|
### 4. Database Schema Updates
|
|
|
|
Add `metrics_performance` table:
|
|
```sql
|
|
CREATE TABLE metrics_performance (
|
|
id SERIAL PRIMARY KEY,
|
|
official_id INTEGER REFERENCES officials(id),
|
|
security_id INTEGER REFERENCES securities(id),
|
|
trade_id INTEGER REFERENCES trades(id),
|
|
|
|
-- Return metrics
|
|
window_days INTEGER NOT NULL,
|
|
entry_price DECIMAL(15, 2),
|
|
exit_price DECIMAL(15, 2),
|
|
return_pct DECIMAL(10, 4),
|
|
return_abs DECIMAL(15, 2),
|
|
|
|
-- Benchmark comparison
|
|
benchmark_ticker VARCHAR(10),
|
|
benchmark_return_pct DECIMAL(10, 4),
|
|
abnormal_return_pct DECIMAL(10, 4), -- alpha
|
|
|
|
-- Calculated at
|
|
calculated_at TIMESTAMP,
|
|
|
|
INDEX(official_id, window_days),
|
|
INDEX(security_id, window_days),
|
|
INDEX(trade_id)
|
|
);
|
|
```
|
|
|
|
## Implementation Steps
|
|
|
|
1. **Create analytics module structure**
|
|
```
|
|
src/pote/analytics/
|
|
├── __init__.py
|
|
├── returns.py # Return calculations
|
|
├── metrics.py # Aggregate metrics
|
|
├── benchmarks.py # Benchmark comparisons
|
|
└── utils.py # Helper functions
|
|
```
|
|
|
|
2. **Add database migration**
|
|
```bash
|
|
alembic revision -m "add_performance_metrics_table"
|
|
```
|
|
|
|
3. **Implement return calculator**
|
|
- Fetch prices from database
|
|
- Calculate returns for various windows (30, 60, 90, 180 days)
|
|
- Handle edge cases (IPOs, delisting, missing data)
|
|
|
|
4. **Implement benchmark comparisons**
|
|
- Fetch benchmark data (SPY, QQQ, etc.)
|
|
- Calculate abnormal returns
|
|
- Statistical significance tests
|
|
|
|
5. **Create calculation scripts**
|
|
```bash
|
|
scripts/calculate_returns.py # Calculate all returns
|
|
scripts/update_metrics.py # Update performance table
|
|
scripts/analyze_official.py # Analyze specific official
|
|
```
|
|
|
|
6. **Add tests**
|
|
- Unit tests for calculators
|
|
- Integration tests with sample data
|
|
- Edge case handling
|
|
|
|
## Example Usage
|
|
|
|
```python
|
|
# Calculate returns for all trades
|
|
from pote.analytics.returns import ReturnCalculator
|
|
from pote.db import get_session
|
|
|
|
calculator = ReturnCalculator()
|
|
|
|
with next(get_session()) as session:
|
|
trades = session.query(Trade).all()
|
|
|
|
for trade in trades:
|
|
for window in [30, 60, 90]:
|
|
result = calculator.calculate_trade_return(trade, window)
|
|
print(f"{trade.official.name} {trade.security.ticker}: "
|
|
f"{result['return_pct']:.1f}% ({window}d)")
|
|
```
|
|
|
|
```python
|
|
# Get official performance summary
|
|
from pote.analytics.metrics import PerformanceMetrics
|
|
|
|
metrics = PerformanceMetrics()
|
|
pelosi_stats = metrics.official_performance(official_id=1, window_days=90)
|
|
|
|
print(f"Average Return: {pelosi_stats['avg_return']:.1f}%")
|
|
print(f"Alpha: {pelosi_stats['avg_abnormal_return']:.1f}%")
|
|
print(f"Win Rate: {pelosi_stats['win_rate']:.1%}")
|
|
```
|
|
|
|
## Success Criteria
|
|
|
|
- ✅ Can calculate returns for any trade + window
|
|
- ✅ Can compare to S&P 500 benchmark
|
|
- ✅ Can generate official performance summaries
|
|
- ✅ All calculations tested and accurate
|
|
- ✅ Performance data stored efficiently
|
|
- ✅ Documentation complete
|
|
|
|
## Timeline
|
|
|
|
- Implementation: 2-3 hours
|
|
- Testing: 1 hour
|
|
- Documentation: 30 minutes
|
|
- **Total: ~4 hours**
|
|
|
|
## Next Steps After PR4
|
|
|
|
**PR5**: Clustering & Behavioral Analysis
|
|
**PR6**: Research Signals (follow_research, avoid_risk, watch)
|
|
**PR7**: API & Dashboard
|
|
|
|
|