Humanize READMEs and guides for clearer, professional tone
Align with project-template docs/writing-docs.md: plain openings, no emoji decoration, archive scratch plans where they cluttered live docs.
This commit is contained in:
@@ -0,0 +1,425 @@
|
||||
# POTE Monitoring System - ALL PHASES COMPLETE!
|
||||
|
||||
## **What Was Built (3 Phases)**
|
||||
|
||||
### **Phase 1: Real-Time Market Monitoring**
|
||||
**Detects unusual market activity in congressional tickers**
|
||||
|
||||
**Features:**
|
||||
- Auto-detect most-traded congressional stocks (top 50)
|
||||
- Monitor for unusual volume (3x average)
|
||||
- Detect price spikes/drops (>5%)
|
||||
- Track high volatility (2x normal)
|
||||
- Log all alerts to database
|
||||
- Severity scoring (1-10 scale)
|
||||
- Generate activity reports
|
||||
|
||||
**Components:**
|
||||
- `MarketMonitor` - Core monitoring engine
|
||||
- `AlertManager` - Alert formatting & filtering
|
||||
- `MarketAlert` model - Database storage
|
||||
- `monitor_market.py` - CLI tool
|
||||
|
||||
**Tests:** 14 passing
|
||||
|
||||
---
|
||||
|
||||
### **Phase 2: Disclosure Timing Correlation**
|
||||
**Matches trades to prior market alerts when disclosures appear**
|
||||
|
||||
**Features:**
|
||||
- Find alerts before each trade (30-day lookback)
|
||||
- Calculate timing advantage scores (0-100 scale)
|
||||
- Identify suspicious timing patterns
|
||||
- Analyze individual trades
|
||||
- Batch analysis of recent disclosures
|
||||
- Official historical patterns
|
||||
- Per-ticker timing analysis
|
||||
|
||||
**Scoring Algorithm:**
|
||||
- Base: alert count × 5 + avg severity × 2
|
||||
- Recency bonus: +10 per alert within 7 days
|
||||
- Severity bonus: +15 per high-severity (7+) alert
|
||||
- **Thresholds:**
|
||||
- 80-100: Highly suspicious
|
||||
- 60-79: Suspicious
|
||||
- 40-59: Notable
|
||||
- 0-39: Normal
|
||||
|
||||
**Components:**
|
||||
- `DisclosureCorrelator` - Correlation engine
|
||||
- `analyze_disclosure_timing.py` - CLI tool
|
||||
|
||||
**Tests:** 13 passing
|
||||
|
||||
---
|
||||
|
||||
### **Phase 3: Pattern Detection & Rankings**
|
||||
**Cross-official analysis and comparative rankings**
|
||||
|
||||
**Features:**
|
||||
- Rank officials by timing scores
|
||||
- Identify repeat offenders (50%+ suspicious)
|
||||
- Analyze ticker patterns
|
||||
- Sector-level analysis
|
||||
- Party comparison (Democrat vs Republican)
|
||||
- Comprehensive pattern reports
|
||||
- Top 10 rankings
|
||||
- Statistical summaries
|
||||
|
||||
**Components:**
|
||||
- `PatternDetector` - Pattern analysis engine
|
||||
- `generate_pattern_report.py` - CLI tool
|
||||
|
||||
**Tests:** 11 passing
|
||||
|
||||
---
|
||||
|
||||
## **Complete System Architecture**
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PHASE 1: Real-Time Monitoring │
|
||||
│ ──────────────────────────────────── │
|
||||
│ Monitor congressional tickers │
|
||||
│ Detect unusual activity │
|
||||
│ Log alerts to database │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
[30-45 days pass]
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PHASE 2: Disclosure Correlation │
|
||||
│ ─────────────────────────────── │
|
||||
│ New congressional trades filed │
|
||||
│ Match to prior alerts │
|
||||
│ Calculate timing scores │
|
||||
│ Flag suspicious trades │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PHASE 3: Pattern Detection │
|
||||
│ ────────────────────────── │
|
||||
│ Rank officials by timing │
|
||||
│ Identify repeat offenders │
|
||||
│ Compare parties, sectors, tickers │
|
||||
│ Generate comprehensive reports │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Usage Guide**
|
||||
|
||||
### **1. Set Up Monitoring (Run Daily)**
|
||||
|
||||
```bash
|
||||
# Monitor congressional tickers (5-minute intervals)
|
||||
python scripts/monitor_market.py --interval 300
|
||||
|
||||
# Or run once
|
||||
python scripts/monitor_market.py --once
|
||||
|
||||
# Monitor specific tickers
|
||||
python scripts/monitor_market.py --tickers NVDA,MSFT,AAPL --once
|
||||
```
|
||||
|
||||
**Automation:**
|
||||
```bash
|
||||
# Add to cron for continuous monitoring
|
||||
crontab -e
|
||||
# Add: */5 * * * * /path/to/pote/scripts/monitor_market.py --once
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### **2. Analyze Timing When Disclosures Appear**
|
||||
|
||||
```bash
|
||||
# Find suspicious trades filed recently
|
||||
python scripts/analyze_disclosure_timing.py --days 30 --min-score 60
|
||||
|
||||
# Analyze specific official
|
||||
python scripts/analyze_disclosure_timing.py --official "Nancy Pelosi"
|
||||
|
||||
# Analyze specific ticker
|
||||
python scripts/analyze_disclosure_timing.py --ticker NVDA
|
||||
|
||||
# Save report
|
||||
python scripts/analyze_disclosure_timing.py --days 30 --output report.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### **3. Generate Pattern Reports (Monthly/Quarterly)**
|
||||
|
||||
```bash
|
||||
# Comprehensive pattern analysis
|
||||
python scripts/generate_pattern_report.py --days 365
|
||||
|
||||
# Last 90 days
|
||||
python scripts/generate_pattern_report.py --days 90
|
||||
|
||||
# Save to file
|
||||
python scripts/generate_pattern_report.py --days 365 --output patterns.txt
|
||||
|
||||
# JSON format
|
||||
python scripts/generate_pattern_report.py --days 365 --format json --output patterns.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Example Reports**
|
||||
|
||||
### **Timing Analysis Report**
|
||||
|
||||
```
|
||||
================================================================================
|
||||
SUSPICIOUS TRADING TIMING ANALYSIS
|
||||
3 Trades with Timing Advantages Detected
|
||||
================================================================================
|
||||
|
||||
#1 - HIGHLY SUSPICIOUS (Timing Score: 85/100)
|
||||
────────────────────────────────────────────────────────────────────────────────
|
||||
Official: Nancy Pelosi
|
||||
Ticker: NVDA
|
||||
Side: BUY
|
||||
Trade Date: 2024-01-15
|
||||
Value: $15,001-$50,000
|
||||
|
||||
Timing Analysis:
|
||||
Prior Alerts: 3
|
||||
Recent Alerts (7d): 2
|
||||
High Severity: 2
|
||||
Avg Severity: 7.5/10
|
||||
|
||||
Assessment: Trade occurred after 3 alerts, including 2 high-severity.
|
||||
High likelihood of timing advantage.
|
||||
|
||||
Prior Market Alerts:
|
||||
Timestamp Type Severity Timing
|
||||
2024-01-12 10:30:00 Unusual Volume 8/10 3 days before
|
||||
2024-01-13 14:15:00 Price Spike 7/10 2 days before
|
||||
2024-01-14 16:20:00 High Volatility 6/10 1 day before
|
||||
```
|
||||
|
||||
### **Pattern Analysis Report**
|
||||
|
||||
```
|
||||
================================================================================
|
||||
CONGRESSIONAL TRADING PATTERN ANALYSIS
|
||||
Period: 365 days
|
||||
================================================================================
|
||||
|
||||
SUMMARY
|
||||
────────────────────────────────────────────────────────────────────────────────
|
||||
Officials Analyzed: 45
|
||||
Repeat Offenders: 8
|
||||
Average Timing Score: 42.3/100
|
||||
|
||||
TOP 10 MOST SUSPICIOUS OFFICIALS (By Timing Score)
|
||||
================================================================================
|
||||
|
||||
Rank Official Party-State Chamber Trades Suspicious Rate Avg Score
|
||||
──── ─────────────────────── ─────────── ─────── ────── ────────── ────── ─────────
|
||||
1 Tommy Tuberville R-AL Senate 47 35/47 74.5% 72.5/100
|
||||
2 Nancy Pelosi D-CA House 38 28/38 73.7% 71.2/100
|
||||
3 Dan Crenshaw R-TX House 25 15/25 60.0% 65.8/100
|
||||
4 Marjorie Taylor Greene R-GA House 19 11/19 57.9% 63.2/100
|
||||
5 Josh Gottheimer D-NJ House 31 14/31 45.2% 58.7/100
|
||||
|
||||
REPEAT OFFENDERS (50%+ Suspicious Trades)
|
||||
================================================================================
|
||||
|
||||
Tommy Tuberville (R-AL, Senate)
|
||||
Trades: 47 | Suspicious: 35 (74.5%)
|
||||
Avg Timing Score: 72.5/100
|
||||
Pattern: HIGHLY SUSPICIOUS - Majority of trades show timing advantage
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Test Coverage**
|
||||
|
||||
**Total: 93 tests, all passing **
|
||||
|
||||
- **Phase 1 (Monitoring):** 14 tests
|
||||
- **Phase 2 (Correlation):** 13 tests
|
||||
- **Phase 3 (Patterns):** 11 tests
|
||||
- **Previous (Analytics, etc.):** 55 tests
|
||||
|
||||
**Coverage:** ~85% overall
|
||||
|
||||
---
|
||||
|
||||
## **Key Insights the System Provides**
|
||||
|
||||
### **1. Individual Official Analysis**
|
||||
- Which officials consistently trade before unusual activity?
|
||||
- Historical timing patterns
|
||||
- Suspicious trade percentage
|
||||
- Repeat offender identification
|
||||
|
||||
### **2. Stock-Specific Analysis**
|
||||
- Which stocks show most suspicious patterns?
|
||||
- Congressional trading concentration
|
||||
- Alert frequency before trades
|
||||
|
||||
### **3. Sector Analysis**
|
||||
- Which sectors have highest timing scores?
|
||||
- Technology vs Energy vs Financial
|
||||
- Sector-specific patterns
|
||||
|
||||
### **4. Party Comparison**
|
||||
- Democrats vs Republicans timing scores
|
||||
- Cross-party patterns
|
||||
- Statistical comparisons
|
||||
|
||||
### **5. Temporal Patterns**
|
||||
- When do suspicious trades cluster?
|
||||
- Seasonal patterns
|
||||
- Event-driven trading
|
||||
|
||||
---
|
||||
|
||||
## **Automated Workflow**
|
||||
|
||||
### **Daily Routine (Recommended)**
|
||||
|
||||
```bash
|
||||
# 1. Morning: Monitor market (every 5 minutes)
|
||||
*/5 9-16 * * 1-5 /path/to/scripts/monitor_market.py --once
|
||||
|
||||
# 2. Evening: Analyze new disclosures
|
||||
0 18 * * 1-5 /path/to/scripts/analyze_disclosure_timing.py --days 7 --min-score 60
|
||||
|
||||
# 3. Weekly: Pattern report
|
||||
0 8 * * 1 /path/to/scripts/generate_pattern_report.py --days 90
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Database Schema**
|
||||
|
||||
**New Table: `market_alerts`**
|
||||
```sql
|
||||
- id (PK)
|
||||
- ticker
|
||||
- alert_type (unusual_volume, price_spike, etc.)
|
||||
- timestamp
|
||||
- details (JSON)
|
||||
- price, volume, change_pct
|
||||
- severity (1-10)
|
||||
- Indexes on ticker, timestamp, alert_type
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Interpretation Guide**
|
||||
|
||||
### **Timing Scores**
|
||||
- **80-100:** Highly suspicious - Multiple high-severity alerts before trade
|
||||
- **60-79:** Suspicious - Clear pattern of alerts before trade
|
||||
- **40-59:** Notable - Some unusual activity before trade
|
||||
- **0-39:** Normal - No significant prior activity
|
||||
|
||||
### **Suspicious Rates**
|
||||
- **>70%:** Systematic pattern - Likely intentional timing
|
||||
- **50-70%:** High concern - Warrants investigation
|
||||
- **25-50%:** Moderate - Some questionable trades
|
||||
- **<25%:** Within normal range
|
||||
|
||||
### **Alert Types (By Suspicion Level)**
|
||||
1. **Most Suspicious:** Unusual volume + high severity + recent
|
||||
2. **Very Suspicious:** Price spike + multiple alerts + pre-news
|
||||
3. **Suspicious:** High volatility + clustering
|
||||
4. **Moderate:** Single low-severity alert
|
||||
|
||||
---
|
||||
|
||||
## **Important Disclaimers**
|
||||
|
||||
### **Legal & Ethical**
|
||||
1. All data is public and legally obtained
|
||||
2. Analysis is retrospective (30-45 day lag)
|
||||
3. For research and transparency only
|
||||
4. NOT investment advice
|
||||
5. NOT proof of illegal activity (requires investigation)
|
||||
6. Statistical patterns ≠ legal evidence
|
||||
|
||||
### **Technical Limitations**
|
||||
1. Cannot identify WHO is trading in real-time
|
||||
2. 30-45 day disclosure lag is built into system
|
||||
3. Relies on yfinance data (15-min delay on free tier)
|
||||
4. Alert detection uses statistical thresholds (not perfect)
|
||||
5. High timing scores indicate patterns, not certainty
|
||||
|
||||
---
|
||||
|
||||
## **Deployment Checklist**
|
||||
|
||||
### **On Proxmox Container**
|
||||
|
||||
```bash
|
||||
# 1. Update database
|
||||
alembic upgrade head
|
||||
|
||||
# 2. Add watchlist
|
||||
python scripts/fetch_congress_members.py --create
|
||||
|
||||
# 3. Test monitoring
|
||||
python scripts/monitor_market.py --once
|
||||
|
||||
# 4. Setup automation
|
||||
crontab -e
|
||||
# Add monitoring schedule
|
||||
|
||||
# 5. Test timing analysis
|
||||
python scripts/analyze_disclosure_timing.py --days 90
|
||||
|
||||
# 6. Generate baseline report
|
||||
python scripts/generate_pattern_report.py --days 365
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Documentation**
|
||||
|
||||
- **`docs/11_live_market_monitoring.md`** - Deep dive into monitoring
|
||||
- **`LOCAL_TEST_GUIDE.md`** - Testing instructions
|
||||
- **`WATCHLIST_GUIDE.md`** - Managing watchlists
|
||||
- **`QUICKSTART.md`** - General usage
|
||||
|
||||
---
|
||||
|
||||
## **Achievement Unlocked!**
|
||||
|
||||
**You now have a complete system that:**
|
||||
|
||||
Monitors real-time market activity
|
||||
Correlates trades to prior alerts
|
||||
Calculates timing advantage scores
|
||||
Identifies repeat offenders
|
||||
Ranks officials by suspicion
|
||||
Generates comprehensive reports
|
||||
93 tests confirming it works
|
||||
|
||||
**This is a production-ready transparency and research tool!**
|
||||
|
||||
---
|
||||
|
||||
## **Potential Future Enhancements**
|
||||
|
||||
### **Phase 4 Ideas (Optional)**
|
||||
- Email/SMS alerts for high-severity patterns
|
||||
- Web dashboard (FastAPI + React)
|
||||
- Machine learning for pattern prediction
|
||||
- Options flow integration (paid APIs)
|
||||
- Social media sentiment correlation
|
||||
- Legislative event correlation
|
||||
- Automated PDF reports
|
||||
- Historical performance tracking
|
||||
|
||||
**But the core system is COMPLETE and FUNCTIONAL now!**
|
||||
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# PR1 Summary: Project Scaffold + DB + Price Loader
|
||||
|
||||
**Status**: Complete
|
||||
**Date**: 2025-12-13
|
||||
|
||||
## What was built
|
||||
|
||||
### 1. Project scaffold
|
||||
- `pyproject.toml` with all dependencies (SQLAlchemy, Alembic, yfinance, pandas, pytest, ruff, black, etc.)
|
||||
- `src/pote/` layout with config, db, and ingestion modules
|
||||
- `.gitignore`, `.env.example`, `Makefile` for dev workflow
|
||||
- Docs: `README.md` + 6 `.md` files in `docs/` covering MVP, architecture, schema, sources, safety/ethics, and dev setup
|
||||
|
||||
### 2. Database models (SQLAlchemy 2.0)
|
||||
- **Officials**: Congress members (name, chamber, party, state, bioguide_id)
|
||||
- **Securities**: stocks/bonds (ticker, name, exchange, sector)
|
||||
- **Trades**: disclosed transactions (official_id, security_id, transaction_date, filing_date, side, value ranges)
|
||||
- **Prices**: daily OHLCV (security_id, date, open/high/low/close/volume)
|
||||
- **Metrics stubs**: `metrics_official` and `metrics_trade` (Phase 2)
|
||||
|
||||
Includes proper indexes, unique constraints, and relationships.
|
||||
|
||||
### 3. Alembic migrations
|
||||
- Initialized Alembic with `env.py` wired to our config
|
||||
- Generated and applied initial migration (`66fd166195e8`)
|
||||
- DB file: `pote.db` (SQLite for dev)
|
||||
|
||||
### 4. Price loader (`PriceLoader`)
|
||||
- Fetches daily price data from **yfinance**
|
||||
- Idempotent: skips existing dates, resumes from gaps
|
||||
- Upsert logic (insert or update on conflict)
|
||||
- Handles single ticker or bulk fetches
|
||||
- Logging + basic error handling
|
||||
|
||||
### 5. Tests (pytest)
|
||||
- `tests/conftest.py`: fixtures for in-memory DB, sample officials/securities/trades/prices
|
||||
- `tests/test_models.py`: model creation, relationships, unique constraints, queries (7 tests)
|
||||
- `tests/test_price_loader.py`: loader logic, idempotency, upsert, mocking yfinance (8 tests)
|
||||
- **Result**: 15 tests, all passing
|
||||
|
||||
### 6. Tooling
|
||||
- **Black** + **ruff** configured and run (all code formatted + linted)
|
||||
- `Makefile` with targets: `install`, `test`, `lint`, `format`, `migrate`, `clean`
|
||||
- Smoke-test script: `scripts/fetch_sample_prices.py` (verified live with AAPL/MSFT/TSLA)
|
||||
|
||||
## What works now
|
||||
- You can spin up the DB, run migrations, fetch price data, and query it
|
||||
- All core Phase 1 foundations are in place
|
||||
- Tests confirm models and ingestion work correctly
|
||||
|
||||
## Next steps (PR2+)
|
||||
Per `docs/00_mvp.md`:
|
||||
- **PR2**: QuiverQuant or FMP client for Congress trades
|
||||
- **PR3**: ETL job to populate `officials` and `trades` tables
|
||||
- **PR4+**: Analytics (abnormal returns, clustering, signals)
|
||||
|
||||
## How to run
|
||||
```bash
|
||||
# Install
|
||||
make install
|
||||
source venv/bin/activate
|
||||
|
||||
# Run migrations
|
||||
make migrate
|
||||
|
||||
# Fetch sample prices
|
||||
python scripts/fetch_sample_prices.py
|
||||
|
||||
# Run tests
|
||||
make test
|
||||
|
||||
# Lint + format
|
||||
make lint
|
||||
make format
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Research-only reminder**: This tool is for transparency and descriptive analytics using public data. Not investment advice.
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
# PR2 Summary: Congressional Trade Ingestion
|
||||
|
||||
**Status**: Complete
|
||||
**Date**: 2025-12-14
|
||||
|
||||
## What was built
|
||||
|
||||
### 1. House Stock Watcher Client (`src/pote/ingestion/house_watcher.py`)
|
||||
- Free API client for https://housestockwatcher.com
|
||||
- No authentication required
|
||||
- Methods:
|
||||
- `fetch_all_transactions(limit)`: Get all recent transactions
|
||||
- `fetch_recent_transactions(days)`: Filter to last N days
|
||||
- Helper functions:
|
||||
- `parse_amount_range()`: Parse "$1,001 - $15,000" → (min, max)
|
||||
- `normalize_transaction_type()`: "Purchase" → "buy", "Sale" → "sell"
|
||||
|
||||
### 2. Trade Loader ETL (`src/pote/ingestion/trade_loader.py`)
|
||||
- `TradeLoader.ingest_transactions()`: Full ETL pipeline
|
||||
- Get-or-create logic for officials and securities (deduplication)
|
||||
- Upsert trades by source + external_id (no duplicates)
|
||||
- Returns counts: `{"officials": N, "securities": N, "trades": N}`
|
||||
- Proper error handling and logging
|
||||
|
||||
### 3. Test Fixtures
|
||||
- `tests/fixtures/sample_house_watcher.json`: 5 realistic sample transactions
|
||||
- Includes House + Senate, Democrats + Republicans, various tickers
|
||||
|
||||
### 4. Tests (13 new tests, all passing )
|
||||
**`tests/test_house_watcher.py` (8 tests)**:
|
||||
- Amount range parsing (with range, single value, invalid)
|
||||
- Transaction type normalization
|
||||
- Fetching all/recent transactions (mocked)
|
||||
- Client context manager
|
||||
|
||||
**`tests/test_trade_loader.py` (5 tests)**:
|
||||
- Ingest from fixture file (full integration)
|
||||
- Duplicate transaction handling (idempotency)
|
||||
- Missing ticker handling (skip gracefully)
|
||||
- Senate vs House official creation
|
||||
- Multiple trades for same official
|
||||
|
||||
### 5. Smoke-test Script (`scripts/fetch_congressional_trades.py`)
|
||||
- CLI tool to fetch live data from House Stock Watcher
|
||||
- Options: `--days N`, `--limit N`, `--all`
|
||||
- Ingests into DB and shows summary stats
|
||||
- Usage:
|
||||
```bash
|
||||
python scripts/fetch_congressional_trades.py --days 30
|
||||
python scripts/fetch_congressional_trades.py --all --limit 100
|
||||
```
|
||||
|
||||
## What works now
|
||||
|
||||
### Live Data Ingestion (FREE!)
|
||||
```bash
|
||||
# Fetch last 30 days of congressional trades
|
||||
python scripts/fetch_congressional_trades.py --days 30
|
||||
|
||||
# Sample output:
|
||||
# Officials created/updated: 47
|
||||
# Securities created/updated: 89
|
||||
# Trades ingested: 234
|
||||
```
|
||||
|
||||
### Database Queries
|
||||
```python
|
||||
from pote.db import SessionLocal
|
||||
from pote.db.models import Official, Trade
|
||||
from sqlalchemy import select
|
||||
|
||||
with SessionLocal() as session:
|
||||
# Find Nancy Pelosi's trades
|
||||
stmt = select(Official).where(Official.name == "Nancy Pelosi")
|
||||
pelosi = session.scalars(stmt).first()
|
||||
|
||||
stmt = select(Trade).where(Trade.official_id == pelosi.id)
|
||||
trades = session.scalars(stmt).all()
|
||||
print(f"Pelosi has {len(trades)} trades")
|
||||
```
|
||||
|
||||
### Test Coverage
|
||||
```bash
|
||||
make test
|
||||
# 28 tests passed in 1.23s
|
||||
# Coverage: 87%+
|
||||
```
|
||||
|
||||
## Data Model Updates
|
||||
|
||||
No schema changes! Existing tables work perfectly:
|
||||
- `officials`: Populated from House Stock Watcher API
|
||||
- `securities`: Tickers from trades (name=ticker for now, will enrich later)
|
||||
- `trades`: Full trade records with transaction_date, filing_date, side, value ranges
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Free API First**: House Stock Watcher = $0, no rate limits
|
||||
2. **Idempotency**: Re-running ingestion won't create duplicates
|
||||
3. **Graceful Degradation**: Skip trades with missing tickers, log warnings
|
||||
4. **Tuple Returns**: `_get_or_create_*` methods return `(entity, is_new)` for accurate counting
|
||||
5. **External IDs**: `official_id_security_id_date_side` for deduplication
|
||||
|
||||
## Performance
|
||||
|
||||
- Fetches 100+ transactions in ~2 seconds
|
||||
- Ingest 100 transactions in ~0.5 seconds (SQLite)
|
||||
- Tests run in 1.2 seconds (28 tests)
|
||||
|
||||
## Next Steps (PR3+)
|
||||
|
||||
Per `docs/00_mvp.md`:
|
||||
- **PR3**: Enrich securities with yfinance (fetch names, sectors, exchanges)
|
||||
- **PR4**: Abnormal return calculations
|
||||
- **PR5**: Clustering & signals
|
||||
- **PR6**: Optional FastAPI + dashboard
|
||||
|
||||
## How to Use
|
||||
|
||||
### 1. Fetch Live Data
|
||||
```bash
|
||||
# Recent trades (last 7 days)
|
||||
python scripts/fetch_congressional_trades.py --days 7
|
||||
|
||||
# All trades, limited to 50
|
||||
python scripts/fetch_congressional_trades.py --all --limit 50
|
||||
```
|
||||
|
||||
### 2. Programmatic Usage
|
||||
```python
|
||||
from pote.db import SessionLocal
|
||||
from pote.ingestion.house_watcher import HouseWatcherClient
|
||||
from pote.ingestion.trade_loader import TradeLoader
|
||||
|
||||
with HouseWatcherClient() as client:
|
||||
txns = client.fetch_recent_transactions(days=30)
|
||||
|
||||
with SessionLocal() as session:
|
||||
loader = TradeLoader(session)
|
||||
counts = loader.ingest_transactions(txns)
|
||||
print(f"Ingested {counts['trades']} trades")
|
||||
```
|
||||
|
||||
### 3. Run Tests
|
||||
```bash
|
||||
# All tests
|
||||
make test
|
||||
|
||||
# Just trade ingestion tests
|
||||
pytest tests/test_trade_loader.py -v
|
||||
|
||||
# With coverage
|
||||
pytest tests/ --cov=pote --cov-report=term-missing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Cost**: $0 (uses free House Stock Watcher API)
|
||||
**Dependencies**: `httpx` (already in `pyproject.toml`)
|
||||
**Research-only reminder**: This tool is for transparency and descriptive analytics. Not investment advice.
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
# PR3 Summary: Security Enrichment + Deployment
|
||||
|
||||
**Status**: Complete
|
||||
**Date**: 2025-12-14
|
||||
|
||||
## What was built
|
||||
|
||||
### 1. Security Enrichment (`src/pote/ingestion/security_enricher.py`)
|
||||
- `SecurityEnricher` class for enriching securities with yfinance data
|
||||
- Fetches: company names, sectors, industries, exchanges
|
||||
- Detects asset type: stock, ETF, mutual fund, index
|
||||
- Methods:
|
||||
- `enrich_security(security, force)`: Enrich single security
|
||||
- `enrich_all_securities(limit, force)`: Batch enrichment
|
||||
- `enrich_by_ticker(ticker)`: Enrich specific ticker
|
||||
- Smart skipping: only enriches unenriched securities (unless `force=True`)
|
||||
|
||||
### 2. Enrichment Script (`scripts/enrich_securities.py`)
|
||||
- CLI tool for enriching securities
|
||||
- Usage:
|
||||
```bash
|
||||
# Enrich all unenriched securities
|
||||
python scripts/enrich_securities.py
|
||||
|
||||
# Enrich specific ticker
|
||||
python scripts/enrich_securities.py --ticker AAPL
|
||||
|
||||
# Limit batch size
|
||||
python scripts/enrich_securities.py --limit 10
|
||||
|
||||
# Force re-enrichment
|
||||
python scripts/enrich_securities.py --force
|
||||
```
|
||||
|
||||
### 3. Tests (9 new tests, all passing )
|
||||
**`tests/test_security_enricher.py`**:
|
||||
- Successful enrichment with complete data
|
||||
- ETF detection and classification
|
||||
- Skip already enriched securities
|
||||
- Force refresh functionality
|
||||
- Handle missing/invalid data gracefully
|
||||
- Batch enrichment
|
||||
- Enrichment with limit
|
||||
- Enrich by specific ticker
|
||||
- Handle ticker not found
|
||||
|
||||
### 4. Deployment Infrastructure
|
||||
- **`Dockerfile`**: Production-ready container image
|
||||
- **`docker-compose.yml`**: Full stack (app + PostgreSQL)
|
||||
- **`.dockerignore`**: Optimize image size
|
||||
- **`docs/07_deployment.md`**: Comprehensive deployment guide
|
||||
- Local development (SQLite)
|
||||
- Single server (PostgreSQL + cron)
|
||||
- Docker deployment
|
||||
- Cloud deployment (AWS, Fly.io, Railway)
|
||||
- Cost estimates
|
||||
- Production checklist
|
||||
|
||||
## What works now
|
||||
|
||||
### Enrich Securities from Fixtures
|
||||
```bash
|
||||
# Our existing fixtures have these tickers: NVDA, MSFT, AAPL, TSLA, GOOGL
|
||||
# They're created as "unenriched" (name == ticker)
|
||||
|
||||
python scripts/enrich_securities.py
|
||||
|
||||
# Output:
|
||||
# Enriching 5 securities
|
||||
# Enriched NVDA: NVIDIA Corporation (Technology)
|
||||
# Enriched MSFT: Microsoft Corporation (Technology)
|
||||
# Enriched AAPL: Apple Inc. (Technology)
|
||||
# Enriched TSLA: Tesla, Inc. (Consumer Cyclical)
|
||||
# Enriched GOOGL: Alphabet Inc. (Communication Services)
|
||||
# Successfully enriched: 5
|
||||
```
|
||||
|
||||
### Query Enriched Data
|
||||
```python
|
||||
from pote.db import SessionLocal
|
||||
from pote.db.models import Security
|
||||
from sqlalchemy import select
|
||||
|
||||
with SessionLocal() as session:
|
||||
stmt = select(Security).where(Security.sector.isnot(None))
|
||||
enriched = session.scalars(stmt).all()
|
||||
|
||||
for sec in enriched:
|
||||
print(f"{sec.ticker}: {sec.name} ({sec.sector})")
|
||||
```
|
||||
|
||||
### Docker Deployment
|
||||
```bash
|
||||
# Quick start
|
||||
docker-compose up -d
|
||||
|
||||
# Run migrations
|
||||
docker-compose exec pote alembic upgrade head
|
||||
|
||||
# Ingest trades from fixtures (offline)
|
||||
docker-compose exec pote python scripts/ingest_from_fixtures.py
|
||||
|
||||
# Enrich securities (needs network in container)
|
||||
docker-compose exec pote python scripts/enrich_securities.py
|
||||
```
|
||||
|
||||
## Data Model Updates
|
||||
|
||||
No schema changes! The `securities` table already had all necessary fields:
|
||||
- `name`: Now populated with full company name
|
||||
- `sector`: Technology, Healthcare, Finance, etc.
|
||||
- `industry`: Specific industry within sector
|
||||
- `exchange`: NASDAQ, NYSE, etc.
|
||||
- `asset_type`: stock, etf, mutual_fund, index
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Smart Skipping**: Only enrich securities where `name == ticker` (unenriched)
|
||||
2. **Force Option**: Can re-enrich with `--force` flag
|
||||
3. **Graceful Degradation**: Skip/log if yfinance data unavailable
|
||||
4. **Batch Control**: `--limit` for rate limiting or testing
|
||||
5. **Asset Type Detection**: Automatically classify ETFs, mutual funds, indexes
|
||||
|
||||
## Performance
|
||||
|
||||
- Enrich single security: ~1 second (yfinance API call)
|
||||
- Batch enrichment: ~1-2 seconds per security
|
||||
- Recommendation: Run weekly or when new tickers appear
|
||||
- yfinance is free but rate-limited (be reasonable!)
|
||||
|
||||
## Integration with Existing System
|
||||
|
||||
### After Trade Ingestion
|
||||
```python
|
||||
# In production cron job:
|
||||
# 1. Fetch trades
|
||||
python scripts/fetch_congressional_trades.py --days 7
|
||||
|
||||
# 2. Enrich any new securities
|
||||
python scripts/enrich_securities.py
|
||||
|
||||
# 3. Fetch prices for all securities
|
||||
python scripts/update_all_prices.py # To be built in PR4
|
||||
```
|
||||
|
||||
### Cron Schedule (Production)
|
||||
```bash
|
||||
# Daily at 6 AM: Fetch trades
|
||||
0 6 * * * cd /path/to/pote && venv/bin/python scripts/fetch_congressional_trades.py --days 7
|
||||
|
||||
# Daily at 6:15 AM: Enrich new securities
|
||||
15 6 * * * cd /path/to/pote && venv/bin/python scripts/enrich_securities.py
|
||||
|
||||
# Daily at 6:30 AM: Update prices
|
||||
30 6 * * * cd /path/to/pote && venv/bin/python scripts/update_all_prices.py
|
||||
```
|
||||
|
||||
## Deployment Options
|
||||
|
||||
| Option | Complexity | Cost/month | Best For |
|
||||
|--------|-----------|------------|----------|
|
||||
| **Local** | | $0 | Development |
|
||||
| **VPS + Docker** | | $10-20 | Personal deployment |
|
||||
| **Railway/Fly.io** | | $5-15 | Easy cloud |
|
||||
| **AWS** | | $20-50 | Scalable production |
|
||||
|
||||
See [`docs/07_deployment.md`](07_deployment.md) for detailed guides.
|
||||
|
||||
## Next Steps (PR4+)
|
||||
|
||||
Per `docs/00_mvp.md`:
|
||||
- **PR4**: Analytics - abnormal returns, benchmarks
|
||||
- **PR5**: Clustering & signals
|
||||
- **PR6**: FastAPI + dashboard
|
||||
|
||||
## How to Use
|
||||
|
||||
### 1. Enrich All Securities
|
||||
```bash
|
||||
python scripts/enrich_securities.py
|
||||
```
|
||||
|
||||
### 2. Enrich Specific Ticker
|
||||
```bash
|
||||
python scripts/enrich_securities.py --ticker NVDA
|
||||
```
|
||||
|
||||
### 3. Re-enrich Everything
|
||||
```bash
|
||||
python scripts/enrich_securities.py --force
|
||||
```
|
||||
|
||||
### 4. Programmatic Usage
|
||||
```python
|
||||
from pote.db import SessionLocal
|
||||
from pote.ingestion.security_enricher import SecurityEnricher
|
||||
|
||||
with SessionLocal() as session:
|
||||
enricher = SecurityEnricher(session)
|
||||
|
||||
# Enrich all unenriched
|
||||
counts = enricher.enrich_all_securities()
|
||||
print(f"Enriched {counts['enriched']} securities")
|
||||
|
||||
# Enrich specific ticker
|
||||
enricher.enrich_by_ticker("AAPL")
|
||||
```
|
||||
|
||||
## Test Coverage
|
||||
|
||||
```bash
|
||||
pytest tests/ -v
|
||||
|
||||
# 37 tests passing
|
||||
# Coverage: 87%+
|
||||
|
||||
# New tests:
|
||||
# - test_security_enricher.py (9 tests)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Cost**: Still $0 (yfinance is free!)
|
||||
**Dependencies**: yfinance (already in `pyproject.toml`)
|
||||
**Research-only reminder**: This tool is for transparency and descriptive analytics. Not investment advice.
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
# PR4 Summary: Phase 2 Analytics Foundation
|
||||
|
||||
## Completed
|
||||
|
||||
**Date**: December 15, 2025
|
||||
**Status**: Complete
|
||||
**Tests**: All passing
|
||||
|
||||
## What Was Built
|
||||
|
||||
### 1. Analytics Module (`src/pote/analytics/`)
|
||||
|
||||
#### ReturnCalculator (`returns.py`)
|
||||
- Calculate returns for trades over various time windows (30/60/90/180 days)
|
||||
- Handle buy and sell trades appropriately
|
||||
- Find closest price data when exact dates unavailable
|
||||
- Export price series as pandas DataFrames
|
||||
|
||||
**Key Methods:**
|
||||
- `calculate_trade_return()` - Single trade return
|
||||
- `calculate_multiple_windows()` - Multiple time windows
|
||||
- `calculate_all_trades()` - Batch calculation
|
||||
- `get_price_series()` - Historical price data
|
||||
|
||||
#### BenchmarkComparison (`benchmarks.py`)
|
||||
- Calculate benchmark returns (SPY, QQQ, DIA, etc.)
|
||||
- Compute abnormal returns (alpha)
|
||||
- Compare trades to market performance
|
||||
- Batch comparison operations
|
||||
|
||||
**Key Methods:**
|
||||
- `calculate_benchmark_return()` - Market index returns
|
||||
- `calculate_abnormal_return()` - Alpha calculation
|
||||
- `compare_trade_to_benchmark()` - Single trade comparison
|
||||
- `calculate_aggregate_alpha()` - Portfolio-level metrics
|
||||
|
||||
#### PerformanceMetrics (`metrics.py`)
|
||||
- Aggregate statistics by official
|
||||
- Sector-level analysis
|
||||
- Top performer rankings
|
||||
- Disclosure timing analysis
|
||||
|
||||
**Key Methods:**
|
||||
- `official_performance()` - Comprehensive official stats
|
||||
- `sector_analysis()` - Performance by sector
|
||||
- `top_performers()` - Leaderboard
|
||||
- `timing_analysis()` - Disclosure lag stats
|
||||
- `summary_statistics()` - System-wide metrics
|
||||
|
||||
### 2. Analysis Scripts (`scripts/`)
|
||||
|
||||
#### `analyze_official.py`
|
||||
Interactive tool to analyze a specific official:
|
||||
```bash
|
||||
python scripts/analyze_official.py "Nancy Pelosi" --window 90 --benchmark SPY
|
||||
```
|
||||
|
||||
**Output Includes:**
|
||||
- Trading activity summary
|
||||
- Return metrics (avg, median, max, min)
|
||||
- Alpha (vs market benchmark)
|
||||
- Win rates
|
||||
- Best/worst trades
|
||||
- Research signals (FOLLOW, AVOID, WATCH)
|
||||
|
||||
#### `calculate_all_returns.py`
|
||||
System-wide performance analysis:
|
||||
```bash
|
||||
python scripts/calculate_all_returns.py --window 90 --benchmark SPY --top 10
|
||||
```
|
||||
|
||||
**Output Includes:**
|
||||
- Overall statistics
|
||||
- Aggregate performance
|
||||
- Top 10 performers by alpha
|
||||
- Sector analysis
|
||||
- Disclosure timing
|
||||
|
||||
### 3. Tests (`tests/test_analytics.py`)
|
||||
|
||||
- Return calculator with sample data
|
||||
- Buy vs sell trade handling
|
||||
- Missing data edge cases
|
||||
- Benchmark comparisons
|
||||
- Official performance metrics
|
||||
- Multiple time windows
|
||||
- Sector analysis
|
||||
- Timing analysis
|
||||
|
||||
**Test Coverage**: Analytics module fully tested
|
||||
|
||||
## Example Usage
|
||||
|
||||
### Analyze an Official
|
||||
|
||||
```python
|
||||
from pote.analytics.metrics import PerformanceMetrics
|
||||
from pote.db import get_session
|
||||
|
||||
with next(get_session()) as session:
|
||||
metrics = PerformanceMetrics(session)
|
||||
|
||||
# Get performance for official ID 1
|
||||
perf = metrics.official_performance(
|
||||
official_id=1,
|
||||
window_days=90,
|
||||
benchmark="SPY"
|
||||
)
|
||||
|
||||
print(f"{perf['name']}")
|
||||
print(f"Average Return: {perf['avg_return']:.2f}%")
|
||||
print(f"Alpha: {perf['avg_alpha']:.2f}%")
|
||||
print(f"Win Rate: {perf['win_rate']:.1%}")
|
||||
```
|
||||
|
||||
### Calculate Trade Returns
|
||||
|
||||
```python
|
||||
from pote.analytics.returns import ReturnCalculator
|
||||
from pote.db import get_session
|
||||
from pote.db.models import Trade
|
||||
|
||||
with next(get_session()) as session:
|
||||
calculator = ReturnCalculator(session)
|
||||
|
||||
# Get a trade
|
||||
trade = session.query(Trade).first()
|
||||
|
||||
# Calculate returns for multiple windows
|
||||
results = calculator.calculate_multiple_windows(
|
||||
trade,
|
||||
windows=[30, 60, 90]
|
||||
)
|
||||
|
||||
for window, data in results.items():
|
||||
print(f"{window}d: {data['return_pct']:.2f}%")
|
||||
```
|
||||
|
||||
### Compare to Benchmark
|
||||
|
||||
```python
|
||||
from pote.analytics.benchmarks import BenchmarkComparison
|
||||
from pote.db import get_session
|
||||
|
||||
with next(get_session()) as session:
|
||||
benchmark = BenchmarkComparison(session)
|
||||
|
||||
# Get aggregate alpha for all officials
|
||||
stats = benchmark.calculate_aggregate_alpha(
|
||||
official_id=None, # All officials
|
||||
window_days=90,
|
||||
benchmark="SPY"
|
||||
)
|
||||
|
||||
print(f"Average Alpha: {stats['avg_alpha']:.2f}%")
|
||||
print(f"Beat Market Rate: {stats['beat_market_rate']:.1%}")
|
||||
```
|
||||
|
||||
## Command Line Usage
|
||||
|
||||
### Analyze Specific Official
|
||||
```bash
|
||||
# In container
|
||||
cd ~/pote && source venv/bin/activate
|
||||
|
||||
# Analyze Nancy Pelosi's trades
|
||||
python scripts/analyze_official.py "Nancy Pelosi"
|
||||
|
||||
# With custom parameters
|
||||
python scripts/analyze_official.py "Tommy Tuberville" --window 180 --benchmark QQQ
|
||||
```
|
||||
|
||||
### System-Wide Analysis
|
||||
```bash
|
||||
# Calculate all returns and show top 10
|
||||
python scripts/calculate_all_returns.py
|
||||
|
||||
# Custom parameters
|
||||
python scripts/calculate_all_returns.py --window 60 --benchmark SPY --top 20
|
||||
```
|
||||
|
||||
## What You Can Do Now
|
||||
|
||||
### 1. Analyze Your Existing Data
|
||||
```bash
|
||||
# On your Proxmox container (10.0.10.95)
|
||||
ssh root@10.0.10.95
|
||||
su - poteapp
|
||||
cd pote && source venv/bin/activate
|
||||
|
||||
# Analyze each official
|
||||
python scripts/analyze_official.py "Nancy Pelosi"
|
||||
python scripts/analyze_official.py "Dan Crenshaw"
|
||||
|
||||
# System-wide view
|
||||
python scripts/calculate_all_returns.py
|
||||
```
|
||||
|
||||
### 2. Compare Officials
|
||||
```python
|
||||
from pote.analytics.metrics import PerformanceMetrics
|
||||
from pote.db import get_session
|
||||
|
||||
with next(get_session()) as session:
|
||||
metrics = PerformanceMetrics(session)
|
||||
|
||||
# Get top 5 by alpha
|
||||
top = metrics.top_performers(window_days=90, limit=5)
|
||||
|
||||
for i, perf in enumerate(top, 1):
|
||||
print(f"{i}. {perf['name']}: {perf['avg_alpha']:.2f}% alpha")
|
||||
```
|
||||
|
||||
### 3. Sector Analysis
|
||||
```python
|
||||
from pote.analytics.metrics import PerformanceMetrics
|
||||
from pote.db import get_session
|
||||
|
||||
with next(get_session()) as session:
|
||||
metrics = PerformanceMetrics(session)
|
||||
|
||||
sectors = metrics.sector_analysis(window_days=90)
|
||||
|
||||
print("Performance by Sector:")
|
||||
for s in sectors:
|
||||
print(f"{s['sector']:20s} | {s['avg_alpha']:+6.2f}% alpha | {s['win_rate']:.1%} win rate")
|
||||
```
|
||||
|
||||
## Limitations & Notes
|
||||
|
||||
### Current Limitations
|
||||
1. **Requires Price Data**: Need historical prices in database
|
||||
- Run `python scripts/fetch_sample_prices.py` first
|
||||
- Or manually add prices for your securities
|
||||
|
||||
2. **Limited Sample**: Only 5 trades currently
|
||||
- Add more trades for meaningful analysis
|
||||
- Use `scripts/add_custom_trades.py`
|
||||
|
||||
3. **No Risk-Adjusted Metrics Yet**
|
||||
- Sharpe ratio (coming in next PR)
|
||||
- Drawdowns
|
||||
- Volatility measures
|
||||
|
||||
### Data Quality
|
||||
- Handles missing price data gracefully (returns None)
|
||||
- Finds closest price within 5-day window
|
||||
- Adjusts returns for buy vs sell trades
|
||||
- Logs warnings for data issues
|
||||
|
||||
## Files Changed/Added
|
||||
|
||||
**New Files:**
|
||||
- `src/pote/analytics/__init__.py`
|
||||
- `src/pote/analytics/returns.py` (245 lines)
|
||||
- `src/pote/analytics/benchmarks.py` (195 lines)
|
||||
- `src/pote/analytics/metrics.py` (265 lines)
|
||||
- `scripts/analyze_official.py` (145 lines)
|
||||
- `scripts/calculate_all_returns.py` (130 lines)
|
||||
- `tests/test_analytics.py` (230 lines)
|
||||
|
||||
**Total New Code:** ~1,210 lines
|
||||
|
||||
## Next Steps (PR5: Signals & Clustering)
|
||||
|
||||
### Planned Features:
|
||||
1. **Research Signals**
|
||||
- `FOLLOW_RESEARCH`: Officials with consistent alpha > 5%
|
||||
- `AVOID_RISK`: Suspicious patterns or negative alpha
|
||||
- `WATCH`: Unusual activity or limited data
|
||||
|
||||
2. **Behavioral Clustering**
|
||||
- Group officials by trading patterns
|
||||
- k-means clustering on features:
|
||||
- Trade frequency
|
||||
- Average position size
|
||||
- Sector preferences
|
||||
- Timing patterns
|
||||
|
||||
3. **Risk Metrics**
|
||||
- Sharpe ratio
|
||||
- Max drawdown
|
||||
- Win/loss streaks
|
||||
- Volatility
|
||||
|
||||
4. **Event Analysis**
|
||||
- Trades near earnings
|
||||
- Trades near policy events
|
||||
- Unusual timing flags
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Can calculate returns for any trade + window
|
||||
- Can compare to S&P 500 benchmark
|
||||
- Can generate official performance summaries
|
||||
- All calculations tested and accurate
|
||||
- Performance data calculated on-the-fly
|
||||
- Documentation complete
|
||||
- Command-line tools working
|
||||
|
||||
## Testing
|
||||
|
||||
Run tests:
|
||||
```bash
|
||||
pytest tests/test_analytics.py -v
|
||||
```
|
||||
|
||||
All analytics tests should pass (may have warnings if no price data).
|
||||
|
||||
---
|
||||
|
||||
**Phase 2 Analytics Foundation: COMPLETE**
|
||||
**Ready for**: PR5 (Signals), PR6 (API), PR7 (Dashboard)
|
||||
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
# Archive
|
||||
|
||||
PR summaries and one-shot status writeups. Prefer the numbered guides and QUICKSTART files.
|
||||
Reference in New Issue
Block a user