Humanize README and docs tone #8

Merged
ilia merged 1 commits from docs/humanize-prose into main 2026-08-05 14:18:02 -05:00
29 changed files with 613 additions and 747 deletions
+37 -174
View File
@@ -1,204 +1,67 @@
# POTE Public Officials Trading Explorer
# POTE (Public Officials Trading Explorer)
**Research-only tool for tracking and analyzing public stock trades by government officials.**
Research tool for tracking publicly disclosed stock trades by government
officials (starting with U.S. Congress). Computes descriptive metrics and
risk/ethics flags from lawfully available public data.
⚠️ **Important**: This project is for personal research and transparency analysis only. It is **NOT** for investment advice or live trading.
Not investment advice. Not for live trading. Public disclosures only; data
may be delayed or incomplete. No claims about inside information.
## What is this?
POTE tracks stock trading activity of government officials (starting with U.S. Congress) using lawfully available public data sources. It computes research metrics, descriptive signals, and risk/ethics flags to help understand trading patterns.
## Key constraints
- **Public data only**: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- **Research framing**: All outputs are descriptive analytics, not trading recommendations
- **No inside information claims**: We use public disclosures that may be delayed or incomplete
## Current Status
**PR1 Complete**: Project scaffold, DB models, price loader
**PR2 Complete**: Congressional trade ingestion (House Stock Watcher)
**PR3 Complete**: Security enrichment + deployment infrastructure
**PR4 Complete**: Phase 2 analytics - returns, benchmarks, performance metrics
**45+ passing tests, 88%+ coverage**
Status: active. Homelab LXC deploy documented under `docs/`.
## Quick start
**🚀 Already deployed?** See **[QUICKSTART.md](docs/QUICKSTART.md)** for full usage guide!
**📦 Deploying?** See **[PROXMOX_QUICKSTART.md](docs/PROXMOX_QUICKSTART.md)** for Proxmox LXC deployment (recommended).
**📧 Want automated reports?** See **[AUTOMATION_QUICKSTART.md](docs/AUTOMATION_QUICKSTART.md)** for email reporting setup!
**🏠 Homelab deploy (LXC 236)?** See **[docs/HANDOFF-2026-05-27.md](docs/HANDOFF-2026-05-27.md)** for ops handoff and next steps.
### Local Development
```bash
# Install
git clone <your-repo>
cd pote
git clone <repository-url>
cd pote # or POTE
make install
source venv/bin/activate
# Run migrations
make migrate
python scripts/ingest_from_fixtures.py # offline sample
make test
```
# Ingest sample data (offline, for testing)
python scripts/ingest_from_fixtures.py
With network:
# Enrich securities with company info
python scripts/enrich_securities.py
# With internet:
```bash
python scripts/fetch_congressional_trades.py
python scripts/fetch_sample_prices.py
# Run tests
make test
# Lint & format
make lint format
```
### Production Deployment
Deploy: `bash scripts/proxmox_setup.sh` or `docker-compose up -d`.
Ops handoff: [docs/HANDOFF-2026-05-27.md](docs/HANDOFF-2026-05-27.md).
## Useful commands
```bash
# Proxmox LXC (Recommended - 5 minutes)
bash scripts/proxmox_setup.sh
# Docker
docker-compose up -d
```
## Tech stack
- **Language**: Python 3.10+
- **Database**: PostgreSQL or SQLite (dev)
- **Data**: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- **Libraries**: SQLAlchemy, Alembic, pandas, numpy, httpx, yfinance, scikit-learn
- **Testing**: pytest (28 tests, 87%+ coverage)
## Documentation
**Getting Started**:
- [`README.md`](README.md) This file
- [`QUICKSTART.md`](docs/QUICKSTART.md) **How to use your deployed POTE instance**
- [`STATUS.md`](docs/STATUS.md) Current project status
- [`FREE_TESTING_QUICKSTART.md`](docs/FREE_TESTING_QUICKSTART.md) Test for $0
- [`OFFLINE_DEMO.md`](docs/OFFLINE_DEMO.md) Works without internet!
**Deployment**:
- [`PROXMOX_QUICKSTART.md`](docs/PROXMOX_QUICKSTART.md) **Proxmox quick deployment (5 min)**
- [`AUTOMATION_QUICKSTART.md`](docs/AUTOMATION_QUICKSTART.md) **Automated reporting setup (5 min)**
- [`docs/07_deployment.md`](docs/07_deployment.md) Full deployment guide (all platforms)
- [`docs/08_proxmox_deployment.md`](docs/08_proxmox_deployment.md) Proxmox detailed guide
- [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md) Automation & CI/CD guide
- [`Dockerfile`](Dockerfile) + [`docker-compose.yml`](docker-compose.yml) Docker setup
**Technical**:
- [`docs/00_mvp.md`](docs/00_mvp.md) MVP roadmap
- [`docs/01_architecture.md`](docs/01_architecture.md) Architecture
- [`docs/02_data_model.md`](docs/02_data_model.md) Database schema
- [`docs/03_data_sources.md`](docs/03_data_sources.md) Data sources
- [`docs/04_safety_ethics.md`](docs/04_safety_ethics.md) Research-only guardrails
- [`docs/05_dev_setup.md`](docs/05_dev_setup.md) Dev conventions
- [`docs/06_free_testing_data.md`](docs/06_free_testing_data.md) Testing strategies
**PR Summaries**:
- [`docs/PR1_SUMMARY.md`](docs/PR1_SUMMARY.md) Scaffold + price loader
- [`docs/PR2_SUMMARY.md`](docs/PR2_SUMMARY.md) Congressional trades
- [`docs/PR3_SUMMARY.md`](docs/PR3_SUMMARY.md) Enrichment + deployment
- [`docs/PR4_SUMMARY.md`](docs/PR4_SUMMARY.md) **Analytics foundation (returns, benchmarks, metrics)**
## What's Working Now
- ✅ SQLAlchemy models for officials, securities, trades, prices
- ✅ Alembic migrations
- ✅ Price loader with yfinance (idempotent, upsert)
- ✅ Congressional trade ingestion from House Stock Watcher (FREE!)
- ✅ Security enrichment (company names, sectors, industries)
- ✅ ETL to populate officials & trades tables
- ✅ Docker + deployment infrastructure
- ✅ 93 passing tests with 88%+ coverage
- ✅ Linting (ruff + mypy) all green
- ✅ Works 100% offline with fixtures
- ✅ Real-time market monitoring & alert system
- ✅ Disclosure timing correlation engine
- ✅ Pattern detection & comparative analysis
- ✅ Automated email reporting (daily/weekly)
- ✅ CI/CD pipeline (GitHub/Gitea Actions)
## What You Can Do Now
### Analyze Performance
```bash
# Analyze specific official
python scripts/analyze_official.py "Nancy Pelosi" --window 90
# System-wide analysis
python scripts/calculate_all_returns.py
```
### Market Monitoring
```bash
# Run market scan
python scripts/monitor_market.py --scan
# Analyze timing of recent disclosures
python scripts/analyze_disclosure_timing.py --recent 7
# Generate pattern report
python scripts/generate_pattern_report.py --days 365
```
### Automated Reporting
```bash
# Set up daily/weekly email reports (5 minutes!)
./scripts/setup_cron.sh
# Send manual report
python scripts/send_daily_report.py --to your@email.com
```
### Add More Data
```bash
# Manual entry
python scripts/add_custom_trades.py
## Stack
# CSV import
python scripts/scrape_alternative_sources.py import trades.csv
```
Python 3.10+, SQLAlchemy, Alembic, pandas/numpy, httpx, yfinance.
PostgreSQL or SQLite for local. pytest + ruff + mypy.
## System Architecture
Data sources: House Stock Watcher, yfinance; QuiverQuant/FMP optional.
POTE now includes a complete 3-phase monitoring system:
## Docs
**Phase 1: Real-Time Market Monitoring**
- Tracks ~50 most-traded congressional stocks
- Detects unusual volume, price spikes, volatility
- Logs all alerts with timestamps and severity
| Doc | Purpose |
|-----|---------|
| [docs/QUICKSTART.md](docs/QUICKSTART.md) | Using a deployed instance |
| [docs/PROXMOX_QUICKSTART.md](docs/PROXMOX_QUICKSTART.md) | Proxmox LXC |
| [docs/AUTOMATION_QUICKSTART.md](docs/AUTOMATION_QUICKSTART.md) | Email reports |
| [docs/01_architecture.md](docs/01_architecture.md) | Architecture |
| [docs/02_data_model.md](docs/02_data_model.md) | Schema |
| [docs/04_safety_ethics.md](docs/04_safety_ethics.md) | Research guardrails |
| [docs/07_deployment.md](docs/07_deployment.md) | Full deploy |
| [docs/00_mvp.md](docs/00_mvp.md) | Roadmap |
**Phase 2: Disclosure Correlation**
- Matches trades with prior market alerts (30-45 day lookback)
- Calculates "timing advantage score" (0-100)
- Identifies suspicious timing patterns
PR writeups live under `docs/archive/` when moved.
**Phase 3: Pattern Detection**
- Ranks officials by consistent suspicious timing
- Analyzes by ticker, sector, and political party
- Generates comprehensive reports
## License
**Full Documentation**: See [`MONITORING_SYSTEM_COMPLETE.md`](docs/MONITORING_SYSTEM_COMPLETE.md)
## Next Steps
- [ ] Signals: "follow_research", "avoid_risk", "watch" with confidence scores
- [ ] Clustering: group officials by trading behavior patterns
- [ ] API: FastAPI backend for queries
- [ ] Dashboard: React/Streamlit visualization
See [`docs/00_mvp.md`](docs/00_mvp.md) for the full roadmap.
---
**License**: MIT (for research/educational use only)
**Disclaimer**: Not investment advice. Use public data only. No claims about inside information.
MIT for research/educational use. Not investment advice.
+4 -4
View File
@@ -6,7 +6,7 @@ Great question! Here are multiple strategies for testing the full pipeline **wit
---
## Strategy 1: Mock/Fixture Data (Current Approach )
## Strategy 1: Mock/Fixture Data (Current Approach )
**What we already have:**
- `tests/conftest.py` creates in-memory SQLite DB with sample officials, securities, trades
@@ -99,7 +99,7 @@ def test_etl_with_real_data():
---
## Strategy 4: Hybrid Testing (Recommended 🌟)
## Strategy 4: Hybrid Testing (Recommended )
**Combine all strategies**:
@@ -164,7 +164,7 @@ def test_etl_with_real_data():
| Source | Free Tier | Paid Tier | Best For |
|--------|-----------|-----------|----------|
| **yfinance** | Unlimited | N/A | Prices (already working ) |
| **yfinance** | Unlimited | N/A | Prices (already working ) |
| **House Stock Watcher** | Unlimited scraping | N/A | Free trades (best option) |
| **Quiver Free** | 500 calls/mo | $30/mo (5k calls) | Testing, not production |
| **FMP Free** | 250 calls/day | $15/mo | Alternative for trades |
@@ -222,5 +222,5 @@ class HouseWatcherClient:
}
```
Let me know if you want me to implement this scraper now for PR2! 🚀
Let me know if you want me to implement this scraper now for PR2!
+2 -2
View File
@@ -4,14 +4,14 @@
POTE can be deployed in several ways depending on your needs:
1. **Local Development** (SQLite) - What you have now
1. **Local Development** (SQLite) - What you have now
2. **Single Server** (PostgreSQL + cron jobs)
3. **Docker** (Containerized, easy to move)
4. **Cloud** (AWS/GCP/Azure with managed DB)
---
## Option 1: Local Development (Current Setup)
## Option 1: Local Development (Current Setup)
**You're already running this!**
+16 -16
View File
@@ -2,17 +2,17 @@
## Why Proxmox is Perfect for POTE
**Full control** - Your hardware, your rules
**No monthly costs** - Just electricity
**Isolated VMs/LXC** - Clean environments
**Snapshots** - Easy rollback if needed
**Resource efficient** - Run alongside other services
**Full control** - Your hardware, your rules
**No monthly costs** - Just electricity
**Isolated VMs/LXC** - Clean environments
**Snapshots** - Easy rollback if needed
**Resource efficient** - Run alongside other services
---
## Deployment Options on Proxmox
### Option 1: LXC Container (Recommended)
### Option 1: LXC Container (Recommended)
**Pros**: Lightweight, fast, efficient resource usage
**Cons**: Linux only (fine for POTE)
@@ -501,14 +501,14 @@ vs.
## Next Steps
1. Create LXC container
2. Install dependencies
3. Setup PostgreSQL
4. Deploy POTE
5. Configure cron jobs
6. Setup backups
7. Build Phase 2 (Analytics)
8. Add FastAPI dashboard (optional)
1. Create LXC container
2. Install dependencies
3. Setup PostgreSQL
4. Deploy POTE
5. Configure cron jobs
6. Setup backups
7. ⏭ Build Phase 2 (Analytics)
8. ⏭ Add FastAPI dashboard (optional)
---
@@ -583,7 +583,7 @@ EOF
sudo -u poteapp mkdir -p /home/poteapp/logs
echo ""
echo " Setup complete!"
echo " Setup complete!"
echo ""
echo "Next steps:"
echo "1. su - poteapp"
@@ -600,5 +600,5 @@ chmod +x proxmox_setup.sh
---
**Your Proxmox setup gives you enterprise-grade infrastructure at hobby costs!** 🚀
**Your Proxmox setup gives you enterprise-grade infrastructure at hobby costs!**
+5 -5
View File
@@ -137,13 +137,13 @@ python scripts/fetch_sample_prices.py
## Data Sources
### Currently Working:
- yfinance (prices, company info)
- Manual entry
- CSV import
- Fixture files (testing)
- yfinance (prices, company info)
- Manual entry
- CSV import
- Fixture files (testing)
### Currently Down:
- House Stock Watcher API (domain issues)
- House Stock Watcher API (domain issues)
### Future Options:
- QuiverQuant (requires $30/month subscription)
+19 -19
View File
@@ -8,10 +8,10 @@
### **Reality Check: No Real-Time Data Exists**
**Federal Law (STOCK Act):**
- 📅 Congress members have **30-45 days** to disclose trades
- 📅 Disclosures are filed as **Periodic Transaction Reports (PTRs)**
- 📅 Public databases update **after** filing (usually next day)
- 📅 **No real-time feed exists by design**
- Congress members have **30-45 days** to disclose trades
- Disclosures are filed as **Periodic Transaction Reports (PTRs)**
- Public databases update **after** filing (usually next day)
- **No real-time feed exists by design**
**Example Timeline:**
```
@@ -25,15 +25,15 @@ Feb 17, 2024 → Your system fetches it
Since trades appear in batches (not continuously), **running once per day is optimal**:
**Daily (7 AM)** - Catches overnight filings
**After market close** - Prices are final
**Low server load** - Off-peak hours
**Hourly** - Wasteful, no new data
**Real-time** - Impossible, not how disclosures work
**Daily (7 AM)** - Catches overnight filings
**After market close** - Prices are final
**Low server load** - Off-peak hours
**Hourly** - Wasteful, no new data
**Real-time** - Impossible, not how disclosures work
---
## 🤖 Automated Setup Options
## Automated Setup Options
### **Option 1: Cron Job (Linux/Proxmox) - Recommended**
@@ -149,7 +149,7 @@ Or from anywhere:
---
## 📊 What Gets Updated?
## What Gets Updated?
### **1. Congressional Trades**
**Script:** `fetch_congressional_trades.py`
@@ -182,7 +182,7 @@ Or from anywhere:
---
## ⚙️ Customizing the Schedule
## Customizing the Schedule
### **Different Frequencies**
@@ -215,7 +215,7 @@ Or from anywhere:
---
## 📧 Email Notifications (Optional)
## Email Notifications (Optional)
### **Setup Email Alerts**
@@ -310,7 +310,7 @@ python scripts/email_summary.py
---
## 🔍 Monitoring & Logging
## Monitoring & Logging
### **Check Cron Job Status**
@@ -355,7 +355,7 @@ Add to `/etc/logrotate.d/pote`:
---
## 🚨 Handling Failures
## Handling Failures
### **What If House Stock Watcher Is Down?**
@@ -363,7 +363,7 @@ The script is designed to continue even if one step fails:
```bash
# Script continues and logs warnings
⚠️ WARNING: Failed to fetch congressional trades
WARNING: Failed to fetch congressional trades
This is likely because House Stock Watcher API is down
Continuing with other steps...
```
@@ -399,7 +399,7 @@ for attempt in range(MAX_RETRIES):
---
## 📈 Performance Optimization
## Performance Optimization
### **Batch Processing**
@@ -439,7 +439,7 @@ CREATE INDEX IF NOT EXISTS ix_prices_security_id ON prices(security_id);
---
## 🎯 Recommended Setup
## Recommended Setup
### **For Proxmox Production:**
@@ -475,7 +475,7 @@ pote-update
---
## 📝 Summary
## Summary
### **Key Points:**
+67 -67
View File
@@ -1,13 +1,13 @@
# Live Market Monitoring + Congressional Trading Analysis
## 🎯 What's Possible vs Impossible
## What's Possible vs Impossible
### **NOT Possible:**
### **NOT Possible:**
- Identify WHO is buying/selling in real-time
- Match live trades to specific Congress members
- See congressional trades before they're disclosed
### **IS Possible:**
### **IS Possible:**
- Track unusual market activity in real-time
- Monitor stocks Congress members historically trade
- Compare unusual activity to later disclosures
@@ -15,7 +15,7 @@
---
## 🔄 **Two-Phase Monitoring System**
## **Two-Phase Monitoring System**
### **Phase 1: Real-Time Market Monitoring**
Monitor unusual activity in stocks Congress trades:
@@ -33,7 +33,7 @@ When disclosures come in:
---
## 📊 **Implementation: Watchlist-Based Monitoring**
## **Implementation: Watchlist-Based Monitoring**
### **Concept:**
@@ -57,38 +57,38 @@ Step 3: When Disclosure Appears (30-45 days later)
---
## 🛠️ **Data Sources for Live Market Monitoring**
## **Data Sources for Live Market Monitoring**
### **Free/Low-Cost Options:**
1. **Yahoo Finance (yfinance)**
- Real-time quotes (15-min delay free)
- Historical options data
- Volume data
- Not true real-time for options flow
- Real-time quotes (15-min delay free)
- Historical options data
- Volume data
- Not true real-time for options flow
2. **Unusual Whales API**
- Options flow data
- Unusual activity alerts
- 💰 Paid ($50-200/month)
- Options flow data
- Unusual activity alerts
- Paid ($50-200/month)
- https://unusualwhales.com/
3. **Tradier API**
- Real-time market data
- Options chains
- 💰 Paid but affordable ($10-50/month)
- Real-time market data
- Options chains
- Paid but affordable ($10-50/month)
- https://tradier.com/
4. **FlowAlgo**
- Options flow tracking
- Dark pool data
- 💰 Paid ($99-399/month)
- Options flow tracking
- Dark pool data
- Paid ($99-399/month)
- https://www.flowalgo.com/
5. **Polygon.io**
- Real-time stock data
- Options data
- 💰 Free tier + paid plans
- Real-time stock data
- Options data
- Free tier + paid plans
- https://polygon.io/
### **Best Free Option: Build Your Own with yfinance**
@@ -97,7 +97,7 @@ Track volume/price changes every 5 minutes for congressional watchlist tickers.
---
## 💡 **Practical Hybrid System**
## **Practical Hybrid System**
### **What We Can Build:**
@@ -141,37 +141,37 @@ for disclosure in new_disclosures:
---
## 📈 **Example: Nancy Pelosi NVDA Trade Analysis**
## **Example: Nancy Pelosi NVDA Trade Analysis**
### **Timeline:**
```
Nov 10, 2024:
🔔 ALERT: NVDA unusual call options activity
ALERT: NVDA unusual call options activity
Volume: 10x average
Strike: $500 (2 weeks out)
Nov 15, 2024:
💰 Someone buys NVDA (unknown who at the time)
Someone buys NVDA (unknown who at the time)
Nov 18, 2024:
📰 NVDA announces new AI chip
📈 Stock jumps 15%
NVDA announces new AI chip
Stock jumps 15%
Dec 15, 2024:
📋 Disclosure: Nancy Pelosi bought NVDA on Nov 15
Disclosure: Nancy Pelosi bought NVDA on Nov 15
Value: $15,001-$50,000
ANALYSIS:
She bought AFTER unusual options activity (Nov 10)
She bought BEFORE announcement (Nov 18)
Timing: 3 days before major news
🚩 Flag: Investigate if announcement was public knowledge
She bought AFTER unusual options activity (Nov 10)
She bought BEFORE announcement (Nov 18)
⏱ Timing: 3 days before major news
Flag: Investigate if announcement was public knowledge
```
---
## 🎯 **Recommended Approach**
## **Recommended Approach**
### **Phase 1: Build Congressional Ticker Watchlist**
@@ -230,12 +230,12 @@ def monitor_tickers(tickers, interval_minutes=5):
# Check for unusual volume
avg_volume = current['Volume'].mean()
if latest['Volume'] > avg_volume * 3:
alert(f"🔔 {ticker}: Unusual volume spike!")
alert(f" {ticker}: Unusual volume spike!")
# Check for price movement
price_change = (latest['Close'] - current['Open'].iloc[0]) / current['Open'].iloc[0]
if abs(price_change) > 0.05: # 5% move
alert(f"📈 {ticker}: {price_change:.2%} move today!")
alert(f" {ticker}: {price_change:.2%} move today!")
except Exception as e:
print(f"Error monitoring {ticker}: {e}")
@@ -290,31 +290,31 @@ def analyze_disclosure_timing(disclosure):
---
## 🚨 **Realistic Expectations**
## **Realistic Expectations**
### **What This System Will Do:**
Monitor stocks Congress members historically trade
Alert on unusual market activity in those stocks
Retroactively correlate disclosures with earlier alerts
Identify timing patterns and potential advantages
Build database of congressional trading patterns
Monitor stocks Congress members historically trade
Alert on unusual market activity in those stocks
Retroactively correlate disclosures with earlier alerts
Identify timing patterns and potential advantages
Build database of congressional trading patterns
### **What This System WON'T Do:**
Identify WHO is buying in real-time
Give you advance notice of congressional trades
Provide real-time inside information
Allow you to "front-run" Congress
Identify WHO is buying in real-time
Give you advance notice of congressional trades
Provide real-time inside information
Allow you to "front-run" Congress
### **Legal & Ethical:**
All data is public
Analysis is retrospective
For research and transparency
Not market manipulation
Cannot and should not be used to replicate potentially illegal trades
All data is public
Analysis is retrospective
For research and transparency
Not market manipulation
Cannot and should not be used to replicate potentially illegal trades
---
## 📊 **Proposed Implementation**
## **Proposed Implementation**
### **New Scripts to Create:**
@@ -365,15 +365,15 @@ CREATE TABLE disclosure_timing_analysis (
---
## 🎯 **Summary**
## **Summary**
### **Your Question:**
> "Can we read live trades being made and compare them to a name?"
### **Answer:**
**No** - Live trades are anonymous, can't identify individuals
**No** - Live trades are anonymous, can't identify individuals
**BUT** - You CAN:
**BUT** - You CAN:
1. Monitor unusual activity in stocks Congress trades
2. Log these alerts in real-time
3. When disclosures appear (30-45 days later), correlate them
@@ -381,25 +381,25 @@ CREATE TABLE disclosure_timing_analysis (
5. Build patterns database of timing and performance
### **This Gives You:**
- Transparency on timing advantages
- Pattern detection across officials
- Research-grade analysis
- Historical correlation data
- Transparency on timing advantages
- Pattern detection across officials
- Research-grade analysis
- Historical correlation data
### **This Does NOT Give You:**
- Real-time identity of traders
- Advance notice of congressional trades
- Ability to "front-run" disclosures
- Real-time identity of traders
- Advance notice of congressional trades
- Ability to "front-run" disclosures
---
## 🚀 **Would You Like Me To Build This?**
## **Would You Like Me To Build This?**
I can create:
1. Real-time monitoring system for congressional tickers
2. Alert logging and analysis
3. Timing correlation when disclosures appear
4. Pattern detection and reporting
1. Real-time monitoring system for congressional tickers
2. Alert logging and analysis
3. Timing correlation when disclosures appear
4. Pattern detection and reporting
This would be **Phase 2.5** of POTE - the "timing analysis" module.
+5 -5
View File
@@ -205,18 +205,18 @@ Output:
POTE HEALTH CHECK
============================================================
Timestamp: 2025-12-15T10:30:00
Overall Status: OK
Overall Status: OK
Database Connection: Database connection successful
Data Freshness: Data is fresh (2 days old)
Database Connection: Database connection successful
Data Freshness: Data is fresh (2 days old)
latest_trade_date: 2025-12-13
Data Counts: Database has 1,234 trades
Data Counts: Database has 1,234 trades
officials: 45
securities: 123
trades: 1,234
prices: 12,345
market_alerts: 567
Recent Alerts: 23 alerts in last 24 hours
Recent Alerts: 23 alerts in last 24 hours
============================================================
```
+35 -35
View File
@@ -22,18 +22,18 @@ chmod 600 .env
chown poteapp:poteapp .env
```
### Pros
### Pros
- Simple, works immediately
- No additional setup
- Standard practice for Python projects
### ⚠️ Cons
### Cons
- Secrets stored in plain text on disk
- Risk if server is compromised
- No audit trail
### 🔒 Security Checklist
- [ ] `.env` in `.gitignore` (already done )
### Security Checklist
- [ ] `.env` in `.gitignore` (already done )
- [ ] File permissions: `chmod 600 .env`
- [ ] Never commit to git
- [ ] Backup securely (encrypted)
@@ -76,12 +76,12 @@ sudo chmod 600 /etc/systemd/system/pote.service
sudo systemctl daemon-reload
```
### Pros
### Pros
- Secrets not in git or project directory
- Standard Linux practice
- Works with systemd timers
### ⚠️ Cons
### Cons
- Still visible in `systemctl show`
- Requires root to edit
@@ -128,12 +128,12 @@ source .env
python scripts/send_daily_report.py
```
### Pros
### Pros
- Secrets separate from code
- Easy to rotate
- Can be backed up separately
### ⚠️ Cons
### Cons
- Extra file to manage
- Still plain text
@@ -191,12 +191,12 @@ class Settings(BaseSettings):
smtp_password: str = Field(default_factory=lambda: get_secret("SMTP_PASSWORD"))
```
### Pros
### Pros
- Docker-native solution
- Encrypted in Swarm mode
- Never in logs
### ⚠️ Cons
### Cons
- Requires Docker
- More complex setup
@@ -228,13 +228,13 @@ secrets = client.secrets.kv.v2.read_secret_version(path='pote')
smtp_password = secrets['data']['data']['smtp_password']
```
### Pros
### Pros
- Centralized secrets management
- Audit logs
- Dynamic secrets
- Access control
### ⚠️ Cons
### Cons
- Complex setup
- Requires Vault infrastructure
- Overkill for single user
@@ -260,14 +260,14 @@ env:
DATABASE_URL: postgresql://user:${{ secrets.DB_PASSWORD }}@postgres/db
```
### ⚠️ Important
### Important
- **Only for CI/CD pipelines**
- **NOT for deployed servers**
- Secrets are injected during workflow runs
---
## 🎯 Recommendation for Your Setup
## Recommendation for Your Setup
### Personal/Research Use (Current)
@@ -305,9 +305,9 @@ gpg -c .env # Creates .env.gpg
---
## 🔒 General Security Best Practices
## General Security Best Practices
### DO
### DO
- Use strong, unique passwords
- Restrict file permissions (`chmod 600`)
@@ -316,7 +316,7 @@ gpg -c .env # Creates .env.gpg
- Use encrypted backups
- Audit who has server access
### DON'T
### DON'T
- Commit secrets to git (even private repos)
- Store passwords in code
@@ -327,7 +327,7 @@ gpg -c .env # Creates .env.gpg
---
## 🧪 Test Your Security
## Test Your Security
### Check if `.env` is protected
@@ -352,7 +352,7 @@ git log --all --full-history --source --pickaxe-all -S 'smtp_password'
---
## 🔄 Password Rotation Procedure
## Password Rotation Procedure
### Every 90 days (or if compromised):
@@ -369,32 +369,32 @@ git log --all --full-history --source --pickaxe-all -S 'smtp_password'
---
## 📊 Security Level Comparison
## Security Level Comparison
| Level | Method | Effort | Protection |
|-------|--------|--------|------------|
| 🔓 Basic | `.env` (default perms) | None | Low |
| 🔒 Good | `.env` (chmod 600) | 1 min | Medium |
| 🔒 Better | Environment variables | 10 min | Good |
| 🔒 Better | Separate secrets file | 10 min | Good |
| 🔐 Best | Docker Secrets | 30 min | Very Good |
| 🔐 Best | Vault | 2+ hours | Excellent |
| Basic | `.env` (default perms) | None | Low |
| Good | `.env` (chmod 600) | 1 min | Medium |
| Better | Environment variables | 10 min | Good |
| Better | Separate secrets file | 10 min | Good |
| Best | Docker Secrets | 30 min | Very Good |
| Best | Vault | 2+ hours | Excellent |
---
## 🎯 Your Current Status
## Your Current Status
**Already secure enough for personal use:**
- `.env` in `.gitignore`
- Not committed to git
- Local server only
**Already secure enough for personal use:**
- `.env` in `.gitignore`
- Not committed to git
- Local server only
⚠️ **Recommended improvement (2 minutes):**
**Recommended improvement (2 minutes):**
```bash
chmod 600 .env
```
🔐 **Optional (if paranoid):**
**Optional (if paranoid):**
- Use separate secrets file in `/etc/pote/`
- Encrypt backups with GPG
- Set up password rotation schedule
@@ -405,9 +405,9 @@ chmod 600 .env
**For your levkin.ca setup:**
1. **Current approach (`.env` file) is fine**
1. **Current approach (`.env` file) is fine**
2. **Add `chmod 600 .env`** for better security (2 minutes)
3. **Don't commit `.env` to git** (already protected )
3. **Don't commit `.env` to git** (already protected )
4. **Consider upgrading to environment variables** if you deploy to production
Your current setup is **appropriate for a personal research project**. Don't over-engineer it unless you have specific compliance requirements or a team.
+3 -3
View File
@@ -62,7 +62,7 @@ Follow the prompts:
2. Choose daily report time (recommend 6 AM)
3. Confirm
That's it! 🎉
That's it!
---
@@ -112,7 +112,7 @@ Run the daily script manually to test:
./scripts/automated_daily_run.sh
```
Check if email arrived! 📧
Check if email arrived!
---
@@ -244,5 +244,5 @@ Add to cron for regular health checks:
---
**You're all set! POTE will now run automatically and send you daily/weekly reports. 🚀**
**You're all set! POTE will now run automatically and send you daily/weekly reports. **
+35 -35
View File
@@ -1,17 +1,17 @@
# POTE Deployment & Automation Guide
## 🎯 Quick Answer to Your Questions
## Quick Answer to Your Questions
### After Deployment, What Happens?
**By default: NOTHING automatic happens.** You need to set up automation.
The deployed system is:
- Running (database, code installed)
- Accessible via SSH at your Proxmox IP
- NOT fetching data automatically
- NOT sending reports automatically
- NOT monitoring markets automatically
- Running (database, code installed)
- Accessible via SSH at your Proxmox IP
- NOT fetching data automatically
- NOT sending reports automatically
- NOT monitoring markets automatically
**You must either:**
1. **Run scripts manually** when you want updates, OR
@@ -19,7 +19,7 @@ The deployed system is:
---
## 🚀 Option 1: Automated Email Reports (Recommended)
## Option 1: Automated Email Reports (Recommended)
### What You Get
@@ -53,13 +53,13 @@ Run the interactive setup:
Follow prompts:
1. Enter your email address
2. Choose report time (default: 6 AM)
3. Done!
3. Done!
**See full guide:** [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md)
---
## 📍 Option 2: Access Reports via IP (No Email)
## Option 2: Access Reports via IP (No Email)
If you don't want email, you can:
@@ -107,7 +107,7 @@ python scripts/health_check.py
---
## 🌐 Option 3: Build a Web Interface (Future)
## Option 3: Build a Web Interface (Future)
Currently, POTE is **command-line only**. No web UI yet.
@@ -122,16 +122,16 @@ For now, use SSH or email reports.
---
## 🔄 Do You Need CI/CD Pipelines?
## Do You Need CI/CD Pipelines?
### What the Pipeline Does
The included CI/CD pipeline (`.github/workflows/ci.yml`) runs on **every git push**:
1. Lint & test (93 tests)
2. Security scanning
3. Dependency scanning
4. Docker build test
1. Lint & test (93 tests)
2. Security scanning
3. Dependency scanning
4. Docker build test
### Should You Use It?
@@ -174,18 +174,18 @@ docker build -t pote:test .
---
## 📊 Comparison of Options
## Comparison of Options
| Method | Pros | Cons | Best For |
|--------|------|------|----------|
| **Automated Email** | Convenient<br> No SSH needed<br> Daily/weekly updates | Requires SMTP setup | Most users |
| **SSH + Manual Scripts** | Full control<br> No email needed | Manual work<br> Must remember to run | Power users |
| **Saved Reports (SSH access)** | Automated<br> No email | Must SSH to view | Users without email |
| **Web Interface** | User-friendly | Not implemented yet | Future |
| **Automated Email** | Convenient<br> No SSH needed<br> Daily/weekly updates | Requires SMTP setup | Most users |
| **SSH + Manual Scripts** | Full control<br> No email needed | Manual work<br> Must remember to run | Power users |
| **Saved Reports (SSH access)** | Automated<br> No email | Must SSH to view | Users without email |
| **Web Interface** | User-friendly | Not implemented yet | Future |
---
## 🛠️ Your Ansible Pipeline
## Your Ansible Pipeline
Your existing Ansible CI/CD pipeline is **NOT directly usable** for POTE because:
@@ -197,11 +197,11 @@ Your existing Ansible CI/CD pipeline is **NOT directly usable** for POTE because
**Concepts from your pipeline that ARE used in POTE's CI/CD:**
- Security scanning (Trivy, Bandit instead of Gitleaks)
- Dependency scanning (Trivy instead of npm audit)
- SAST scanning (Bandit instead of Semgrep)
- Container scanning (Docker build test)
- Workflow summary generation
- Security scanning (Trivy, Bandit instead of Gitleaks)
- Dependency scanning (Trivy instead of npm audit)
- SAST scanning (Bandit instead of Semgrep)
- Container scanning (Docker build test)
- Workflow summary generation
**The POTE pipeline (`.github/workflows/ci.yml`) already includes all of these!**
@@ -217,7 +217,7 @@ Your Gitea server can run the POTE pipeline using Gitea Actions:
---
## 📧 Email Setup Examples
## Email Setup Examples
### Gmail (Most Common)
@@ -258,7 +258,7 @@ REPORT_RECIPIENTS=admin@yourdomain.com
---
## Recommended Setup for Most Users
## Recommended Setup for Most Users
1. **Deploy to Proxmox** (5 min)
```bash
@@ -283,7 +283,7 @@ REPORT_RECIPIENTS=admin@yourdomain.com
---
## 🔍 Monitoring & Health Checks
## Monitoring & Health Checks
### Add System Health Monitoring
@@ -310,17 +310,17 @@ python scripts/health_check.py
---
## 📚 Full Documentation
## Full Documentation
- **Automation Setup**: [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md)
- **Deployment**: [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md)
- **Usage**: [`QUICKSTART.md`](QUICKSTART.md)
- **Automation Setup**: [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md)
- **Deployment**: [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md)
- **Usage**: [`QUICKSTART.md`](QUICKSTART.md)
- **Detailed Automation Guide**: [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md)
- **Monitoring System**: [`MONITORING_SYSTEM_COMPLETE.md`](MONITORING_SYSTEM_COMPLETE.md)
---
## 🎉 Summary
## Summary
**After deployment:**
- Reports are NOT sent automatically by default
@@ -334,5 +334,5 @@ python scripts/health_check.py
1. Deploy to Proxmox
2. Run `./scripts/setup_cron.sh`
3. Receive daily/weekly email reports
4. Done! 🚀
4. Done!
+10 -10
View File
@@ -2,7 +2,7 @@
Homelab POTE sends via Mailcow **`mail.levkine.ca`** using the shared **`alerts@levkine.ca`** mailbox (same as Kuma/Beszel). See ansible `docs/guides/smtp-inventory.md`.
## Configuration Done
## Configuration Done
The `.env` file has been created with these settings:
@@ -15,7 +15,7 @@ FROM_EMAIL=alerts@levkine.ca
REPORT_RECIPIENTS=idobkin@gmail.com
```
## 🔑 Next Steps
## Next Steps
### 1. Add Your Password
@@ -38,7 +38,7 @@ python scripts/send_daily_report.py --to test@levkin.ca --test-smtp
If successful, you'll see:
```
SMTP connection test successful!
Daily report sent successfully!
Daily report sent successfully!
```
And you should receive a test email at `test@levkin.ca`!
@@ -57,7 +57,7 @@ This will:
- Schedule daily reports (default: 6 AM)
- Schedule weekly reports (Sundays at 8 AM)
## 📧 Email Server Details (For Reference)
## Email Server Details (For Reference)
Based on your Thunderbird setup:
@@ -76,10 +76,10 @@ Based on your Thunderbird setup:
POTE only uses **SMTP (outgoing)** to send reports.
## 🔒 Security Notes
## Security Notes
1. **Never commit `.env` to git!**
- Already in `.gitignore`
- Already in `.gitignore`
- Contains sensitive password
2. **Password Security:**
@@ -92,7 +92,7 @@ POTE only uses **SMTP (outgoing)** to send reports.
chmod 600 .env # Only owner can read/write
```
## 🎯 Change Recipients
## Change Recipients
To send reports to different email addresses (not just test@levkin.ca):
@@ -109,14 +109,14 @@ python scripts/send_daily_report.py --to someone-else@example.com
# The FROM address will still be test@levkin.ca
```
## Testing Checklist
## Testing Checklist
- [ ] Updated `.env` with your actual password
- [ ] Run `python scripts/send_daily_report.py --to test@levkin.ca --test-smtp`
- [ ] Checked inbox at test@levkin.ca (check spam folder!)
- [ ] If successful, run `./scripts/setup_cron.sh` to automate
## 🐛 Troubleshooting
## Troubleshooting
### Error: "SMTP connection failed"
@@ -139,5 +139,5 @@ Check:
---
**You're all set! POTE will send reports from test@levkin.ca 📧**
**You're all set! POTE will send reports from test@levkin.ca **
+7 -7
View File
@@ -2,14 +2,14 @@
## TL;DR: You can test everything for $0
### Already Working (PR1 )
### Already Working (PR1 )
- **Price data**: `yfinance` (free, unlimited)
- **Unit tests**: Mocked data in `tests/` (15 passing tests)
- **Coverage**: 87% without any paid APIs
### For PR2 (Congressional Trades) - FREE Options
#### Best Option: House Stock Watcher 🌟
#### Best Option: House Stock Watcher
```bash
# No API key needed, just scrape their public JSON
curl https://housestockwatcher.com/api/all_transactions
@@ -44,10 +44,10 @@ python scripts/fetch_house_watcher_sample.py # We'll build this in PR2
### What You DON'T Need to Pay For
QuiverQuant Pro ($30/mo) - free tier is enough for dev/testing
Financial Modeling Prep paid tier - free tier works
Any paid database hosting - SQLite works great locally
Any cloud services - runs 100% locally
QuiverQuant Pro ($30/mo) - free tier is enough for dev/testing
Financial Modeling Prep paid tier - free tier works
Any paid database hosting - SQLite works great locally
Any cloud services - runs 100% locally
### When You MIGHT Want Paid (Way Later)
@@ -56,7 +56,7 @@ python scripts/fetch_house_watcher_sample.py # We'll build this in PR2
- Multiple concurrent users on a dashboard
- Commercial use (check each API's terms)
**For personal research? Stay free forever. 🎉**
**For personal research? Stay free forever. **
---
+58 -58
View File
@@ -1,20 +1,20 @@
# 🔐 Gitea Secrets Guide for POTE
# Gitea Secrets Guide for POTE
## YES! You Can Store Passwords in Gitea
## YES! You Can Store Passwords in Gitea
Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you store passwords securely and use them in:
1. **CI/CD pipelines** (Gitea Actions workflows)
2. **Deployment workflows**
1. **CI/CD pipelines** (Gitea Actions workflows)
2. **Deployment workflows**
**BUT NOT:**
- Directly in your running application on Proxmox
- Accessed by scripts outside of workflows
- Directly in your running application on Proxmox
- Accessed by scripts outside of workflows
---
## 🎯 What Gitea Secrets Are Good For
## What Gitea Secrets Are Good For
### Perfect Use Cases
### Perfect Use Cases
1. **CI/CD Testing** - Run tests with real credentials
2. **Automated Deployment** - Deploy to Proxmox with SSH keys
@@ -22,7 +22,7 @@ Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you stor
4. **Docker Registry** - Push images with credentials
5. **API Keys** - Access external services during builds
### NOT Good For
### NOT Good For
1. **Runtime secrets** - Your deployed app on Proxmox can't access them
2. **Local development** - Can't use secrets on your laptop
@@ -30,7 +30,7 @@ Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you stor
---
## 🔧 How to Set Up Gitea Secrets
## How to Set Up Gitea Secrets
### Step 1: Add Secrets to Gitea
@@ -57,7 +57,7 @@ Secrets are accessed with `${{ secrets.SECRET_NAME }}` syntax.
---
## 📝 Example: CI Pipeline with Secrets
## Example: CI Pipeline with Secrets
**File:** `.github/workflows/ci.yml`
@@ -92,11 +92,11 @@ jobs:
--smtp-password "${{ secrets.SMTP_PASSWORD }}"
```
** I've already updated your CI pipeline to use secrets!**
** I've already updated your CI pipeline to use secrets!**
---
## 🚀 Example: Automated Deployment Workflow
## Example: Automated Deployment Workflow
Create `.github/workflows/deploy.yml`:
@@ -152,7 +152,7 @@ jobs:
---
## 🔄 How Secrets Flow to Your Server
## How Secrets Flow to Your Server
### Option 1: Deploy Workflow Updates `.env` (Recommended)
@@ -201,11 +201,11 @@ jobs:
---
## 🎯 Recommended Setup for Your POTE Project
## Recommended Setup for Your POTE Project
### For CI/CD (Testing):
**Use Gitea Secrets**
**Use Gitea Secrets**
```yaml
# .github/workflows/ci.yml (already updated!)
@@ -216,7 +216,7 @@ env:
### For Deployed Server (Proxmox):
**Keep using `.env` file**
**Keep using `.env` file**
Why?
- Simpler for manual SSH access
@@ -227,7 +227,7 @@ Why?
---
## 🚀 Complete Workflow: Gitea → Proxmox
## Complete Workflow: Gitea → Proxmox
### 1. Store Secrets in Gitea
@@ -264,27 +264,27 @@ git push origin main
---
## ⚠️ Important Limitations
## Important Limitations
### Gitea Secrets CAN'T:
Be accessed outside of workflows
Be used in local `python script.py` runs
Be read by cron jobs on Proxmox (directly)
Replace `.env` for runtime application config
Be accessed outside of workflows
Be used in local `python script.py` runs
Be read by cron jobs on Proxmox (directly)
Replace `.env` for runtime application config
### Gitea Secrets CAN:
Secure your CI/CD pipeline
Deploy safely without exposing passwords in git
Update `.env` on server during deployment
Run automated tests with real credentials
Secure your CI/CD pipeline
Deploy safely without exposing passwords in git
Update `.env` on server during deployment
Run automated tests with real credentials
---
## 🔒 Security Best Practices
## Security Best Practices
### DO:
### DO:
1. **Store ALL sensitive data as Gitea secrets**
- SMTP passwords
@@ -300,10 +300,10 @@ git push origin main
3. **Never echo secrets**
```yaml
# ❌ BAD - exposes in logs
# BAD - exposes in logs
- run: echo "${{ secrets.PASSWORD }}"
# ✅ GOOD - masked automatically
# GOOD - masked automatically
- run: use_password "${{ secrets.PASSWORD }}"
```
@@ -311,7 +311,7 @@ git push origin main
- Update in Gitea UI
- Re-run deployment workflow
### DON'T:
### DON'T:
1. **Commit secrets to git** (even private repos)
2. **Share secrets via Slack/email**
@@ -320,18 +320,18 @@ git push origin main
---
## 📊 Comparison: Where to Store Secrets
## Comparison: Where to Store Secrets
| Storage | CI/CD | Deployed App | Easy Updates | Security |
|---------|-------|--------------|--------------|----------|
| **Gitea Secrets** | Perfect | No | Via workflow | ⭐⭐⭐⭐⭐ |
| **`.env` file** | No | Perfect | `nano .env` | ⭐⭐⭐ |
| **Environment Vars** | Yes | Yes | Harder | ⭐⭐⭐⭐ |
| **Both (Recommended)** | Yes | Yes | Automated | ⭐⭐⭐⭐⭐ |
| **Gitea Secrets** | Perfect | No | Via workflow | |
| **`.env` file** | No | Perfect | `nano .env` | |
| **Environment Vars** | Yes | Yes | Harder | |
| **Both (Recommended)** | Yes | Yes | Automated | |
---
## 🎯 My Recommendation for You
## My Recommendation for You
### Use BOTH:
@@ -345,21 +345,21 @@ git push origin main
2. Commit code changes
3. Push to Gitea
4. Workflow runs:
- Tests with Gitea secrets
- Deploys to Proxmox
- Updates .env with secrets
5. Proxmox app reads from .env
- Tests with Gitea secrets
- Deploys to Proxmox
- Updates .env with secrets
5. Proxmox app reads from .env
```
**This gives you:**
- Secure CI/CD
- Easy manual SSH access
- Automated deployments
- No passwords in git
- Secure CI/CD
- Easy manual SSH access
- Automated deployments
- No passwords in git
---
## 🚀 Next Steps
## Next Steps
### 1. Add Secrets to Gitea (5 minutes)
@@ -387,7 +387,7 @@ I can create `.github/workflows/deploy.yml` if you want automated deployments!
---
## 💡 Quick Commands
## Quick Commands
### Add SSH Key to Gitea (for deployment):
@@ -409,12 +409,12 @@ git commit --allow-empty -m "Test secrets"
git push
# Check Gitea Actions tab
# Look for green checkmarks
# Look for green checkmarks
```
---
## 📚 See Also
## See Also
- **[docs/13_secrets_management.md](docs/13_secrets_management.md)** - All secrets options
- **[.github/workflows/ci.yml](.github/workflows/ci.yml)** - Updated with secrets support
@@ -422,16 +422,16 @@ git push
---
## Summary
## Summary
**YES, use Gitea secrets!** They're perfect for:
- CI/CD pipelines
- Automated deployments
- Keeping passwords out of git
- CI/CD pipelines
- Automated deployments
- Keeping passwords out of git
**But ALSO keep `.env` on Proxmox** for:
- Runtime application config
- Manual SSH access
- Cron jobs
- Runtime application config
- Manual SSH access
- Cron jobs
**Best of both worlds:** Gitea secrets deploy and update the `.env` file automatically! 🚀
**Best of both worlds:** Gitea secrets deploy and update the `.env` file automatically!
+38 -38
View File
@@ -1,6 +1,6 @@
# Local Testing Guide for POTE
## Testing Locally Before Deployment
## Testing Locally Before Deployment
### Quick Test - Run Full Suite
@@ -10,18 +10,18 @@ source venv/bin/activate
pytest -v
```
**Expected Result:** All 55 tests should pass
**Expected Result:** All 55 tests should pass
---
## 📊 Current Data Status
## Current Data Status
### Live Data Status: **NOT LIVE YET**
### Live Data Status: **NOT LIVE YET**
**Why?**
- 🔴 **House Stock Watcher API is DOWN** (domain issues, unreachable)
- 🟢 **yfinance works** (for price data)
- 🟡 **Sample data available** (5 trades from fixtures)
- **House Stock Watcher API is DOWN** (domain issues, unreachable)
- **yfinance works** (for price data)
- **Sample data available** (5 trades from fixtures)
### What Data Do You Have?
@@ -40,7 +40,7 @@ This will show:
---
## 🧪 Testing Analytics Locally
## Testing Analytics Locally
### 1. Unit Tests (Fast, No External Dependencies)
@@ -53,10 +53,10 @@ pytest tests/test_analytics_integration.py -v
```
These tests:
- Create synthetic price data
- Simulate trades with known returns
- Verify calculations are correct
- Test edge cases (missing data, sell trades, etc.)
- Create synthetic price data
- Simulate trades with known returns
- Verify calculations are correct
- Test edge cases (missing data, sell trades, etc.)
### 2. Manual Test with Local Database
@@ -92,27 +92,27 @@ calc = ReturnCalculator(session)
---
## 📦 What Gets Tested?
## What Gets Tested?
### Core Functionality (All Working )
### Core Functionality (All Working )
1. **Database Models** - Officials, Securities, Trades, Prices
2. **Data Ingestion** - Trade loading, security enrichment
3. **Analytics Engine** - Returns, benchmarks, metrics
4. **Edge Cases** - Missing data, sell trades, disclosure lags
### Integration Tests Cover:
- Return calculations over multiple time windows (30/60/90/180 days)
- Benchmark comparisons (stock vs SPY/QQQ)
- Abnormal return (alpha) calculations
- Official performance summaries
- Sector analysis
- Disclosure timing analysis
- Top performer rankings
- System-wide statistics
- Return calculations over multiple time windows (30/60/90/180 days)
- Benchmark comparisons (stock vs SPY/QQQ)
- Abnormal return (alpha) calculations
- Official performance summaries
- Sector analysis
- Disclosure timing analysis
- Top performer rankings
- System-wide statistics
---
## 🔄 Getting Live Data
## Getting Live Data
### Option 1: Wait for House Stock Watcher API
The API is currently down. Once it's back up:
@@ -164,7 +164,7 @@ export QUIVER_API_KEY="your_key_here"
---
## 📈 After Adding Data, Fetch Prices
## After Adding Data, Fetch Prices
```bash
# This will fetch prices for all securities in your database
@@ -176,12 +176,12 @@ python scripts/enrich_securities.py
---
## 🎯 Complete Local Test Workflow
## Complete Local Test Workflow
```bash
# 1. Run all tests
pytest -v
# All 55 tests should pass
# All 55 tests should pass
# 2. Check local database
python -c "
@@ -210,7 +210,7 @@ python scripts/calculate_all_returns.py --window 90
---
## 🚀 Deploy to Proxmox
## Deploy to Proxmox
Once local tests pass:
@@ -240,7 +240,7 @@ alembic upgrade head
---
## 🐛 Common Issues
## Common Issues
### "No price data found"
**Fix:** Run `python scripts/fetch_sample_prices.py`
@@ -260,7 +260,7 @@ sudo -u postgres psql -c "\l"
---
## 📊 Test Coverage
## Test Coverage
Run tests with coverage report:
@@ -277,19 +277,19 @@ firefox htmlcov/index.html # View coverage report
---
## Summary
## Summary
**Before Deploying:**
1. Run `pytest -v` - all tests pass
2. Run `make lint` - no errors
3. Test locally with sample data
4. Verify analytics work with synthetic prices
1. Run `pytest -v` - all tests pass
2. Run `make lint` - no errors
3. Test locally with sample data
4. Verify analytics work with synthetic prices
**Getting Live Data:**
- 🔴 House Stock Watcher API is down (external issue)
- 🟢 Manual CSV import works NOW
- 🟢 yfinance for prices works NOW
- 🟡 QuiverQuant available (requires free API key)
- House Stock Watcher API is down (external issue)
- Manual CSV import works NOW
- yfinance for prices works NOW
- QuiverQuant available (requires free API key)
**You can deploy and use the system NOW with:**
- Manual data entry
+16 -16
View File
@@ -1,6 +1,6 @@
# Offline Demo - Works Without Internet!
## Full System Working Without Network Access
## Full System Working Without Network Access
Even though your environment doesn't have external internet, **everything works perfectly** using fixture files.
@@ -10,9 +10,9 @@ Even though your environment doesn't have external internet, **everything works
python scripts/ingest_from_fixtures.py
# Output:
# Officials created/updated: 4
# Securities created/updated: 2
# Trades ingested: 5
# Officials created/updated: 4
# Securities created/updated: 2
# Trades ingested: 5
#
# Database totals:
# Total officials: 4
@@ -33,7 +33,7 @@ python scripts/ingest_from_fixtures.py
- NVDA, MSFT, AAPL, TSLA, GOOGL tickers
2. **Offline Scripts**
- `scripts/ingest_from_fixtures.py` - Ingest sample trades ( works now!)
- `scripts/ingest_from_fixtures.py` - Ingest sample trades ( works now!)
- `scripts/fetch_sample_prices.py` - Would need network (yfinance)
3. **28 Passing Tests** - All use mocks, no network required
@@ -67,14 +67,14 @@ with SessionLocal() as session:
### What You Can Do Offline
**Run all tests**: `make test`
**Ingest fixture data**: `python scripts/ingest_from_fixtures.py`
**Query the database**: Use Python REPL or SQLite browser
**Lint & format**: `make lint format`
**Run migrations**: `make migrate`
**Build analytics** (Phase 2): All math/ML works offline!
**Run all tests**: `make test`
**Ingest fixture data**: `python scripts/ingest_from_fixtures.py`
**Query the database**: Use Python REPL or SQLite browser
**Lint & format**: `make lint format`
**Run migrations**: `make migrate`
**Build analytics** (Phase 2): All math/ML works offline!
**Can't do (needs network)**:
**Can't do (needs network)**:
- Fetch live congressional trades from House Stock Watcher
- Fetch stock prices from yfinance
- (But you can add more fixture files to simulate this!)
@@ -108,9 +108,9 @@ You can expand the fixtures for offline development:
## Summary
**The network error is not a problem!** The entire system is designed to work with:
- Fixtures for development/testing
- Real APIs for production (when network available)
- Same code paths for both
- Fixtures for development/testing
- Real APIs for production (when network available)
- Same code paths for both
This is **by design** - makes development fast and tests reliable! 🚀
This is **by design** - makes development fast and tests reliable!
+6 -6
View File
@@ -223,12 +223,12 @@ print(f"Win Rate: {pelosi_stats['win_rate']:.1%}")
## Success Criteria
- Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark
- Can generate official performance summaries
- All calculations tested and accurate
- Performance data stored efficiently
- Documentation complete
- Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark
- Can generate official performance summaries
- All calculations tested and accurate
- Performance data stored efficiently
- Documentation complete
## Timeline
+20 -20
View File
@@ -1,4 +1,4 @@
# Proxmox Quick Start
# Proxmox Quick Start
**Got Proxmox? Deploy POTE in 5 minutes!**
@@ -17,7 +17,7 @@ su - poteapp
cd pote && source venv/bin/activate
python scripts/ingest_from_fixtures.py
# Done!
# Done!
```
---
@@ -81,8 +81,8 @@ source venv/bin/activate
python scripts/ingest_from_fixtures.py
# Should see:
# Officials created: 4
# Trades ingested: 5
# Officials created: 4
# Trades ingested: 5
```
### 5. Setup Cron Jobs
@@ -96,7 +96,7 @@ crontab -e
15 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/enrich_securities.py >> /home/poteapp/logs/enrich.log 2>&1
```
### 6. Done! 🎉
### 6. Done!
Your POTE instance is now running and will:
- Fetch congressional trades daily at 6 AM
@@ -107,12 +107,12 @@ Your POTE instance is now running and will:
## What You Get
**Full PostgreSQL database**
**Automated daily updates** (via cron)
**Isolated environment** (LXC container)
**Easy backups** (Proxmox snapshots)
**Low resource usage** (~500MB RAM)
**Cost**: Just electricity (~$5-10/mo)
**Full PostgreSQL database**
**Automated daily updates** (via cron)
**Isolated environment** (LXC container)
**Easy backups** (Proxmox snapshots)
**Low resource usage** (~500MB RAM)
**Cost**: Just electricity (~$5-10/mo)
---
@@ -231,13 +231,13 @@ pip install -e .
## Next Steps
1. Container running
2. POTE installed
3. Data ingested
4. Setup Proxmox backups (Web UI → Datacenter → Backup)
5. Configure static IP (if needed)
6. Build Phase 2 analytics
7. Add FastAPI dashboard
1. Container running
2. POTE installed
3. Data ingested
4. ⏭ Setup Proxmox backups (Web UI → Datacenter → Backup)
5. ⏭ Configure static IP (if needed)
6. ⏭ Build Phase 2 analytics
7. ⏭ Add FastAPI dashboard
---
@@ -264,10 +264,10 @@ pct restart 100
---
**Your Proxmox = Enterprise infrastructure at hobby prices!** 🚀
**Your Proxmox = Enterprise infrastructure at hobby prices!**
Cost breakdown:
- Cloud VPS: $20/mo
- Your Proxmox: ~$10/mo (power)
- **Savings: $120/year**
- **Savings: $120/year**
+19 -19
View File
@@ -1,15 +1,15 @@
# POTE Quick Start Guide
## 🚀 Your System is Ready!
## Your System is Ready!
**Container IP**: Check with `ip addr show eth0 | grep "inet"`
**Database**: PostgreSQL on port 5432
**Username**: `poteuser`
**Password**: `changeme123` (⚠️ change in production!)
**Password**: `changeme123` ( change in production!)
---
## 📊 How to Use POTE
## How to Use POTE
### Option 1: Command Line (SSH into container)
@@ -51,7 +51,7 @@ with engine.connect() as conn:
---
## 🎯 Common Tasks
## Common Tasks
### 1. Check System Status
@@ -145,7 +145,7 @@ ORDER BY trade_count DESC;
---
## 📈 Example Workflows
## Example Workflows
### Workflow 1: Daily Update
@@ -237,7 +237,7 @@ print(f"Exported {len(df)} trades to trades_export.csv")
---
## 🔧 Maintenance
## Maintenance
### Update POTE Code
@@ -287,7 +287,7 @@ nano ~/pote/.env
---
## 🌐 Access Methods Summary
## Access Methods Summary
| Method | From Where | Command |
|--------|-----------|---------|
@@ -298,22 +298,22 @@ nano ~/pote/.env
---
## 📚 What Data Do You Have?
## What Data Do You Have?
Right now (Phase 1 complete):
- **Congressional trading data** (from House Stock Watcher)
- **Security information** (tickers, names, sectors)
- **Historical prices** (OHLCV data from yfinance)
- **Official profiles** (name, party, chamber, state)
- **Congressional trading data** (from House Stock Watcher)
- **Security information** (tickers, names, sectors)
- **Historical prices** (OHLCV data from yfinance)
- **Official profiles** (name, party, chamber, state)
Coming next (Phase 2):
- 📊 **Abnormal return calculations**
- 🤖 **Behavioral clustering**
- 🚨 **Research signals** (follow_research, avoid_risk, watch)
- **Abnormal return calculations**
- **Behavioral clustering**
- **Research signals** (follow_research, avoid_risk, watch)
---
## 🎓 Learning SQL for POTE
## Learning SQL for POTE
### Count Records
```sql
@@ -349,7 +349,7 @@ GROUP BY o.party;
---
## Troubleshooting
## Troubleshooting
### Can't connect remotely?
```bash
@@ -380,12 +380,12 @@ pip install -e .
---
## 🚀 Next Steps
## Next Steps
1. **Populate with real data**: Run `fetch_congressional_trades.py` regularly
2. **Set up cron job** for automatic daily updates
3. **Build analytics** (Phase 2) - abnormal returns, signals
4. **Create dashboard** (Phase 3) - web interface for exploration
Ready to build Phase 2 analytics? Just ask! 📈
Ready to build Phase 2 analytics? Just ask!
+31 -31
View File
@@ -1,15 +1,15 @@
# 🚀 POTE Quick Setup Card
# POTE Quick Setup Card
## 📍 Your Configuration
## Your Configuration
**Email Server:** `mail.levkin.ca`
**Email Account:** `test@levkin.ca`
**Database:** PostgreSQL (configured)
**Status:** Ready for deployment
**Status:** Ready for deployment
---
## 3-Step Setup
## 3-Step Setup
### Step 1: Add Your Password (30 seconds)
@@ -27,7 +27,7 @@ source venv/bin/activate
python scripts/send_daily_report.py --to test@levkin.ca --test-smtp
```
**Check test@levkin.ca inbox** - you should receive a test email!
**Check test@levkin.ca inbox** - you should receive a test email!
### Step 3: Automate (2 minutes)
@@ -37,13 +37,13 @@ python scripts/send_daily_report.py --to test@levkin.ca --test-smtp
# Choose time: 6 AM (recommended)
```
**Done!** 🎉 You'll now receive:
**Done!** You'll now receive:
- Daily reports at 6 AM
- Weekly reports on Sundays
---
## 📦 Deployment to Proxmox (5 minutes)
## Deployment to Proxmox (5 minutes)
### On Proxmox Host:
@@ -79,7 +79,7 @@ bash scripts/proxmox_setup.sh
---
## 🔍 Quick Commands
## Quick Commands
### On Deployed Server (SSH)
@@ -105,7 +105,7 @@ ls -lh ~/logs/*.txt
---
## 📧 Email Configuration (.env)
## Email Configuration (.env)
```env
SMTP_HOST=mail.levkin.ca
@@ -123,27 +123,27 @@ REPORT_RECIPIENTS=test@levkin.ca,user2@example.com,user3@example.com
---
## 📊 What You'll Receive
## What You'll Receive
### Daily Report (6 AM)
```
New congressional trades
Market alerts (unusual activity)
Suspicious timing detections
Summary statistics
New congressional trades
Market alerts (unusual activity)
Suspicious timing detections
Summary statistics
```
### Weekly Report (Sunday 8 AM)
```
Most active officials
Most traded securities
Repeat offenders
Pattern analysis
Most active officials
Most traded securities
Repeat offenders
Pattern analysis
```
---
## 🔧 Troubleshooting
## Troubleshooting
### Email Not Working?
@@ -191,19 +191,19 @@ tail -50 ~/logs/daily_run.log
---
## 📚 Documentation
## Documentation
| Document | Purpose |
|----------|---------|
| **[EMAIL_SETUP.md](EMAIL_SETUP.md)** | Your levkin.ca setup guide |
| **[DEPLOYMENT_AND_AUTOMATION.md](DEPLOYMENT_AND_AUTOMATION.md)** | Answers all questions |
| **[EMAIL_SETUP.md](EMAIL_SETUP.md)** | Your levkin.ca setup guide |
| **[DEPLOYMENT_AND_AUTOMATION.md](DEPLOYMENT_AND_AUTOMATION.md)** | Answers all questions |
| **[AUTOMATION_QUICKSTART.md](AUTOMATION_QUICKSTART.md)** | Quick automation guide |
| **[PROXMOX_QUICKSTART.md](PROXMOX_QUICKSTART.md)** | Proxmox deployment |
| **[QUICKSTART.md](QUICKSTART.md)** | Usage guide |
---
## Checklist
## Checklist
**Local Development:**
- [ ] `.env` file created with password
@@ -227,18 +227,18 @@ tail -50 ~/logs/daily_run.log
---
## 🎯 Your Current Status
## Your Current Status
**Code:** Complete (93 tests passing)
**Monitoring:** 3-phase system operational
**CI/CD:** Pipeline ready (.github/workflows/ci.yml)
**Email:** Configured for test@levkin.ca
**Code:** Complete (93 tests passing)
**Monitoring:** 3-phase system operational
**CI/CD:** Pipeline ready (.github/workflows/ci.yml)
**Email:** Configured for test@levkin.ca
⏳ **Deployment:** Ready to deploy to Proxmox
⏳ **Automation:** Ready to set up with `setup_cron.sh`
---
## 🚀 Next Action
## Next Action
**Right now (local testing):**
```bash
@@ -255,9 +255,9 @@ cd ~/pote
./scripts/setup_cron.sh
```
**That's it! 🎉**
**That's it! **
---
**Everything is ready - just add your password and test!** 📧
**Everything is ready - just add your password and test!**
+27 -27
View File
@@ -3,25 +3,25 @@
**Last Updated**: 2025-12-14
**Version**: Phase 1 Complete (PR1 + PR2)
## 🎉 What's Working Now
## What's Working Now
### Data Ingestion (FREE!)
**Congressional Trades**: Live ingestion from House Stock Watcher
**Stock Prices**: Daily OHLCV from yfinance
**Officials**: Auto-populated from trade disclosures
**Securities**: Auto-created, ready for enrichment
**Congressional Trades**: Live ingestion from House Stock Watcher
**Stock Prices**: Daily OHLCV from yfinance
**Officials**: Auto-populated from trade disclosures
**Securities**: Auto-created, ready for enrichment
### Database
**Schema**: Normalized (officials, securities, trades, prices, metrics stubs)
**Migrations**: Alembic configured and applied
**DB**: SQLite for dev, PostgreSQL-ready
**Schema**: Normalized (officials, securities, trades, prices, metrics stubs)
**Migrations**: Alembic configured and applied
**DB**: SQLite for dev, PostgreSQL-ready
### Code Quality
**Tests**: 28 passing (86% coverage)
**Linting**: ruff + mypy all green
**Format**: black applied consistently
**Tests**: 28 passing (86% coverage)
**Linting**: ruff + mypy all green
**Format**: black applied consistently
## 📊 Current Stats
## Current Stats
```bash
# Test Suite
@@ -43,7 +43,7 @@ All free/open-source:
- pytest (testing)
```
## 🚀 Quick Commands
## Quick Commands
### Fetch Live Data (FREE!)
```bash
@@ -75,7 +75,7 @@ make format # Format with black
make migrate # Run Alembic migrations
```
## 🏠 Deployment
## Deployment
**Your Proxmox?** Perfect! See [`docs/08_proxmox_deployment.md`](docs/08_proxmox_deployment.md) for:
- LXC container setup (lightweight, recommended)
@@ -90,7 +90,7 @@ Other options in [`docs/07_deployment.md`](docs/07_deployment.md):
- Railway/Fly.io - $5-15/mo
- AWS/GCP - $20-50/mo
## 📂 Project Structure
## Project Structure
```
pote/
@@ -138,7 +138,7 @@ pote/
└── fetch_sample_prices.py # Live price fetch
```
## 💰 Cost Breakdown
## Cost Breakdown
| Component | Cost | Notes |
|-----------|------|-------|
@@ -154,7 +154,7 @@ Optional paid upgrades (NOT needed):
- Financial Modeling Prep: $15/mo (250 calls/day free tier available)
- PostgreSQL hosting: $7+/mo (only if deploying)
## Completed PRs
## Completed PRs
### PR1: Project Scaffold + Price Loader
- [x] Project structure (`src/`, `tests/`, docs)
@@ -176,7 +176,7 @@ Optional paid upgrades (NOT needed):
**See**: [`docs/PR2_SUMMARY.md`](docs/PR2_SUMMARY.md)
## 📋 Next Steps (Phase 2 - Analytics)
## Next Steps (Phase 2 - Analytics)
### PR3: Security Enrichment
- [ ] Enrich securities table with yfinance (names, sectors, exchanges)
@@ -206,20 +206,20 @@ Optional paid upgrades (NOT needed):
**See**: [`docs/00_mvp.md`](docs/00_mvp.md) for full roadmap
## 🔬 Research-Only Reminder
## Research-Only Reminder
**This tool is for private research and transparency analysis only.**
- Not investment advice
- Not a trading system
- No claims about inside information
- Public data only
- Descriptive analytics
- Research transparency
- Not investment advice
- Not a trading system
- No claims about inside information
- Public data only
- Descriptive analytics
- Research transparency
See [`docs/04_safety_ethics.md`](docs/04_safety_ethics.md) for guardrails.
## 🤝 Contributing
## Contributing
This is a personal research project, but if you want to use it:
@@ -229,7 +229,7 @@ This is a personal research project, but if you want to use it:
4. `python scripts/fetch_congressional_trades.py --days 7`
5. Start exploring!
## 📄 License
## License
MIT License (for research/educational use only)
+39 -39
View File
@@ -1,12 +1,12 @@
# POTE Testing Status Report
**Date:** December 15, 2025
**Status:** All Systems Operational - Ready for Deployment
**Status:** All Systems Operational - Ready for Deployment
---
## 🎯 Test Suite Summary
## Test Suite Summary
### **55 Tests - All Passing **
### **55 Tests - All Passing **
```
Platform: Python 3.13.5, pytest-9.0.2
@@ -18,17 +18,17 @@ Coverage: ~85% overall
| Module | Tests | Status | Coverage |
|--------|-------|--------|----------|
| **Analytics** | 18 tests | PASS | 80% |
| **Models** | 7 tests | PASS | 90% |
| **Ingestion** | 14 tests | PASS | 85% |
| **Price Loader** | 8 tests | PASS | 90% |
| **Security Enricher** | 8 tests | PASS | 85% |
| **Analytics** | 18 tests | PASS | 80% |
| **Models** | 7 tests | PASS | 90% |
| **Ingestion** | 14 tests | PASS | 85% |
| **Price Loader** | 8 tests | PASS | 90% |
| **Security Enricher** | 8 tests | PASS | 85% |
---
## 📊 What's Been Tested?
## What's Been Tested?
### Core Database Operations
### Core Database Operations
- [x] Creating and querying Officials
- [x] Creating and querying Securities
- [x] Creating and querying Trades
@@ -36,7 +36,7 @@ Coverage: ~85% overall
- [x] Unique constraints and relationships
- [x] Database migrations (Alembic)
### Data Ingestion
### Data Ingestion
- [x] House Stock Watcher client (with fixtures)
- [x] Trade loading from JSON
- [x] Security enrichment from yfinance
@@ -44,7 +44,7 @@ Coverage: ~85% overall
- [x] Idempotent operations (no duplicates)
- [x] Error handling for missing/invalid data
### Analytics Engine
### Analytics Engine
- [x] Return calculations (buy trades)
- [x] Return calculations (sell trades)
- [x] Multiple time windows (30/60/90/180 days)
@@ -56,7 +56,7 @@ Coverage: ~85% overall
- [x] Top performer rankings
- [x] System-wide statistics
### Edge Cases
### Edge Cases
- [x] Missing price data handling
- [x] Trades with no exit price yet
- [x] Sell trades (inverted returns)
@@ -67,7 +67,7 @@ Coverage: ~85% overall
---
## 🧪 Test Types
## Test Types
### 1. Unit Tests (Fast, Isolated)
**Location:** `tests/test_*.py` (excluding integration)
@@ -100,7 +100,7 @@ Coverage: ~85% overall
---
## 🔧 How to Run Tests Locally
## How to Run Tests Locally
### Quick Test
```bash
@@ -136,7 +136,7 @@ ptw
---
## 🚨 Known Limitations
## Known Limitations
### 1. External API Dependency
**Issue:** House Stock Watcher API is currently DOWN
@@ -161,7 +161,7 @@ ptw
---
## 📈 Performance Benchmarks
## Performance Benchmarks
### Test Execution Time
- **Full suite:** 1.8 seconds
@@ -178,7 +178,7 @@ ptw
---
## 🎯 Pre-Deployment Checklist
## Pre-Deployment Checklist
### Before Deploying to Proxmox:
@@ -215,7 +215,7 @@ python scripts/enrich_securities.py
---
## 🔄 Continuous Testing
## Continuous Testing
### Git Pre-Commit Hook (Optional)
```bash
@@ -247,7 +247,7 @@ jobs:
---
## 📝 Test Maintenance
## Test Maintenance
### Adding New Tests
@@ -281,36 +281,36 @@ Fixtures are in `tests/conftest.py`:
---
## 🎉 Summary
## Summary
### Current Status: **PRODUCTION READY**
### Current Status: **PRODUCTION READY**
**What Works:**
- All 55 tests passing
- Full analytics pipeline functional
- Database operations solid
- Data ingestion from multiple sources
- Price fetching from yfinance
- Security enrichment
- Return calculations
- Benchmark comparisons
- Performance metrics
- CLI scripts operational
- All 55 tests passing
- Full analytics pipeline functional
- Database operations solid
- Data ingestion from multiple sources
- Price fetching from yfinance
- Security enrichment
- Return calculations
- Benchmark comparisons
- Performance metrics
- CLI scripts operational
**What's Missing:**
- Live congressional trade API (external issue - House Stock Watcher down)
- Live congressional trade API (external issue - House Stock Watcher down)
- **Workaround:** Manual import, CSV, or alternative APIs available
**Next Steps:**
1. Tests are complete
2. Code is ready
3. ➡️ **Deploy to Proxmox** (or continue with Phase 2 features)
4. ➡️ Add more data sources
5. ➡️ Build dashboard (Phase 3)
1. Tests are complete
2. Code is ready
3. **Deploy to Proxmox** (or continue with Phase 2 features)
4. Add more data sources
5. Build dashboard (Phase 3)
---
## 📞 Need Help?
## Need Help?
See:
- `LOCAL_TEST_GUIDE.md` - Detailed local testing instructions
+18 -18
View File
@@ -1,6 +1,6 @@
# POTE Watchlist & Trading Reports
## 🎯 Get Trading Reports 1 Hour Before Market Close
## Get Trading Reports 1 Hour Before Market Close
### Quick Setup
@@ -25,7 +25,7 @@ crontab -e
---
## 📋 Watchlist System
## Watchlist System
### Who's on the Default Watchlist?
@@ -48,7 +48,7 @@ crontab -e
---
## 🔧 Managing Your Watchlist
## Managing Your Watchlist
### View Current Watchlist
@@ -104,7 +104,7 @@ This fetches all 535 members of Congress (100 Senate + 435 House).
---
## 📊 Generating Reports
## Generating Reports
### Manual Report Generation
@@ -135,24 +135,24 @@ python scripts/generate_trading_report.py --format json --output report.json
================================================================================
────────────────────────────────────────────────────────────────────────────────
👤 Nancy Pelosi (D-CA, House)
Nancy Pelosi (D-CA, House)
────────────────────────────────────────────────────────────────────────────────
Side Ticker Company Sector Value Trade Date Filed
-------- ------ -------------------------- ---------- ------------------- ---------- ----------
🟢 BUY NVDA NVIDIA Corporation Technology $15,001 - $50,000 2024-11-15 2024-12-01
🔴 SELL MSFT Microsoft Corporation Technology $50,001 - $100,000 2024-11-20 2024-12-01
BUY NVDA NVIDIA Corporation Technology $15,001 - $50,000 2024-11-15 2024-12-01
SELL MSFT Microsoft Corporation Technology $50,001 - $100,000 2024-11-20 2024-12-01
────────────────────────────────────────────────────────────────────────────────
👤 Tommy Tuberville (R-AL, Senate)
Tommy Tuberville (R-AL, Senate)
────────────────────────────────────────────────────────────────────────────────
Side Ticker Company Sector Value Trade Date Filed
-------- ------ -------------------------- ---------- ------------------- ---------- ----------
🟢 BUY SPY SPDR S&P 500 ETF Financial $100,001 - $250,000 2024-11-18 2024-12-02
🟢 BUY AAPL Apple Inc. Technology $50,001 - $100,000 2024-11-22 2024-12-02
🔴 SELL TSLA Tesla, Inc. Automotive $15,001 - $50,000 2024-11-25 2024-12-02
BUY SPY SPDR S&P 500 ETF Financial $100,001 - $250,000 2024-11-18 2024-12-02
BUY AAPL Apple Inc. Technology $50,001 - $100,000 2024-11-22 2024-12-02
SELL TSLA Tesla, Inc. Automotive $15,001 - $50,000 2024-11-25 2024-12-02
================================================================================
📊 SUMMARY
SUMMARY
================================================================================
Total Trades: 5
@@ -223,7 +223,7 @@ Runs at 8 AM and 3 PM daily (weekdays).
---
## 📧 Email Reports (Optional)
## Email Reports (Optional)
### Setup Email Notifications
@@ -256,7 +256,7 @@ python scripts/send_email.py /tmp/report.html your-email@example.com
---
## 🎯 Typical Workflow
## Typical Workflow
### Daily Routine (3 PM ET)
@@ -282,7 +282,7 @@ python scripts/send_email.py /tmp/report.html your-email@example.com
---
## 🔍 Finding More Officials
## Finding More Officials
### Public Resources
@@ -318,7 +318,7 @@ Add committee members to your watchlist.
---
## 📈 Example Cron Setup
## Example Cron Setup
```bash
# Edit crontab
@@ -341,7 +341,7 @@ This gives you:
---
## 🚀 Quick Start Summary
## Quick Start Summary
```bash
# 1. Create watchlist
@@ -363,7 +363,7 @@ cat reports/trading_report_$(date +%Y%m%d).txt
---
## FAQ
## FAQ
**Q: Why are all the trades old (30-45 days)?**
**A:** Federal law (STOCK Act) gives Congress 30-45 days to file. This is normal.
@@ -1,8 +1,8 @@
# 🎉 POTE Monitoring System - ALL PHASES COMPLETE!
# POTE Monitoring System - ALL PHASES COMPLETE!
## **What Was Built (3 Phases)**
## **What Was Built (3 Phases)**
### **Phase 1: Real-Time Market Monitoring**
### **Phase 1: Real-Time Market Monitoring**
**Detects unusual market activity in congressional tickers**
**Features:**
@@ -20,11 +20,11 @@
- `MarketAlert` model - Database storage
- `monitor_market.py` - CLI tool
**Tests:** 14 passing
**Tests:** 14 passing
---
### **Phase 2: Disclosure Timing Correlation**
### **Phase 2: Disclosure Timing Correlation**
**Matches trades to prior market alerts when disclosures appear**
**Features:**
@@ -50,11 +50,11 @@
- `DisclosureCorrelator` - Correlation engine
- `analyze_disclosure_timing.py` - CLI tool
**Tests:** 13 passing
**Tests:** 13 passing
---
### **Phase 3: Pattern Detection & Rankings**
### **Phase 3: Pattern Detection & Rankings**
**Cross-official analysis and comparative rankings**
**Features:**
@@ -71,19 +71,19 @@
- `PatternDetector` - Pattern analysis engine
- `generate_pattern_report.py` - CLI tool
**Tests:** 11 passing
**Tests:** 11 passing
---
## 📊 **Complete System Architecture**
## **Complete System Architecture**
```
┌─────────────────────────────────────────────────────────────┐
│ PHASE 1: Real-Time Monitoring │
│ ──────────────────────────────────── │
🔔 Monitor congressional tickers
📊 Detect unusual activity
💾 Log alerts to database
│ Monitor congressional tickers │
│ Detect unusual activity │
│ Log alerts to database │
└─────────────────────────────────────────────────────────────┘
[30-45 days pass]
@@ -91,25 +91,25 @@
┌─────────────────────────────────────────────────────────────┐
│ PHASE 2: Disclosure Correlation │
│ ─────────────────────────────── │
📋 New congressional trades filed
🔗 Match to prior alerts
📈 Calculate timing scores
🚩 Flag suspicious trades
│ New congressional trades filed │
│ Match to prior alerts │
│ Calculate timing scores │
│ Flag suspicious trades │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ PHASE 3: Pattern Detection │
│ ────────────────────────── │
📊 Rank officials by timing
🔥 Identify repeat offenders
📈 Compare parties, sectors, tickers
📋 Generate comprehensive reports
│ Rank officials by timing │
│ Identify repeat offenders │
│ Compare parties, sectors, tickers │
│ Generate comprehensive reports │
└─────────────────────────────────────────────────────────────┘
```
---
## 🚀 **Usage Guide**
## **Usage Guide**
### **1. Set Up Monitoring (Run Daily)**
@@ -169,7 +169,7 @@ python scripts/generate_pattern_report.py --days 365 --format json --output patt
---
## 📋 **Example Reports**
## **Example Reports**
### **Timing Analysis Report**
@@ -179,7 +179,7 @@ python scripts/generate_pattern_report.py --days 365 --format json --output patt
3 Trades with Timing Advantages Detected
================================================================================
🚨 #1 - HIGHLY SUSPICIOUS (Timing Score: 85/100)
#1 - HIGHLY SUSPICIOUS (Timing Score: 85/100)
────────────────────────────────────────────────────────────────────────────────
Official: Nancy Pelosi
Ticker: NVDA
@@ -187,16 +187,16 @@ Side: BUY
Trade Date: 2024-01-15
Value: $15,001-$50,000
📊 Timing Analysis:
Timing Analysis:
Prior Alerts: 3
Recent Alerts (7d): 2
High Severity: 2
Avg Severity: 7.5/10
💡 Assessment: Trade occurred after 3 alerts, including 2 high-severity.
Assessment: Trade occurred after 3 alerts, including 2 high-severity.
High likelihood of timing advantage.
🔔 Prior Market Alerts:
Prior Market Alerts:
Timestamp Type Severity Timing
2024-01-12 10:30:00 Unusual Volume 8/10 3 days before
2024-01-13 14:15:00 Price Spike 7/10 2 days before
@@ -211,27 +211,27 @@ Timestamp Type Severity Timing
Period: 365 days
================================================================================
📊 SUMMARY
SUMMARY
────────────────────────────────────────────────────────────────────────────────
Officials Analyzed: 45
Repeat Offenders: 8
Average Timing Score: 42.3/100
🚨 TOP 10 MOST SUSPICIOUS OFFICIALS (By Timing Score)
TOP 10 MOST SUSPICIOUS OFFICIALS (By Timing Score)
================================================================================
Rank Official Party-State Chamber Trades Suspicious Rate Avg Score
──── ─────────────────────── ─────────── ─────── ────── ────────── ────── ─────────
🚨 1 Tommy Tuberville R-AL Senate 47 35/47 74.5% 72.5/100
🚨 2 Nancy Pelosi D-CA House 38 28/38 73.7% 71.2/100
🔴 3 Dan Crenshaw R-TX House 25 15/25 60.0% 65.8/100
🔴 4 Marjorie Taylor Greene R-GA House 19 11/19 57.9% 63.2/100
🟡 5 Josh Gottheimer D-NJ House 31 14/31 45.2% 58.7/100
1 Tommy Tuberville R-AL Senate 47 35/47 74.5% 72.5/100
2 Nancy Pelosi D-CA House 38 28/38 73.7% 71.2/100
3 Dan Crenshaw R-TX House 25 15/25 60.0% 65.8/100
4 Marjorie Taylor Greene R-GA House 19 11/19 57.9% 63.2/100
5 Josh Gottheimer D-NJ House 31 14/31 45.2% 58.7/100
🔥 REPEAT OFFENDERS (50%+ Suspicious Trades)
REPEAT OFFENDERS (50%+ Suspicious Trades)
================================================================================
🚨 Tommy Tuberville (R-AL, Senate)
Tommy Tuberville (R-AL, Senate)
Trades: 47 | Suspicious: 35 (74.5%)
Avg Timing Score: 72.5/100
Pattern: HIGHLY SUSPICIOUS - Majority of trades show timing advantage
@@ -239,9 +239,9 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 📈 **Test Coverage**
## **Test Coverage**
**Total: 93 tests, all passing **
**Total: 93 tests, all passing **
- **Phase 1 (Monitoring):** 14 tests
- **Phase 2 (Correlation):** 13 tests
@@ -252,7 +252,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 🎯 **Key Insights the System Provides**
## **Key Insights the System Provides**
### **1. Individual Official Analysis**
- Which officials consistently trade before unusual activity?
@@ -282,7 +282,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 🔧 **Automated Workflow**
## **Automated Workflow**
### **Daily Routine (Recommended)**
@@ -299,7 +299,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 📊 **Database Schema**
## **Database Schema**
**New Table: `market_alerts`**
```sql
@@ -315,7 +315,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 🎓 **Interpretation Guide**
## **Interpretation Guide**
### **Timing Scores**
- **80-100:** Highly suspicious - Multiple high-severity alerts before trade
@@ -337,15 +337,15 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## ⚠️ **Important Disclaimers**
## **Important Disclaimers**
### **Legal & Ethical**
1. All data is public and legally obtained
2. Analysis is retrospective (30-45 day lag)
3. For research and transparency only
4. NOT investment advice
5. NOT proof of illegal activity (requires investigation)
6. Statistical patterns ≠ legal evidence
1. All data is public and legally obtained
2. Analysis is retrospective (30-45 day lag)
3. For research and transparency only
4. NOT investment advice
5. NOT proof of illegal activity (requires investigation)
6. Statistical patterns ≠ legal evidence
### **Technical Limitations**
1. Cannot identify WHO is trading in real-time
@@ -356,7 +356,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 🚀 **Deployment Checklist**
## **Deployment Checklist**
### **On Proxmox Container**
@@ -383,7 +383,7 @@ python scripts/generate_pattern_report.py --days 365
---
## 📚 **Documentation**
## **Documentation**
- **`docs/11_live_market_monitoring.md`** - Deep dive into monitoring
- **`LOCAL_TEST_GUIDE.md`** - Testing instructions
@@ -392,23 +392,23 @@ python scripts/generate_pattern_report.py --days 365
---
## 🎉 **Achievement Unlocked!**
## **Achievement Unlocked!**
**You now have a complete system that:**
Monitors real-time market activity
Correlates trades to prior alerts
Calculates timing advantage scores
Identifies repeat offenders
Ranks officials by suspicion
Generates comprehensive reports
93 tests confirming it works
Monitors real-time market activity
Correlates trades to prior alerts
Calculates timing advantage scores
Identifies repeat offenders
Ranks officials by suspicion
Generates comprehensive reports
93 tests confirming it works
**This is a production-ready transparency and research tool!** 🚀
**This is a production-ready transparency and research tool!**
---
## 🔜 **Potential Future Enhancements**
## **Potential Future Enhancements**
### **Phase 4 Ideas (Optional)**
- Email/SMS alerts for high-severity patterns
@@ -420,6 +420,6 @@ python scripts/generate_pattern_report.py --days 365
- Automated PDF reports
- Historical performance tracking
**But the core system is COMPLETE and FUNCTIONAL now!**
**But the core system is COMPLETE and FUNCTIONAL now!**
@@ -1,6 +1,6 @@
# PR1 Summary: Project Scaffold + DB + Price Loader
**Status**: Complete
**Status**: Complete
**Date**: 2025-12-13
## What was built
@@ -36,7 +36,7 @@ Includes proper indexes, unique constraints, and relationships.
- `tests/conftest.py`: fixtures for in-memory DB, sample officials/securities/trades/prices
- `tests/test_models.py`: model creation, relationships, unique constraints, queries (7 tests)
- `tests/test_price_loader.py`: loader logic, idempotency, upsert, mocking yfinance (8 tests)
- **Result**: 15 tests, all passing
- **Result**: 15 tests, all passing
### 6. Tooling
- **Black** + **ruff** configured and run (all code formatted + linted)
@@ -1,6 +1,6 @@
# PR2 Summary: Congressional Trade Ingestion
**Status**: Complete
**Status**: Complete
**Date**: 2025-12-14
## What was built
@@ -26,7 +26,7 @@
- `tests/fixtures/sample_house_watcher.json`: 5 realistic sample transactions
- Includes House + Senate, Democrats + Republicans, various tickers
### 4. Tests (13 new tests, all passing )
### 4. Tests (13 new tests, all passing )
**`tests/test_house_watcher.py` (8 tests)**:
- Amount range parsing (with range, single value, invalid)
- Transaction type normalization
@@ -58,9 +58,9 @@
python scripts/fetch_congressional_trades.py --days 30
# Sample output:
# Officials created/updated: 47
# Securities created/updated: 89
# Trades ingested: 234
# Officials created/updated: 47
# Securities created/updated: 89
# Trades ingested: 234
```
### Database Queries
@@ -1,6 +1,6 @@
# PR3 Summary: Security Enrichment + Deployment
**Status**: Complete
**Status**: Complete
**Date**: 2025-12-14
## What was built
@@ -32,7 +32,7 @@
python scripts/enrich_securities.py --force
```
### 3. Tests (9 new tests, all passing )
### 3. Tests (9 new tests, all passing )
**`tests/test_security_enricher.py`**:
- Successful enrichment with complete data
- ETF detection and classification
@@ -72,7 +72,7 @@ python scripts/enrich_securities.py
# Enriched AAPL: Apple Inc. (Technology)
# Enriched TSLA: Tesla, Inc. (Consumer Cyclical)
# Enriched GOOGL: Alphabet Inc. (Communication Services)
# Successfully enriched: 5
# Successfully enriched: 5
```
### Query Enriched Data
@@ -159,10 +159,10 @@ python scripts/update_all_prices.py # To be built in PR4
| Option | Complexity | Cost/month | Best For |
|--------|-----------|------------|----------|
| **Local** | | $0 | Development |
| **VPS + Docker** | ⭐⭐ | $10-20 | Personal deployment |
| **Railway/Fly.io** | | $5-15 | Easy cloud |
| **AWS** | ⭐⭐⭐ | $20-50 | Scalable production |
| **Local** | | $0 | Development |
| **VPS + Docker** | | $10-20 | Personal deployment |
| **Railway/Fly.io** | | $5-15 | Easy cloud |
| **AWS** | | $20-50 | Scalable production |
See [`docs/07_deployment.md`](07_deployment.md) for detailed guides.
@@ -1,6 +1,6 @@
# PR4 Summary: Phase 2 Analytics Foundation
## Completed
## Completed
**Date**: December 15, 2025
**Status**: Complete
@@ -78,14 +78,14 @@ python scripts/calculate_all_returns.py --window 90 --benchmark SPY --top 10
### 3. Tests (`tests/test_analytics.py`)
- Return calculator with sample data
- Buy vs sell trade handling
- Missing data edge cases
- Benchmark comparisons
- Official performance metrics
- Multiple time windows
- Sector analysis
- Timing analysis
- Return calculator with sample data
- Buy vs sell trade handling
- Missing data edge cases
- Benchmark comparisons
- Official performance metrics
- Multiple time windows
- Sector analysis
- Timing analysis
**Test Coverage**: Analytics module fully tested
@@ -288,15 +288,15 @@ with next(get_session()) as session:
- Trades near policy events
- Unusual timing flags
## Success Criteria
## Success Criteria
- Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark
- Can generate official performance summaries
- All calculations tested and accurate
- Performance data calculated on-the-fly
- Documentation complete
- Command-line tools working
- Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark
- Can generate official performance summaries
- All calculations tested and accurate
- Performance data calculated on-the-fly
- Documentation complete
- Command-line tools working
## Testing
@@ -309,7 +309,7 @@ All analytics tests should pass (may have warnings if no price data).
---
**Phase 2 Analytics Foundation: COMPLETE**
**Phase 2 Analytics Foundation: COMPLETE**
**Ready for**: PR5 (Signals), PR6 (API), PR7 (Dashboard)
+3
View File
@@ -0,0 +1,3 @@
# Archive
PR summaries and one-shot status writeups. Prefer the numbered guides and QUICKSTART files.