Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready
6.9 KiB
POTE – Public Officials Trading Explorer
Research-only tool for tracking and analyzing public stock trades by government officials.
⚠️ Important: This project is for personal research and transparency analysis only. It is NOT for investment advice or live trading.
What is this?
POTE tracks stock trading activity of government officials (starting with U.S. Congress) using lawfully available public data sources. It computes research metrics, descriptive signals, and risk/ethics flags to help understand trading patterns.
Key constraints
- Public data only: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- Research framing: All outputs are descriptive analytics, not trading recommendations
- No inside information claims: We use public disclosures that may be delayed or incomplete
Current Status
✅ PR1 Complete: Project scaffold, DB models, price loader
✅ PR2 Complete: Congressional trade ingestion (House Stock Watcher)
✅ PR3 Complete: Security enrichment + deployment infrastructure
✅ PR4 Complete: Phase 2 analytics - returns, benchmarks, performance metrics
45+ passing tests, 88%+ coverage
Quick start
🚀 Already deployed? See QUICKSTART.md for full usage guide!
📦 Deploying? See PROXMOX_QUICKSTART.md for Proxmox LXC deployment (recommended).
📧 Want automated reports? See AUTOMATION_QUICKSTART.md for email reporting setup!
Local Development
# Install
git clone <your-repo>
cd pote
make install
source venv/bin/activate
# Run migrations
make migrate
# Ingest sample data (offline, for testing)
python scripts/ingest_from_fixtures.py
# Enrich securities with company info
python scripts/enrich_securities.py
# With internet:
python scripts/fetch_congressional_trades.py
python scripts/fetch_sample_prices.py
# Run tests
make test
# Lint & format
make lint format
Production Deployment
# Proxmox LXC (Recommended - 5 minutes)
bash scripts/proxmox_setup.sh
# Docker
docker-compose up -d
Tech stack
- Language: Python 3.10+
- Database: PostgreSQL or SQLite (dev)
- Data: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- Libraries: SQLAlchemy, Alembic, pandas, numpy, httpx, yfinance, scikit-learn
- Testing: pytest (28 tests, 87%+ coverage)
Documentation
Getting Started:
README.md– This fileQUICKSTART.md– ⭐ How to use your deployed POTE instanceSTATUS.md– Current project statusFREE_TESTING_QUICKSTART.md– Test for $0OFFLINE_DEMO.md– Works without internet!
Deployment:
PROXMOX_QUICKSTART.md– ⭐ Proxmox quick deployment (5 min)AUTOMATION_QUICKSTART.md– ⭐ Automated reporting setup (5 min)docs/07_deployment.md– Full deployment guide (all platforms)docs/08_proxmox_deployment.md– Proxmox detailed guidedocs/12_automation_and_reporting.md– Automation & CI/CD guideDockerfile+docker-compose.yml– Docker setup
Technical:
docs/00_mvp.md– MVP roadmapdocs/01_architecture.md– Architecturedocs/02_data_model.md– Database schemadocs/03_data_sources.md– Data sourcesdocs/04_safety_ethics.md– Research-only guardrailsdocs/05_dev_setup.md– Dev conventionsdocs/06_free_testing_data.md– Testing strategies
PR Summaries:
docs/PR1_SUMMARY.md– Scaffold + price loaderdocs/PR2_SUMMARY.md– Congressional tradesdocs/PR3_SUMMARY.md– Enrichment + deploymentdocs/PR4_SUMMARY.md– ⭐ Analytics foundation (returns, benchmarks, metrics)
What's Working Now
- ✅ SQLAlchemy models for officials, securities, trades, prices
- ✅ Alembic migrations
- ✅ Price loader with yfinance (idempotent, upsert)
- ✅ Congressional trade ingestion from House Stock Watcher (FREE!)
- ✅ Security enrichment (company names, sectors, industries)
- ✅ ETL to populate officials & trades tables
- ✅ Docker + deployment infrastructure
- ✅ 93 passing tests with 88%+ coverage
- ✅ Linting (ruff + mypy) all green
- ✅ Works 100% offline with fixtures
- ✅ Real-time market monitoring & alert system
- ✅ Disclosure timing correlation engine
- ✅ Pattern detection & comparative analysis
- ✅ Automated email reporting (daily/weekly)
- ✅ CI/CD pipeline (GitHub/Gitea Actions)
What You Can Do Now
Analyze Performance
# Analyze specific official
python scripts/analyze_official.py "Nancy Pelosi" --window 90
# System-wide analysis
python scripts/calculate_all_returns.py
Market Monitoring
# Run market scan
python scripts/monitor_market.py --scan
# Analyze timing of recent disclosures
python scripts/analyze_disclosure_timing.py --recent 7
# Generate pattern report
python scripts/generate_pattern_report.py --days 365
Automated Reporting
# Set up daily/weekly email reports (5 minutes!)
./scripts/setup_cron.sh
# Send manual report
python scripts/send_daily_report.py --to your@email.com
Add More Data
# Manual entry
python scripts/add_custom_trades.py
# CSV import
python scripts/scrape_alternative_sources.py import trades.csv
System Architecture
POTE now includes a complete 3-phase monitoring system:
Phase 1: Real-Time Market Monitoring
- Tracks ~50 most-traded congressional stocks
- Detects unusual volume, price spikes, volatility
- Logs all alerts with timestamps and severity
Phase 2: Disclosure Correlation
- Matches trades with prior market alerts (30-45 day lookback)
- Calculates "timing advantage score" (0-100)
- Identifies suspicious timing patterns
Phase 3: Pattern Detection
- Ranks officials by consistent suspicious timing
- Analyzes by ticker, sector, and political party
- Generates comprehensive reports
Full Documentation: See MONITORING_SYSTEM_COMPLETE.md
Next Steps
- Signals: "follow_research", "avoid_risk", "watch" with confidence scores
- Clustering: group officials by trading behavior patterns
- API: FastAPI backend for queries
- Dashboard: React/Streamlit visualization
See docs/00_mvp.md for the full roadmap.
License: MIT (for research/educational use only)
Disclaimer: Not investment advice. Use public data only. No claims about inside information.