ilia/POTE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
POTE/README.md
ilia 0d8d85adc1 Add complete automation, reporting, and CI/CD system 
Features Added:
==============

📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py

🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup

🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration

🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI

📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide

🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron

WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
   - New trades, market alerts, suspicious timing
   - Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)

USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh

# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com

# Check system health
python scripts/health_check.py

See AUTOMATION_QUICKSTART.md for full instructions.

93 tests passing | Full CI/CD | Email reports ready
2025-12-15 15:34:31 -05:00

203 lines
6.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# POTE – Public Officials Trading Explorer
**Research-only tool for tracking and analyzing public stock trades by government officials.**
⚠️ **Important**: This project is for personal research and transparency analysis only. It is **NOT** for investment advice or live trading.
## What is this?
POTE tracks stock trading activity of government officials (starting with U.S. Congress) using lawfully available public data sources. It computes research metrics, descriptive signals, and risk/ethics flags to help understand trading patterns.
## Key constraints
- **Public data only**: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- **Research framing**: All outputs are descriptive analytics, not trading recommendations
- **No inside information claims**: We use public disclosures that may be delayed or incomplete
## Current Status
✅ **PR1 Complete**: Project scaffold, DB models, price loader
✅ **PR2 Complete**: Congressional trade ingestion (House Stock Watcher)
✅ **PR3 Complete**: Security enrichment + deployment infrastructure
✅ **PR4 Complete**: Phase 2 analytics - returns, benchmarks, performance metrics
**45+ passing tests, 88%+ coverage**
## Quick start
**🚀 Already deployed?** See **[QUICKSTART.md](QUICKSTART.md)** for full usage guide!
**📦 Deploying?** See **[PROXMOX_QUICKSTART.md](PROXMOX_QUICKSTART.md)** for Proxmox LXC deployment (recommended).
**📧 Want automated reports?** See **[AUTOMATION_QUICKSTART.md](AUTOMATION_QUICKSTART.md)** for email reporting setup!
### Local Development
```bash
# Install
git clone <your-repo>
cd pote
make install
source venv/bin/activate
# Run migrations
make migrate
# Ingest sample data (offline, for testing)
python scripts/ingest_from_fixtures.py
# Enrich securities with company info
python scripts/enrich_securities.py
# With internet:
python scripts/fetch_congressional_trades.py
python scripts/fetch_sample_prices.py
# Run tests
make test
# Lint & format
make lint format
```
### Production Deployment
```bash
# Proxmox LXC (Recommended - 5 minutes)
bash scripts/proxmox_setup.sh
# Docker
docker-compose up -d
```
## Tech stack
- **Language**: Python 3.10+
- **Database**: PostgreSQL or SQLite (dev)
- **Data**: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- **Libraries**: SQLAlchemy, Alembic, pandas, numpy, httpx, yfinance, scikit-learn
- **Testing**: pytest (28 tests, 87%+ coverage)
## Documentation
**Getting Started**:
- [`README.md`](README.md) – This file
- [`QUICKSTART.md`](QUICKSTART.md) – ⭐ **How to use your deployed POTE instance**
- [`STATUS.md`](STATUS.md) – Current project status
- [`FREE_TESTING_QUICKSTART.md`](FREE_TESTING_QUICKSTART.md) – Test for $0
- [`OFFLINE_DEMO.md`](OFFLINE_DEMO.md) – Works without internet!
**Deployment**:
- [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md) – ⭐ **Proxmox quick deployment (5 min)**
- [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md) – ⭐ **Automated reporting setup (5 min)**
- [`docs/07_deployment.md`](docs/07_deployment.md) – Full deployment guide (all platforms)
- [`docs/08_proxmox_deployment.md`](docs/08_proxmox_deployment.md) – Proxmox detailed guide
- [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md) – Automation & CI/CD guide
- [`Dockerfile`](Dockerfile) + [`docker-compose.yml`](docker-compose.yml) – Docker setup
**Technical**:
- [`docs/00_mvp.md`](docs/00_mvp.md) – MVP roadmap
- [`docs/01_architecture.md`](docs/01_architecture.md) – Architecture
- [`docs/02_data_model.md`](docs/02_data_model.md) – Database schema
- [`docs/03_data_sources.md`](docs/03_data_sources.md) – Data sources
- [`docs/04_safety_ethics.md`](docs/04_safety_ethics.md) – Research-only guardrails
- [`docs/05_dev_setup.md`](docs/05_dev_setup.md) – Dev conventions
- [`docs/06_free_testing_data.md`](docs/06_free_testing_data.md) – Testing strategies
**PR Summaries**:
- [`docs/PR1_SUMMARY.md`](docs/PR1_SUMMARY.md) – Scaffold + price loader
- [`docs/PR2_SUMMARY.md`](docs/PR2_SUMMARY.md) – Congressional trades
- [`docs/PR3_SUMMARY.md`](docs/PR3_SUMMARY.md) – Enrichment + deployment
- [`docs/PR4_SUMMARY.md`](docs/PR4_SUMMARY.md) – ⭐ **Analytics foundation (returns, benchmarks, metrics)**
## What's Working Now
- ✅ SQLAlchemy models for officials, securities, trades, prices
- ✅ Alembic migrations
- ✅ Price loader with yfinance (idempotent, upsert)
- ✅ Congressional trade ingestion from House Stock Watcher (FREE!)
- ✅ Security enrichment (company names, sectors, industries)
- ✅ ETL to populate officials & trades tables
- ✅ Docker + deployment infrastructure
- ✅ 93 passing tests with 88%+ coverage
- ✅ Linting (ruff + mypy) all green
- ✅ Works 100% offline with fixtures
- ✅ Real-time market monitoring & alert system
- ✅ Disclosure timing correlation engine
- ✅ Pattern detection & comparative analysis
- ✅ Automated email reporting (daily/weekly)
- ✅ CI/CD pipeline (GitHub/Gitea Actions)
## What You Can Do Now
### Analyze Performance
```bash
# Analyze specific official
python scripts/analyze_official.py "Nancy Pelosi" --window 90
# System-wide analysis
python scripts/calculate_all_returns.py
```
### Market Monitoring
```bash
# Run market scan
python scripts/monitor_market.py --scan
# Analyze timing of recent disclosures
python scripts/analyze_disclosure_timing.py --recent 7
# Generate pattern report
python scripts/generate_pattern_report.py --days 365
```
### Automated Reporting
```bash
# Set up daily/weekly email reports (5 minutes!)
./scripts/setup_cron.sh
# Send manual report
python scripts/send_daily_report.py --to your@email.com
```
### Add More Data
```bash
# Manual entry
python scripts/add_custom_trades.py
# CSV import
python scripts/scrape_alternative_sources.py import trades.csv
```
## System Architecture
POTE now includes a complete 3-phase monitoring system:
**Phase 1: Real-Time Market Monitoring**
- Tracks ~50 most-traded congressional stocks
- Detects unusual volume, price spikes, volatility
- Logs all alerts with timestamps and severity
**Phase 2: Disclosure Correlation**
- Matches trades with prior market alerts (30-45 day lookback)
- Calculates "timing advantage score" (0-100)
- Identifies suspicious timing patterns
**Phase 3: Pattern Detection**
- Ranks officials by consistent suspicious timing
- Analyzes by ticker, sector, and political party
- Generates comprehensive reports
**Full Documentation**: See [`MONITORING_SYSTEM_COMPLETE.md`](MONITORING_SYSTEM_COMPLETE.md)
## Next Steps
- [ ] Signals: "follow_research", "avoid_risk", "watch" with confidence scores
- [ ] Clustering: group officials by trading behavior patterns
- [ ] API: FastAPI backend for queries
- [ ] Dashboard: React/Streamlit visualization
See [`docs/00_mvp.md`](docs/00_mvp.md) for the full roadmap.
---
**License**: MIT (for research/educational use only)
**Disclaimer**: Not investment advice. Use public data only. No claims about inside information.
Reference in New Issue View Git Blame Copy Permalink