Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready
396 lines
9.9 KiB
Markdown
396 lines
9.9 KiB
Markdown
# POTE Watchlist & Trading Reports
|
|
|
|
## 🎯 Get Trading Reports 1 Hour Before Market Close
|
|
|
|
### Quick Setup
|
|
|
|
```bash
|
|
# 1. Create watchlist of officials to monitor
|
|
python scripts/fetch_congress_members.py --create
|
|
|
|
# 2. Setup cron job for 3 PM ET (1 hour before close)
|
|
crontab -e
|
|
|
|
# Add this line:
|
|
0 15 * * 1-5 /home/poteapp/pote/scripts/pre_market_close_update.sh
|
|
|
|
# Save and exit
|
|
```
|
|
|
|
**What happens at 3 PM daily:**
|
|
1. Fetches latest trade disclosures
|
|
2. Enriches new securities
|
|
3. **Generates report** showing what was bought/sold
|
|
4. Saves to `reports/trading_report_YYYYMMDD.txt`
|
|
|
|
---
|
|
|
|
## 📋 Watchlist System
|
|
|
|
### Who's on the Default Watchlist?
|
|
|
|
**29 known active traders** based on 2023-2024 public reporting:
|
|
|
|
**Top Senate Traders:**
|
|
- Tommy Tuberville (R-AL) - Very active trader
|
|
- Rand Paul (R-KY) - Consistent activity
|
|
- Mark Warner (D-VA) - Tech sector focus
|
|
- Rick Scott (R-FL) - High volume
|
|
|
|
**Top House Traders:**
|
|
- Nancy Pelosi (D-CA) - Tech stocks, options
|
|
- Dan Crenshaw (R-TX) - Energy, defense
|
|
- Marjorie Taylor Greene (R-GA) - Various sectors
|
|
- Josh Gottheimer (D-NJ) - Financial services
|
|
- Brian Higgins (D-NY) - High frequency
|
|
|
|
[See full list: `python scripts/fetch_congress_members.py`]
|
|
|
|
---
|
|
|
|
## 🔧 Managing Your Watchlist
|
|
|
|
### View Current Watchlist
|
|
|
|
```bash
|
|
python scripts/fetch_congress_members.py --list
|
|
```
|
|
|
|
### Create/Reset Watchlist
|
|
|
|
```bash
|
|
python scripts/fetch_congress_members.py --create
|
|
```
|
|
|
|
Creates `config/watchlist.json` with default 29 active traders.
|
|
|
|
### Customize Watchlist
|
|
|
|
Edit `config/watchlist.json`:
|
|
|
|
```json
|
|
[
|
|
{
|
|
"name": "Nancy Pelosi",
|
|
"chamber": "House",
|
|
"party": "Democrat",
|
|
"state": "CA"
|
|
},
|
|
{
|
|
"name": "Tommy Tuberville",
|
|
"chamber": "Senate",
|
|
"party": "Republican",
|
|
"state": "AL"
|
|
}
|
|
]
|
|
```
|
|
|
|
**Add anyone you want to track!**
|
|
|
|
### Get ALL Members (Not Just Active Traders)
|
|
|
|
```bash
|
|
# 1. Get free API key from ProPublica:
|
|
# https://www.propublica.org/datastore/api/propublica-congress-api
|
|
|
|
# 2. Edit scripts/fetch_congress_members.py
|
|
# Add your API key
|
|
|
|
# 3. Run:
|
|
python scripts/fetch_congress_members.py --propublica
|
|
```
|
|
|
|
This fetches all 535 members of Congress (100 Senate + 435 House).
|
|
|
|
---
|
|
|
|
## 📊 Generating Reports
|
|
|
|
### Manual Report Generation
|
|
|
|
```bash
|
|
# Report from last 7 days (watchlist only)
|
|
python scripts/generate_trading_report.py --days 7 --watchlist-only
|
|
|
|
# Report from last 30 days (all officials)
|
|
python scripts/generate_trading_report.py --days 30
|
|
|
|
# Save to file
|
|
python scripts/generate_trading_report.py --output report.txt
|
|
|
|
# HTML format (for email)
|
|
python scripts/generate_trading_report.py --format html --output report.html
|
|
|
|
# JSON format (for programmatic use)
|
|
python scripts/generate_trading_report.py --format json --output report.json
|
|
```
|
|
|
|
### Example Report Output
|
|
|
|
```
|
|
================================================================================
|
|
CONGRESSIONAL TRADING REPORT
|
|
5 New Trades
|
|
Generated: 2024-12-15
|
|
================================================================================
|
|
|
|
────────────────────────────────────────────────────────────────────────────────
|
|
👤 Nancy Pelosi (D-CA, House)
|
|
────────────────────────────────────────────────────────────────────────────────
|
|
Side Ticker Company Sector Value Trade Date Filed
|
|
-------- ------ -------------------------- ---------- ------------------- ---------- ----------
|
|
🟢 BUY NVDA NVIDIA Corporation Technology $15,001 - $50,000 2024-11-15 2024-12-01
|
|
🔴 SELL MSFT Microsoft Corporation Technology $50,001 - $100,000 2024-11-20 2024-12-01
|
|
|
|
────────────────────────────────────────────────────────────────────────────────
|
|
👤 Tommy Tuberville (R-AL, Senate)
|
|
────────────────────────────────────────────────────────────────────────────────
|
|
Side Ticker Company Sector Value Trade Date Filed
|
|
-------- ------ -------------------------- ---------- ------------------- ---------- ----------
|
|
🟢 BUY SPY SPDR S&P 500 ETF Financial $100,001 - $250,000 2024-11-18 2024-12-02
|
|
🟢 BUY AAPL Apple Inc. Technology $50,001 - $100,000 2024-11-22 2024-12-02
|
|
🔴 SELL TSLA Tesla, Inc. Automotive $15,001 - $50,000 2024-11-25 2024-12-02
|
|
|
|
================================================================================
|
|
📊 SUMMARY
|
|
================================================================================
|
|
|
|
Total Trades: 5
|
|
Buys: 3
|
|
Sells: 2
|
|
Unique Officials: 2
|
|
Unique Tickers: 5
|
|
|
|
Top Tickers:
|
|
NVDA - 1 trades
|
|
MSFT - 1 trades
|
|
SPY - 1 trades
|
|
AAPL - 1 trades
|
|
TSLA - 1 trades
|
|
|
|
================================================================================
|
|
```
|
|
|
|
---
|
|
|
|
## ⏰ Automated Schedule Options
|
|
|
|
### Option 1: Pre-Market Close (3 PM ET) - Recommended
|
|
|
|
```bash
|
|
crontab -e
|
|
# Add: 0 15 * * 1-5 /home/poteapp/pote/scripts/pre_market_close_update.sh
|
|
```
|
|
|
|
**Why 3 PM?**
|
|
- 1 hour before market close (4 PM ET)
|
|
- Time to review report and make decisions
|
|
- Disclosures often appear during business hours
|
|
- Weekdays only (no weekends)
|
|
|
|
### Option 2: Pre-Market Open (8 AM ET)
|
|
|
|
```bash
|
|
crontab -e
|
|
# Add: 0 8 * * 1-5 /home/poteapp/pote/scripts/pre_market_close_update.sh
|
|
```
|
|
|
|
**Why 8 AM?**
|
|
- 30 minutes before market opens (9:30 AM ET)
|
|
- Catch overnight filings
|
|
- Review before trading day
|
|
|
|
### Option 3: After Market Close (5 PM ET)
|
|
|
|
```bash
|
|
crontab -e
|
|
# Add: 0 17 * * 1-5 /home/poteapp/pote/scripts/pre_market_close_update.sh
|
|
```
|
|
|
|
**Why 5 PM?**
|
|
- After market closes
|
|
- No trading pressure
|
|
- Full day of potential filings captured
|
|
|
|
### Option 4: Multiple Times Per Day
|
|
|
|
```bash
|
|
crontab -e
|
|
# Add: 0 8,15 * * 1-5 /home/poteapp/pote/scripts/pre_market_close_update.sh
|
|
```
|
|
|
|
Runs at 8 AM and 3 PM daily (weekdays).
|
|
|
|
---
|
|
|
|
## 📧 Email Reports (Optional)
|
|
|
|
### Setup Email Notifications
|
|
|
|
Edit `scripts/pre_market_close_update.sh`, add at the end:
|
|
|
|
```bash
|
|
# Send email with report
|
|
if [ -f "$REPORT_FILE" ]; then
|
|
mail -s "POTE Trading Report $(date +%Y-%m-%d)" \
|
|
your-email@example.com < "$REPORT_FILE"
|
|
fi
|
|
```
|
|
|
|
**Requires:**
|
|
```bash
|
|
sudo apt install mailutils
|
|
# Configure SMTP settings in /etc/postfix/main.cf
|
|
```
|
|
|
|
### HTML Email Reports
|
|
|
|
```bash
|
|
python scripts/generate_trading_report.py \
|
|
--format html \
|
|
--output /tmp/report.html
|
|
|
|
# Send HTML email
|
|
python scripts/send_email.py /tmp/report.html your-email@example.com
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 Typical Workflow
|
|
|
|
### Daily Routine (3 PM ET)
|
|
|
|
1. **Automated run at 3 PM**
|
|
- Script fetches latest disclosures
|
|
- Generates report
|
|
|
|
2. **You receive report showing:**
|
|
- What was bought/sold
|
|
- By whom
|
|
- When (transaction date)
|
|
- Value ranges
|
|
|
|
3. **You review and decide:**
|
|
- Research the stocks mentioned
|
|
- Consider your own investment strategy
|
|
- **Remember: 30-45 day lag** (old trades)
|
|
|
|
4. **Important:**
|
|
- This is for research/transparency
|
|
- Not investment advice
|
|
- Trades are 30-45 days old by law
|
|
|
|
---
|
|
|
|
## 🔍 Finding More Officials
|
|
|
|
### Public Resources
|
|
|
|
**High-Volume Traders:**
|
|
- https://housestockwatcher.com/
|
|
- https://senatestockwatcher.com/
|
|
- https://www.capitoltrades.com/
|
|
|
|
**Official Sources:**
|
|
- House Clerk: https://disclosures.house.gov/
|
|
- Senate: https://efdsearch.senate.gov/
|
|
|
|
**News Coverage:**
|
|
- "Unusual Whales" on Twitter/X
|
|
- Financial news sites
|
|
- ProPublica investigations
|
|
|
|
### Research Specific Committees
|
|
|
|
Members of certain committees tend to trade more:
|
|
|
|
**Senate:**
|
|
- Banking Committee (financial regulations)
|
|
- Armed Services (defense contracts)
|
|
- Energy Committee (energy stocks)
|
|
|
|
**House:**
|
|
- Financial Services
|
|
- Energy and Commerce
|
|
- Armed Services
|
|
|
|
Add committee members to your watchlist.
|
|
|
|
---
|
|
|
|
## 📈 Example Cron Setup
|
|
|
|
```bash
|
|
# Edit crontab
|
|
crontab -e
|
|
|
|
# Add these lines:
|
|
|
|
# Pre-market report (8 AM ET weekdays)
|
|
0 8 * * 1-5 /home/poteapp/pote/scripts/pre_market_close_update.sh
|
|
|
|
# Weekly full update (Sunday night)
|
|
0 0 * * 0 /home/poteapp/pote/scripts/daily_fetch.sh
|
|
|
|
# Save and exit
|
|
```
|
|
|
|
This gives you:
|
|
- Daily reports at 8 AM (weekdays)
|
|
- Weekly full system update (prices, analytics)
|
|
|
|
---
|
|
|
|
## 🚀 Quick Start Summary
|
|
|
|
```bash
|
|
# 1. Create watchlist
|
|
python scripts/fetch_congress_members.py --create
|
|
|
|
# 2. Test report generation
|
|
python scripts/generate_trading_report.py --days 7 --watchlist-only
|
|
|
|
# 3. Setup automation (3 PM daily)
|
|
crontab -e
|
|
# Add: 0 15 * * 1-5 /home/poteapp/pote/scripts/pre_market_close_update.sh
|
|
|
|
# 4. Check logs
|
|
tail -f logs/pre_market_*.log
|
|
|
|
# 5. View reports
|
|
cat reports/trading_report_$(date +%Y%m%d).txt
|
|
```
|
|
|
|
---
|
|
|
|
## ❓ FAQ
|
|
|
|
**Q: Why are all the trades old (30-45 days)?**
|
|
**A:** Federal law (STOCK Act) gives Congress 30-45 days to file. This is normal.
|
|
|
|
**Q: Can I track specific senators/representatives?**
|
|
**A:** Yes! Edit `config/watchlist.json` and add anyone.
|
|
|
|
**Q: Where's the full list of Congress members?**
|
|
**A:** Use `--propublica` option with a free API key to get all 535 members.
|
|
|
|
**Q: Can I get alerts for specific stocks?**
|
|
**A:** Yes! Modify `generate_trading_report.py` to filter by ticker.
|
|
|
|
**Q: What if House Stock Watcher is down?**
|
|
**A:** Reports will show existing data. Use CSV import for new data manually.
|
|
|
|
**Q: Can I track past trades?**
|
|
**A:** Yes! Adjust `--days` parameter: `--days 365` for full year.
|
|
|
|
---
|
|
|
|
**Ready to start tracking?**
|
|
|
|
```bash
|
|
python scripts/fetch_congress_members.py --create
|
|
python scripts/generate_trading_report.py --watchlist-only
|
|
```
|
|
|
|
|