Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready
11 KiB
Live Market Monitoring + Congressional Trading Analysis
🎯 What's Possible vs Impossible
❌ NOT Possible:
- Identify WHO is buying/selling in real-time
- Match live trades to specific Congress members
- See congressional trades before they're disclosed
✅ IS Possible:
- Track unusual market activity in real-time
- Monitor stocks Congress members historically trade
- Compare unusual activity to later disclosures
- Detect patterns (timing, sectors, etc.)
🔄 Two-Phase Monitoring System
Phase 1: Real-Time Market Monitoring
Monitor unusual activity in stocks Congress trades:
- Unusual options flow
- Large block trades
- Volatility spikes
- Volume anomalies
Phase 2: Retroactive Analysis (30-45 days later)
When disclosures come in:
- Match disclosed trades to earlier unusual activity
- Identify if Congress bought BEFORE or AFTER spikes
- Calculate timing advantage (if any)
- Build pattern database
📊 Implementation: Watchlist-Based Monitoring
Concept:
Step 1: Congress Member Trades (Historical)
Nancy Pelosi often trades: NVDA, MSFT, GOOGL, AAPL
Dan Crenshaw often trades: XOM, CVX, LMT, BA
Step 2: Create Monitoring Watchlist
Monitor these tickers in real-time for:
- Unusual options activity
- Large block trades
- Price/volume anomalies
Step 3: When Disclosure Appears (30-45 days later)
Compare:
- Did they buy BEFORE unusual activity? (Suspicious)
- Did they buy AFTER? (Following market)
- What was the timing advantage?
🛠️ Data Sources for Live Market Monitoring
Free/Low-Cost Options:
-
Yahoo Finance (yfinance)
- ✅ Real-time quotes (15-min delay free)
- ✅ Historical options data
- ✅ Volume data
- ❌ Not true real-time for options flow
-
Unusual Whales API
- ✅ Options flow data
- ✅ Unusual activity alerts
- 💰 Paid ($50-200/month)
- https://unusualwhales.com/
-
Tradier API
- ✅ Real-time market data
- ✅ Options chains
- 💰 Paid but affordable ($10-50/month)
- https://tradier.com/
-
FlowAlgo
- ✅ Options flow tracking
- ✅ Dark pool data
- 💰 Paid ($99-399/month)
- https://www.flowalgo.com/
-
Polygon.io
- ✅ Real-time stock data
- ✅ Options data
- 💰 Free tier + paid plans
- https://polygon.io/
Best Free Option: Build Your Own with yfinance
Track volume/price changes every 5 minutes for congressional watchlist tickers.
💡 Practical Hybrid System
What We Can Build:
# Pseudo-code for hybrid monitoring
# 1. Get stocks Congress trades
congress_tickers = get_tickers_congress_trades()
# Result: ["NVDA", "MSFT", "TSLA", "AAPL", "SPY", ...]
# 2. Monitor these tickers for unusual activity
while market_open():
for ticker in congress_tickers:
current_data = get_realtime_data(ticker)
if is_unusual_activity(current_data):
log_alert({
"ticker": ticker,
"type": "unusual_volume", # or "price_spike", "options_flow"
"timestamp": now(),
"details": current_data
})
# 3. When disclosures appear (30-45 days later)
new_disclosures = fetch_congressional_trades()
for disclosure in new_disclosures:
# Check if we saw unusual activity BEFORE their trade
prior_alerts = get_alerts_before_date(
ticker=disclosure.ticker,
before_date=disclosure.transaction_date
)
if prior_alerts:
# They bought BEFORE unusual activity = Potential inside info
flag_suspicious(disclosure, prior_alerts)
else:
# They bought AFTER unusual activity = Following market
flag_following(disclosure)
📈 Example: Nancy Pelosi NVDA Trade Analysis
Timeline:
Nov 10, 2024:
🔔 ALERT: NVDA unusual call options activity
Volume: 10x average
Strike: $500 (2 weeks out)
Nov 15, 2024:
💰 Someone buys NVDA (unknown who at the time)
Nov 18, 2024:
📰 NVDA announces new AI chip
📈 Stock jumps 15%
Dec 15, 2024:
📋 Disclosure: Nancy Pelosi bought NVDA on Nov 15
Value: $15,001-$50,000
ANALYSIS:
✅ She bought AFTER unusual options activity (Nov 10)
❓ She bought BEFORE announcement (Nov 18)
⏱️ Timing: 3 days before major news
🚩 Flag: Investigate if announcement was public knowledge
🎯 Recommended Approach
Phase 1: Build Congressional Ticker Watchlist
# scripts/build_ticker_watchlist.py
from pote.db import get_session
from pote.db.models import Trade, Security
from sqlalchemy import func
def get_most_traded_tickers(limit=50):
"""Get tickers Congress trades most frequently."""
session = next(get_session())
results = (
session.query(
Security.ticker,
func.count(Trade.id).label('trade_count')
)
.join(Trade)
.group_by(Security.ticker)
.order_by(func.count(Trade.id).desc())
.limit(limit)
.all()
)
return [r[0] for r in results]
# Result: Top 50 tickers Congress trades
# Use these for real-time monitoring
Phase 2: Real-Time Monitoring (Simple)
# scripts/monitor_congressional_tickers.py
import yfinance as yf
from datetime import datetime, timedelta
import time
def monitor_tickers(tickers, interval_minutes=5):
"""Monitor tickers for unusual activity."""
baseline = {} # Store baseline metrics
while True:
for ticker in tickers:
try:
stock = yf.Ticker(ticker)
current = stock.history(period="1d", interval="1m")
if len(current) > 0:
latest = current.iloc[-1]
# Check for unusual volume
avg_volume = current['Volume'].mean()
if latest['Volume'] > avg_volume * 3:
alert(f"🔔 {ticker}: Unusual volume spike!")
# Check for price movement
price_change = (latest['Close'] - current['Open'].iloc[0]) / current['Open'].iloc[0]
if abs(price_change) > 0.05: # 5% move
alert(f"📈 {ticker}: {price_change:.2%} move today!")
except Exception as e:
print(f"Error monitoring {ticker}: {e}")
time.sleep(interval_minutes * 60)
Phase 3: Retroactive Analysis
When disclosures appear, analyze timing:
# scripts/analyze_trade_timing.py
def analyze_disclosure_timing(disclosure):
"""
When a disclosure appears, check if there was unusual
activity BEFORE the trade date.
"""
# Get alerts from 7 days before trade
lookback_start = disclosure.transaction_date - timedelta(days=7)
lookback_end = disclosure.transaction_date
alerts = get_alerts_in_range(
ticker=disclosure.ticker,
start=lookback_start,
end=lookback_end
)
if alerts:
return {
"suspicious": True,
"reason": "Unusual activity before trade",
"alerts": alerts
}
# Check if trade was before major price movement
post_trade_price = get_price_change(
ticker=disclosure.ticker,
start=disclosure.transaction_date,
days=30
)
if post_trade_price > 0.10: # 10% gain
return {
"notable": True,
"reason": f"Stock up {post_trade_price:.1%} after trade",
"gain": post_trade_price
}
🚨 Realistic Expectations
What This System Will Do:
✅ Monitor stocks Congress members historically trade ✅ Alert on unusual market activity in those stocks ✅ Retroactively correlate disclosures with earlier alerts ✅ Identify timing patterns and potential advantages ✅ Build database of congressional trading patterns
What This System WON'T Do:
❌ Identify WHO is buying in real-time ❌ Give you advance notice of congressional trades ❌ Provide real-time inside information ❌ Allow you to "front-run" Congress
Legal & Ethical:
✅ All data is public ✅ Analysis is retrospective ✅ For research and transparency ✅ Not market manipulation ❌ Cannot and should not be used to replicate potentially illegal trades
📊 Proposed Implementation
New Scripts to Create:
-
scripts/build_congressional_watchlist.py- Analyzes historical trades
- Identifies most-traded tickers
- Creates monitoring watchlist
-
scripts/monitor_market_live.py- Monitors watchlist tickers
- Detects unusual activity
- Logs alerts to database
-
scripts/analyze_disclosure_timing.py- When new disclosures appear
- Checks for prior unusual activity
- Flags suspicious timing
-
scripts/generate_timing_report.py- Shows disclosures with unusual timing
- Calculates timing advantage
- Identifies patterns
New Database Tables:
-- Track unusual market activity
CREATE TABLE market_alerts (
id SERIAL PRIMARY KEY,
ticker VARCHAR(20),
alert_type VARCHAR(50), -- 'unusual_volume', 'price_spike', 'options_flow'
timestamp TIMESTAMP,
details JSONB,
created_at TIMESTAMP DEFAULT NOW()
);
-- Link disclosures to prior alerts
CREATE TABLE disclosure_timing_analysis (
id SERIAL PRIMARY KEY,
trade_id INTEGER REFERENCES trades(id),
suspicious_flag BOOLEAN,
timing_score DECIMAL(5,2), -- 0-100 score
prior_alerts JSONB,
post_trade_performance DECIMAL(10,4),
created_at TIMESTAMP DEFAULT NOW()
);
🎯 Summary
Your Question:
"Can we read live trades being made and compare them to a name?"
Answer:
❌ No - Live trades are anonymous, can't identify individuals
✅ BUT - You CAN:
- Monitor unusual activity in stocks Congress trades
- Log these alerts in real-time
- When disclosures appear (30-45 days later), correlate them
- Identify if Congress bought BEFORE or AFTER unusual activity
- Build patterns database of timing and performance
This Gives You:
- ✅ Transparency on timing advantages
- ✅ Pattern detection across officials
- ✅ Research-grade analysis
- ✅ Historical correlation data
This Does NOT Give You:
- ❌ Real-time identity of traders
- ❌ Advance notice of congressional trades
- ❌ Ability to "front-run" disclosures
🚀 Would You Like Me To Build This?
I can create:
- ✅ Real-time monitoring system for congressional tickers
- ✅ Alert logging and analysis
- ✅ Timing correlation when disclosures appear
- ✅ Pattern detection and reporting
This would be Phase 2.5 of POTE - the "timing analysis" module.
Should I proceed with implementation?