POTE/docs/11_live_market_monitoring.md
ilia 0d8d85adc1 Add complete automation, reporting, and CI/CD system
Features Added:
==============

📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py

🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup

🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration

🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI

📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide

🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron

WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
   - New trades, market alerts, suspicious timing
   - Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)

USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh

# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com

# Check system health
python scripts/health_check.py

See AUTOMATION_QUICKSTART.md for full instructions.

93 tests passing | Full CI/CD | Email reports ready
2025-12-15 15:34:31 -05:00

11 KiB

Live Market Monitoring + Congressional Trading Analysis

🎯 What's Possible vs Impossible

NOT Possible:

  • Identify WHO is buying/selling in real-time
  • Match live trades to specific Congress members
  • See congressional trades before they're disclosed

IS Possible:

  • Track unusual market activity in real-time
  • Monitor stocks Congress members historically trade
  • Compare unusual activity to later disclosures
  • Detect patterns (timing, sectors, etc.)

🔄 Two-Phase Monitoring System

Phase 1: Real-Time Market Monitoring

Monitor unusual activity in stocks Congress trades:

  • Unusual options flow
  • Large block trades
  • Volatility spikes
  • Volume anomalies

Phase 2: Retroactive Analysis (30-45 days later)

When disclosures come in:

  • Match disclosed trades to earlier unusual activity
  • Identify if Congress bought BEFORE or AFTER spikes
  • Calculate timing advantage (if any)
  • Build pattern database

📊 Implementation: Watchlist-Based Monitoring

Concept:

Step 1: Congress Member Trades (Historical)
  Nancy Pelosi often trades: NVDA, MSFT, GOOGL, AAPL
  Dan Crenshaw often trades: XOM, CVX, LMT, BA
  
Step 2: Create Monitoring Watchlist
  Monitor these tickers in real-time for:
  - Unusual options activity
  - Large block trades
  - Price/volume anomalies
  
Step 3: When Disclosure Appears (30-45 days later)
  Compare:
  - Did they buy BEFORE unusual activity? (Suspicious)
  - Did they buy AFTER? (Following market)
  - What was the timing advantage?

🛠️ Data Sources for Live Market Monitoring

Free/Low-Cost Options:

  1. Yahoo Finance (yfinance)

    • Real-time quotes (15-min delay free)
    • Historical options data
    • Volume data
    • Not true real-time for options flow
  2. Unusual Whales API

  3. Tradier API

    • Real-time market data
    • Options chains
    • 💰 Paid but affordable ($10-50/month)
    • https://tradier.com/
  4. FlowAlgo

  5. Polygon.io

Best Free Option: Build Your Own with yfinance

Track volume/price changes every 5 minutes for congressional watchlist tickers.


💡 Practical Hybrid System

What We Can Build:

# Pseudo-code for hybrid monitoring

# 1. Get stocks Congress trades
congress_tickers = get_tickers_congress_trades()
# Result: ["NVDA", "MSFT", "TSLA", "AAPL", "SPY", ...]

# 2. Monitor these tickers for unusual activity
while market_open():
    for ticker in congress_tickers:
        current_data = get_realtime_data(ticker)
        
        if is_unusual_activity(current_data):
            log_alert({
                "ticker": ticker,
                "type": "unusual_volume",  # or "price_spike", "options_flow"
                "timestamp": now(),
                "details": current_data
            })

# 3. When disclosures appear (30-45 days later)
new_disclosures = fetch_congressional_trades()

for disclosure in new_disclosures:
    # Check if we saw unusual activity BEFORE their trade
    prior_alerts = get_alerts_before_date(
        ticker=disclosure.ticker,
        before_date=disclosure.transaction_date
    )
    
    if prior_alerts:
        # They bought BEFORE unusual activity = Potential inside info
        flag_suspicious(disclosure, prior_alerts)
    else:
        # They bought AFTER unusual activity = Following market
        flag_following(disclosure)

📈 Example: Nancy Pelosi NVDA Trade Analysis

Timeline:

Nov 10, 2024:
  🔔 ALERT: NVDA unusual call options activity
     Volume: 10x average
     Strike: $500 (2 weeks out)
     
Nov 15, 2024:
  💰 Someone buys NVDA (unknown who at the time)
  
Nov 18, 2024:
  📰 NVDA announces new AI chip
  📈 Stock jumps 15%
  
Dec 15, 2024:
  📋 Disclosure: Nancy Pelosi bought NVDA on Nov 15
     Value: $15,001-$50,000
     
ANALYSIS:
  ✅ She bought AFTER unusual options activity (Nov 10)
  ❓ She bought BEFORE announcement (Nov 18)
  ⏱️ Timing: 3 days before major news
  🚩 Flag: Investigate if announcement was public knowledge

Phase 1: Build Congressional Ticker Watchlist

# scripts/build_ticker_watchlist.py

from pote.db import get_session
from pote.db.models import Trade, Security
from sqlalchemy import func

def get_most_traded_tickers(limit=50):
    """Get tickers Congress trades most frequently."""
    session = next(get_session())
    
    results = (
        session.query(
            Security.ticker,
            func.count(Trade.id).label('trade_count')
        )
        .join(Trade)
        .group_by(Security.ticker)
        .order_by(func.count(Trade.id).desc())
        .limit(limit)
        .all()
    )
    
    return [r[0] for r in results]

# Result: Top 50 tickers Congress trades
# Use these for real-time monitoring

Phase 2: Real-Time Monitoring (Simple)

# scripts/monitor_congressional_tickers.py

import yfinance as yf
from datetime import datetime, timedelta
import time

def monitor_tickers(tickers, interval_minutes=5):
    """Monitor tickers for unusual activity."""
    
    baseline = {}  # Store baseline metrics
    
    while True:
        for ticker in tickers:
            try:
                stock = yf.Ticker(ticker)
                current = stock.history(period="1d", interval="1m")
                
                if len(current) > 0:
                    latest = current.iloc[-1]
                    
                    # Check for unusual volume
                    avg_volume = current['Volume'].mean()
                    if latest['Volume'] > avg_volume * 3:
                        alert(f"🔔 {ticker}: Unusual volume spike!")
                    
                    # Check for price movement
                    price_change = (latest['Close'] - current['Open'].iloc[0]) / current['Open'].iloc[0]
                    if abs(price_change) > 0.05:  # 5% move
                        alert(f"📈 {ticker}: {price_change:.2%} move today!")
                        
            except Exception as e:
                print(f"Error monitoring {ticker}: {e}")
        
        time.sleep(interval_minutes * 60)

Phase 3: Retroactive Analysis

When disclosures appear, analyze timing:

# scripts/analyze_trade_timing.py

def analyze_disclosure_timing(disclosure):
    """
    When a disclosure appears, check if there was unusual
    activity BEFORE the trade date.
    """
    
    # Get alerts from 7 days before trade
    lookback_start = disclosure.transaction_date - timedelta(days=7)
    lookback_end = disclosure.transaction_date
    
    alerts = get_alerts_in_range(
        ticker=disclosure.ticker,
        start=lookback_start,
        end=lookback_end
    )
    
    if alerts:
        return {
            "suspicious": True,
            "reason": "Unusual activity before trade",
            "alerts": alerts
        }
    
    # Check if trade was before major price movement
    post_trade_price = get_price_change(
        ticker=disclosure.ticker,
        start=disclosure.transaction_date,
        days=30
    )
    
    if post_trade_price > 0.10:  # 10% gain
        return {
            "notable": True,
            "reason": f"Stock up {post_trade_price:.1%} after trade",
            "gain": post_trade_price
        }

🚨 Realistic Expectations

What This System Will Do:

Monitor stocks Congress members historically trade Alert on unusual market activity in those stocks Retroactively correlate disclosures with earlier alerts Identify timing patterns and potential advantages Build database of congressional trading patterns

What This System WON'T Do:

Identify WHO is buying in real-time Give you advance notice of congressional trades Provide real-time inside information Allow you to "front-run" Congress

All data is public Analysis is retrospective For research and transparency Not market manipulation Cannot and should not be used to replicate potentially illegal trades


📊 Proposed Implementation

New Scripts to Create:

  1. scripts/build_congressional_watchlist.py

    • Analyzes historical trades
    • Identifies most-traded tickers
    • Creates monitoring watchlist
  2. scripts/monitor_market_live.py

    • Monitors watchlist tickers
    • Detects unusual activity
    • Logs alerts to database
  3. scripts/analyze_disclosure_timing.py

    • When new disclosures appear
    • Checks for prior unusual activity
    • Flags suspicious timing
  4. scripts/generate_timing_report.py

    • Shows disclosures with unusual timing
    • Calculates timing advantage
    • Identifies patterns

New Database Tables:

-- Track unusual market activity
CREATE TABLE market_alerts (
    id SERIAL PRIMARY KEY,
    ticker VARCHAR(20),
    alert_type VARCHAR(50),  -- 'unusual_volume', 'price_spike', 'options_flow'
    timestamp TIMESTAMP,
    details JSONB,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Link disclosures to prior alerts
CREATE TABLE disclosure_timing_analysis (
    id SERIAL PRIMARY KEY,
    trade_id INTEGER REFERENCES trades(id),
    suspicious_flag BOOLEAN,
    timing_score DECIMAL(5,2),  -- 0-100 score
    prior_alerts JSONB,
    post_trade_performance DECIMAL(10,4),
    created_at TIMESTAMP DEFAULT NOW()
);

🎯 Summary

Your Question:

"Can we read live trades being made and compare them to a name?"

Answer:

No - Live trades are anonymous, can't identify individuals

BUT - You CAN:

  1. Monitor unusual activity in stocks Congress trades
  2. Log these alerts in real-time
  3. When disclosures appear (30-45 days later), correlate them
  4. Identify if Congress bought BEFORE or AFTER unusual activity
  5. Build patterns database of timing and performance

This Gives You:

  • Transparency on timing advantages
  • Pattern detection across officials
  • Research-grade analysis
  • Historical correlation data

This Does NOT Give You:

  • Real-time identity of traders
  • Advance notice of congressional trades
  • Ability to "front-run" disclosures

🚀 Would You Like Me To Build This?

I can create:

  1. Real-time monitoring system for congressional tickers
  2. Alert logging and analysis
  3. Timing correlation when disclosures appear
  4. Pattern detection and reporting

This would be Phase 2.5 of POTE - the "timing analysis" module.

Should I proceed with implementation?