ilia/POTE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
19 Commits 4 Branches 0 Tags
Commit Graph

19 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
ilia
07af492026 Add email setup guide for levkin.ca mail server 
Configuration for test@levkin.ca:
- SMTP: mail.levkin.ca:587 (STARTTLS)
- Includes setup instructions
- Testing checklist
- Troubleshooting guide

Note: .env file created locally (not committed, in .gitignore)
 2025-12-15 15:43:44 -05:00
ilia
d8f723bafb Add comprehensive deployment and automation FAQ 
Answers user's questions:
- What happens after deployment? (nothing automatic by default)
- How to get reports? (3 options: email, SSH, future web UI)
- Where are reports sent? (email or saved to ~/logs/)
- Do you need to check IP? (depends on setup method)
- Can we setup email reports? (YES! 5-minute setup)
- Do we need CI/CD pipelines? (optional, but included)
- Can we use existing Ansible pipeline? (concepts reused, not directly)

This document ties everything together and provides clear next steps.
 2025-12-15 15:35:33 -05:00
ilia
0d8d85adc1 Add complete automation, reporting, and CI/CD system 
Features Added:
==============

📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py

🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup

🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration

🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI

📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide

🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron

WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
   - New trades, market alerts, suspicious timing
   - Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)

USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh

# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com

# Check system health
python scripts/health_check.py

See AUTOMATION_QUICKSTART.md for full instructions.

93 tests passing | Full CI/CD | Email reports ready
 2025-12-15 15:34:31 -05:00
ilia
53d631a903 Add comprehensive monitoring system documentation 
Complete summary of all 3 phases:
- Phase 1: Real-time market monitoring
- Phase 2: Disclosure timing correlation
- Phase 3: Pattern detection & rankings

Documentation includes:
- System architecture diagram
- Usage guide for all phases
- Example reports
- Test coverage summary
- Deployment checklist
- Interpretation guide
- Legal/ethical disclaimers
- Automated workflow examples

Total Achievement:
✅ 93 tests passing
✅ All 3 phases complete
✅ Production-ready system
✅ Full documentation

The POTE monitoring system is now complete!
 2025-12-15 15:25:07 -05:00
ilia
2ec4a8e373 Phase 3: Pattern Detection & Comparative Analysis - COMPLETE 
COMPLETE: Cross-official pattern detection and ranking system

New Module:
- src/pote/monitoring/pattern_detector.py: Pattern analysis engine
  * rank_officials_by_timing(): Rank all officials by suspicion
  * identify_repeat_offenders(): Find systematic offenders
  * analyze_ticker_patterns(): Per-stock suspicious patterns
  * get_sector_timing_analysis(): Sector-level analysis
  * get_party_comparison(): Democrat vs Republican comparison
  * generate_pattern_report(): Comprehensive report

Analysis Features:
- Official Rankings:
  * By average timing score
  * Suspicious trade percentage
  * Alert rates
  * Pattern classification

- Repeat Offender Detection:
  * Identifies officials with 50%+ suspicious trades
  * Historical pattern tracking
  * Systematic timing advantage detection

- Comparative Analysis:
  * Cross-party comparison
  * Sector analysis
  * Ticker-specific patterns
  * Statistical aggregations

New Script:
- scripts/generate_pattern_report.py: Comprehensive reports
  * Top 10 most suspicious officials
  * Repeat offenders list
  * Most suspiciously traded stocks
  * Sector breakdowns
  * Party comparison stats
  * Text/JSON formats

New Tests (11 total, all passing):
- test_rank_officials_by_timing
- test_identify_repeat_offenders
- test_analyze_ticker_patterns
- test_get_sector_timing_analysis
- test_get_party_comparison
- test_generate_pattern_report
- test_rank_officials_min_trades_filter
- test_empty_data_handling
- test_ranking_score_accuracy
- test_sector_stats_accuracy
- test_party_stats_completeness

Usage:
  python scripts/generate_pattern_report.py --days 365

Report Includes:
- Top suspicious officials ranked
- Repeat offenders (50%+ suspicious rate)
- Most suspiciously traded tickers
- Sector analysis
- Party comparison
- Interpretation guide

Total Test Suite: 93 tests passing ✅

ALL 3 PHASES COMPLETE!
 2025-12-15 15:23:40 -05:00
ilia
a52313145b Add comprehensive tests for Phase 2 correlation engine 
New Tests (13 total, all passing):
- test_get_alerts_before_trade: Retrieve prior alerts
- test_get_alerts_before_trade_no_alerts: Handle no alerts
- test_calculate_timing_score_high_suspicion: High score logic
- test_calculate_timing_score_no_alerts: Zero score handling
- test_calculate_timing_score_factors: Multi-factor scoring
- test_analyze_trade_full: Complete trade analysis
- test_analyze_recent_disclosures: Batch processing
- test_get_official_timing_pattern: Historical patterns
- test_get_official_timing_pattern_no_trades: Edge case
- test_get_ticker_timing_analysis: Per-ticker analysis
- test_get_ticker_timing_analysis_no_trades: Edge case
- test_alerts_outside_lookback_window: Date filtering
- test_different_ticker_alerts_excluded: Ticker filtering

Test Coverage:
- Alert-to-trade correlation
- Timing score calculation (all factors)
- Pattern analysis (officials & tickers)
- Batch analysis
- Edge cases & filtering
- Date range handling

Total Test Suite: 82 tests passing ✅
 2025-12-15 15:20:40 -05:00
ilia
6b62ae96f7 Phase 2: Disclosure Timing Correlation Engine 
COMPLETE: Match congressional trades to prior market alerts

New Module:
- src/pote/monitoring/disclosure_correlator.py: Core correlation engine
  * get_alerts_before_trade(): Find alerts before trade date
  * calculate_timing_score(): Score suspicious timing (0-100 scale)
    - Factors: alert count, severity, recency, type
    - Thresholds: 60+ = suspicious, 80+ = highly suspicious
  * analyze_trade(): Complete trade analysis with timing
  * analyze_recent_disclosures(): Batch analysis of new filings
  * get_official_timing_pattern(): Historical pattern analysis
  * get_ticker_timing_analysis(): Per-stock timing patterns

Timing Score Algorithm:
- Base score: alert count × 5 + avg severity × 2
- Recency bonus: +10 per alert within 7 days
- Severity bonus: +15 per high-severity (7+) alert
- Total score: 0-100 (capped)
- Interpretation:
  * 80-100: Highly suspicious (likely timing advantage)
  * 60-79: Suspicious (possible timing advantage)
  * 40-59: Notable (some unusual activity)
  * 0-39: Normal (no significant pattern)

New Script:
- scripts/analyze_disclosure_timing.py: CLI analysis tool
  * Analyze recent disclosures (--days N)
  * Filter by timing score (--min-score)
  * Analyze specific official (--official NAME)
  * Analyze specific ticker (--ticker SYMBOL)
  * Text/JSON output formats
  * Detailed reports with prior alerts

Usage Examples:
  # Find suspicious trades filed recently
  python scripts/analyze_disclosure_timing.py --days 30 --min-score 60

  # Analyze specific official
  python scripts/analyze_disclosure_timing.py --official "Nancy Pelosi"

  # Analyze specific ticker
  python scripts/analyze_disclosure_timing.py --ticker NVDA

Report Includes:
- Timing score and suspicion level
- Prior alert details (count, severity, timing)
- Official name, ticker, trade details
- Assessment and reasoning
- Top suspicious trades ranked

Next: Phase 3 - Pattern Detection across officials/stocks
 2025-12-15 15:17:09 -05:00
ilia
db34f26cdc Add comprehensive tests for Phase 1 monitoring system 
New Tests (14 total, all passing):
- test_get_congressional_watchlist: Auto-detect most-traded tickers
- test_check_ticker_basic: Single ticker analysis
- test_scan_watchlist_with_mock: Batch scanning with controlled data
- test_save_alerts: Database persistence
- test_get_recent_alerts: Query filtering (ticker, type, severity, date)
- test_get_ticker_alert_summary: Aggregated statistics
- test_alert_manager_format_text: Text formatting
- test_alert_manager_format_html: HTML formatting
- test_alert_manager_filter_alerts: Multi-criteria filtering
- test_alert_manager_generate_summary_text: Report generation
- test_alert_manager_generate_summary_html: HTML reports
- test_alert_manager_empty_alerts: Edge case handling
- test_market_alert_model: ORM model validation
- test_alert_timestamp_filtering: Time-based queries

Test Coverage:
- Market monitoring core logic
- Alert detection algorithms
- Database operations
- Filtering and querying
- Report generation (text/HTML)
- Edge cases and error handling

Total Test Suite: 69 tests passing ✅
 2025-12-15 15:14:58 -05:00
ilia
cfaf38b0be Phase 1: Real-Time Market Monitoring System 
COMPLETE: Real-time unusual activity detection for congressional tickers

New Database Model:
- MarketAlert: Stores unusual market activity alerts
  * Tracks volume spikes, price movements, volatility
  * JSON details field for flexible data storage
  * Severity scoring (1-10 scale)
  * Indexed for efficient queries by ticker/timestamp

New Modules:
- src/pote/monitoring/market_monitor.py: Core monitoring engine
  * get_congressional_watchlist(): Top 50 most-traded tickers
  * check_ticker(): Analyze single stock for unusual activity
  * scan_watchlist(): Batch analysis of multiple tickers
  * Detection logic:
    - Unusual volume (3x average)
    - Price spikes/drops (>5%)
    - High volatility (2x normal)
  * save_alerts(): Persist to database
  * get_recent_alerts(): Query historical alerts

- src/pote/monitoring/alert_manager.py: Alert formatting & filtering
  * format_alert_text(): Human-readable output
  * format_alert_html(): HTML email format
  * filter_alerts(): By severity, ticker, type
  * generate_summary_report(): Text/HTML reports

Scripts:
- scripts/monitor_market.py: CLI monitoring tool
  * Continuous monitoring mode (--interval)
  * One-time scan (--once)
  * Custom ticker lists or auto-detect congressional watchlist
  * Severity filtering (--min-severity)
  * Report generation and saving

Migrations:
- alembic/versions/f44014715b40_add_market_alerts_table.py

Documentation:
- docs/11_live_market_monitoring.md: Complete explanation
  * Why you can't track WHO is trading
  * What IS possible (timing analysis)
  * How hybrid monitoring works
  * Data sources and APIs

Usage:
  # Monitor congressional tickers (one-time scan)
  python scripts/monitor_market.py --once

  # Continuous monitoring (every 5 minutes)
  python scripts/monitor_market.py --interval 300

  # Monitor specific tickers
  python scripts/monitor_market.py --tickers NVDA,MSFT,AAPL --once

Next Steps (Phase 2):
- Disclosure correlation engine
- Timing advantage calculator
- Suspicious trade flagging
 2025-12-15 15:10:49 -05:00
ilia
8ba9d7ffdd Add watchlist system and pre-market trading reports 
New Features:
- Watchlist system for tracking specific Congress members
- Trading report generation with multiple formats
- Pre-market-close automated updates (3 PM)

New Scripts:
- scripts/fetch_congress_members.py: Manage watchlist
  * 29 known active traders (curated list)
  * Optional ProPublica API integration (all 535 members)
  * Create/view/manage watchlist

- scripts/generate_trading_report.py: Generate trading reports
  * Filter by watchlist or show all
  * Multiple formats: text, HTML, JSON
  * Summary statistics (buys/sells, top tickers)
  * Color-coded output (🟢 BUY, 🔴 SELL)

- scripts/pre_market_close_update.sh: 3 PM automation
  * Quick fetch of latest trades
  * Enrichment of new securities
  * Generate and display report
  * Saves to reports/ directory

Documentation:
- WATCHLIST_GUIDE.md: Complete guide
  * List of 29 known active traders
  * How to create/customize watchlist
  * Schedule options (pre-market, post-market)
  * Email setup (optional)
  * FAQ and examples

Known Active Traders Include:
Senate: Tuberville, Rand Paul, Mark Warner, Rick Scott
House: Pelosi, Crenshaw, MTG, Gottheimer, Brian Higgins

Use Cases:
✅ Daily reports at 3 PM (1 hour before close)
✅ See what Congress bought/sold recently
✅ Track specific members you care about
✅ Export to HTML/JSON for further analysis
 2025-12-15 15:00:42 -05:00
ilia
3a89c1e6d2 Add comprehensive automation system 
New Scripts:
- scripts/daily_fetch.sh: Automated daily data updates
  * Fetches congressional trades (last 7 days)
  * Enriches securities (name, sector, industry)
  * Updates price data for all securities
  * Calculates returns and metrics
  * Logs everything to logs/ directory

- scripts/setup_automation.sh: Interactive automation setup
  * Makes scripts executable
  * Creates log directories
  * Configures cron jobs (multiple schedule options)
  * Guides user through setup

Documentation:
- docs/10_automation.md: Complete automation guide
  * Explains disclosure timing (30-45 day legal lag)
  * Why daily updates are optimal (not hourly/real-time)
  * Cron job setup instructions
  * Systemd timer alternative
  * Email notifications (optional)
  * Monitoring and logging
  * Failure handling
  * Performance optimization

Key Insights:
❌ No real-time data possible (STOCK Act = 30-45 day lag)
✅ Daily updates are optimal
✅ Automated via cron jobs
✅ Handles API failures gracefully
✅ Logs everything for debugging
 2025-12-15 14:55:05 -05:00
ilia
77bd69b85c Add comprehensive testing status documentation 2025-12-15 14:43:52 -05:00
ilia
b4e6a7c340 Fix analytics tests and add comprehensive testing guide 
Critical Fixes:
- Fixed Price model query to use security_id join with Security
- Added Security import to returns.py module
- Fixed all test fixtures to use test_db_session correctly
- Added AAPL price data to sample_prices fixture

New Tests:
- tests/test_analytics_integration.py: 10 comprehensive integration tests
  * Real-world scenarios with synthetic price data
  * Return calculations, benchmark comparisons, performance metrics
  * Edge cases: missing data, sell trades, disclosure timing

Documentation:
- LOCAL_TEST_GUIDE.md: Complete guide for local testing
  * How to test before deploying
  * Current data status (live vs fixtures)
  * Multiple options for getting real data
  * Common issues and fixes

Test Results:
✅ All 55 tests passing
✅ Analytics fully functional
✅ Ready for deployment

Live Data Status:
❌ House Stock Watcher API still down (external issue)
✅ Manual CSV import works
✅ yfinance for prices works
✅ Can use system NOW with manual data
 2025-12-15 14:42:20 -05:00
ilia
34aebb1c2e PR4: Phase 2 Analytics Foundation 
Complete analytics module with returns, benchmarks, and performance metrics.

New Modules:
- src/pote/analytics/returns.py: Return calculator for trades
- src/pote/analytics/benchmarks.py: Benchmark comparison & alpha
- src/pote/analytics/metrics.py: Performance aggregations

Scripts:
- scripts/analyze_official.py: Analyze specific official
- scripts/calculate_all_returns.py: System-wide analysis

Tests:
- tests/test_analytics.py: Full coverage of analytics

Features:
✅ Calculate returns over 30/60/90/180 day windows
✅ Compare to market benchmarks (SPY, QQQ, etc.)
✅ Calculate abnormal returns (alpha)
✅ Aggregate stats by official, sector
✅ Top performer rankings
✅ Disclosure timing analysis
✅ Command-line analysis tools

~1,210 lines of new code, all tested
 2025-12-15 11:33:21 -05:00
ilia
02c10c85d6 Add data update tools and Phase 2 plan 
- scripts/add_custom_trades.py: Manual trade entry
- scripts/scrape_alternative_sources.py: CSV import
- scripts/daily_update.sh: Automated daily updates
- docs/09_data_updates.md: Complete update guide
- docs/PR4_PLAN.md: Phase 2 analytics plan

Enables users to add representatives and set up auto-updates
 2025-12-15 10:39:18 -05:00
ilia
895c34e2c1 Add psycopg2-binary to dependencies for PostgreSQL support 
- Required for SQLAlchemy PostgreSQL connections
- Fixes: ModuleNotFoundError: No module named 'psycopg2'
 2025-12-14 21:12:36 -05:00
ilia
44ddd88879 Complete fix: Replace all sudo commands with su for LXC compatibility 
- Replace 'sudo -u' with 'su -' throughout the script
- Works perfectly in LXC containers running as root (no sudo installed)
- Also works on regular VMs/servers where sudo is available
- Fixes all remaining: sudo: command not found errors
 2025-12-14 21:05:35 -05:00
ilia
9bb39c9913 Fix proxmox_setup.sh to work when running as root in LXC 
- Detect if running as root and use 'su' instead of 'sudo' for postgres
- Fixes: sudo: command not found error in LXC containers
 2025-12-14 21:01:30 -05:00
ilia
204cd0e75b Initial commit: POTE Phase 1 complete 
- PR1: Project scaffold, DB models, price loader
- PR2: Congressional trade ingestion (House Stock Watcher)
- PR3: Security enrichment + deployment infrastructure
- 37 passing tests, 87%+ coverage
- Docker + Proxmox deployment ready
- Complete documentation
- Works 100% offline with fixtures
 2025-12-14 20:45:34 -05:00