ilia/POTE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23 Commits 4 Branches 0 Tags
Commit Graph

7 Commits

Author SHA1 Message Date
ilia
0d8d85adc1 Add complete automation, reporting, and CI/CD system 
Features Added:
==============

📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py

🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup

🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration

🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI

📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide

🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron

WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
   - New trades, market alerts, suspicious timing
   - Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)

USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh

# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com

# Check system health
python scripts/health_check.py

See AUTOMATION_QUICKSTART.md for full instructions.

93 tests passing | Full CI/CD | Email reports ready
 2025-12-15 15:34:31 -05:00
ilia
2ec4a8e373 Phase 3: Pattern Detection & Comparative Analysis - COMPLETE 
COMPLETE: Cross-official pattern detection and ranking system

New Module:
- src/pote/monitoring/pattern_detector.py: Pattern analysis engine
  * rank_officials_by_timing(): Rank all officials by suspicion
  * identify_repeat_offenders(): Find systematic offenders
  * analyze_ticker_patterns(): Per-stock suspicious patterns
  * get_sector_timing_analysis(): Sector-level analysis
  * get_party_comparison(): Democrat vs Republican comparison
  * generate_pattern_report(): Comprehensive report

Analysis Features:
- Official Rankings:
  * By average timing score
  * Suspicious trade percentage
  * Alert rates
  * Pattern classification

- Repeat Offender Detection:
  * Identifies officials with 50%+ suspicious trades
  * Historical pattern tracking
  * Systematic timing advantage detection

- Comparative Analysis:
  * Cross-party comparison
  * Sector analysis
  * Ticker-specific patterns
  * Statistical aggregations

New Script:
- scripts/generate_pattern_report.py: Comprehensive reports
  * Top 10 most suspicious officials
  * Repeat offenders list
  * Most suspiciously traded stocks
  * Sector breakdowns
  * Party comparison stats
  * Text/JSON formats

New Tests (11 total, all passing):
- test_rank_officials_by_timing
- test_identify_repeat_offenders
- test_analyze_ticker_patterns
- test_get_sector_timing_analysis
- test_get_party_comparison
- test_generate_pattern_report
- test_rank_officials_min_trades_filter
- test_empty_data_handling
- test_ranking_score_accuracy
- test_sector_stats_accuracy
- test_party_stats_completeness

Usage:
  python scripts/generate_pattern_report.py --days 365

Report Includes:
- Top suspicious officials ranked
- Repeat offenders (50%+ suspicious rate)
- Most suspiciously traded tickers
- Sector analysis
- Party comparison
- Interpretation guide

Total Test Suite: 93 tests passing ✅

ALL 3 PHASES COMPLETE!
 2025-12-15 15:23:40 -05:00
ilia
a52313145b Add comprehensive tests for Phase 2 correlation engine 
New Tests (13 total, all passing):
- test_get_alerts_before_trade: Retrieve prior alerts
- test_get_alerts_before_trade_no_alerts: Handle no alerts
- test_calculate_timing_score_high_suspicion: High score logic
- test_calculate_timing_score_no_alerts: Zero score handling
- test_calculate_timing_score_factors: Multi-factor scoring
- test_analyze_trade_full: Complete trade analysis
- test_analyze_recent_disclosures: Batch processing
- test_get_official_timing_pattern: Historical patterns
- test_get_official_timing_pattern_no_trades: Edge case
- test_get_ticker_timing_analysis: Per-ticker analysis
- test_get_ticker_timing_analysis_no_trades: Edge case
- test_alerts_outside_lookback_window: Date filtering
- test_different_ticker_alerts_excluded: Ticker filtering

Test Coverage:
- Alert-to-trade correlation
- Timing score calculation (all factors)
- Pattern analysis (officials & tickers)
- Batch analysis
- Edge cases & filtering
- Date range handling

Total Test Suite: 82 tests passing ✅
 2025-12-15 15:20:40 -05:00
ilia
db34f26cdc Add comprehensive tests for Phase 1 monitoring system 
New Tests (14 total, all passing):
- test_get_congressional_watchlist: Auto-detect most-traded tickers
- test_check_ticker_basic: Single ticker analysis
- test_scan_watchlist_with_mock: Batch scanning with controlled data
- test_save_alerts: Database persistence
- test_get_recent_alerts: Query filtering (ticker, type, severity, date)
- test_get_ticker_alert_summary: Aggregated statistics
- test_alert_manager_format_text: Text formatting
- test_alert_manager_format_html: HTML formatting
- test_alert_manager_filter_alerts: Multi-criteria filtering
- test_alert_manager_generate_summary_text: Report generation
- test_alert_manager_generate_summary_html: HTML reports
- test_alert_manager_empty_alerts: Edge case handling
- test_market_alert_model: ORM model validation
- test_alert_timestamp_filtering: Time-based queries

Test Coverage:
- Market monitoring core logic
- Alert detection algorithms
- Database operations
- Filtering and querying
- Report generation (text/HTML)
- Edge cases and error handling

Total Test Suite: 69 tests passing ✅
 2025-12-15 15:14:58 -05:00
ilia
b4e6a7c340 Fix analytics tests and add comprehensive testing guide 
Critical Fixes:
- Fixed Price model query to use security_id join with Security
- Added Security import to returns.py module
- Fixed all test fixtures to use test_db_session correctly
- Added AAPL price data to sample_prices fixture

New Tests:
- tests/test_analytics_integration.py: 10 comprehensive integration tests
  * Real-world scenarios with synthetic price data
  * Return calculations, benchmark comparisons, performance metrics
  * Edge cases: missing data, sell trades, disclosure timing

Documentation:
- LOCAL_TEST_GUIDE.md: Complete guide for local testing
  * How to test before deploying
  * Current data status (live vs fixtures)
  * Multiple options for getting real data
  * Common issues and fixes

Test Results:
✅ All 55 tests passing
✅ Analytics fully functional
✅ Ready for deployment

Live Data Status:
❌ House Stock Watcher API still down (external issue)
✅ Manual CSV import works
✅ yfinance for prices works
✅ Can use system NOW with manual data
 2025-12-15 14:42:20 -05:00
ilia
34aebb1c2e PR4: Phase 2 Analytics Foundation 
Complete analytics module with returns, benchmarks, and performance metrics.

New Modules:
- src/pote/analytics/returns.py: Return calculator for trades
- src/pote/analytics/benchmarks.py: Benchmark comparison & alpha
- src/pote/analytics/metrics.py: Performance aggregations

Scripts:
- scripts/analyze_official.py: Analyze specific official
- scripts/calculate_all_returns.py: System-wide analysis

Tests:
- tests/test_analytics.py: Full coverage of analytics

Features:
✅ Calculate returns over 30/60/90/180 day windows
✅ Compare to market benchmarks (SPY, QQQ, etc.)
✅ Calculate abnormal returns (alpha)
✅ Aggregate stats by official, sector
✅ Top performer rankings
✅ Disclosure timing analysis
✅ Command-line analysis tools

~1,210 lines of new code, all tested
 2025-12-15 11:33:21 -05:00
ilia
204cd0e75b Initial commit: POTE Phase 1 complete 
- PR1: Project scaffold, DB models, price loader
- PR2: Congressional trade ingestion (House Stock Watcher)
- PR3: Security enrichment + deployment infrastructure
- 37 passing tests, 87%+ coverage
- Docker + Proxmox deployment ready
- Complete documentation
- Works 100% offline with fixtures
 2025-12-14 20:45:34 -05:00