ilia
|
0d8d85adc1
|
Add complete automation, reporting, and CI/CD system
Features Added:
==============
📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py
🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup
🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration
🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI
📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide
🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron
WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
- New trades, market alerts, suspicious timing
- Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)
USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh
# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com
# Check system health
python scripts/health_check.py
See AUTOMATION_QUICKSTART.md for full instructions.
93 tests passing | Full CI/CD | Email reports ready
|
2025-12-15 15:34:31 -05:00 |
|
ilia
|
6b62ae96f7
|
Phase 2: Disclosure Timing Correlation Engine
COMPLETE: Match congressional trades to prior market alerts
New Module:
- src/pote/monitoring/disclosure_correlator.py: Core correlation engine
* get_alerts_before_trade(): Find alerts before trade date
* calculate_timing_score(): Score suspicious timing (0-100 scale)
- Factors: alert count, severity, recency, type
- Thresholds: 60+ = suspicious, 80+ = highly suspicious
* analyze_trade(): Complete trade analysis with timing
* analyze_recent_disclosures(): Batch analysis of new filings
* get_official_timing_pattern(): Historical pattern analysis
* get_ticker_timing_analysis(): Per-stock timing patterns
Timing Score Algorithm:
- Base score: alert count × 5 + avg severity × 2
- Recency bonus: +10 per alert within 7 days
- Severity bonus: +15 per high-severity (7+) alert
- Total score: 0-100 (capped)
- Interpretation:
* 80-100: Highly suspicious (likely timing advantage)
* 60-79: Suspicious (possible timing advantage)
* 40-59: Notable (some unusual activity)
* 0-39: Normal (no significant pattern)
New Script:
- scripts/analyze_disclosure_timing.py: CLI analysis tool
* Analyze recent disclosures (--days N)
* Filter by timing score (--min-score)
* Analyze specific official (--official NAME)
* Analyze specific ticker (--ticker SYMBOL)
* Text/JSON output formats
* Detailed reports with prior alerts
Usage Examples:
# Find suspicious trades filed recently
python scripts/analyze_disclosure_timing.py --days 30 --min-score 60
# Analyze specific official
python scripts/analyze_disclosure_timing.py --official "Nancy Pelosi"
# Analyze specific ticker
python scripts/analyze_disclosure_timing.py --ticker NVDA
Report Includes:
- Timing score and suspicion level
- Prior alert details (count, severity, timing)
- Official name, ticker, trade details
- Assessment and reasoning
- Top suspicious trades ranked
Next: Phase 3 - Pattern Detection across officials/stocks
|
2025-12-15 15:17:09 -05:00 |
|