ilia
|
0d8d85adc1
|
Add complete automation, reporting, and CI/CD system
Features Added:
==============
📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py
🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup
🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration
🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI
📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide
🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron
WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
- New trades, market alerts, suspicious timing
- Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)
USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh
# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com
# Check system health
python scripts/health_check.py
See AUTOMATION_QUICKSTART.md for full instructions.
93 tests passing | Full CI/CD | Email reports ready
|
2025-12-15 15:34:31 -05:00 |
|
ilia
|
cfaf38b0be
|
Phase 1: Real-Time Market Monitoring System
COMPLETE: Real-time unusual activity detection for congressional tickers
New Database Model:
- MarketAlert: Stores unusual market activity alerts
* Tracks volume spikes, price movements, volatility
* JSON details field for flexible data storage
* Severity scoring (1-10 scale)
* Indexed for efficient queries by ticker/timestamp
New Modules:
- src/pote/monitoring/market_monitor.py: Core monitoring engine
* get_congressional_watchlist(): Top 50 most-traded tickers
* check_ticker(): Analyze single stock for unusual activity
* scan_watchlist(): Batch analysis of multiple tickers
* Detection logic:
- Unusual volume (3x average)
- Price spikes/drops (>5%)
- High volatility (2x normal)
* save_alerts(): Persist to database
* get_recent_alerts(): Query historical alerts
- src/pote/monitoring/alert_manager.py: Alert formatting & filtering
* format_alert_text(): Human-readable output
* format_alert_html(): HTML email format
* filter_alerts(): By severity, ticker, type
* generate_summary_report(): Text/HTML reports
Scripts:
- scripts/monitor_market.py: CLI monitoring tool
* Continuous monitoring mode (--interval)
* One-time scan (--once)
* Custom ticker lists or auto-detect congressional watchlist
* Severity filtering (--min-severity)
* Report generation and saving
Migrations:
- alembic/versions/f44014715b40_add_market_alerts_table.py
Documentation:
- docs/11_live_market_monitoring.md: Complete explanation
* Why you can't track WHO is trading
* What IS possible (timing analysis)
* How hybrid monitoring works
* Data sources and APIs
Usage:
# Monitor congressional tickers (one-time scan)
python scripts/monitor_market.py --once
# Continuous monitoring (every 5 minutes)
python scripts/monitor_market.py --interval 300
# Monitor specific tickers
python scripts/monitor_market.py --tickers NVDA,MSFT,AAPL --once
Next Steps (Phase 2):
- Disclosure correlation engine
- Timing advantage calculator
- Suspicious trade flagging
|
2025-12-15 15:10:49 -05:00 |
|