ilia
|
0d8d85adc1
|
Add complete automation, reporting, and CI/CD system
Features Added:
==============
📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py
🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup
🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration
🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI
📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide
🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron
WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
- New trades, market alerts, suspicious timing
- Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)
USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh
# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com
# Check system health
python scripts/health_check.py
See AUTOMATION_QUICKSTART.md for full instructions.
93 tests passing | Full CI/CD | Email reports ready
|
2025-12-15 15:34:31 -05:00 |
|
ilia
|
3a89c1e6d2
|
Add comprehensive automation system
New Scripts:
- scripts/daily_fetch.sh: Automated daily data updates
* Fetches congressional trades (last 7 days)
* Enriches securities (name, sector, industry)
* Updates price data for all securities
* Calculates returns and metrics
* Logs everything to logs/ directory
- scripts/setup_automation.sh: Interactive automation setup
* Makes scripts executable
* Creates log directories
* Configures cron jobs (multiple schedule options)
* Guides user through setup
Documentation:
- docs/10_automation.md: Complete automation guide
* Explains disclosure timing (30-45 day legal lag)
* Why daily updates are optimal (not hourly/real-time)
* Cron job setup instructions
* Systemd timer alternative
* Email notifications (optional)
* Monitoring and logging
* Failure handling
* Performance optimization
Key Insights:
❌ No real-time data possible (STOCK Act = 30-45 day lag)
✅ Daily updates are optimal
✅ Automated via cron jobs
✅ Handles API failures gracefully
✅ Logs everything for debugging
|
2025-12-15 14:55:05 -05:00 |
|