ilia/POTE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
29 Commits 4 Branches 0 Tags
Commit Graph

9 Commits

Author SHA1 Message Date
ilia
5161f6c421 Organize documentation: move setup guides to docs/ folder
Some checks failed
CI / lint-and-test (push) Has been cancelled
Details
CI / security-scan (push) Has been cancelled
Details
CI / dependency-scan (push) Has been cancelled
Details
CI / docker-build-test (push) Has been cancelled
Details
CI / workflow-summary (push) Has been cancelled
Details 
Moved:
- BRANCH_SETUP_COMPLETE.md → docs/15_branch_setup_checklist.md
- PIPELINE_SETUP_GUIDE.md → docs/16_pipeline_setup.md

Reason: Keep root directory clean, organize all docs in docs/ folder

Documentation structure now:
docs/
├── 14_branch_strategy_and_deployment.md (comprehensive guide)
├── 15_branch_setup_checklist.md (quick checklist)
└── 16_pipeline_setup.md (CI setup guide)
 2025-12-24 21:55:24 -05:00
ilia
01597f608f Add comprehensive branch strategy and multi-environment deployment guide 
NEW DOCUMENTATION:
==================
docs/14_branch_strategy_and_deployment.md

COVERS:
=======
✅ Branch Strategy (main/qa/dev)
✅ Branch Protection Rules for Gitea
✅ Multi-environment deployment workflows
✅ Integration with Ansible auto-deploy
✅ Gitea secrets per environment
✅ Deployment flow diagram
✅ Rollback strategies
✅ Complete setup checklist

BRANCH STRUCTURE:
=================
main (prod) ← qa (staging) ← dev (development)

PROTECTION LEVELS:
==================
- main: MOST PROTECTED (require PR + 2 approvals + CI)
- qa: MODERATELY PROTECTED (require PR + CI)
- dev: LIGHTLY PROTECTED (require CI only)

DEPLOYMENT WORKFLOWS:
=====================
- deploy-dev.yml: Auto-deploy on push to dev
- deploy-qa.yml: Auto-deploy on push to qa + smoke tests
- deploy-prod.yml: Manual approval + rollback on failure

ANSIBLE INTEGRATION:
====================
- Webhook-based deployment
- Gitea Actions → Ansible API
- Environment-specific inventory
- Automated health checks

WHAT'S MISSING (ANSWERED):
==========================
✅ Environment variables per env
✅ Database migration strategy
✅ Rollback procedures
✅ Monitoring & alerts
✅ Feature flags
✅ Changelog management

BRANCHES CREATED:
=================
✅ dev branch created and pushed
✅ qa branch created and pushed
✅ main branch (existing)

Ready for Ansible auto-configure integration!
 2025-12-24 21:46:37 -05:00
ilia
0c183fb28c Add comprehensive secrets management guide 
Covers 6 options for storing passwords securely:
1. .env file (current, good for personal use)
2. Environment variables (better for production)
3. Separate secrets file
4. Docker secrets
5. HashiCorp Vault (enterprise)
6. Git secrets (CI/CD only)

Recommendation: Current .env setup is fine for personal/research use
Improvement: chmod 600 .env (done)

Includes security checklist, rotation procedures, and testing
 2025-12-15 15:47:12 -05:00
ilia
0d8d85adc1 Add complete automation, reporting, and CI/CD system 
Features Added:
==============

📧 EMAIL REPORTING SYSTEM:
- EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom)
- ReportGenerator: Generate daily/weekly summaries with HTML/text formatting
- Configurable via .env (SMTP_HOST, SMTP_PORT, etc.)
- Scripts: send_daily_report.py, send_weekly_report.py

🤖 AUTOMATED RUNS:
- automated_daily_run.sh: Full daily ETL pipeline + reporting
- automated_weekly_run.sh: Weekly pattern analysis + reports
- setup_cron.sh: Interactive cron job setup (5-minute setup)
- Logs saved to ~/logs/ with automatic cleanup

🔍 HEALTH CHECKS:
- health_check.py: System health monitoring
- Checks: DB connection, data freshness, counts, recent alerts
- JSON output for programmatic use
- Exit codes for monitoring integration

🚀 CI/CD PIPELINE:
- .github/workflows/ci.yml: Full CI/CD pipeline
- GitHub Actions / Gitea Actions compatible
- Jobs: lint & test, security scan, dependency scan, Docker build
- PostgreSQL service for integration tests
- 93 tests passing in CI

📚 COMPREHENSIVE DOCUMENTATION:
- AUTOMATION_QUICKSTART.md: 5-minute email setup guide
- docs/12_automation_and_reporting.md: Full automation guide
- Updated README.md with automation links
- Deployment → Production workflow guide

🛠️ IMPROVEMENTS:
- All shell scripts made executable
- Environment variable examples in .env.example
- Report logs saved with timestamps
- 30-day log retention with auto-cleanup
- Health checks can be scheduled via cron

WHAT THIS ENABLES:
==================
After deployment, users can:
1. Set up automated daily/weekly email reports (5 min)
2. Receive HTML+text emails with:
   - New trades, market alerts, suspicious timing
   - Weekly patterns, rankings, repeat offenders
3. Monitor system health automatically
4. Run full CI/CD pipeline on every commit
5. Deploy with confidence (tests + security scans)

USAGE:
======
# One-time setup (on deployed server)
./scripts/setup_cron.sh

# Or manually send reports
python scripts/send_daily_report.py --to user@example.com
python scripts/send_weekly_report.py --to user@example.com

# Check system health
python scripts/health_check.py

See AUTOMATION_QUICKSTART.md for full instructions.

93 tests passing | Full CI/CD | Email reports ready
 2025-12-15 15:34:31 -05:00
ilia
cfaf38b0be Phase 1: Real-Time Market Monitoring System 
COMPLETE: Real-time unusual activity detection for congressional tickers

New Database Model:
- MarketAlert: Stores unusual market activity alerts
  * Tracks volume spikes, price movements, volatility
  * JSON details field for flexible data storage
  * Severity scoring (1-10 scale)
  * Indexed for efficient queries by ticker/timestamp

New Modules:
- src/pote/monitoring/market_monitor.py: Core monitoring engine
  * get_congressional_watchlist(): Top 50 most-traded tickers
  * check_ticker(): Analyze single stock for unusual activity
  * scan_watchlist(): Batch analysis of multiple tickers
  * Detection logic:
    - Unusual volume (3x average)
    - Price spikes/drops (>5%)
    - High volatility (2x normal)
  * save_alerts(): Persist to database
  * get_recent_alerts(): Query historical alerts

- src/pote/monitoring/alert_manager.py: Alert formatting & filtering
  * format_alert_text(): Human-readable output
  * format_alert_html(): HTML email format
  * filter_alerts(): By severity, ticker, type
  * generate_summary_report(): Text/HTML reports

Scripts:
- scripts/monitor_market.py: CLI monitoring tool
  * Continuous monitoring mode (--interval)
  * One-time scan (--once)
  * Custom ticker lists or auto-detect congressional watchlist
  * Severity filtering (--min-severity)
  * Report generation and saving

Migrations:
- alembic/versions/f44014715b40_add_market_alerts_table.py

Documentation:
- docs/11_live_market_monitoring.md: Complete explanation
  * Why you can't track WHO is trading
  * What IS possible (timing analysis)
  * How hybrid monitoring works
  * Data sources and APIs

Usage:
  # Monitor congressional tickers (one-time scan)
  python scripts/monitor_market.py --once

  # Continuous monitoring (every 5 minutes)
  python scripts/monitor_market.py --interval 300

  # Monitor specific tickers
  python scripts/monitor_market.py --tickers NVDA,MSFT,AAPL --once

Next Steps (Phase 2):
- Disclosure correlation engine
- Timing advantage calculator
- Suspicious trade flagging
 2025-12-15 15:10:49 -05:00
ilia
3a89c1e6d2 Add comprehensive automation system 
New Scripts:
- scripts/daily_fetch.sh: Automated daily data updates
  * Fetches congressional trades (last 7 days)
  * Enriches securities (name, sector, industry)
  * Updates price data for all securities
  * Calculates returns and metrics
  * Logs everything to logs/ directory

- scripts/setup_automation.sh: Interactive automation setup
  * Makes scripts executable
  * Creates log directories
  * Configures cron jobs (multiple schedule options)
  * Guides user through setup

Documentation:
- docs/10_automation.md: Complete automation guide
  * Explains disclosure timing (30-45 day legal lag)
  * Why daily updates are optimal (not hourly/real-time)
  * Cron job setup instructions
  * Systemd timer alternative
  * Email notifications (optional)
  * Monitoring and logging
  * Failure handling
  * Performance optimization

Key Insights:
❌ No real-time data possible (STOCK Act = 30-45 day lag)
✅ Daily updates are optimal
✅ Automated via cron jobs
✅ Handles API failures gracefully
✅ Logs everything for debugging
 2025-12-15 14:55:05 -05:00
ilia
34aebb1c2e PR4: Phase 2 Analytics Foundation 
Complete analytics module with returns, benchmarks, and performance metrics.

New Modules:
- src/pote/analytics/returns.py: Return calculator for trades
- src/pote/analytics/benchmarks.py: Benchmark comparison & alpha
- src/pote/analytics/metrics.py: Performance aggregations

Scripts:
- scripts/analyze_official.py: Analyze specific official
- scripts/calculate_all_returns.py: System-wide analysis

Tests:
- tests/test_analytics.py: Full coverage of analytics

Features:
✅ Calculate returns over 30/60/90/180 day windows
✅ Compare to market benchmarks (SPY, QQQ, etc.)
✅ Calculate abnormal returns (alpha)
✅ Aggregate stats by official, sector
✅ Top performer rankings
✅ Disclosure timing analysis
✅ Command-line analysis tools

~1,210 lines of new code, all tested
 2025-12-15 11:33:21 -05:00
ilia
02c10c85d6 Add data update tools and Phase 2 plan 
- scripts/add_custom_trades.py: Manual trade entry
- scripts/scrape_alternative_sources.py: CSV import
- scripts/daily_update.sh: Automated daily updates
- docs/09_data_updates.md: Complete update guide
- docs/PR4_PLAN.md: Phase 2 analytics plan

Enables users to add representatives and set up auto-updates
 2025-12-15 10:39:18 -05:00
ilia
204cd0e75b Initial commit: POTE Phase 1 complete 
- PR1: Project scaffold, DB models, price loader
- PR2: Congressional trade ingestion (House Stock Watcher)
- PR3: Security enrichment + deployment infrastructure
- 37 passing tests, 87%+ coverage
- Docker + Proxmox deployment ready
- Complete documentation
- Works 100% offline with fixtures
 2025-12-14 20:45:34 -05:00