26 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
ilia	659896f096	Add pipeline setup guide for branch protection	2025-12-24 21:54:25 -05:00
ilia	3910ca9d04	Update CI to run on all branches (main, qa, dev)	2025-12-24 21:53:14 -05:00
ilia	9940100239	Add branch setup completion guide and checklist	2025-12-24 21:47:37 -05:00
ilia	01597f608f	Add comprehensive branch strategy and multi-environment deployment guide ... NEW DOCUMENTATION: ================== docs/14_branch_strategy_and_deployment.md COVERS: ======= ✅ Branch Strategy (main/qa/dev) ✅ Branch Protection Rules for Gitea ✅ Multi-environment deployment workflows ✅ Integration with Ansible auto-deploy ✅ Gitea secrets per environment ✅ Deployment flow diagram ✅ Rollback strategies ✅ Complete setup checklist BRANCH STRUCTURE: ================= main (prod) ← qa (staging) ← dev (development) PROTECTION LEVELS: ================== - main: MOST PROTECTED (require PR + 2 approvals + CI) - qa: MODERATELY PROTECTED (require PR + CI) - dev: LIGHTLY PROTECTED (require CI only) DEPLOYMENT WORKFLOWS: ===================== - deploy-dev.yml: Auto-deploy on push to dev - deploy-qa.yml: Auto-deploy on push to qa + smoke tests - deploy-prod.yml: Manual approval + rollback on failure ANSIBLE INTEGRATION: ==================== - Webhook-based deployment - Gitea Actions → Ansible API - Environment-specific inventory - Automated health checks WHAT'S MISSING (ANSWERED): ========================== ✅ Environment variables per env ✅ Database migration strategy ✅ Rollback procedures ✅ Monitoring & alerts ✅ Feature flags ✅ Changelog management BRANCHES CREATED: ================= ✅ dev branch created and pushed ✅ qa branch created and pushed ✅ main branch (existing) Ready for Ansible auto-configure integration!	2025-12-24 21:46:37 -05:00
ilia	ead0820cf9	Add Gitea Secrets integration for CI/CD and deployment ... NEW FEATURES: ============ 📁 GITEA_SECRETS_GUIDE.md: - Comprehensive guide on using Gitea secrets - Store passwords in Gitea (not in git!) - Use in CI/CD and deployment workflows - Best practices and security recommendations 🔧 .github/workflows/ci.yml (UPDATED): - Now uses Gitea secrets with fallbacks - ${{ secrets.SMTP_PASSWORD || 'testpass123' }} - ${{ secrets.DB_PASSWORD || 'testpass123' }} - Tests run with real credentials from Gitea 🚀 .github/workflows/deploy.yml (NEW): - Automated deployment to Proxmox - Manual trigger via Gitea UI - Steps: 1. SSH to Proxmox with secrets.PROXMOX_SSH_KEY 2. Pull latest code 3. Update .env with secrets from Gitea 4. Run migrations 5. Health check 6. Test email 7. Rollback on failure HOW IT WORKS: ============= 1. Store passwords in Gitea (Settings → Secrets) 2. CI/CD uses secrets automatically 3. Deployment workflow updates .env on Proxmox 4. Best of both worlds: secure CI + simple runtime SECRETS TO ADD IN GITEA: ======================== - SMTP_PASSWORD: your mail password - DB_PASSWORD: changeme123 - PROXMOX_HOST: 10.0.10.95 - PROXMOX_USER: poteapp - PROXMOX_SSH_KEY: (SSH private key) - SMTP_HOST: mail.levkin.ca - SMTP_USER: test@levkin.ca - FROM_EMAIL: test@levkin.ca USAGE: ====== # In Gitea UI: Actions → Deploy to Proxmox → Run workflow # Or push commits: git push origin main # CI runs with secrets automatically See GITEA_SECRETS_GUIDE.md for full instructions!	2025-12-15 15:52:19 -05:00
ilia	0c183fb28c	Add comprehensive secrets management guide ... Covers 6 options for storing passwords securely: 1. .env file (current, good for personal use) 2. Environment variables (better for production) 3. Separate secrets file 4. Docker secrets 5. HashiCorp Vault (enterprise) 6. Git secrets (CI/CD only) Recommendation: Current .env setup is fine for personal/research use Improvement: chmod 600 .env (done) Includes security checklist, rotation procedures, and testing	2025-12-15 15:47:12 -05:00
ilia	5613d7f894	Add quick setup card with levkin.ca configuration ... One-page reference with: - Your specific configuration (test@levkin.ca) - 3-step setup (30 seconds + 1 min + 2 min) - Quick commands for deployed server - Troubleshooting checklist - Current status tracker - Next actions Everything ready - just needs password in .env	2025-12-15 15:44:30 -05:00
ilia	07af492026	Add email setup guide for levkin.ca mail server ... Configuration for test@levkin.ca: - SMTP: mail.levkin.ca:587 (STARTTLS) - Includes setup instructions - Testing checklist - Troubleshooting guide Note: .env file created locally (not committed, in .gitignore)	2025-12-15 15:43:44 -05:00
ilia	d8f723bafb	Add comprehensive deployment and automation FAQ ... Answers user's questions: - What happens after deployment? (nothing automatic by default) - How to get reports? (3 options: email, SSH, future web UI) - Where are reports sent? (email or saved to ~/logs/) - Do you need to check IP? (depends on setup method) - Can we setup email reports? (YES! 5-minute setup) - Do we need CI/CD pipelines? (optional, but included) - Can we use existing Ansible pipeline? (concepts reused, not directly) This document ties everything together and provides clear next steps.	2025-12-15 15:35:33 -05:00
ilia	0d8d85adc1	Add complete automation, reporting, and CI/CD system ... Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready	2025-12-15 15:34:31 -05:00
ilia	53d631a903	Add comprehensive monitoring system documentation ... Complete summary of all 3 phases: - Phase 1: Real-time market monitoring - Phase 2: Disclosure timing correlation - Phase 3: Pattern detection & rankings Documentation includes: - System architecture diagram - Usage guide for all phases - Example reports - Test coverage summary - Deployment checklist - Interpretation guide - Legal/ethical disclaimers - Automated workflow examples Total Achievement: ✅ 93 tests passing ✅ All 3 phases complete ✅ Production-ready system ✅ Full documentation The POTE monitoring system is now complete!	2025-12-15 15:25:07 -05:00
ilia	2ec4a8e373	Phase 3: Pattern Detection & Comparative Analysis - COMPLETE ... COMPLETE: Cross-official pattern detection and ranking system New Module: - src/pote/monitoring/pattern_detector.py: Pattern analysis engine * rank_officials_by_timing(): Rank all officials by suspicion * identify_repeat_offenders(): Find systematic offenders * analyze_ticker_patterns(): Per-stock suspicious patterns * get_sector_timing_analysis(): Sector-level analysis * get_party_comparison(): Democrat vs Republican comparison * generate_pattern_report(): Comprehensive report Analysis Features: - Official Rankings: * By average timing score * Suspicious trade percentage * Alert rates * Pattern classification - Repeat Offender Detection: * Identifies officials with 50%+ suspicious trades * Historical pattern tracking * Systematic timing advantage detection - Comparative Analysis: * Cross-party comparison * Sector analysis * Ticker-specific patterns * Statistical aggregations New Script: - scripts/generate_pattern_report.py: Comprehensive reports * Top 10 most suspicious officials * Repeat offenders list * Most suspiciously traded stocks * Sector breakdowns * Party comparison stats * Text/JSON formats New Tests (11 total, all passing): - test_rank_officials_by_timing - test_identify_repeat_offenders - test_analyze_ticker_patterns - test_get_sector_timing_analysis - test_get_party_comparison - test_generate_pattern_report - test_rank_officials_min_trades_filter - test_empty_data_handling - test_ranking_score_accuracy - test_sector_stats_accuracy - test_party_stats_completeness Usage: python scripts/generate_pattern_report.py --days 365 Report Includes: - Top suspicious officials ranked - Repeat offenders (50%+ suspicious rate) - Most suspiciously traded tickers - Sector analysis - Party comparison - Interpretation guide Total Test Suite: 93 tests passing ✅ ALL 3 PHASES COMPLETE!	2025-12-15 15:23:40 -05:00
ilia	a52313145b	Add comprehensive tests for Phase 2 correlation engine ... New Tests (13 total, all passing): - test_get_alerts_before_trade: Retrieve prior alerts - test_get_alerts_before_trade_no_alerts: Handle no alerts - test_calculate_timing_score_high_suspicion: High score logic - test_calculate_timing_score_no_alerts: Zero score handling - test_calculate_timing_score_factors: Multi-factor scoring - test_analyze_trade_full: Complete trade analysis - test_analyze_recent_disclosures: Batch processing - test_get_official_timing_pattern: Historical patterns - test_get_official_timing_pattern_no_trades: Edge case - test_get_ticker_timing_analysis: Per-ticker analysis - test_get_ticker_timing_analysis_no_trades: Edge case - test_alerts_outside_lookback_window: Date filtering - test_different_ticker_alerts_excluded: Ticker filtering Test Coverage: - Alert-to-trade correlation - Timing score calculation (all factors) - Pattern analysis (officials & tickers) - Batch analysis - Edge cases & filtering - Date range handling Total Test Suite: 82 tests passing ✅	2025-12-15 15:20:40 -05:00
ilia	6b62ae96f7	Phase 2: Disclosure Timing Correlation Engine ... COMPLETE: Match congressional trades to prior market alerts New Module: - src/pote/monitoring/disclosure_correlator.py: Core correlation engine * get_alerts_before_trade(): Find alerts before trade date * calculate_timing_score(): Score suspicious timing (0-100 scale) - Factors: alert count, severity, recency, type - Thresholds: 60+ = suspicious, 80+ = highly suspicious * analyze_trade(): Complete trade analysis with timing * analyze_recent_disclosures(): Batch analysis of new filings * get_official_timing_pattern(): Historical pattern analysis * get_ticker_timing_analysis(): Per-stock timing patterns Timing Score Algorithm: - Base score: alert count × 5 + avg severity × 2 - Recency bonus: +10 per alert within 7 days - Severity bonus: +15 per high-severity (7+) alert - Total score: 0-100 (capped) - Interpretation: * 80-100: Highly suspicious (likely timing advantage) * 60-79: Suspicious (possible timing advantage) * 40-59: Notable (some unusual activity) * 0-39: Normal (no significant pattern) New Script: - scripts/analyze_disclosure_timing.py: CLI analysis tool * Analyze recent disclosures (--days N) * Filter by timing score (--min-score) * Analyze specific official (--official NAME) * Analyze specific ticker (--ticker SYMBOL) * Text/JSON output formats * Detailed reports with prior alerts Usage Examples: # Find suspicious trades filed recently python scripts/analyze_disclosure_timing.py --days 30 --min-score 60 # Analyze specific official python scripts/analyze_disclosure_timing.py --official "Nancy Pelosi" # Analyze specific ticker python scripts/analyze_disclosure_timing.py --ticker NVDA Report Includes: - Timing score and suspicion level - Prior alert details (count, severity, timing) - Official name, ticker, trade details - Assessment and reasoning - Top suspicious trades ranked Next: Phase 3 - Pattern Detection across officials/stocks	2025-12-15 15:17:09 -05:00
ilia	db34f26cdc	Add comprehensive tests for Phase 1 monitoring system ... New Tests (14 total, all passing): - test_get_congressional_watchlist: Auto-detect most-traded tickers - test_check_ticker_basic: Single ticker analysis - test_scan_watchlist_with_mock: Batch scanning with controlled data - test_save_alerts: Database persistence - test_get_recent_alerts: Query filtering (ticker, type, severity, date) - test_get_ticker_alert_summary: Aggregated statistics - test_alert_manager_format_text: Text formatting - test_alert_manager_format_html: HTML formatting - test_alert_manager_filter_alerts: Multi-criteria filtering - test_alert_manager_generate_summary_text: Report generation - test_alert_manager_generate_summary_html: HTML reports - test_alert_manager_empty_alerts: Edge case handling - test_market_alert_model: ORM model validation - test_alert_timestamp_filtering: Time-based queries Test Coverage: - Market monitoring core logic - Alert detection algorithms - Database operations - Filtering and querying - Report generation (text/HTML) - Edge cases and error handling Total Test Suite: 69 tests passing ✅	2025-12-15 15:14:58 -05:00
ilia	cfaf38b0be	Phase 1: Real-Time Market Monitoring System ... COMPLETE: Real-time unusual activity detection for congressional tickers New Database Model: - MarketAlert: Stores unusual market activity alerts * Tracks volume spikes, price movements, volatility * JSON details field for flexible data storage * Severity scoring (1-10 scale) * Indexed for efficient queries by ticker/timestamp New Modules: - src/pote/monitoring/market_monitor.py: Core monitoring engine * get_congressional_watchlist(): Top 50 most-traded tickers * check_ticker(): Analyze single stock for unusual activity * scan_watchlist(): Batch analysis of multiple tickers * Detection logic: - Unusual volume (3x average) - Price spikes/drops (>5%) - High volatility (2x normal) * save_alerts(): Persist to database * get_recent_alerts(): Query historical alerts - src/pote/monitoring/alert_manager.py: Alert formatting & filtering * format_alert_text(): Human-readable output * format_alert_html(): HTML email format * filter_alerts(): By severity, ticker, type * generate_summary_report(): Text/HTML reports Scripts: - scripts/monitor_market.py: CLI monitoring tool * Continuous monitoring mode (--interval) * One-time scan (--once) * Custom ticker lists or auto-detect congressional watchlist * Severity filtering (--min-severity) * Report generation and saving Migrations: - alembic/versions/f44014715b40_add_market_alerts_table.py Documentation: - docs/11_live_market_monitoring.md: Complete explanation * Why you can't track WHO is trading * What IS possible (timing analysis) * How hybrid monitoring works * Data sources and APIs Usage: # Monitor congressional tickers (one-time scan) python scripts/monitor_market.py --once # Continuous monitoring (every 5 minutes) python scripts/monitor_market.py --interval 300 # Monitor specific tickers python scripts/monitor_market.py --tickers NVDA,MSFT,AAPL --once Next Steps (Phase 2): - Disclosure correlation engine - Timing advantage calculator - Suspicious trade flagging	2025-12-15 15:10:49 -05:00
ilia	8ba9d7ffdd	Add watchlist system and pre-market trading reports ... New Features: - Watchlist system for tracking specific Congress members - Trading report generation with multiple formats - Pre-market-close automated updates (3 PM) New Scripts: - scripts/fetch_congress_members.py: Manage watchlist * 29 known active traders (curated list) * Optional ProPublica API integration (all 535 members) * Create/view/manage watchlist - scripts/generate_trading_report.py: Generate trading reports * Filter by watchlist or show all * Multiple formats: text, HTML, JSON * Summary statistics (buys/sells, top tickers) * Color-coded output (🟢 BUY, 🔴 SELL) - scripts/pre_market_close_update.sh: 3 PM automation * Quick fetch of latest trades * Enrichment of new securities * Generate and display report * Saves to reports/ directory Documentation: - WATCHLIST_GUIDE.md: Complete guide * List of 29 known active traders * How to create/customize watchlist * Schedule options (pre-market, post-market) * Email setup (optional) * FAQ and examples Known Active Traders Include: Senate: Tuberville, Rand Paul, Mark Warner, Rick Scott House: Pelosi, Crenshaw, MTG, Gottheimer, Brian Higgins Use Cases: ✅ Daily reports at 3 PM (1 hour before close) ✅ See what Congress bought/sold recently ✅ Track specific members you care about ✅ Export to HTML/JSON for further analysis	2025-12-15 15:00:42 -05:00
ilia	3a89c1e6d2	Add comprehensive automation system ... New Scripts: - scripts/daily_fetch.sh: Automated daily data updates * Fetches congressional trades (last 7 days) * Enriches securities (name, sector, industry) * Updates price data for all securities * Calculates returns and metrics * Logs everything to logs/ directory - scripts/setup_automation.sh: Interactive automation setup * Makes scripts executable * Creates log directories * Configures cron jobs (multiple schedule options) * Guides user through setup Documentation: - docs/10_automation.md: Complete automation guide * Explains disclosure timing (30-45 day legal lag) * Why daily updates are optimal (not hourly/real-time) * Cron job setup instructions * Systemd timer alternative * Email notifications (optional) * Monitoring and logging * Failure handling * Performance optimization Key Insights: ❌ No real-time data possible (STOCK Act = 30-45 day lag) ✅ Daily updates are optimal ✅ Automated via cron jobs ✅ Handles API failures gracefully ✅ Logs everything for debugging	2025-12-15 14:55:05 -05:00
ilia	77bd69b85c	Add comprehensive testing status documentation	2025-12-15 14:43:52 -05:00
ilia	b4e6a7c340	Fix analytics tests and add comprehensive testing guide ... Critical Fixes: - Fixed Price model query to use security_id join with Security - Added Security import to returns.py module - Fixed all test fixtures to use test_db_session correctly - Added AAPL price data to sample_prices fixture New Tests: - tests/test_analytics_integration.py: 10 comprehensive integration tests * Real-world scenarios with synthetic price data * Return calculations, benchmark comparisons, performance metrics * Edge cases: missing data, sell trades, disclosure timing Documentation: - LOCAL_TEST_GUIDE.md: Complete guide for local testing * How to test before deploying * Current data status (live vs fixtures) * Multiple options for getting real data * Common issues and fixes Test Results: ✅ All 55 tests passing ✅ Analytics fully functional ✅ Ready for deployment Live Data Status: ❌ House Stock Watcher API still down (external issue) ✅ Manual CSV import works ✅ yfinance for prices works ✅ Can use system NOW with manual data	2025-12-15 14:42:20 -05:00
ilia	34aebb1c2e	PR4: Phase 2 Analytics Foundation ... Complete analytics module with returns, benchmarks, and performance metrics. New Modules: - src/pote/analytics/returns.py: Return calculator for trades - src/pote/analytics/benchmarks.py: Benchmark comparison & alpha - src/pote/analytics/metrics.py: Performance aggregations Scripts: - scripts/analyze_official.py: Analyze specific official - scripts/calculate_all_returns.py: System-wide analysis Tests: - tests/test_analytics.py: Full coverage of analytics Features: ✅ Calculate returns over 30/60/90/180 day windows ✅ Compare to market benchmarks (SPY, QQQ, etc.) ✅ Calculate abnormal returns (alpha) ✅ Aggregate stats by official, sector ✅ Top performer rankings ✅ Disclosure timing analysis ✅ Command-line analysis tools ~1,210 lines of new code, all tested	2025-12-15 11:33:21 -05:00
ilia	02c10c85d6	Add data update tools and Phase 2 plan ... - scripts/add_custom_trades.py: Manual trade entry - scripts/scrape_alternative_sources.py: CSV import - scripts/daily_update.sh: Automated daily updates - docs/09_data_updates.md: Complete update guide - docs/PR4_PLAN.md: Phase 2 analytics plan Enables users to add representatives and set up auto-updates	2025-12-15 10:39:18 -05:00
ilia	895c34e2c1	Add psycopg2-binary to dependencies for PostgreSQL support ... - Required for SQLAlchemy PostgreSQL connections - Fixes: ModuleNotFoundError: No module named 'psycopg2'	2025-12-14 21:12:36 -05:00
ilia	44ddd88879	Complete fix: Replace all sudo commands with su for LXC compatibility ... - Replace 'sudo -u' with 'su -' throughout the script - Works perfectly in LXC containers running as root (no sudo installed) - Also works on regular VMs/servers where sudo is available - Fixes all remaining: sudo: command not found errors	2025-12-14 21:05:35 -05:00
ilia	9bb39c9913	Fix proxmox_setup.sh to work when running as root in LXC ... - Detect if running as root and use 'su' instead of 'sudo' for postgres - Fixes: sudo: command not found error in LXC containers	2025-12-14 21:01:30 -05:00
ilia	204cd0e75b	Initial commit: POTE Phase 1 complete ... - PR1: Project scaffold, DB models, price loader - PR2: Congressional trade ingestion (House Stock Watcher) - PR3: Security enrichment + deployment infrastructure - 37 passing tests, 87%+ coverage - Docker + Proxmox deployment ready - Complete documentation - Works 100% offline with fixtures	2025-12-14 20:45:34 -05:00