35 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
ilia	f94ca17b39	Fix CI: Install Node.js in containers for checkout action ... ISSUE: ====== Gitea Actions containers need Node.js to run actions/checkout@v4 Error: 'exec: "node": executable file not found in $PATH' FIX: ==== Added Node.js installation step before checkout in all jobs: For Debian/Ubuntu containers (python:3.11-bullseye, ubuntu:22.04): apt-get update && apt-get install -y curl git curl -fsSL https://deb.nodesource.com/setup_20.x | bash - apt-get install -y nodejs For Alpine containers (gitleaks, trivy): apk add --no-cache nodejs npm curl git JOBS FIXED: =========== ✅ lint-and-test ✅ secret-scanning ✅ security-scan ✅ dependency-scan ✅ sast-scan ✅ container-scan ✅ docker-build-test (no container, already works) This is a common requirement for Gitea Actions when using custom containers.	2025-12-24 22:53:33 -05:00
ilia	bb0bce40c9	Clean up Ansible-specific and outdated documentation ... REMOVED: ======== ❌ ANSIBLE_HANDOFF.md - Ansible team will get from git history ❌ ANSIBLE_TECHNICAL_REFERENCE.md - Ansible team will get from git history ❌ MOVE_ANSIBLE_TO_SEPARATE_REPO.md - Migration complete, no longer needed ❌ SETUP_COMPLETE.md - Outdated summary ❌ TESTING_STATUS.md - Outdated status RATIONALE: ========== Ansible code has been removed from this repo and belongs in infrastructure repo. These docs were for the migration process and are no longer relevant to the POTE application repo. Ansible team can retrieve these files from git history if needed: git show d40b412:ANSIBLE_HANDOFF.md git show d40b412:ANSIBLE_TECHNICAL_REFERENCE.md KEPT: ===== ✅ CUSTOMIZATION_CHECKLIST.md - Still useful for config reference ✅ CI_PIPELINE_COMPLETE.md - Current CI documentation ✅ All quickstart guides ✅ All deployment docs	2025-12-24 22:36:50 -05:00
ilia	fd392b976f	Add comprehensive CI pipeline documentation ... NEW: Complete documentation for enhanced security pipeline 📄 CI_PIPELINE_COMPLETE.md =========================== SECURITY LAYERS (5): ==================== 1. 🔐 Secret Scanning (Gitleaks) - Exposed credentials, API keys, tokens - Scans entire git history - Redacted output 2. 🔒 Security Scan (Safety + Bandit) - Safety: Known CVEs in dependencies - Bandit: Python security linter - SQL injection, hardcoded passwords, etc. 3. 📦 Dependency Scan (Trivy) - Python packages + system libraries - CVE database lookup - Comprehensive vulnerability scanning 4. 🔍 SAST Scan (Semgrep) - Static Application Security Testing - Language-aware pattern matching - OWASP Top 10 detection 5. 🐳 Container Scan (Trivy) - Dockerfile misconfigurations - Filesystem vulnerabilities - HIGH/CRITICAL severity focus FEATURES: ========= ✅ 8 parallel jobs (fast execution) ✅ Non-blocking security scans (informational) ✅ Comprehensive workflow summary ✅ Comparison with Ansible pipeline ✅ Local testing instructions ✅ Troubleshooting guide ✅ Best practices COMPARISON: =========== Kept from Ansible pipeline: - Secret scanning (Gitleaks) - Dependency scanning (Trivy) - SAST scanning (Semgrep) - Container scanning (Trivy) Removed (Ansible-specific): - Ansible linting - Vault validation - Playbook syntax checks Added (Python-specific): - Python linting (ruff, black, mypy) - pytest with coverage - Safety (Python CVE check) - Bandit (Python security) RESULT: ======= Production-ready CI pipeline with multiple security layers providing comprehensive vulnerability detection without blocking development workflow.	2025-12-24 22:34:40 -05:00
ilia	d40b412f67	Remove Ansible code and enhance CI pipeline with security scanning ... REMOVED: ======== ❌ ansible/ directory (moved to infrastructure repo) ❌ ANSIBLE_INTEGRATION.md (redundant) KEPT (for reference): ===================== ✅ ANSIBLE_HANDOFF.md - Integration guide for Ansible team ✅ ANSIBLE_TECHNICAL_REFERENCE.md - Exact commands/paths for role ✅ CUSTOMIZATION_CHECKLIST.md - Configuration reference ✅ MOVE_ANSIBLE_TO_SEPARATE_REPO.md - Migration guide ENHANCED CI PIPELINE: ===================== Added comprehensive security scanning: 🔐 Secret Scanning (Gitleaks) - Scans for exposed credentials, API keys, tokens - Checks entire git history - Redacted output for safety 🔒 Security Scan (Safety + Bandit) - Safety: Known vulnerabilities in Python dependencies - Bandit: Static security analysis of Python code - Detects common security issues 📦 Dependency Scan (Trivy) - Scans all dependencies for vulnerabilities - Checks Python packages and system libraries - CVE database lookup 🔍 SAST Scan (Semgrep) - Static Application Security Testing - Language-aware pattern matching - Detects security anti-patterns 🐳 Container Scan (Trivy) - Scans Dockerfile for misconfigurations - Filesystem vulnerability scanning - HIGH/CRITICAL severity focus 🐋 Docker Build Test - Ensures Docker image builds successfully - Tests basic import functionality - Uses build cache for speed 📊 Workflow Summary - Comprehensive status report - Shows all security layers - Easy-to-read summary RATIONALE: ========== Ansible code belongs in infrastructure repo, not app repo. This eliminates circular dependency and follows best practices. Enhanced CI provides multiple layers of security validation.	2025-12-24 22:33:20 -05:00
ilia	7924c3bdc7	Add comprehensive Ansible technical reference ... NEW: Complete technical documentation for Ansible role implementation 📄 ANSIBLE_TECHNICAL_REFERENCE.md ==================================== INSTALLATION: ✅ Uses pyproject.toml (pip install -e .) ✅ NOT requirements.txt ✅ Python 3.11+ required MIGRATIONS: ✅ Command: alembic upgrade head ✅ Run from project root (where alembic.ini is) ✅ Reads DATABASE_URL from .env ✅ Idempotent (safe to re-run) SCRIPT PATHS (all in scripts/): ================================ Data Ingestion: - fetch_congressional_trades.py - enrich_securities.py - fetch_sample_prices.py - ingest_from_fixtures.py - add_custom_trades.py - scrape_alternative_sources.py Analytics: - analyze_official.py - calculate_all_returns.py - generate_trading_report.py Monitoring: - monitor_market.py - analyze_disclosure_timing.py - generate_pattern_report.py Reporting: - send_daily_report.py - send_weekly_report.py - health_check.py Automation (Shell): - automated_daily_run.sh ⭐ (for cron) - automated_weekly_run.sh ⭐ (for cron) - setup_cron.sh - setup_automation.sh COMPLETE DEPLOYMENT SEQUENCE: ============================= Phase 1: System prep (user, packages, PostgreSQL) Phase 2: App deployment (clone, venv, install, migrate) Phase 3: Automation (logs, cron jobs) Phase 4: Verification (health check, tests) ANSIBLE TASK CHECKLIST: ======================= ✅ Full task examples for tasks/main.yml ✅ .env template (templates/env.j2) ✅ Variable reference (minimal vs full) ✅ Update procedure ✅ Testing commands ✅ Common issues & solutions This doc has EVERYTHING needed to write the Ansible role!	2025-12-24 22:30:38 -05:00
ilia	e9fd12d949	Add documentation for Ansible separation and customization ... NEW FILES: ========== 📄 CUSTOMIZATION_CHECKLIST.md - Complete list of everything that needs customization - Organized by priority: Critical, Important, Optional - Covers .env, Ansible, Gitea secrets, email, etc. - Quick action checklist for deployment 📄 ANSIBLE_HANDOFF.md - Guide for integrating POTE with existing Ansible system - Explains what Ansible needs to know - Variable reference and secrets management - Multi-environment deployment strategy - Example playbook and testing instructions 📄 MOVE_ANSIBLE_TO_SEPARATE_REPO.md - Explains why ansible/ should be in infrastructure repo - Step-by-step migration guide - Final directory structure for both repos - Benefits and workflow after migration KEY INSIGHT: ============ The ansible/ directory doesn't belong in the POTE app repo because: - Ansible runs BEFORE the app exists (creates container, deploys app) - Creates circular dependency (Ansible clones repo that contains Ansible) - Should live in centralized infrastructure repository NEXT STEPS: =========== 1. Review CUSTOMIZATION_CHECKLIST.md for deployment config 2. Copy ansible/ to infrastructure repo 3. Remove ansible/ from POTE repo (keep handoff docs) 4. Deploy via centralized Ansible system	2025-12-24 22:26:27 -05:00
ilia	d2ae095fcf	Add complete Ansible integration configuration ... NEW: Complete Ansible role for automated POTE deployment Files Added: ============ 📁 ansible/ ├── README.md - Overview and usage ├── roles/pote/defaults/main.yml - ⭐ 200+ variables defined ├── group_vars/ │ ├── all.yml - Common variables │ ├── development.yml - Dev environment (branch: dev) │ ├── staging.yml - QA environment (branch: qa) │ └── production.yml - Prod environment (branch: main) ├── inventory.example.yml - Example inventory └── vault.example.yml - Example secrets 📄 ANSIBLE_INTEGRATION.md - Complete integration guide What Ansible Needs to Know: ============================ ✅ Git repository & branch (per environment) ✅ Application user & paths ✅ Python & system dependencies ✅ Database configuration (per environment) ✅ Email/SMTP settings ✅ Monitoring configuration ✅ Cron/automation schedules ✅ Deployment options ✅ Security settings ✅ Feature flags ✅ Environment-specific overrides Variable Categories (11): ========================== 1. Project basics (5 vars) 2. User & paths (7 vars) 3. Python & dependencies (3 lists) 4. Database (8 vars) 5. Email/SMTP (6 vars) 6. Monitoring (2 vars) 7. Logging (2 vars) 8. Cron/automation (7 vars) 9. Deployment (6 vars) 10. Security (4 vars) 11. Feature flags (4 vars) Integration: ============ ✅ Compatible with base_os role ✅ Multi-environment support (dev/qa/prod) ✅ Branch-based deployment (dev→qa→main) ✅ Ansible Vault for secrets ✅ Sensible defaults for everything ✅ Minimal required config (3 vars!) Usage: ====== ansible-playbook deploy-pote.yml --limit development ansible-playbook deploy-pote.yml --limit staging ansible-playbook deploy-pote.yml --limit production Ready for your Ansible auto-configure system!	2025-12-24 22:04:36 -05:00
ilia	0313ec1de1	Add setup completion summary ... ✅ POTE SETUP COMPLETE! What's Done: - Branch strategy (main/qa/dev) - Branch protection with CI checks - Complete CI/CD pipeline - Comprehensive documentation - Email reporting system - Market monitoring (3 phases) - Ready for Ansible integration Next Steps: 1. Merge documentation to main 2. Add Gitea secrets 3. Test the workflow 4. Set up email automation Total Achievement: - 93 tests passing - Professional dev workflow - Production-ready system	2025-12-24 22:01:08 -05:00
ilia	5161f6c421	Organize documentation: move setup guides to docs/ folder ... Moved: - BRANCH_SETUP_COMPLETE.md → docs/15_branch_setup_checklist.md - PIPELINE_SETUP_GUIDE.md → docs/16_pipeline_setup.md Reason: Keep root directory clean, organize all docs in docs/ folder Documentation structure now: docs/ ├── 14_branch_strategy_and_deployment.md (comprehensive guide) ├── 15_branch_setup_checklist.md (quick checklist) └── 16_pipeline_setup.md (CI setup guide)	2025-12-24 21:55:24 -05:00
ilia	659896f096	Add pipeline setup guide for branch protection	2025-12-24 21:54:25 -05:00
ilia	3910ca9d04	Update CI to run on all branches (main, qa, dev)	2025-12-24 21:53:14 -05:00
ilia	9940100239	Add branch setup completion guide and checklist	2025-12-24 21:47:37 -05:00
ilia	01597f608f	Add comprehensive branch strategy and multi-environment deployment guide ... NEW DOCUMENTATION: ================== docs/14_branch_strategy_and_deployment.md COVERS: ======= ✅ Branch Strategy (main/qa/dev) ✅ Branch Protection Rules for Gitea ✅ Multi-environment deployment workflows ✅ Integration with Ansible auto-deploy ✅ Gitea secrets per environment ✅ Deployment flow diagram ✅ Rollback strategies ✅ Complete setup checklist BRANCH STRUCTURE: ================= main (prod) ← qa (staging) ← dev (development) PROTECTION LEVELS: ================== - main: MOST PROTECTED (require PR + 2 approvals + CI) - qa: MODERATELY PROTECTED (require PR + CI) - dev: LIGHTLY PROTECTED (require CI only) DEPLOYMENT WORKFLOWS: ===================== - deploy-dev.yml: Auto-deploy on push to dev - deploy-qa.yml: Auto-deploy on push to qa + smoke tests - deploy-prod.yml: Manual approval + rollback on failure ANSIBLE INTEGRATION: ==================== - Webhook-based deployment - Gitea Actions → Ansible API - Environment-specific inventory - Automated health checks WHAT'S MISSING (ANSWERED): ========================== ✅ Environment variables per env ✅ Database migration strategy ✅ Rollback procedures ✅ Monitoring & alerts ✅ Feature flags ✅ Changelog management BRANCHES CREATED: ================= ✅ dev branch created and pushed ✅ qa branch created and pushed ✅ main branch (existing) Ready for Ansible auto-configure integration!	2025-12-24 21:46:37 -05:00
ilia	ead0820cf9	Add Gitea Secrets integration for CI/CD and deployment ... NEW FEATURES: ============ 📁 GITEA_SECRETS_GUIDE.md: - Comprehensive guide on using Gitea secrets - Store passwords in Gitea (not in git!) - Use in CI/CD and deployment workflows - Best practices and security recommendations 🔧 .github/workflows/ci.yml (UPDATED): - Now uses Gitea secrets with fallbacks - ${{ secrets.SMTP_PASSWORD || 'testpass123' }} - ${{ secrets.DB_PASSWORD || 'testpass123' }} - Tests run with real credentials from Gitea 🚀 .github/workflows/deploy.yml (NEW): - Automated deployment to Proxmox - Manual trigger via Gitea UI - Steps: 1. SSH to Proxmox with secrets.PROXMOX_SSH_KEY 2. Pull latest code 3. Update .env with secrets from Gitea 4. Run migrations 5. Health check 6. Test email 7. Rollback on failure HOW IT WORKS: ============= 1. Store passwords in Gitea (Settings → Secrets) 2. CI/CD uses secrets automatically 3. Deployment workflow updates .env on Proxmox 4. Best of both worlds: secure CI + simple runtime SECRETS TO ADD IN GITEA: ======================== - SMTP_PASSWORD: your mail password - DB_PASSWORD: changeme123 - PROXMOX_HOST: 10.0.10.95 - PROXMOX_USER: poteapp - PROXMOX_SSH_KEY: (SSH private key) - SMTP_HOST: mail.levkin.ca - SMTP_USER: test@levkin.ca - FROM_EMAIL: test@levkin.ca USAGE: ====== # In Gitea UI: Actions → Deploy to Proxmox → Run workflow # Or push commits: git push origin main # CI runs with secrets automatically See GITEA_SECRETS_GUIDE.md for full instructions!	2025-12-15 15:52:19 -05:00
ilia	0c183fb28c	Add comprehensive secrets management guide ... Covers 6 options for storing passwords securely: 1. .env file (current, good for personal use) 2. Environment variables (better for production) 3. Separate secrets file 4. Docker secrets 5. HashiCorp Vault (enterprise) 6. Git secrets (CI/CD only) Recommendation: Current .env setup is fine for personal/research use Improvement: chmod 600 .env (done) Includes security checklist, rotation procedures, and testing	2025-12-15 15:47:12 -05:00
ilia	5613d7f894	Add quick setup card with levkin.ca configuration ... One-page reference with: - Your specific configuration (test@levkin.ca) - 3-step setup (30 seconds + 1 min + 2 min) - Quick commands for deployed server - Troubleshooting checklist - Current status tracker - Next actions Everything ready - just needs password in .env	2025-12-15 15:44:30 -05:00
ilia	07af492026	Add email setup guide for levkin.ca mail server ... Configuration for test@levkin.ca: - SMTP: mail.levkin.ca:587 (STARTTLS) - Includes setup instructions - Testing checklist - Troubleshooting guide Note: .env file created locally (not committed, in .gitignore)	2025-12-15 15:43:44 -05:00
ilia	d8f723bafb	Add comprehensive deployment and automation FAQ ... Answers user's questions: - What happens after deployment? (nothing automatic by default) - How to get reports? (3 options: email, SSH, future web UI) - Where are reports sent? (email or saved to ~/logs/) - Do you need to check IP? (depends on setup method) - Can we setup email reports? (YES! 5-minute setup) - Do we need CI/CD pipelines? (optional, but included) - Can we use existing Ansible pipeline? (concepts reused, not directly) This document ties everything together and provides clear next steps.	2025-12-15 15:35:33 -05:00
ilia	0d8d85adc1	Add complete automation, reporting, and CI/CD system ... Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready	2025-12-15 15:34:31 -05:00
ilia	53d631a903	Add comprehensive monitoring system documentation ... Complete summary of all 3 phases: - Phase 1: Real-time market monitoring - Phase 2: Disclosure timing correlation - Phase 3: Pattern detection & rankings Documentation includes: - System architecture diagram - Usage guide for all phases - Example reports - Test coverage summary - Deployment checklist - Interpretation guide - Legal/ethical disclaimers - Automated workflow examples Total Achievement: ✅ 93 tests passing ✅ All 3 phases complete ✅ Production-ready system ✅ Full documentation The POTE monitoring system is now complete!	2025-12-15 15:25:07 -05:00
ilia	2ec4a8e373	Phase 3: Pattern Detection & Comparative Analysis - COMPLETE ... COMPLETE: Cross-official pattern detection and ranking system New Module: - src/pote/monitoring/pattern_detector.py: Pattern analysis engine * rank_officials_by_timing(): Rank all officials by suspicion * identify_repeat_offenders(): Find systematic offenders * analyze_ticker_patterns(): Per-stock suspicious patterns * get_sector_timing_analysis(): Sector-level analysis * get_party_comparison(): Democrat vs Republican comparison * generate_pattern_report(): Comprehensive report Analysis Features: - Official Rankings: * By average timing score * Suspicious trade percentage * Alert rates * Pattern classification - Repeat Offender Detection: * Identifies officials with 50%+ suspicious trades * Historical pattern tracking * Systematic timing advantage detection - Comparative Analysis: * Cross-party comparison * Sector analysis * Ticker-specific patterns * Statistical aggregations New Script: - scripts/generate_pattern_report.py: Comprehensive reports * Top 10 most suspicious officials * Repeat offenders list * Most suspiciously traded stocks * Sector breakdowns * Party comparison stats * Text/JSON formats New Tests (11 total, all passing): - test_rank_officials_by_timing - test_identify_repeat_offenders - test_analyze_ticker_patterns - test_get_sector_timing_analysis - test_get_party_comparison - test_generate_pattern_report - test_rank_officials_min_trades_filter - test_empty_data_handling - test_ranking_score_accuracy - test_sector_stats_accuracy - test_party_stats_completeness Usage: python scripts/generate_pattern_report.py --days 365 Report Includes: - Top suspicious officials ranked - Repeat offenders (50%+ suspicious rate) - Most suspiciously traded tickers - Sector analysis - Party comparison - Interpretation guide Total Test Suite: 93 tests passing ✅ ALL 3 PHASES COMPLETE!	2025-12-15 15:23:40 -05:00
ilia	a52313145b	Add comprehensive tests for Phase 2 correlation engine ... New Tests (13 total, all passing): - test_get_alerts_before_trade: Retrieve prior alerts - test_get_alerts_before_trade_no_alerts: Handle no alerts - test_calculate_timing_score_high_suspicion: High score logic - test_calculate_timing_score_no_alerts: Zero score handling - test_calculate_timing_score_factors: Multi-factor scoring - test_analyze_trade_full: Complete trade analysis - test_analyze_recent_disclosures: Batch processing - test_get_official_timing_pattern: Historical patterns - test_get_official_timing_pattern_no_trades: Edge case - test_get_ticker_timing_analysis: Per-ticker analysis - test_get_ticker_timing_analysis_no_trades: Edge case - test_alerts_outside_lookback_window: Date filtering - test_different_ticker_alerts_excluded: Ticker filtering Test Coverage: - Alert-to-trade correlation - Timing score calculation (all factors) - Pattern analysis (officials & tickers) - Batch analysis - Edge cases & filtering - Date range handling Total Test Suite: 82 tests passing ✅	2025-12-15 15:20:40 -05:00
ilia	6b62ae96f7	Phase 2: Disclosure Timing Correlation Engine ... COMPLETE: Match congressional trades to prior market alerts New Module: - src/pote/monitoring/disclosure_correlator.py: Core correlation engine * get_alerts_before_trade(): Find alerts before trade date * calculate_timing_score(): Score suspicious timing (0-100 scale) - Factors: alert count, severity, recency, type - Thresholds: 60+ = suspicious, 80+ = highly suspicious * analyze_trade(): Complete trade analysis with timing * analyze_recent_disclosures(): Batch analysis of new filings * get_official_timing_pattern(): Historical pattern analysis * get_ticker_timing_analysis(): Per-stock timing patterns Timing Score Algorithm: - Base score: alert count × 5 + avg severity × 2 - Recency bonus: +10 per alert within 7 days - Severity bonus: +15 per high-severity (7+) alert - Total score: 0-100 (capped) - Interpretation: * 80-100: Highly suspicious (likely timing advantage) * 60-79: Suspicious (possible timing advantage) * 40-59: Notable (some unusual activity) * 0-39: Normal (no significant pattern) New Script: - scripts/analyze_disclosure_timing.py: CLI analysis tool * Analyze recent disclosures (--days N) * Filter by timing score (--min-score) * Analyze specific official (--official NAME) * Analyze specific ticker (--ticker SYMBOL) * Text/JSON output formats * Detailed reports with prior alerts Usage Examples: # Find suspicious trades filed recently python scripts/analyze_disclosure_timing.py --days 30 --min-score 60 # Analyze specific official python scripts/analyze_disclosure_timing.py --official "Nancy Pelosi" # Analyze specific ticker python scripts/analyze_disclosure_timing.py --ticker NVDA Report Includes: - Timing score and suspicion level - Prior alert details (count, severity, timing) - Official name, ticker, trade details - Assessment and reasoning - Top suspicious trades ranked Next: Phase 3 - Pattern Detection across officials/stocks	2025-12-15 15:17:09 -05:00
ilia	db34f26cdc	Add comprehensive tests for Phase 1 monitoring system ... New Tests (14 total, all passing): - test_get_congressional_watchlist: Auto-detect most-traded tickers - test_check_ticker_basic: Single ticker analysis - test_scan_watchlist_with_mock: Batch scanning with controlled data - test_save_alerts: Database persistence - test_get_recent_alerts: Query filtering (ticker, type, severity, date) - test_get_ticker_alert_summary: Aggregated statistics - test_alert_manager_format_text: Text formatting - test_alert_manager_format_html: HTML formatting - test_alert_manager_filter_alerts: Multi-criteria filtering - test_alert_manager_generate_summary_text: Report generation - test_alert_manager_generate_summary_html: HTML reports - test_alert_manager_empty_alerts: Edge case handling - test_market_alert_model: ORM model validation - test_alert_timestamp_filtering: Time-based queries Test Coverage: - Market monitoring core logic - Alert detection algorithms - Database operations - Filtering and querying - Report generation (text/HTML) - Edge cases and error handling Total Test Suite: 69 tests passing ✅	2025-12-15 15:14:58 -05:00
ilia	cfaf38b0be	Phase 1: Real-Time Market Monitoring System ... COMPLETE: Real-time unusual activity detection for congressional tickers New Database Model: - MarketAlert: Stores unusual market activity alerts * Tracks volume spikes, price movements, volatility * JSON details field for flexible data storage * Severity scoring (1-10 scale) * Indexed for efficient queries by ticker/timestamp New Modules: - src/pote/monitoring/market_monitor.py: Core monitoring engine * get_congressional_watchlist(): Top 50 most-traded tickers * check_ticker(): Analyze single stock for unusual activity * scan_watchlist(): Batch analysis of multiple tickers * Detection logic: - Unusual volume (3x average) - Price spikes/drops (>5%) - High volatility (2x normal) * save_alerts(): Persist to database * get_recent_alerts(): Query historical alerts - src/pote/monitoring/alert_manager.py: Alert formatting & filtering * format_alert_text(): Human-readable output * format_alert_html(): HTML email format * filter_alerts(): By severity, ticker, type * generate_summary_report(): Text/HTML reports Scripts: - scripts/monitor_market.py: CLI monitoring tool * Continuous monitoring mode (--interval) * One-time scan (--once) * Custom ticker lists or auto-detect congressional watchlist * Severity filtering (--min-severity) * Report generation and saving Migrations: - alembic/versions/f44014715b40_add_market_alerts_table.py Documentation: - docs/11_live_market_monitoring.md: Complete explanation * Why you can't track WHO is trading * What IS possible (timing analysis) * How hybrid monitoring works * Data sources and APIs Usage: # Monitor congressional tickers (one-time scan) python scripts/monitor_market.py --once # Continuous monitoring (every 5 minutes) python scripts/monitor_market.py --interval 300 # Monitor specific tickers python scripts/monitor_market.py --tickers NVDA,MSFT,AAPL --once Next Steps (Phase 2): - Disclosure correlation engine - Timing advantage calculator - Suspicious trade flagging	2025-12-15 15:10:49 -05:00
ilia	8ba9d7ffdd	Add watchlist system and pre-market trading reports ... New Features: - Watchlist system for tracking specific Congress members - Trading report generation with multiple formats - Pre-market-close automated updates (3 PM) New Scripts: - scripts/fetch_congress_members.py: Manage watchlist * 29 known active traders (curated list) * Optional ProPublica API integration (all 535 members) * Create/view/manage watchlist - scripts/generate_trading_report.py: Generate trading reports * Filter by watchlist or show all * Multiple formats: text, HTML, JSON * Summary statistics (buys/sells, top tickers) * Color-coded output (🟢 BUY, 🔴 SELL) - scripts/pre_market_close_update.sh: 3 PM automation * Quick fetch of latest trades * Enrichment of new securities * Generate and display report * Saves to reports/ directory Documentation: - WATCHLIST_GUIDE.md: Complete guide * List of 29 known active traders * How to create/customize watchlist * Schedule options (pre-market, post-market) * Email setup (optional) * FAQ and examples Known Active Traders Include: Senate: Tuberville, Rand Paul, Mark Warner, Rick Scott House: Pelosi, Crenshaw, MTG, Gottheimer, Brian Higgins Use Cases: ✅ Daily reports at 3 PM (1 hour before close) ✅ See what Congress bought/sold recently ✅ Track specific members you care about ✅ Export to HTML/JSON for further analysis	2025-12-15 15:00:42 -05:00
ilia	3a89c1e6d2	Add comprehensive automation system ... New Scripts: - scripts/daily_fetch.sh: Automated daily data updates * Fetches congressional trades (last 7 days) * Enriches securities (name, sector, industry) * Updates price data for all securities * Calculates returns and metrics * Logs everything to logs/ directory - scripts/setup_automation.sh: Interactive automation setup * Makes scripts executable * Creates log directories * Configures cron jobs (multiple schedule options) * Guides user through setup Documentation: - docs/10_automation.md: Complete automation guide * Explains disclosure timing (30-45 day legal lag) * Why daily updates are optimal (not hourly/real-time) * Cron job setup instructions * Systemd timer alternative * Email notifications (optional) * Monitoring and logging * Failure handling * Performance optimization Key Insights: ❌ No real-time data possible (STOCK Act = 30-45 day lag) ✅ Daily updates are optimal ✅ Automated via cron jobs ✅ Handles API failures gracefully ✅ Logs everything for debugging	2025-12-15 14:55:05 -05:00
ilia	77bd69b85c	Add comprehensive testing status documentation	2025-12-15 14:43:52 -05:00
ilia	b4e6a7c340	Fix analytics tests and add comprehensive testing guide ... Critical Fixes: - Fixed Price model query to use security_id join with Security - Added Security import to returns.py module - Fixed all test fixtures to use test_db_session correctly - Added AAPL price data to sample_prices fixture New Tests: - tests/test_analytics_integration.py: 10 comprehensive integration tests * Real-world scenarios with synthetic price data * Return calculations, benchmark comparisons, performance metrics * Edge cases: missing data, sell trades, disclosure timing Documentation: - LOCAL_TEST_GUIDE.md: Complete guide for local testing * How to test before deploying * Current data status (live vs fixtures) * Multiple options for getting real data * Common issues and fixes Test Results: ✅ All 55 tests passing ✅ Analytics fully functional ✅ Ready for deployment Live Data Status: ❌ House Stock Watcher API still down (external issue) ✅ Manual CSV import works ✅ yfinance for prices works ✅ Can use system NOW with manual data	2025-12-15 14:42:20 -05:00
ilia	34aebb1c2e	PR4: Phase 2 Analytics Foundation ... Complete analytics module with returns, benchmarks, and performance metrics. New Modules: - src/pote/analytics/returns.py: Return calculator for trades - src/pote/analytics/benchmarks.py: Benchmark comparison & alpha - src/pote/analytics/metrics.py: Performance aggregations Scripts: - scripts/analyze_official.py: Analyze specific official - scripts/calculate_all_returns.py: System-wide analysis Tests: - tests/test_analytics.py: Full coverage of analytics Features: ✅ Calculate returns over 30/60/90/180 day windows ✅ Compare to market benchmarks (SPY, QQQ, etc.) ✅ Calculate abnormal returns (alpha) ✅ Aggregate stats by official, sector ✅ Top performer rankings ✅ Disclosure timing analysis ✅ Command-line analysis tools ~1,210 lines of new code, all tested	2025-12-15 11:33:21 -05:00
ilia	02c10c85d6	Add data update tools and Phase 2 plan ... - scripts/add_custom_trades.py: Manual trade entry - scripts/scrape_alternative_sources.py: CSV import - scripts/daily_update.sh: Automated daily updates - docs/09_data_updates.md: Complete update guide - docs/PR4_PLAN.md: Phase 2 analytics plan Enables users to add representatives and set up auto-updates	2025-12-15 10:39:18 -05:00
ilia	895c34e2c1	Add psycopg2-binary to dependencies for PostgreSQL support ... - Required for SQLAlchemy PostgreSQL connections - Fixes: ModuleNotFoundError: No module named 'psycopg2'	2025-12-14 21:12:36 -05:00
ilia	44ddd88879	Complete fix: Replace all sudo commands with su for LXC compatibility ... - Replace 'sudo -u' with 'su -' throughout the script - Works perfectly in LXC containers running as root (no sudo installed) - Also works on regular VMs/servers where sudo is available - Fixes all remaining: sudo: command not found errors	2025-12-14 21:05:35 -05:00
ilia	9bb39c9913	Fix proxmox_setup.sh to work when running as root in LXC ... - Detect if running as root and use 'su' instead of 'sudo' for postgres - Fixes: sudo: command not found error in LXC containers	2025-12-14 21:01:30 -05:00
ilia	204cd0e75b	Initial commit: POTE Phase 1 complete ... - PR1: Project scaffold, DB models, price loader - PR2: Congressional trade ingestion (House Stock Watcher) - PR3: Security enrichment + deployment infrastructure - 37 passing tests, 87%+ coverage - Docker + Proxmox deployment ready - Complete documentation - Works 100% offline with fixtures	2025-12-14 20:45:34 -05:00