diff --git a/ANSIBLE_HANDOFF.md b/ANSIBLE_HANDOFF.md deleted file mode 100644 index 7090fd0..0000000 --- a/ANSIBLE_HANDOFF.md +++ /dev/null @@ -1,376 +0,0 @@ -# POTE Ansible Integration - Handoff Guide - -## ๐ŸŽฏ Purpose - -This document explains what your Ansible auto-configure system needs to know to deploy POTE across different environments. - -**Note:** The Ansible code in `ansible/` directory should be **moved to your Ansible repository**, not kept here in the POTE app repo. - ---- - -## ๐Ÿ“ฆ What to Move to Your Ansible Repository - -### Files to Extract from POTE Repo - -```bash -# Copy these to your Ansible repository: -ansible/ -โ”œโ”€โ”€ README.md โ†’ roles/pote/README.md -โ”œโ”€โ”€ roles/pote/defaults/main.yml โ†’ roles/pote/defaults/main.yml -โ”œโ”€โ”€ group_vars/ -โ”‚ โ”œโ”€โ”€ all.yml โ†’ group_vars/all.yml (merge with existing) -โ”‚ โ”œโ”€โ”€ development.yml โ†’ group_vars/development.yml (merge) -โ”‚ โ”œโ”€โ”€ staging.yml โ†’ group_vars/staging.yml (merge) -โ”‚ โ””โ”€โ”€ production.yml โ†’ group_vars/production.yml (merge) -โ”œโ”€โ”€ inventory.example.yml โ†’ inventory.example.yml (reference) -โ””โ”€โ”€ vault.example.yml โ†’ vault.example.yml (reference) -``` - -### What to Keep in POTE Repo - -```bash -# Keep these for deployment reference: -docs/08_proxmox_deployment.md # Manual deployment guide -scripts/proxmox_setup.sh # Bootstrap script (if needed) -CUSTOMIZATION_CHECKLIST.md # Configuration reference -``` - ---- - -## ๐Ÿ”ง Integration with Your Ansible System - -### 1. Create POTE Role in Your Ansible Repo - -```bash -# In your Ansible repository: -roles/ -โ””โ”€โ”€ pote/ - โ”œโ”€โ”€ defaults/ - โ”‚ โ””โ”€โ”€ main.yml # Copy from POTE repo - โ”œโ”€โ”€ tasks/ - โ”‚ โ””โ”€โ”€ main.yml # Create based on requirements below - โ”œโ”€โ”€ templates/ - โ”‚ โ””โ”€โ”€ env.j2 # Create for .env file - โ”œโ”€โ”€ handlers/ - โ”‚ โ””โ”€โ”€ main.yml # Restart services, etc. - โ””โ”€โ”€ README.md # Copy from POTE repo -``` - -### 2. Required Ansible Tasks - -Your `roles/pote/tasks/main.yml` should handle: - -#### Phase 1: System Preparation -- Create application user (`poteapp`) -- Install system dependencies (Python 3.11, PostgreSQL, build-essential, etc.) -- Configure PostgreSQL database and user - -#### Phase 2: Application Deployment -- Clone POTE repository from Git (using SSH key) -- Create Python virtual environment -- Install Python dependencies -- Create `.env` file from template -- Run database migrations (`alembic upgrade head`) - -#### Phase 3: Automation Setup -- Create log directories -- Install cron jobs for daily/weekly reports -- Set up health check monitoring -- Configure email reporting - -#### Phase 4: Security & Cleanup -- Set proper file permissions -- Secure `.env` file (600) -- Clean up temporary files - ---- - -## ๐Ÿ“‹ Variables Your Ansible System Needs - -### Minimal Required Variables (3) - -```yaml -# Absolute minimum to deploy POTE: -pote_git_repo: "gitea@10.0.30.169:ilia/POTE.git" -pote_git_branch: "main" # or "dev", "qa" per environment -vault_smtp_password: "{{ vault_smtp_password }}" -``` - -### Recommended Variables (Environment-Specific) - -```yaml -# Per-environment configuration: -environment: "production" # or "development", "staging" -pote_git_branch: "main" # "dev" for development, "qa" for staging - -# Database -db_name: "potedb" # "potedb_dev" for dev, "potedb_qa" for staging -db_user: "poteuser" -db_password: "{{ vault_db_password_prod }}" - -# Email -smtp_host: "mail.levkin.ca" -smtp_port: 587 -smtp_user: "test@levkin.ca" -from_email: "test@levkin.ca" -smtp_password: "{{ vault_smtp_password }}" -report_recipients: "ilia@levkin.ca" # Your actual email - -# Monitoring -market_tickers: "NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX" -alert_severity: 5 - -# Automation -pote_enable_cron: true -pote_daily_report_time: "0 6" # 6:00 AM -pote_weekly_report_time: "0 8" # 8:00 AM Sunday -``` - -### Full Variable List - -See `ansible/roles/pote/defaults/main.yml` for all 200+ variables with defaults. - ---- - -## ๐Ÿ” Secrets Management - -### Required Secrets in Ansible Vault - -```yaml -# In your Ansible vault (group_vars/all/vault.yml): -vault_git_ssh_key: | - -----BEGIN OPENSSH PRIVATE KEY----- - your_ssh_private_key_for_gitea - -----END OPENSSH PRIVATE KEY----- - -vault_ssh_public_key: "ssh-rsa AAAAB3NzaC1yc2E..." - -vault_smtp_password: "your_email_password" - -vault_db_password_dev: "dev_password_123" -vault_db_password_qa: "qa_password_123" -vault_db_password_prod: "strong_production_password" - -# Optional: -vault_quiverquant_key: "" -vault_fmp_key: "" -``` - ---- - -## ๐ŸŽญ Multi-Environment Strategy - -### Branch โ†’ Environment Mapping - -```yaml -development: - git_branch: dev - db_name: potedb_dev - log_level: DEBUG - cron_enabled: false # Manual testing - -staging: - git_branch: qa - db_name: potedb_qa - log_level: INFO - cron_enabled: true - -production: - git_branch: main - db_name: potedb - log_level: WARNING - cron_enabled: true -``` - -### Deployment Commands - -```bash -# Deploy to development -ansible-playbook deploy-pote.yml --limit development - -# Deploy to staging -ansible-playbook deploy-pote.yml --limit staging - -# Deploy to production -ansible-playbook deploy-pote.yml --limit production -``` - ---- - -## ๐Ÿ“ Example Playbook - -```yaml ---- -# deploy-pote.yml -- name: Deploy POTE Application - hosts: all - become: yes - - pre_tasks: - - name: Ensure base OS role has run - debug: - msg: "Assuming base_os role has already configured the system" - - roles: - - role: pote - tags: ['pote', 'deploy'] -``` - ---- - -## ๐Ÿ”„ Deployment Workflow - -### Initial Deployment (New Server) - -1. **Provision LXC container** in Proxmox -2. **Run base_os role** (your existing role) -3. **Run pote role** (new role) -4. **Verify deployment** with health check - -### Updates (Existing Server) - -1. **Pull latest code** from Git -2. **Update dependencies** if needed -3. **Run migrations** (`alembic upgrade head`) -4. **Restart services** if needed - -### Rollback (If Needed) - -1. **Checkout previous Git tag/commit** -2. **Run migrations down** if needed -3. **Restore database backup** if needed - ---- - -## ๐Ÿงช Testing the Integration - -### Dry Run - -```bash -# Test without making changes -ansible-playbook deploy-pote.yml --limit development --check --diff -``` - -### Verify Deployment - -```bash -# SSH into server and run: -cd ~/pote -source venv/bin/activate -python scripts/health_check.py - -# Check cron jobs: -crontab -l | grep POTE - -# Check logs: -tail -f ~/logs/daily_run.log -``` - ---- - -## ๐Ÿ“Š What POTE Provides for Ansible - -### 1. Configuration Template - -The `.env` file template your Ansible should generate: - -```bash -# Database -DATABASE_URL=postgresql://{{ db_user }}:{{ db_password }}@localhost:5432/{{ db_name }} - -# Email -SMTP_HOST={{ smtp_host }} -SMTP_PORT={{ smtp_port }} -SMTP_USER={{ smtp_user }} -SMTP_PASSWORD={{ smtp_password }} -FROM_EMAIL={{ from_email }} -REPORT_RECIPIENTS={{ report_recipients }} - -# Monitoring -MARKET_MONITOR_TICKERS={{ market_tickers }} -ALERT_MIN_SEVERITY={{ alert_severity }} - -# Logging -LOG_LEVEL={{ log_level }} -LOG_FILE={{ log_file }} - -# Optional API Keys -QUIVERQUANT_API_KEY={{ quiverquant_api_key | default('') }} -FMP_API_KEY={{ fmp_api_key | default('') }} -``` - -### 2. Health Check Endpoint - -```bash -# Your Ansible can verify deployment with: -python scripts/health_check.py -# Exit code 0 = healthy, non-zero = issues -``` - -### 3. Database Migrations - -```bash -# Your Ansible should run after code updates: -alembic upgrade head -``` - -### 4. Idempotency - -All POTE scripts are idempotent: -- Re-running data fetches won't duplicate records -- Migrations are safe to re-run -- Cron setup checks for existing jobs - ---- - -## ๐Ÿš€ Quick Start for Your Ansible Team - -1. **Copy** `ansible/` directory from POTE repo to your Ansible repo -2. **Customize** `group_vars/` files with your values -3. **Create** `group_vars/all/vault.yml` with secrets -4. **Encrypt** vault: `ansible-vault encrypt group_vars/all/vault.yml` -5. **Update** `inventory.yml` with your server IPs -6. **Test** deployment to dev: `ansible-playbook deploy-pote.yml --limit development` -7. **Verify** with health check -8. **Deploy** to staging, then production - ---- - -## ๐Ÿ”— Dependencies - -### System Packages (via apt) - -```yaml -system_packages: - - python3.11 - - python3.11-venv - - python3.11-dev - - python3-pip - - postgresql - - postgresql-contrib - - libpq-dev - - build-essential - - git - - curl - - htop -``` - -### Python Packages (via pip) - -Installed automatically via `pip install -e .` from `pyproject.toml` - ---- - -## ๐Ÿ“ž Support - -- **POTE Configuration:** See `CUSTOMIZATION_CHECKLIST.md` -- **Ansible Variables:** See `ansible/roles/pote/defaults/main.yml` -- **Deployment Issues:** Check `scripts/health_check.py` output -- **Email Issues:** See `EMAIL_SETUP.md` - ---- - -**Last Updated:** December 2025 -**POTE Version:** 0.1.0 -**Target Ansible Version:** 2.9+ - diff --git a/ANSIBLE_TECHNICAL_REFERENCE.md b/ANSIBLE_TECHNICAL_REFERENCE.md deleted file mode 100644 index dd98e3e..0000000 --- a/ANSIBLE_TECHNICAL_REFERENCE.md +++ /dev/null @@ -1,598 +0,0 @@ -# POTE Ansible Role - Technical Reference - -**Exact commands, paths, and procedures for the Ansible `roles/pote` role.** - ---- - -## ๐Ÿ“ฆ Installation Method - -### โœ… Uses `pyproject.toml` (NOT requirements.txt) - -```bash -# Install in editable mode with all dependencies -pip install -e . - -# For development (includes pytest, ruff, black, mypy) -pip install -e ".[dev]" -``` - -**File:** `pyproject.toml` - -**Key dependencies:** -```toml -dependencies = [ - "sqlalchemy>=2.0", - "alembic>=1.12", - "pydantic>=2.0", - "pydantic-settings>=2.0", - "python-dotenv>=1.0", - "requests>=2.31", - "pandas>=2.0", - "numpy>=1.24", - "yfinance>=0.2", - "psycopg2-binary>=2.9", -] -``` - -**Python version:** `>=3.11` (required) - ---- - -## ๐Ÿ—„๏ธ Database Migrations - -### Command - -```bash -alembic upgrade head -``` - -**Where to run:** From the project root directory (where `alembic.ini` is located) - -**Configuration:** -- Migration scripts: `alembic/versions/` -- Config file: `alembic.ini` (uses `.env` for DATABASE_URL) -- Models: `src/pote/db/models.py` - -**Important:** -- Alembic reads `DATABASE_URL` from `.env` file via `src/pote/config.py` -- Must run AFTER `.env` file is created -- Must run AFTER database and user are created in PostgreSQL - -**Idempotency:** โœ… Safe to re-run (Alembic tracks applied migrations) - ---- - -## ๐Ÿ“ Actual Script Paths - -### All scripts are in `scripts/` directory: - -#### Data Ingestion -```bash -scripts/fetch_congressional_trades.py # Fetch from House Stock Watcher API -scripts/enrich_securities.py # Get company names, sectors from yfinance -scripts/fetch_sample_prices.py # Fetch price data for securities -scripts/ingest_from_fixtures.py # Offline mode - load from fixtures -scripts/add_custom_trades.py # Manual trade entry -scripts/scrape_alternative_sources.py # Import from CSV -scripts/fetch_congress_members.py # Fetch list of current members -``` - -#### Analytics -```bash -scripts/analyze_official.py # Analyze specific official's performance -scripts/calculate_all_returns.py # System-wide return calculations -scripts/generate_trading_report.py # Generate trading activity report -``` - -#### Monitoring (Phase 1-3) -```bash -scripts/monitor_market.py # Real-time market monitoring -scripts/analyze_disclosure_timing.py # Disclosure timing correlation -scripts/generate_pattern_report.py # Pattern detection & rankings -``` - -#### Reporting -```bash -scripts/send_daily_report.py # Send daily email report -scripts/send_weekly_report.py # Send weekly email report -scripts/health_check.py # System health check -``` - -#### Automation (Shell Scripts) -```bash -scripts/automated_daily_run.sh # โญ Daily cron job -scripts/automated_weekly_run.sh # โญ Weekly cron job -scripts/setup_cron.sh # Interactive cron setup -scripts/setup_automation.sh # Alternative automation setup -scripts/daily_fetch.sh # Daily data fetch -scripts/daily_update.sh # Daily update routine -scripts/pre_market_close_update.sh # Pre-market close routine -``` - -#### Deployment -```bash -scripts/proxmox_setup.sh # Bootstrap script for Proxmox LXC -``` - ---- - -## ๐Ÿ”ง Complete Deployment Sequence - -### Phase 1: System Preparation - -```bash -# 1. Create application user -useradd -m -s /bin/bash poteapp - -# 2. Install system dependencies -apt-get update -apt-get install -y \ - python3.11 \ - python3.11-venv \ - python3.11-dev \ - python3-pip \ - postgresql \ - postgresql-contrib \ - libpq-dev \ - build-essential \ - git \ - curl \ - htop - -# 3. Configure PostgreSQL -sudo -u postgres psql << EOF -CREATE USER poteuser WITH PASSWORD 'your_password'; -CREATE DATABASE potedb OWNER poteuser; -GRANT ALL PRIVILEGES ON DATABASE potedb TO poteuser; -EOF - -# 4. Configure PostgreSQL for remote access (if needed) -# Edit /etc/postgresql/15/main/postgresql.conf: -# listen_addresses = '*' -# Edit /etc/postgresql/15/main/pg_hba.conf: -# host all all 0.0.0.0/0 scram-sha-256 -# Restart: systemctl restart postgresql -``` - -### Phase 2: Application Deployment - -```bash -# 1. Switch to application user -su - poteapp - -# 2. Clone repository (requires SSH key setup) -git clone gitea@10.0.30.169:ilia/POTE.git pote -cd pote - -# 3. Checkout appropriate branch -git checkout main # or 'dev', 'qa' depending on environment - -# 4. Create virtual environment -python3.11 -m venv venv - -# 5. Activate virtual environment -source venv/bin/activate - -# 6. Upgrade pip -pip install --upgrade pip - -# 7. Install application -pip install -e . - -# 8. Create .env file (from template or Ansible) -cat > .env << 'EOF' -DATABASE_URL=postgresql://poteuser:your_password@localhost:5432/potedb -SMTP_HOST=mail.levkin.ca -SMTP_PORT=587 -SMTP_USER=test@levkin.ca -SMTP_PASSWORD=your_smtp_password -FROM_EMAIL=test@levkin.ca -REPORT_RECIPIENTS=your@email.com -MARKET_MONITOR_TICKERS=NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX -ALERT_MIN_SEVERITY=5 -LOG_LEVEL=INFO -LOG_FILE=/home/poteapp/logs/pote.log -EOF - -# 9. Secure .env file -chmod 600 .env - -# 10. Run database migrations -alembic upgrade head -``` - -### Phase 3: Automation Setup - -```bash -# 1. Create log directory -mkdir -p /home/poteapp/logs - -# 2. Make scripts executable -chmod +x scripts/*.sh - -# 3. Set up cron jobs -crontab -e -# Add these lines: -# Daily report at 6:00 AM -0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1 -# Weekly report at 8:00 AM on Sunday -0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1 -``` - -### Phase 4: Verification - -```bash -# 1. Run health check -cd /home/poteapp/pote -source venv/bin/activate -python scripts/health_check.py - -# Expected output: -# โœ… Database connection: OK -# โœ… Database has X officials, Y trades, Z securities -# โœ… Latest trade: YYYY-MM-DD -# โœ… Price data: X securities with prices - -# 2. Test data ingestion (optional) -python scripts/ingest_from_fixtures.py # Offline test -# OR -python scripts/fetch_congressional_trades.py # Live API test - -# 3. Test email (optional) -python scripts/send_daily_report.py --to test@example.com --test-smtp - -# 4. Verify cron jobs -crontab -l | grep POTE -``` - ---- - -## ๐Ÿ“‹ Ansible Task Checklist - -### Your `roles/pote/tasks/main.yml` should include: - -```yaml ---- -# 1. System preparation -- name: Create application user - user: - name: "{{ appuser_name }}" - shell: "{{ appuser_shell }}" - create_home: yes - -- name: Install system dependencies - apt: - name: - - python3.11 - - python3.11-venv - - python3.11-dev - - python3-pip - - postgresql - - postgresql-contrib - - libpq-dev - - build-essential - - git - - curl - - htop - state: present - update_cache: yes - -# 2. PostgreSQL setup -- name: Create PostgreSQL user - postgresql_user: - name: "{{ db_user }}" - password: "{{ db_password }}" - state: present - become_user: postgres - -- name: Create PostgreSQL database - postgresql_db: - name: "{{ db_name }}" - owner: "{{ db_user }}" - state: present - become_user: postgres - -# 3. SSH key setup (for git clone) -- name: Deploy SSH private key for git - copy: - content: "{{ pote_git_ssh_key }}" - dest: "/home/{{ appuser_name }}/.ssh/id_rsa" - owner: "{{ appuser_name }}" - mode: '0600' - -- name: Add Gitea to known_hosts - known_hosts: - name: "{{ pote_git_host }}" - key: "{{ lookup('pipe', 'ssh-keyscan ' + pote_git_host) }}" - state: present - become_user: "{{ appuser_name }}" - -# 4. Clone repository -- name: Clone POTE repository - git: - repo: "{{ pote_git_repo }}" - dest: "{{ pote_install_path }}" - version: "{{ pote_git_branch }}" - accept_hostkey: yes - become_user: "{{ appuser_name }}" - -# 5. Python environment -- name: Create virtual environment - command: python3.11 -m venv {{ pote_install_path }}/venv - args: - creates: "{{ pote_install_path }}/venv" - become_user: "{{ appuser_name }}" - -- name: Upgrade pip - pip: - name: pip - state: latest - virtualenv: "{{ pote_install_path }}/venv" - become_user: "{{ appuser_name }}" - -- name: Install POTE application - pip: - name: "{{ pote_install_path }}" - editable: yes - virtualenv: "{{ pote_install_path }}/venv" - become_user: "{{ appuser_name }}" - -# 6. Configuration -- name: Deploy .env file - template: - src: env.j2 - dest: "{{ pote_install_path }}/.env" - owner: "{{ appuser_name }}" - mode: '0600' - become_user: "{{ appuser_name }}" - -# 7. Database migrations -- name: Run Alembic migrations - command: "{{ pote_install_path }}/venv/bin/alembic upgrade head" - args: - chdir: "{{ pote_install_path }}" - become_user: "{{ appuser_name }}" - -# 8. Log directory -- name: Create log directory - file: - path: "/home/{{ appuser_name }}/logs" - state: directory - owner: "{{ appuser_name }}" - mode: '0755' - -# 9. Make scripts executable -- name: Make shell scripts executable - file: - path: "{{ pote_install_path }}/scripts/{{ item }}" - mode: '0755' - loop: - - automated_daily_run.sh - - automated_weekly_run.sh - - setup_cron.sh - become_user: "{{ appuser_name }}" - -# 10. Cron jobs -- name: Set up daily cron job - cron: - name: "POTE Automated Daily Run" - minute: "{{ pote_daily_report_time.split()[1] }}" - hour: "{{ pote_daily_report_time.split()[0] }}" - job: "{{ pote_install_path }}/scripts/automated_daily_run.sh >> /home/{{ appuser_name }}/logs/daily_run.log 2>&1" - user: "{{ appuser_name }}" - when: pote_enable_cron and pote_daily_report_enabled - -- name: Set up weekly cron job - cron: - name: "POTE Automated Weekly Run" - minute: "{{ pote_weekly_report_time.split()[1] }}" - hour: "{{ pote_weekly_report_time.split()[0] }}" - day: "0" # Sunday - job: "{{ pote_install_path }}/scripts/automated_weekly_run.sh >> /home/{{ appuser_name }}/logs/weekly_run.log 2>&1" - user: "{{ appuser_name }}" - when: pote_enable_cron and pote_weekly_report_enabled -``` - ---- - -## ๐ŸŽจ `.env` Template for Ansible - -### Create `roles/pote/templates/env.j2`: - -```jinja2 -# Database -DATABASE_URL=postgresql://{{ db_user }}:{{ db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }} - -# Email -SMTP_HOST={{ smtp_host }} -SMTP_PORT={{ smtp_port }} -SMTP_USER={{ smtp_user }} -SMTP_PASSWORD={{ smtp_password }} -FROM_EMAIL={{ from_email }} -REPORT_RECIPIENTS={{ report_recipients }} - -# Monitoring -MARKET_MONITOR_TICKERS={{ market_tickers }} -ALERT_MIN_SEVERITY={{ alert_severity }} - -# Logging -LOG_LEVEL={{ log_level }} -LOG_FILE={{ log_file }} - -# Optional API Keys -{% if quiverquant_api_key is defined and quiverquant_api_key %} -QUIVERQUANT_API_KEY={{ quiverquant_api_key }} -{% endif %} -{% if fmp_api_key is defined and fmp_api_key %} -FMP_API_KEY={{ fmp_api_key }} -{% endif %} -``` - ---- - -## ๐Ÿ”„ Update Procedure (Existing Deployment) - -```bash -# 1. Pull latest code -cd /home/poteapp/pote -git pull origin main - -# 2. Activate venv -source venv/bin/activate - -# 3. Update dependencies (if pyproject.toml changed) -pip install -e . - -# 4. Run migrations (if models changed) -alembic upgrade head - -# 5. Restart services (if any) -# (Currently no services - cron jobs will pick up changes) -``` - ---- - -## ๐Ÿงช Testing the Deployment - -### Quick Health Check - -```bash -cd /home/poteapp/pote -source venv/bin/activate -python scripts/health_check.py -``` - -**Exit codes:** -- `0` = Healthy -- Non-zero = Issues found - -### Full Test Sequence - -```bash -# 1. Test database connection -python -c "from pote.db import get_session; next(get_session()); print('โœ… DB OK')" - -# 2. Test offline ingestion -python scripts/ingest_from_fixtures.py - -# 3. Test enrichment -python scripts/enrich_securities.py - -# 4. Test price fetch -python scripts/fetch_sample_prices.py - -# 5. Test email (dry run) -python scripts/send_daily_report.py --to test@example.com --test-smtp - -# 6. Verify cron -crontab -l | grep POTE -``` - ---- - -## ๐Ÿ“Š Key Variables Reference - -### Minimal (3 required) - -```yaml -pote_git_repo: "gitea@10.0.30.169:ilia/POTE.git" -pote_git_branch: "main" -vault_smtp_password: "{{ vault_smtp_password }}" -``` - -### Essential (for production) - -```yaml -# Paths -pote_install_path: "/home/poteapp/pote" -appuser_name: "poteapp" - -# Database -db_name: "potedb" -db_user: "poteuser" -db_password: "{{ vault_db_password_prod }}" -db_host: "localhost" -db_port: 5432 - -# Email -smtp_host: "mail.levkin.ca" -smtp_port: 587 -smtp_user: "test@levkin.ca" -smtp_password: "{{ vault_smtp_password }}" -from_email: "test@levkin.ca" -report_recipients: "your@email.com" - -# Monitoring -market_tickers: "NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX" -alert_severity: 5 - -# Automation -pote_enable_cron: true -pote_daily_report_time: "0 6" # 6:00 AM -pote_weekly_report_time: "0 8" # 8:00 AM Sunday - -# Logging -log_level: "INFO" -log_file: "/home/poteapp/logs/pote.log" -``` - ---- - -## ๐Ÿšจ Common Issues & Solutions - -### Issue: `ModuleNotFoundError: No module named 'pote'` -**Solution:** Run `pip install -e .` from project root - -### Issue: `alembic.util.exc.CommandError: Can't locate revision identified by 'head'` -**Solution:** Database is not initialized. Run `alembic upgrade head` - -### Issue: `psycopg2.OperationalError: could not connect to server` -**Solution:** Check PostgreSQL is running and DATABASE_URL is correct - -### Issue: Cron jobs not running -**Solution:** -- Check crontab: `crontab -l` -- Check logs: `tail -f ~/logs/daily_run.log` -- Verify script permissions: `ls -la scripts/*.sh` - -### Issue: Email not sending -**Solution:** -- Test SMTP: `python scripts/send_daily_report.py --test-smtp` -- Check `.env` has correct SMTP settings -- Verify SMTP_PASSWORD is correct - ---- - -## ๐Ÿ“ž Support Commands - -```bash -# Check Python version -python3.11 --version - -# Check pip packages -pip list | grep -E "(alembic|sqlalchemy|pydantic)" - -# Check database -psql -U poteuser -d potedb -c "\dt" - -# Check logs -tail -f ~/logs/daily_run.log -tail -f ~/logs/pote.log - -# Check cron -crontab -l -grep CRON /var/log/syslog | tail -20 - -# Manual test run -cd ~/pote -source venv/bin/activate -./scripts/automated_daily_run.sh -``` - ---- - -**Last Updated:** December 2025 -**POTE Version:** 0.1.0 -**Python Required:** 3.11+ -**PostgreSQL Required:** 15+ - diff --git a/MOVE_ANSIBLE_TO_SEPARATE_REPO.md b/MOVE_ANSIBLE_TO_SEPARATE_REPO.md deleted file mode 100644 index 8d9984a..0000000 --- a/MOVE_ANSIBLE_TO_SEPARATE_REPO.md +++ /dev/null @@ -1,248 +0,0 @@ -# Move Ansible Code to Separate Repository - -## โŒ Problem - -The `ansible/` directory in this repository doesn't make sense here because: - -1. **Ansible runs BEFORE the app exists** - it creates the container and deploys the app -2. **Ansible belongs in your infrastructure repo** - not in the application repo -3. **Chicken-and-egg problem** - Ansible needs to clone this repo, but the Ansible code is inside it - -## โœ… Solution - -Move the Ansible code to your **Ansible/infrastructure repository** where your other roles live. - ---- - -## ๐Ÿšš Migration Steps - -### Step 1: Copy Ansible Code to Your Ansible Repo - -```bash -# From your Ansible repository: -cd /path/to/your/ansible-repo - -# Copy the POTE role -cp -r /path/to/pote/ansible/roles/pote roles/ - -# Merge group_vars (don't overwrite existing files!) -# Manually merge these into your existing group_vars: -# - ansible/group_vars/all.yml -# - ansible/group_vars/development.yml -# - ansible/group_vars/staging.yml -# - ansible/group_vars/production.yml - -# Copy examples for reference -cp /path/to/pote/ansible/inventory.example.yml examples/pote-inventory.example.yml -cp /path/to/pote/ansible/vault.example.yml examples/pote-vault.example.yml -``` - -### Step 2: Remove Ansible Code from POTE Repo - -```bash -# From the POTE repository: -cd /path/to/pote - -# Remove the ansible directory -git rm -r ansible/ - -# Remove Ansible-specific docs (keep handoff guide) -git rm ANSIBLE_INTEGRATION.md - -# Keep these files: -# - ANSIBLE_HANDOFF.md (reference for Ansible team) -# - CUSTOMIZATION_CHECKLIST.md (configuration reference) -# - docs/08_proxmox_deployment.md (manual deployment guide) - -# Commit the changes -git add -A -git commit -m "Move Ansible code to infrastructure repository - -The ansible/ directory has been moved to the Ansible infrastructure -repository where it belongs. This repo now only contains: -- Application code -- Deployment documentation (ANSIBLE_HANDOFF.md) -- Configuration reference (CUSTOMIZATION_CHECKLIST.md) -- Manual deployment guide (docs/08_proxmox_deployment.md) - -Ansible code now lives in: [your-ansible-repo]/roles/pote/" -``` - -### Step 3: Update Your Ansible Repo - -```bash -# In your Ansible repository: - -# Create the playbook -cat > playbooks/deploy-pote.yml << 'EOF' ---- -- name: Deploy POTE Application - hosts: pote_servers - become: yes - - roles: - - role: pote - tags: ['pote', 'deploy'] -EOF - -# Update inventory (add POTE servers) -# Edit inventory/production.yml, inventory/staging.yml, etc. - -# Create/update vault with POTE secrets -ansible-vault edit group_vars/all/vault.yml -# Add: vault_smtp_password, vault_git_ssh_key, vault_db_password_*, etc. - -# Test deployment -ansible-playbook playbooks/deploy-pote.yml --limit development --check -``` - ---- - -## ๐Ÿ“ Final Directory Structure - -### POTE Application Repo (this repo) - -``` -pote/ -โ”œโ”€โ”€ src/pote/ # Application code -โ”œโ”€โ”€ tests/ # Tests -โ”œโ”€โ”€ scripts/ # Deployment scripts -โ”œโ”€โ”€ docs/ # Documentation -โ”œโ”€โ”€ .github/workflows/ # CI/CD pipelines -โ”œโ”€โ”€ ANSIBLE_HANDOFF.md # โœ… Keep - Reference for Ansible team -โ”œโ”€โ”€ CUSTOMIZATION_CHECKLIST.md # โœ… Keep - Configuration guide -โ”œโ”€โ”€ MOVE_ANSIBLE_TO_SEPARATE_REPO.md # โœ… Keep - This file -โ””โ”€โ”€ pyproject.toml # Python dependencies -``` - -### Your Ansible Infrastructure Repo - -``` -ansible-infrastructure/ -โ”œโ”€โ”€ roles/ -โ”‚ โ”œโ”€โ”€ base_os/ # Your existing role -โ”‚ โ”œโ”€โ”€ pote/ # โœ… NEW - Moved from POTE repo -โ”‚ โ”‚ โ”œโ”€โ”€ defaults/main.yml -โ”‚ โ”‚ โ”œโ”€โ”€ tasks/main.yml -โ”‚ โ”‚ โ”œโ”€โ”€ templates/env.j2 -โ”‚ โ”‚ โ””โ”€โ”€ handlers/main.yml -โ”‚ โ””โ”€โ”€ ...other roles... -โ”œโ”€โ”€ playbooks/ -โ”‚ โ”œโ”€โ”€ deploy-pote.yml # โœ… NEW - POTE deployment playbook -โ”‚ โ””โ”€โ”€ ...other playbooks... -โ”œโ”€โ”€ group_vars/ -โ”‚ โ”œโ”€โ”€ all.yml # Merged POTE variables -โ”‚ โ”œโ”€โ”€ all/vault.yml # Added POTE secrets -โ”‚ โ”œโ”€โ”€ development.yml # Merged POTE dev config -โ”‚ โ”œโ”€โ”€ staging.yml # Merged POTE staging config -โ”‚ โ””โ”€โ”€ production.yml # Merged POTE prod config -โ”œโ”€โ”€ inventory/ -โ”‚ โ”œโ”€โ”€ production.yml # Added POTE servers -โ”‚ โ”œโ”€โ”€ staging.yml # Added POTE servers -โ”‚ โ””โ”€โ”€ development.yml # Added POTE servers -โ””โ”€โ”€ examples/ - โ”œโ”€โ”€ pote-inventory.example.yml - โ””โ”€โ”€ pote-vault.example.yml -``` - ---- - -## ๐ŸŽฏ Benefits of This Approach - -1. **Cleaner Separation** - Application code separate from infrastructure code -2. **No Circular Dependency** - Ansible can clone the app repo without needing to be inside it -3. **Centralized Infrastructure** - All your Ansible roles in one place -4. **Easier Management** - Update infrastructure without touching app code -5. **Standard Practice** - Follows industry best practices - ---- - -## ๐Ÿ”„ Workflow After Migration - -### Deploying POTE - -```bash -# From your Ansible repository: -ansible-playbook playbooks/deploy-pote.yml --limit production -``` - -### Updating POTE Application - -```bash -# Option 1: Via Ansible (recommended) -ansible-playbook playbooks/deploy-pote.yml --limit production --tags update - -# Option 2: Via Gitea Actions (CI/CD) -# Push to main branch โ†’ Gitea Actions runs tests โ†’ Auto-deploy - -# Option 3: Manual (SSH to server) -cd ~/pote -git pull origin main -source venv/bin/activate -pip install -e . -alembic upgrade head -``` - ---- - -## ๐Ÿ“ What to Tell Your Ansible Team - -Send them: -1. **ANSIBLE_HANDOFF.md** - Complete integration guide -2. **CUSTOMIZATION_CHECKLIST.md** - What needs to be configured -3. **ansible/** directory - The actual role code - -They need to: -1. Copy the role to their Ansible repo -2. Merge the group_vars into their existing structure -3. Add POTE servers to their inventory -4. Add POTE secrets to their vault -5. Test deployment to dev environment -6. Deploy to staging, then production - ---- - -## โš ๏ธ Important Notes - -### Don't Delete These from POTE Repo - -Keep these files for reference: -- `ANSIBLE_HANDOFF.md` - Ansible integration guide -- `CUSTOMIZATION_CHECKLIST.md` - Configuration reference -- `docs/08_proxmox_deployment.md` - Manual deployment guide -- `scripts/proxmox_setup.sh` - Bootstrap script (if needed) - -### Do Delete These from POTE Repo - -Remove these (they belong in Ansible repo): -- `ansible/` directory (entire thing) -- `ANSIBLE_INTEGRATION.md` (redundant with ANSIBLE_HANDOFF.md) - ---- - -## ๐Ÿš€ Ready to Move? - -Run these commands when ready: - -```bash -# 1. Backup first (just in case) -cp -r ansible /tmp/pote-ansible-backup - -# 2. Remove from POTE repo -git rm -r ansible/ -git rm ANSIBLE_INTEGRATION.md -git add -A -git commit -m "Move Ansible code to infrastructure repository" -git push origin dev - -# 3. Copy to your Ansible repo (do this manually) -# See "Step 1: Copy Ansible Code to Your Ansible Repo" above - -# 4. Test deployment from Ansible repo -# See "Step 3: Update Your Ansible Repo" above -``` - ---- - -**Questions?** See `ANSIBLE_HANDOFF.md` for complete integration details. - diff --git a/SETUP_COMPLETE.md b/SETUP_COMPLETE.md deleted file mode 100644 index 54cb5f3..0000000 --- a/SETUP_COMPLETE.md +++ /dev/null @@ -1,252 +0,0 @@ -# ๐ŸŽ‰ POTE Setup Complete! - -## โœ… What's Done - -### 1. **Branch Strategy** โœ… -- `main` (production) - Protected -- `qa` (staging) - Protected -- `dev` (development) - Protected - -### 2. **Branch Protection** โœ… -All branches now have: -- CI status checks required -- Force push blocked -- Proper merge requirements - -### 3. **CI/CD Pipeline** โœ… -- Runs on all branches (main, qa, dev) -- 93 tests passing -- Security scanning -- Dependency scanning -- Docker build verification - -### 4. **Complete Documentation** โœ… -- Branch strategy guide -- Deployment workflows -- Secrets management -- Email reporting setup -- Automation guides - ---- - -## ๐Ÿš€ Your Workflow Now - -### Development Flow: - -``` -1. Work on dev branch - git checkout dev - git pull origin dev - # make changes - git commit -m "Add feature" - git push origin dev - โ†“ - CI runs automatically - โ†“ - โœ… Tests must pass before push succeeds - -2. Promote to QA - Create PR: dev โ†’ qa in Gitea - โ†“ - CI runs on PR - โ†“ - โœ… Tests + Security + Dependencies must pass - โ†“ - Get 1 approval - โ†“ - Merge to qa - -3. Promote to Production - Create PR: qa โ†’ main in Gitea - โ†“ - CI runs on PR - โ†“ - โœ… All checks must pass (tests, security, deps, docker) - โ†“ - Get approval(s) - โ†“ - Merge to main -``` - ---- - -## ๐ŸŽฏ Next Steps - -### 1. **Merge Documentation to Main** - -Go to: https://git.levkin.ca/ilia/POTE/compare/main...dev - -Create PR with all the new documentation: -- Branch strategy guide -- Pipeline setup guide -- Email configuration -- Gitea secrets guide -- And more! - -### 2. **Add Gitea Secrets** (5 minutes) - -Go to: https://git.levkin.ca/ilia/POTE/settings/secrets - -Add these secrets: - -| Secret Name | Value | Purpose | -|-------------|-------|---------| -| `SMTP_PASSWORD` | your mail password | Email reports | -| `DB_PASSWORD` | `changeme123` | Database | -| `SMTP_HOST` | `mail.levkin.ca` | Email server | -| `SMTP_USER` | `test@levkin.ca` | Email user | -| `FROM_EMAIL` | `test@levkin.ca` | From address | -| `REPORT_RECIPIENTS` | `test@levkin.ca` | Report recipients | - -**For multi-environment (later):** -- `DEV_HOST`, `DEV_USER`, `DEV_SSH_KEY` -- `QA_HOST`, `QA_USER`, `QA_SSH_KEY` -- `PROXMOX_HOST`, `PROXMOX_USER`, `PROXMOX_SSH_KEY` - -### 3. **Set Up Email Reporting** (5 minutes) - -On your deployed server: - -```bash -ssh poteapp@your-proxmox-ip -cd ~/pote - -# Add password to .env -nano .env -# Update: SMTP_PASSWORD=your_actual_password - -# Test email -source venv/bin/activate -python scripts/send_daily_report.py --to test@levkin.ca --test-smtp - -# Set up automation -./scripts/setup_cron.sh -``` - -### 4. **Configure Ansible Integration** (optional) - -Update your Ansible auto-deploy system to: -- Listen for Gitea webhooks -- Deploy based on branch (dev/qa/main) -- Use environment-specific configs - -See: `docs/14_branch_strategy_and_deployment.md` - ---- - -## ๐Ÿ“š Documentation Index - -### Quick Start Guides (Root): -- `README.md` - Project overview -- `QUICKSTART.md` - Usage guide -- `AUTOMATION_QUICKSTART.md` - Email setup (5 min) -- `GITEA_SECRETS_GUIDE.md` - Secrets management -- `PROXMOX_QUICKSTART.md` - Deployment (5 min) -- `DEPLOYMENT_AND_AUTOMATION.md` - Complete deployment FAQ -- `EMAIL_SETUP.md` - levkin.ca email config - -### Technical Documentation (docs/): -- `docs/14_branch_strategy_and_deployment.md` - Branch strategy & multi-env -- `docs/15_branch_setup_checklist.md` - Setup checklist -- `docs/16_pipeline_setup.md` - CI/CD pipeline setup -- `docs/13_secrets_management.md` - Secrets options -- `docs/12_automation_and_reporting.md` - Automation guide -- `docs/11_live_market_monitoring.md` - Monitoring system - -### Other Guides: -- `MONITORING_SYSTEM_COMPLETE.md` - Phase 1-3 monitoring -- `WATCHLIST_GUIDE.md` - Watchlist configuration -- `LOCAL_TEST_GUIDE.md` - Local testing -- `OFFLINE_DEMO.md` - Offline mode - ---- - -## โœ… System Status - -### Code & Tests: -- โœ… 93 tests passing -- โœ… 88%+ code coverage -- โœ… All linters passing -- โœ… Security scans clean - -### Infrastructure: -- โœ… PostgreSQL database -- โœ… Docker support -- โœ… Proxmox deployment ready -- โœ… CI/CD pipeline operational - -### Features: -- โœ… Congressional trade tracking -- โœ… Market monitoring (3-phase system) -- โœ… Disclosure correlation -- โœ… Pattern detection -- โœ… Email reporting -- โœ… Automated daily/weekly reports - -### Branch Protection: -- โœ… main: Fully protected -- โœ… qa: Protected with approvals -- โœ… dev: Protected with CI checks - ---- - -## ๐Ÿ”— Important Links - -- **Repository:** https://git.levkin.ca/ilia/POTE -- **Create PR:** https://git.levkin.ca/ilia/POTE/compare/main...dev -- **Actions:** https://git.levkin.ca/ilia/POTE/actions -- **Secrets:** https://git.levkin.ca/ilia/POTE/settings/secrets -- **Branch Protection:** https://git.levkin.ca/ilia/POTE/settings/branches - ---- - -## ๐ŸŽฏ Immediate Actions - -1. **Merge documentation to main** (2 min) - - Create PR: dev โ†’ main - - Review changes - - Merge - -2. **Add Gitea secrets** (5 min) - - SMTP_PASSWORD - - DB_PASSWORD - - Other email configs - -3. **Test the workflow** (10 min) - - Make a small change in dev - - Push and watch CI run - - Create PR to qa - - Verify status checks work - -4. **Set up email automation** (5 min) - - SSH to Proxmox - - Run setup_cron.sh - - Test daily report - ---- - -## ๐ŸŽ‰ Congratulations! - -You now have a **production-ready, enterprise-grade development workflow** with: - -- โœ… Multi-environment branching strategy -- โœ… Automated CI/CD pipeline -- โœ… Branch protection with status checks -- โœ… Security and quality gates -- โœ… Comprehensive documentation -- โœ… Email reporting system -- โœ… Market monitoring (3 phases) -- โœ… Ready for Ansible integration - -**Total Achievement:** -- 93 tests passing -- 3-branch strategy (dev/qa/main) -- Full CI/CD pipeline -- Complete automation -- Professional documentation - -**The POTE system is production-ready!** ๐Ÿš€ - ---- - -**Next:** Merge this documentation to main and start using your new workflow! diff --git a/TESTING_STATUS.md b/TESTING_STATUS.md deleted file mode 100644 index 4a1ce8b..0000000 --- a/TESTING_STATUS.md +++ /dev/null @@ -1,324 +0,0 @@ -# POTE Testing Status Report -**Date:** December 15, 2025 -**Status:** โœ… All Systems Operational - Ready for Deployment - ---- - -## ๐ŸŽฏ Test Suite Summary - -### **55 Tests - All Passing โœ…** - -``` -Platform: Python 3.13.5, pytest-9.0.2 -Test Duration: ~1.8 seconds -Coverage: ~85% overall -``` - -### Test Breakdown by Module: - -| Module | Tests | Status | Coverage | -|--------|-------|--------|----------| -| **Analytics** | 18 tests | โœ… PASS | 80% | -| **Models** | 7 tests | โœ… PASS | 90% | -| **Ingestion** | 14 tests | โœ… PASS | 85% | -| **Price Loader** | 8 tests | โœ… PASS | 90% | -| **Security Enricher** | 8 tests | โœ… PASS | 85% | - ---- - -## ๐Ÿ“Š What's Been Tested? - -### โœ… Core Database Operations -- [x] Creating and querying Officials -- [x] Creating and querying Securities -- [x] Creating and querying Trades -- [x] Price data storage and retrieval -- [x] Unique constraints and relationships -- [x] Database migrations (Alembic) - -### โœ… Data Ingestion -- [x] House Stock Watcher client (with fixtures) -- [x] Trade loading from JSON -- [x] Security enrichment from yfinance -- [x] Price data fetching and storage -- [x] Idempotent operations (no duplicates) -- [x] Error handling for missing/invalid data - -### โœ… Analytics Engine -- [x] Return calculations (buy trades) -- [x] Return calculations (sell trades) -- [x] Multiple time windows (30/60/90/180 days) -- [x] Benchmark comparisons (SPY, QQQ, etc.) -- [x] Abnormal returns (alpha calculations) -- [x] Official performance summaries -- [x] Sector-level analysis -- [x] Disclosure timing analysis -- [x] Top performer rankings -- [x] System-wide statistics - -### โœ… Edge Cases -- [x] Missing price data handling -- [x] Trades with no exit price yet -- [x] Sell trades (inverted returns) -- [x] Disclosure lags -- [x] Duplicate prevention -- [x] Invalid date ranges -- [x] Empty result sets - ---- - -## ๐Ÿงช Test Types - -### 1. Unit Tests (Fast, Isolated) -**Location:** `tests/test_*.py` (excluding integration) -**Purpose:** Test individual functions and classes -**Database:** In-memory SQLite (fresh for each test) -**Speed:** ~0.5 seconds - -**Examples:** -- `test_parse_amount_range()` - Parse trade amounts -- `test_normalize_transaction_type()` - Trade type normalization -- `test_get_or_create_security()` - Security deduplication - -### 2. Integration Tests (Realistic Scenarios) -**Location:** `tests/test_analytics_integration.py` -**Purpose:** Test complete workflows with synthetic data -**Database:** In-memory SQLite with realistic price data -**Speed:** ~0.7 seconds - -**Examples:** -- `test_return_calculation_with_real_data()` - Full return calc pipeline -- `test_benchmark_comparison_with_real_data()` - Alpha calculations -- `test_official_performance_summary()` - Aggregated metrics - -**Scenarios Tested:** -- Nancy Pelosi buys NVDA early (strong returns) -- Tommy Tuberville buys NVDA later (good but less alpha) -- 120 days of synthetic price data (realistic trends) -- SPY benchmark comparison -- Multiple time window analysis - ---- - -## ๐Ÿ”ง How to Run Tests Locally - -### Quick Test -```bash -cd /home/user/Documents/code/pote -source venv/bin/activate -pytest -v -``` - -### With Coverage Report -```bash -pytest --cov=src/pote --cov-report=html --cov-report=term -# View: firefox htmlcov/index.html -``` - -### Specific Test Modules -```bash -# Just analytics -pytest tests/test_analytics.py -v - -# Just integration tests -pytest tests/test_analytics_integration.py -v - -# Specific test -pytest tests/test_analytics.py::test_return_calculator_basic -v -``` - -### Watch Mode (Re-run on changes) -```bash -pytest-watch -# or -ptw -``` - ---- - -## ๐Ÿšจ Known Limitations - -### 1. External API Dependency -**Issue:** House Stock Watcher API is currently DOWN -**Impact:** Can't fetch live congressional trades automatically -**Workaround:** -- Use fixtures (`scripts/ingest_from_fixtures.py`) -- Manual CSV import (`scripts/scrape_alternative_sources.py`) -- Manual entry (`scripts/add_custom_trades.py`) - -### 2. Market Data Limits -**Issue:** yfinance has rate limits and occasional failures -**Impact:** Bulk price fetching may be slow -**Workaround:** -- Fetch in batches -- Add retry logic (already implemented) -- Use caching (already implemented) - -### 3. No Live Trading API -**Issue:** We only use public disclosure data (inherent lag) -**Impact:** Trades are 30-45 days delayed by law -**This is expected:** POTE is for research, not real-time trading - ---- - -## ๐Ÿ“ˆ Performance Benchmarks - -### Test Execution Time -- **Full suite:** 1.8 seconds -- **Unit tests only:** 0.5 seconds -- **Integration tests:** 0.7 seconds -- **Parallel execution:** ~1.0 second (with pytest-xdist) - -### Database Operations -- **Create official:** < 1ms -- **Create trade:** < 1ms -- **Fetch prices (100 days):** ~50ms (in-memory) -- **Calculate returns:** ~10ms per trade -- **Aggregate metrics:** ~50ms for 100 trades - ---- - -## ๐ŸŽฏ Pre-Deployment Checklist - -### Before Deploying to Proxmox: - -- [x] All tests passing locally -- [x] No linter errors (`make lint`) -- [x] Database migrations work (`alembic upgrade head`) -- [x] Scripts are executable and work -- [x] Environment variables documented -- [x] Sample data available for testing -- [x] Documentation up to date - -### On Proxmox Container: - -```bash -# 1. Pull latest code -cd ~/pote -git pull - -# 2. Update dependencies -pip install -e . - -# 3. Run tests -pytest -v - -# 4. Run migrations -alembic upgrade head - -# 5. Verify system -python ~/status.sh - -# 6. Test a script -python scripts/enrich_securities.py -``` - ---- - -## ๐Ÿ”„ Continuous Testing - -### Git Pre-Commit Hook (Optional) -```bash -#!/bin/bash -# .git/hooks/pre-commit -pytest --tb=short -if [ $? -ne 0 ]; then - echo "Tests failed. Commit aborted." - exit 1 -fi -``` - -### CI/CD Integration (Future) -When you set up GitHub Actions or GitLab CI: - -```yaml -# .github/workflows/test.yml -name: Tests -on: [push, pull_request] -jobs: - test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - uses: actions/setup-python@v2 - - run: pip install -e . - - run: pytest -v --cov -``` - ---- - -## ๐Ÿ“ Test Maintenance - -### Adding New Tests - -**When to add tests:** -- Adding new features -- Fixing bugs (write test that fails, then fix) -- Before refactoring (ensure tests pass before & after) - -**Where to add tests:** -- Unit tests: `tests/test_.py` -- Integration tests: `tests/test__integration.py` - -**Example:** -```python -def test_new_feature(test_db_session): - """Test description.""" - session = test_db_session - # Arrange - # Act - # Assert -``` - -### Updating Fixtures - -Fixtures are in `tests/conftest.py`: -- `test_db_session` - Fresh database -- `sample_official` - Test official -- `sample_security` - Test security (AAPL) -- `sample_trade` - Test trade -- `sample_price` - Test price record - ---- - -## ๐ŸŽ‰ Summary - -### Current Status: **PRODUCTION READY** โœ… - -**What Works:** -- โœ… All 55 tests passing -- โœ… Full analytics pipeline functional -- โœ… Database operations solid -- โœ… Data ingestion from multiple sources -- โœ… Price fetching from yfinance -- โœ… Security enrichment -- โœ… Return calculations -- โœ… Benchmark comparisons -- โœ… Performance metrics -- โœ… CLI scripts operational - -**What's Missing:** -- โŒ Live congressional trade API (external issue - House Stock Watcher down) - - **Workaround:** Manual import, CSV, or alternative APIs available - -**Next Steps:** -1. โœ… Tests are complete -2. โœ… Code is ready -3. โžก๏ธ **Deploy to Proxmox** (or continue with Phase 2 features) -4. โžก๏ธ Add more data sources -5. โžก๏ธ Build dashboard (Phase 3) - ---- - -## ๐Ÿ“ž Need Help? - -See: -- `LOCAL_TEST_GUIDE.md` - Detailed local testing instructions -- `QUICKSTART.md` - Usage guide for deployed system -- `docs/09_data_updates.md` - How to add/update data -- `README.md` - Project overview - -**Questions about testing?** -All tests are documented with docstrings - read the test files! - -