Add comprehensive Ansible technical reference

NEW: Complete technical documentation for Ansible role implementation

📄 ANSIBLE_TECHNICAL_REFERENCE.md
====================================

INSTALLATION:
✅ Uses pyproject.toml (pip install -e .)
✅ NOT requirements.txt
✅ Python 3.11+ required

MIGRATIONS:
✅ Command: alembic upgrade head
✅ Run from project root (where alembic.ini is)
✅ Reads DATABASE_URL from .env
✅ Idempotent (safe to re-run)

SCRIPT PATHS (all in scripts/):
================================
Data Ingestion:
- fetch_congressional_trades.py
- enrich_securities.py
- fetch_sample_prices.py
- ingest_from_fixtures.py
- add_custom_trades.py
- scrape_alternative_sources.py

Analytics:
- analyze_official.py
- calculate_all_returns.py
- generate_trading_report.py

Monitoring:
- monitor_market.py
- analyze_disclosure_timing.py
- generate_pattern_report.py

Reporting:
- send_daily_report.py
- send_weekly_report.py
- health_check.py

Automation (Shell):
- automated_daily_run.sh ⭐ (for cron)
- automated_weekly_run.sh ⭐ (for cron)
- setup_cron.sh
- setup_automation.sh

COMPLETE DEPLOYMENT SEQUENCE:
=============================
Phase 1: System prep (user, packages, PostgreSQL)
Phase 2: App deployment (clone, venv, install, migrate)
Phase 3: Automation (logs, cron jobs)
Phase 4: Verification (health check, tests)

ANSIBLE TASK CHECKLIST:
=======================
✅ Full task examples for tasks/main.yml
✅ .env template (templates/env.j2)
✅ Variable reference (minimal vs full)
✅ Update procedure
✅ Testing commands
✅ Common issues & solutions

This doc has EVERYTHING needed to write the Ansible role!

ANSIBLE_TECHNICAL_REFERENCE.md

@@ -0,0 +1,598 @@
+# POTE Ansible Role - Technical Reference
+
+**Exact commands, paths, and procedures for the Ansible `roles/pote` role.**
+
+---
+
+## 📦 Installation Method
+
+### ✅ Uses `pyproject.toml` (NOT requirements.txt)
+
+```bash
+# Install in editable mode with all dependencies
+pip install -e .
+
+# For development (includes pytest, ruff, black, mypy)
+pip install -e ".[dev]"
+```
+
+**File:** `pyproject.toml`
+
+**Key dependencies:**
+```toml
+dependencies = [
+    "sqlalchemy>=2.0",
+    "alembic>=1.12",
+    "pydantic>=2.0",
+    "pydantic-settings>=2.0",
+    "python-dotenv>=1.0",
+    "requests>=2.31",
+    "pandas>=2.0",
+    "numpy>=1.24",
+    "yfinance>=0.2",
+    "psycopg2-binary>=2.9",
+]
+```
+
+**Python version:** `>=3.11` (required)
+
+---
+
+## 🗄️ Database Migrations
+
+### Command
+
+```bash
+alembic upgrade head
+```
+
+**Where to run:** From the project root directory (where `alembic.ini` is located)
+
+**Configuration:**
+- Migration scripts: `alembic/versions/`
+- Config file: `alembic.ini` (uses `.env` for DATABASE_URL)
+- Models: `src/pote/db/models.py`
+
+**Important:** 
+- Alembic reads `DATABASE_URL` from `.env` file via `src/pote/config.py`
+- Must run AFTER `.env` file is created
+- Must run AFTER database and user are created in PostgreSQL
+
+**Idempotency:** ✅ Safe to re-run (Alembic tracks applied migrations)
+
+---
+
+## 📁 Actual Script Paths
+
+### All scripts are in `scripts/` directory:
+
+#### Data Ingestion
+```bash
+scripts/fetch_congressional_trades.py    # Fetch from House Stock Watcher API
+scripts/enrich_securities.py             # Get company names, sectors from yfinance
+scripts/fetch_sample_prices.py           # Fetch price data for securities
+scripts/ingest_from_fixtures.py          # Offline mode - load from fixtures
+scripts/add_custom_trades.py             # Manual trade entry
+scripts/scrape_alternative_sources.py    # Import from CSV
+scripts/fetch_congress_members.py        # Fetch list of current members
+```
+
+#### Analytics
+```bash
+scripts/analyze_official.py              # Analyze specific official's performance
+scripts/calculate_all_returns.py         # System-wide return calculations
+scripts/generate_trading_report.py       # Generate trading activity report
+```
+
+#### Monitoring (Phase 1-3)
+```bash
+scripts/monitor_market.py                # Real-time market monitoring
+scripts/analyze_disclosure_timing.py     # Disclosure timing correlation
+scripts/generate_pattern_report.py       # Pattern detection & rankings
+```
+
+#### Reporting
+```bash
+scripts/send_daily_report.py             # Send daily email report
+scripts/send_weekly_report.py            # Send weekly email report
+scripts/health_check.py                  # System health check
+```
+
+#### Automation (Shell Scripts)
+```bash
+scripts/automated_daily_run.sh           # ⭐ Daily cron job
+scripts/automated_weekly_run.sh          # ⭐ Weekly cron job
+scripts/setup_cron.sh                    # Interactive cron setup
+scripts/setup_automation.sh              # Alternative automation setup
+scripts/daily_fetch.sh                   # Daily data fetch
+scripts/daily_update.sh                  # Daily update routine
+scripts/pre_market_close_update.sh       # Pre-market close routine
+```
+
+#### Deployment
+```bash
+scripts/proxmox_setup.sh                 # Bootstrap script for Proxmox LXC
+```
+
+---
+
+## 🔧 Complete Deployment Sequence
+
+### Phase 1: System Preparation
+
+```bash
+# 1. Create application user
+useradd -m -s /bin/bash poteapp
+
+# 2. Install system dependencies
+apt-get update
+apt-get install -y \
+    python3.11 \
+    python3.11-venv \
+    python3.11-dev \
+    python3-pip \
+    postgresql \
+    postgresql-contrib \
+    libpq-dev \
+    build-essential \
+    git \
+    curl \
+    htop
+
+# 3. Configure PostgreSQL
+sudo -u postgres psql << EOF
+CREATE USER poteuser WITH PASSWORD 'your_password';
+CREATE DATABASE potedb OWNER poteuser;
+GRANT ALL PRIVILEGES ON DATABASE potedb TO poteuser;
+EOF
+
+# 4. Configure PostgreSQL for remote access (if needed)
+# Edit /etc/postgresql/15/main/postgresql.conf:
+#   listen_addresses = '*'
+# Edit /etc/postgresql/15/main/pg_hba.conf:
+#   host all all 0.0.0.0/0 scram-sha-256
+# Restart: systemctl restart postgresql
+```
+
+### Phase 2: Application Deployment
+
+```bash
+# 1. Switch to application user
+su - poteapp
+
+# 2. Clone repository (requires SSH key setup)
+git clone gitea@10.0.30.169:ilia/POTE.git pote
+cd pote
+
+# 3. Checkout appropriate branch
+git checkout main  # or 'dev', 'qa' depending on environment
+
+# 4. Create virtual environment
+python3.11 -m venv venv
+
+# 5. Activate virtual environment
+source venv/bin/activate
+
+# 6. Upgrade pip
+pip install --upgrade pip
+
+# 7. Install application
+pip install -e .
+
+# 8. Create .env file (from template or Ansible)
+cat > .env << 'EOF'
+DATABASE_URL=postgresql://poteuser:your_password@localhost:5432/potedb
+SMTP_HOST=mail.levkin.ca
+SMTP_PORT=587
+SMTP_USER=test@levkin.ca
+SMTP_PASSWORD=your_smtp_password
+FROM_EMAIL=test@levkin.ca
+REPORT_RECIPIENTS=your@email.com
+MARKET_MONITOR_TICKERS=NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX
+ALERT_MIN_SEVERITY=5
+LOG_LEVEL=INFO
+LOG_FILE=/home/poteapp/logs/pote.log
+EOF
+
+# 9. Secure .env file
+chmod 600 .env
+
+# 10. Run database migrations
+alembic upgrade head
+```
+
+### Phase 3: Automation Setup
+
+```bash
+# 1. Create log directory
+mkdir -p /home/poteapp/logs
+
+# 2. Make scripts executable
+chmod +x scripts/*.sh
+
+# 3. Set up cron jobs
+crontab -e
+# Add these lines:
+# Daily report at 6:00 AM
+0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
+# Weekly report at 8:00 AM on Sunday
+0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1
+```
+
+### Phase 4: Verification
+
+```bash
+# 1. Run health check
+cd /home/poteapp/pote
+source venv/bin/activate
+python scripts/health_check.py
+
+# Expected output:
+# ✅ Database connection: OK
+# ✅ Database has X officials, Y trades, Z securities
+# ✅ Latest trade: YYYY-MM-DD
+# ✅ Price data: X securities with prices
+
+# 2. Test data ingestion (optional)
+python scripts/ingest_from_fixtures.py  # Offline test
+# OR
+python scripts/fetch_congressional_trades.py  # Live API test
+
+# 3. Test email (optional)
+python scripts/send_daily_report.py --to test@example.com --test-smtp
+
+# 4. Verify cron jobs
+crontab -l | grep POTE
+```
+
+---
+
+## 📋 Ansible Task Checklist
+
+### Your `roles/pote/tasks/main.yml` should include:
+
+```yaml
+---
+# 1. System preparation
+- name: Create application user
+  user:
+    name: "{{ appuser_name }}"
+    shell: "{{ appuser_shell }}"
+    create_home: yes
+
+- name: Install system dependencies
+  apt:
+    name:
+      - python3.11
+      - python3.11-venv
+      - python3.11-dev
+      - python3-pip
+      - postgresql
+      - postgresql-contrib
+      - libpq-dev
+      - build-essential
+      - git
+      - curl
+      - htop
+    state: present
+    update_cache: yes
+
+# 2. PostgreSQL setup
+- name: Create PostgreSQL user
+  postgresql_user:
+    name: "{{ db_user }}"
+    password: "{{ db_password }}"
+    state: present
+  become_user: postgres
+
+- name: Create PostgreSQL database
+  postgresql_db:
+    name: "{{ db_name }}"
+    owner: "{{ db_user }}"
+    state: present
+  become_user: postgres
+
+# 3. SSH key setup (for git clone)
+- name: Deploy SSH private key for git
+  copy:
+    content: "{{ pote_git_ssh_key }}"
+    dest: "/home/{{ appuser_name }}/.ssh/id_rsa"
+    owner: "{{ appuser_name }}"
+    mode: '0600'
+
+- name: Add Gitea to known_hosts
+  known_hosts:
+    name: "{{ pote_git_host }}"
+    key: "{{ lookup('pipe', 'ssh-keyscan ' + pote_git_host) }}"
+    state: present
+  become_user: "{{ appuser_name }}"
+
+# 4. Clone repository
+- name: Clone POTE repository
+  git:
+    repo: "{{ pote_git_repo }}"
+    dest: "{{ pote_install_path }}"
+    version: "{{ pote_git_branch }}"
+    accept_hostkey: yes
+  become_user: "{{ appuser_name }}"
+
+# 5. Python environment
+- name: Create virtual environment
+  command: python3.11 -m venv {{ pote_install_path }}/venv
+  args:
+    creates: "{{ pote_install_path }}/venv"
+  become_user: "{{ appuser_name }}"
+
+- name: Upgrade pip
+  pip:
+    name: pip
+    state: latest
+    virtualenv: "{{ pote_install_path }}/venv"
+  become_user: "{{ appuser_name }}"
+
+- name: Install POTE application
+  pip:
+    name: "{{ pote_install_path }}"
+    editable: yes
+    virtualenv: "{{ pote_install_path }}/venv"
+  become_user: "{{ appuser_name }}"
+
+# 6. Configuration
+- name: Deploy .env file
+  template:
+    src: env.j2
+    dest: "{{ pote_install_path }}/.env"
+    owner: "{{ appuser_name }}"
+    mode: '0600'
+  become_user: "{{ appuser_name }}"
+
+# 7. Database migrations
+- name: Run Alembic migrations
+  command: "{{ pote_install_path }}/venv/bin/alembic upgrade head"
+  args:
+    chdir: "{{ pote_install_path }}"
+  become_user: "{{ appuser_name }}"
+
+# 8. Log directory
+- name: Create log directory
+  file:
+    path: "/home/{{ appuser_name }}/logs"
+    state: directory
+    owner: "{{ appuser_name }}"
+    mode: '0755'
+
+# 9. Make scripts executable
+- name: Make shell scripts executable
+  file:
+    path: "{{ pote_install_path }}/scripts/{{ item }}"
+    mode: '0755'
+  loop:
+    - automated_daily_run.sh
+    - automated_weekly_run.sh
+    - setup_cron.sh
+  become_user: "{{ appuser_name }}"
+
+# 10. Cron jobs
+- name: Set up daily cron job
+  cron:
+    name: "POTE Automated Daily Run"
+    minute: "{{ pote_daily_report_time.split()[1] }}"
+    hour: "{{ pote_daily_report_time.split()[0] }}"
+    job: "{{ pote_install_path }}/scripts/automated_daily_run.sh >> /home/{{ appuser_name }}/logs/daily_run.log 2>&1"
+    user: "{{ appuser_name }}"
+  when: pote_enable_cron and pote_daily_report_enabled
+
+- name: Set up weekly cron job
+  cron:
+    name: "POTE Automated Weekly Run"
+    minute: "{{ pote_weekly_report_time.split()[1] }}"
+    hour: "{{ pote_weekly_report_time.split()[0] }}"
+    day: "0"  # Sunday
+    job: "{{ pote_install_path }}/scripts/automated_weekly_run.sh >> /home/{{ appuser_name }}/logs/weekly_run.log 2>&1"
+    user: "{{ appuser_name }}"
+  when: pote_enable_cron and pote_weekly_report_enabled
+```
+
+---
+
+## 🎨 `.env` Template for Ansible
+
+### Create `roles/pote/templates/env.j2`:
+
+```jinja2
+# Database
+DATABASE_URL=postgresql://{{ db_user }}:{{ db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }}
+
+# Email
+SMTP_HOST={{ smtp_host }}
+SMTP_PORT={{ smtp_port }}
+SMTP_USER={{ smtp_user }}
+SMTP_PASSWORD={{ smtp_password }}
+FROM_EMAIL={{ from_email }}
+REPORT_RECIPIENTS={{ report_recipients }}
+
+# Monitoring
+MARKET_MONITOR_TICKERS={{ market_tickers }}
+ALERT_MIN_SEVERITY={{ alert_severity }}
+
+# Logging
+LOG_LEVEL={{ log_level }}
+LOG_FILE={{ log_file }}
+
+# Optional API Keys
+{% if quiverquant_api_key is defined and quiverquant_api_key %}
+QUIVERQUANT_API_KEY={{ quiverquant_api_key }}
+{% endif %}
+{% if fmp_api_key is defined and fmp_api_key %}
+FMP_API_KEY={{ fmp_api_key }}
+{% endif %}
+```
+
+---
+
+## 🔄 Update Procedure (Existing Deployment)
+
+```bash
+# 1. Pull latest code
+cd /home/poteapp/pote
+git pull origin main
+
+# 2. Activate venv
+source venv/bin/activate
+
+# 3. Update dependencies (if pyproject.toml changed)
+pip install -e .
+
+# 4. Run migrations (if models changed)
+alembic upgrade head
+
+# 5. Restart services (if any)
+# (Currently no services - cron jobs will pick up changes)
+```
+
+---
+
+## 🧪 Testing the Deployment
+
+### Quick Health Check
+
+```bash
+cd /home/poteapp/pote
+source venv/bin/activate
+python scripts/health_check.py
+```
+
+**Exit codes:**
+- `0` = Healthy
+- Non-zero = Issues found
+
+### Full Test Sequence
+
+```bash
+# 1. Test database connection
+python -c "from pote.db import get_session; next(get_session()); print('✅ DB OK')"
+
+# 2. Test offline ingestion
+python scripts/ingest_from_fixtures.py
+
+# 3. Test enrichment
+python scripts/enrich_securities.py
+
+# 4. Test price fetch
+python scripts/fetch_sample_prices.py
+
+# 5. Test email (dry run)
+python scripts/send_daily_report.py --to test@example.com --test-smtp
+
+# 6. Verify cron
+crontab -l | grep POTE
+```
+
+---
+
+## 📊 Key Variables Reference
+
+### Minimal (3 required)
+
+```yaml
+pote_git_repo: "gitea@10.0.30.169:ilia/POTE.git"
+pote_git_branch: "main"
+vault_smtp_password: "{{ vault_smtp_password }}"
+```
+
+### Essential (for production)
+
+```yaml
+# Paths
+pote_install_path: "/home/poteapp/pote"
+appuser_name: "poteapp"
+
+# Database
+db_name: "potedb"
+db_user: "poteuser"
+db_password: "{{ vault_db_password_prod }}"
+db_host: "localhost"
+db_port: 5432
+
+# Email
+smtp_host: "mail.levkin.ca"
+smtp_port: 587
+smtp_user: "test@levkin.ca"
+smtp_password: "{{ vault_smtp_password }}"
+from_email: "test@levkin.ca"
+report_recipients: "your@email.com"
+
+# Monitoring
+market_tickers: "NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX"
+alert_severity: 5
+
+# Automation
+pote_enable_cron: true
+pote_daily_report_time: "0 6"    # 6:00 AM
+pote_weekly_report_time: "0 8"   # 8:00 AM Sunday
+
+# Logging
+log_level: "INFO"
+log_file: "/home/poteapp/logs/pote.log"
+```
+
+---
+
+## 🚨 Common Issues & Solutions
+
+### Issue: `ModuleNotFoundError: No module named 'pote'`
+**Solution:** Run `pip install -e .` from project root
+
+### Issue: `alembic.util.exc.CommandError: Can't locate revision identified by 'head'`
+**Solution:** Database is not initialized. Run `alembic upgrade head`
+
+### Issue: `psycopg2.OperationalError: could not connect to server`
+**Solution:** Check PostgreSQL is running and DATABASE_URL is correct
+
+### Issue: Cron jobs not running
+**Solution:** 
+- Check crontab: `crontab -l`
+- Check logs: `tail -f ~/logs/daily_run.log`
+- Verify script permissions: `ls -la scripts/*.sh`
+
+### Issue: Email not sending
+**Solution:**
+- Test SMTP: `python scripts/send_daily_report.py --test-smtp`
+- Check `.env` has correct SMTP settings
+- Verify SMTP_PASSWORD is correct
+
+---
+
+## 📞 Support Commands
+
+```bash
+# Check Python version
+python3.11 --version
+
+# Check pip packages
+pip list | grep -E "(alembic|sqlalchemy|pydantic)"
+
+# Check database
+psql -U poteuser -d potedb -c "\dt"
+
+# Check logs
+tail -f ~/logs/daily_run.log
+tail -f ~/logs/pote.log
+
+# Check cron
+crontab -l
+grep CRON /var/log/syslog | tail -20
+
+# Manual test run
+cd ~/pote
+source venv/bin/activate
+./scripts/automated_daily_run.sh
+```
+
+---
+
+**Last Updated:** December 2025  
+**POTE Version:** 0.1.0  
+**Python Required:** 3.11+  
+**PostgreSQL Required:** 15+
+