# POTE Ansible Role - Technical Reference

**Exact commands, paths, and procedures for the Ansible `roles/pote` role.**

---

## 📦 Installation Method

### ✅ Uses `pyproject.toml` (NOT requirements.txt)

```bash
# Install in editable mode with all dependencies
pip install -e .

# For development (includes pytest, ruff, black, mypy)
pip install -e ".[dev]"
```

**File:** `pyproject.toml`

**Key dependencies:**
```toml
dependencies = [
    "sqlalchemy>=2.0",
    "alembic>=1.12",
    "pydantic>=2.0",
    "pydantic-settings>=2.0",
    "python-dotenv>=1.0",
    "requests>=2.31",
    "pandas>=2.0",
    "numpy>=1.24",
    "yfinance>=0.2",
    "psycopg2-binary>=2.9",
]
```

**Python version:** `>=3.11` (required)

---

## 🗄️ Database Migrations

### Command

```bash
alembic upgrade head
```

**Where to run:** From the project root directory (where `alembic.ini` is located)

**Configuration:**
- Migration scripts: `alembic/versions/`
- Config file: `alembic.ini` (uses `.env` for DATABASE_URL)
- Models: `src/pote/db/models.py`

**Important:** 
- Alembic reads `DATABASE_URL` from `.env` file via `src/pote/config.py`
- Must run AFTER `.env` file is created
- Must run AFTER database and user are created in PostgreSQL

**Idempotency:** ✅ Safe to re-run (Alembic tracks applied migrations)

---

## 📁 Actual Script Paths

### All scripts are in `scripts/` directory:

#### Data Ingestion
```bash
scripts/fetch_congressional_trades.py    # Fetch from House Stock Watcher API
scripts/enrich_securities.py             # Get company names, sectors from yfinance
scripts/fetch_sample_prices.py           # Fetch price data for securities
scripts/ingest_from_fixtures.py          # Offline mode - load from fixtures
scripts/add_custom_trades.py             # Manual trade entry
scripts/scrape_alternative_sources.py    # Import from CSV
scripts/fetch_congress_members.py        # Fetch list of current members
```

#### Analytics
```bash
scripts/analyze_official.py              # Analyze specific official's performance
scripts/calculate_all_returns.py         # System-wide return calculations
scripts/generate_trading_report.py       # Generate trading activity report
```

#### Monitoring (Phase 1-3)
```bash
scripts/monitor_market.py                # Real-time market monitoring
scripts/analyze_disclosure_timing.py     # Disclosure timing correlation
scripts/generate_pattern_report.py       # Pattern detection & rankings
```

#### Reporting
```bash
scripts/send_daily_report.py             # Send daily email report
scripts/send_weekly_report.py            # Send weekly email report
scripts/health_check.py                  # System health check
```

#### Automation (Shell Scripts)
```bash
scripts/automated_daily_run.sh           # ⭐ Daily cron job
scripts/automated_weekly_run.sh          # ⭐ Weekly cron job
scripts/setup_cron.sh                    # Interactive cron setup
scripts/setup_automation.sh              # Alternative automation setup
scripts/daily_fetch.sh                   # Daily data fetch
scripts/daily_update.sh                  # Daily update routine
scripts/pre_market_close_update.sh       # Pre-market close routine
```

#### Deployment
```bash
scripts/proxmox_setup.sh                 # Bootstrap script for Proxmox LXC
```

---

## 🔧 Complete Deployment Sequence

### Phase 1: System Preparation

```bash
# 1. Create application user
useradd -m -s /bin/bash poteapp

# 2. Install system dependencies
apt-get update
apt-get install -y \
    python3.11 \
    python3.11-venv \
    python3.11-dev \
    python3-pip \
    postgresql \
    postgresql-contrib \
    libpq-dev \
    build-essential \
    git \
    curl \
    htop

# 3. Configure PostgreSQL
sudo -u postgres psql << EOF
CREATE USER poteuser WITH PASSWORD 'your_password';
CREATE DATABASE potedb OWNER poteuser;
GRANT ALL PRIVILEGES ON DATABASE potedb TO poteuser;
EOF

# 4. Configure PostgreSQL for remote access (if needed)
# Edit /etc/postgresql/15/main/postgresql.conf:
#   listen_addresses = '*'
# Edit /etc/postgresql/15/main/pg_hba.conf:
#   host all all 0.0.0.0/0 scram-sha-256
# Restart: systemctl restart postgresql
```

### Phase 2: Application Deployment

```bash
# 1. Switch to application user
su - poteapp

# 2. Clone repository (requires SSH key setup)
git clone gitea@10.0.30.169:ilia/POTE.git pote
cd pote

# 3. Checkout appropriate branch
git checkout main  # or 'dev', 'qa' depending on environment

# 4. Create virtual environment
python3.11 -m venv venv

# 5. Activate virtual environment
source venv/bin/activate

# 6. Upgrade pip
pip install --upgrade pip

# 7. Install application
pip install -e .

# 8. Create .env file (from template or Ansible)
cat > .env << 'EOF'
DATABASE_URL=postgresql://poteuser:your_password@localhost:5432/potedb
SMTP_HOST=mail.levkin.ca
SMTP_PORT=587
SMTP_USER=test@levkin.ca
SMTP_PASSWORD=your_smtp_password
FROM_EMAIL=test@levkin.ca
REPORT_RECIPIENTS=your@email.com
MARKET_MONITOR_TICKERS=NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX
ALERT_MIN_SEVERITY=5
LOG_LEVEL=INFO
LOG_FILE=/home/poteapp/logs/pote.log
EOF

# 9. Secure .env file
chmod 600 .env

# 10. Run database migrations
alembic upgrade head
```

### Phase 3: Automation Setup

```bash
# 1. Create log directory
mkdir -p /home/poteapp/logs

# 2. Make scripts executable
chmod +x scripts/*.sh

# 3. Set up cron jobs
crontab -e
# Add these lines:
# Daily report at 6:00 AM
0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
# Weekly report at 8:00 AM on Sunday
0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1
```

### Phase 4: Verification

```bash
# 1. Run health check
cd /home/poteapp/pote
source venv/bin/activate
python scripts/health_check.py

# Expected output:
# ✅ Database connection: OK
# ✅ Database has X officials, Y trades, Z securities
# ✅ Latest trade: YYYY-MM-DD
# ✅ Price data: X securities with prices

# 2. Test data ingestion (optional)
python scripts/ingest_from_fixtures.py  # Offline test
# OR
python scripts/fetch_congressional_trades.py  # Live API test

# 3. Test email (optional)
python scripts/send_daily_report.py --to test@example.com --test-smtp

# 4. Verify cron jobs
crontab -l | grep POTE
```

---

## 📋 Ansible Task Checklist

### Your `roles/pote/tasks/main.yml` should include:

```yaml
---
# 1. System preparation
- name: Create application user
  user:
    name: "{{ appuser_name }}"
    shell: "{{ appuser_shell }}"
    create_home: yes

- name: Install system dependencies
  apt:
    name:
      - python3.11
      - python3.11-venv
      - python3.11-dev
      - python3-pip
      - postgresql
      - postgresql-contrib
      - libpq-dev
      - build-essential
      - git
      - curl
      - htop
    state: present
    update_cache: yes

# 2. PostgreSQL setup
- name: Create PostgreSQL user
  postgresql_user:
    name: "{{ db_user }}"
    password: "{{ db_password }}"
    state: present
  become_user: postgres

- name: Create PostgreSQL database
  postgresql_db:
    name: "{{ db_name }}"
    owner: "{{ db_user }}"
    state: present
  become_user: postgres

# 3. SSH key setup (for git clone)
- name: Deploy SSH private key for git
  copy:
    content: "{{ pote_git_ssh_key }}"
    dest: "/home/{{ appuser_name }}/.ssh/id_rsa"
    owner: "{{ appuser_name }}"
    mode: '0600'

- name: Add Gitea to known_hosts
  known_hosts:
    name: "{{ pote_git_host }}"
    key: "{{ lookup('pipe', 'ssh-keyscan ' + pote_git_host) }}"
    state: present
  become_user: "{{ appuser_name }}"

# 4. Clone repository
- name: Clone POTE repository
  git:
    repo: "{{ pote_git_repo }}"
    dest: "{{ pote_install_path }}"
    version: "{{ pote_git_branch }}"
    accept_hostkey: yes
  become_user: "{{ appuser_name }}"

# 5. Python environment
- name: Create virtual environment
  command: python3.11 -m venv {{ pote_install_path }}/venv
  args:
    creates: "{{ pote_install_path }}/venv"
  become_user: "{{ appuser_name }}"

- name: Upgrade pip
  pip:
    name: pip
    state: latest
    virtualenv: "{{ pote_install_path }}/venv"
  become_user: "{{ appuser_name }}"

- name: Install POTE application
  pip:
    name: "{{ pote_install_path }}"
    editable: yes
    virtualenv: "{{ pote_install_path }}/venv"
  become_user: "{{ appuser_name }}"

# 6. Configuration
- name: Deploy .env file
  template:
    src: env.j2
    dest: "{{ pote_install_path }}/.env"
    owner: "{{ appuser_name }}"
    mode: '0600'
  become_user: "{{ appuser_name }}"

# 7. Database migrations
- name: Run Alembic migrations
  command: "{{ pote_install_path }}/venv/bin/alembic upgrade head"
  args:
    chdir: "{{ pote_install_path }}"
  become_user: "{{ appuser_name }}"

# 8. Log directory
- name: Create log directory
  file:
    path: "/home/{{ appuser_name }}/logs"
    state: directory
    owner: "{{ appuser_name }}"
    mode: '0755'

# 9. Make scripts executable
- name: Make shell scripts executable
  file:
    path: "{{ pote_install_path }}/scripts/{{ item }}"
    mode: '0755'
  loop:
    - automated_daily_run.sh
    - automated_weekly_run.sh
    - setup_cron.sh
  become_user: "{{ appuser_name }}"

# 10. Cron jobs
- name: Set up daily cron job
  cron:
    name: "POTE Automated Daily Run"
    minute: "{{ pote_daily_report_time.split()[1] }}"
    hour: "{{ pote_daily_report_time.split()[0] }}"
    job: "{{ pote_install_path }}/scripts/automated_daily_run.sh >> /home/{{ appuser_name }}/logs/daily_run.log 2>&1"
    user: "{{ appuser_name }}"
  when: pote_enable_cron and pote_daily_report_enabled

- name: Set up weekly cron job
  cron:
    name: "POTE Automated Weekly Run"
    minute: "{{ pote_weekly_report_time.split()[1] }}"
    hour: "{{ pote_weekly_report_time.split()[0] }}"
    day: "0"  # Sunday
    job: "{{ pote_install_path }}/scripts/automated_weekly_run.sh >> /home/{{ appuser_name }}/logs/weekly_run.log 2>&1"
    user: "{{ appuser_name }}"
  when: pote_enable_cron and pote_weekly_report_enabled
```

---

## 🎨 `.env` Template for Ansible

### Create `roles/pote/templates/env.j2`:

```jinja2
# Database
DATABASE_URL=postgresql://{{ db_user }}:{{ db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }}

# Email
SMTP_HOST={{ smtp_host }}
SMTP_PORT={{ smtp_port }}
SMTP_USER={{ smtp_user }}
SMTP_PASSWORD={{ smtp_password }}
FROM_EMAIL={{ from_email }}
REPORT_RECIPIENTS={{ report_recipients }}

# Monitoring
MARKET_MONITOR_TICKERS={{ market_tickers }}
ALERT_MIN_SEVERITY={{ alert_severity }}

# Logging
LOG_LEVEL={{ log_level }}
LOG_FILE={{ log_file }}

# Optional API Keys
{% if quiverquant_api_key is defined and quiverquant_api_key %}
QUIVERQUANT_API_KEY={{ quiverquant_api_key }}
{% endif %}
{% if fmp_api_key is defined and fmp_api_key %}
FMP_API_KEY={{ fmp_api_key }}
{% endif %}
```

---

## 🔄 Update Procedure (Existing Deployment)

```bash
# 1. Pull latest code
cd /home/poteapp/pote
git pull origin main

# 2. Activate venv
source venv/bin/activate

# 3. Update dependencies (if pyproject.toml changed)
pip install -e .

# 4. Run migrations (if models changed)
alembic upgrade head

# 5. Restart services (if any)
# (Currently no services - cron jobs will pick up changes)
```

---

## 🧪 Testing the Deployment

### Quick Health Check

```bash
cd /home/poteapp/pote
source venv/bin/activate
python scripts/health_check.py
```

**Exit codes:**
- `0` = Healthy
- Non-zero = Issues found

### Full Test Sequence

```bash
# 1. Test database connection
python -c "from pote.db import get_session; next(get_session()); print('✅ DB OK')"

# 2. Test offline ingestion
python scripts/ingest_from_fixtures.py

# 3. Test enrichment
python scripts/enrich_securities.py

# 4. Test price fetch
python scripts/fetch_sample_prices.py

# 5. Test email (dry run)
python scripts/send_daily_report.py --to test@example.com --test-smtp

# 6. Verify cron
crontab -l | grep POTE
```

---

## 📊 Key Variables Reference

### Minimal (3 required)

```yaml
pote_git_repo: "gitea@10.0.30.169:ilia/POTE.git"
pote_git_branch: "main"
vault_smtp_password: "{{ vault_smtp_password }}"
```

### Essential (for production)

```yaml
# Paths
pote_install_path: "/home/poteapp/pote"
appuser_name: "poteapp"

# Database
db_name: "potedb"
db_user: "poteuser"
db_password: "{{ vault_db_password_prod }}"
db_host: "localhost"
db_port: 5432

# Email
smtp_host: "mail.levkin.ca"
smtp_port: 587
smtp_user: "test@levkin.ca"
smtp_password: "{{ vault_smtp_password }}"
from_email: "test@levkin.ca"
report_recipients: "your@email.com"

# Monitoring
market_tickers: "NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX"
alert_severity: 5

# Automation
pote_enable_cron: true
pote_daily_report_time: "0 6"    # 6:00 AM
pote_weekly_report_time: "0 8"   # 8:00 AM Sunday

# Logging
log_level: "INFO"
log_file: "/home/poteapp/logs/pote.log"
```

---

## 🚨 Common Issues & Solutions

### Issue: `ModuleNotFoundError: No module named 'pote'`
**Solution:** Run `pip install -e .` from project root

### Issue: `alembic.util.exc.CommandError: Can't locate revision identified by 'head'`
**Solution:** Database is not initialized. Run `alembic upgrade head`

### Issue: `psycopg2.OperationalError: could not connect to server`
**Solution:** Check PostgreSQL is running and DATABASE_URL is correct

### Issue: Cron jobs not running
**Solution:** 
- Check crontab: `crontab -l`
- Check logs: `tail -f ~/logs/daily_run.log`
- Verify script permissions: `ls -la scripts/*.sh`

### Issue: Email not sending
**Solution:**
- Test SMTP: `python scripts/send_daily_report.py --test-smtp`
- Check `.env` has correct SMTP settings
- Verify SMTP_PASSWORD is correct

---

## 📞 Support Commands

```bash
# Check Python version
python3.11 --version

# Check pip packages
pip list | grep -E "(alembic|sqlalchemy|pydantic)"

# Check database
psql -U poteuser -d potedb -c "\dt"

# Check logs
tail -f ~/logs/daily_run.log
tail -f ~/logs/pote.log

# Check cron
crontab -l
grep CRON /var/log/syslog | tail -20

# Manual test run
cd ~/pote
source venv/bin/activate
./scripts/automated_daily_run.sh
```

---

**Last Updated:** December 2025  
**POTE Version:** 0.1.0  
**Python Required:** 3.11+  
**PostgreSQL Required:** 15+