POTE/ANSIBLE_TECHNICAL_REFERENCE.md

POTE Ansible Role - Technical Reference

Exact commands, paths, and procedures for the Ansible roles/pote role.

---

📦 Installation Method

✅ Uses pyproject.toml (NOT requirements.txt)

# Install in editable mode with all dependencies
pip install -e .

# For development (includes pytest, ruff, black, mypy)
pip install -e ".[dev]"

File: pyproject.toml

Key dependencies:

dependencies = [
    "sqlalchemy>=2.0",
    "alembic>=1.12",
    "pydantic>=2.0",
    "pydantic-settings>=2.0",
    "python-dotenv>=1.0",
    "requests>=2.31",
    "pandas>=2.0",
    "numpy>=1.24",
    "yfinance>=0.2",
    "psycopg2-binary>=2.9",
]

Python version: >=3.11 (required)

---

🗄️ Database Migrations

Command

alembic upgrade head

Where to run: From the project root directory (where alembic.ini is located)

Configuration:

• Migration scripts: alembic/versions/
• Config file: alembic.ini (uses .env for DATABASE_URL)
• Models: src/pote/db/models.py

Important:

• Alembic reads DATABASE_URL from .env file via src/pote/config.py
• Must run AFTER .env file is created
• Must run AFTER database and user are created in PostgreSQL

Idempotency: ✅ Safe to re-run (Alembic tracks applied migrations)

---

📁 Actual Script Paths

All scripts are in scripts/ directory:

Data Ingestion

scripts/fetch_congressional_trades.py    # Fetch from House Stock Watcher API
scripts/enrich_securities.py             # Get company names, sectors from yfinance
scripts/fetch_sample_prices.py           # Fetch price data for securities
scripts/ingest_from_fixtures.py          # Offline mode - load from fixtures
scripts/add_custom_trades.py             # Manual trade entry
scripts/scrape_alternative_sources.py    # Import from CSV
scripts/fetch_congress_members.py        # Fetch list of current members

Analytics

scripts/analyze_official.py              # Analyze specific official's performance
scripts/calculate_all_returns.py         # System-wide return calculations
scripts/generate_trading_report.py       # Generate trading activity report

Monitoring (Phase 1-3)

scripts/monitor_market.py                # Real-time market monitoring
scripts/analyze_disclosure_timing.py     # Disclosure timing correlation
scripts/generate_pattern_report.py       # Pattern detection & rankings

Reporting

scripts/send_daily_report.py             # Send daily email report
scripts/send_weekly_report.py            # Send weekly email report
scripts/health_check.py                  # System health check

Automation (Shell Scripts)

scripts/automated_daily_run.sh           # ⭐ Daily cron job
scripts/automated_weekly_run.sh          # ⭐ Weekly cron job
scripts/setup_cron.sh                    # Interactive cron setup
scripts/setup_automation.sh              # Alternative automation setup
scripts/daily_fetch.sh                   # Daily data fetch
scripts/daily_update.sh                  # Daily update routine
scripts/pre_market_close_update.sh       # Pre-market close routine

Deployment

scripts/proxmox_setup.sh                 # Bootstrap script for Proxmox LXC

---

🔧 Complete Deployment Sequence

Phase 1: System Preparation

# 1. Create application user
useradd -m -s /bin/bash poteapp

# 2. Install system dependencies
apt-get update
apt-get install -y \
    python3.11 \
    python3.11-venv \
    python3.11-dev \
    python3-pip \
    postgresql \
    postgresql-contrib \
    libpq-dev \
    build-essential \
    git \
    curl \
    htop

# 3. Configure PostgreSQL
sudo -u postgres psql << EOF
CREATE USER poteuser WITH PASSWORD 'your_password';
CREATE DATABASE potedb OWNER poteuser;
GRANT ALL PRIVILEGES ON DATABASE potedb TO poteuser;
EOF

# 4. Configure PostgreSQL for remote access (if needed)
# Edit /etc/postgresql/15/main/postgresql.conf:
#   listen_addresses = '*'
# Edit /etc/postgresql/15/main/pg_hba.conf:
#   host all all 0.0.0.0/0 scram-sha-256
# Restart: systemctl restart postgresql

Phase 2: Application Deployment

# 1. Switch to application user
su - poteapp

# 2. Clone repository (requires SSH key setup)
git clone gitea@10.0.30.169:ilia/POTE.git pote
cd pote

# 3. Checkout appropriate branch
git checkout main  # or 'dev', 'qa' depending on environment

# 4. Create virtual environment
python3.11 -m venv venv

# 5. Activate virtual environment
source venv/bin/activate

# 6. Upgrade pip
pip install --upgrade pip

# 7. Install application
pip install -e .

# 8. Create .env file (from template or Ansible)
cat > .env << 'EOF'
DATABASE_URL=postgresql://poteuser:your_password@localhost:5432/potedb
SMTP_HOST=mail.levkin.ca
SMTP_PORT=587
SMTP_USER=test@levkin.ca
SMTP_PASSWORD=your_smtp_password
FROM_EMAIL=test@levkin.ca
REPORT_RECIPIENTS=your@email.com
MARKET_MONITOR_TICKERS=NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX
ALERT_MIN_SEVERITY=5
LOG_LEVEL=INFO
LOG_FILE=/home/poteapp/logs/pote.log
EOF

# 9. Secure .env file
chmod 600 .env

# 10. Run database migrations
alembic upgrade head

Phase 3: Automation Setup

# 1. Create log directory
mkdir -p /home/poteapp/logs

# 2. Make scripts executable
chmod +x scripts/*.sh

# 3. Set up cron jobs
crontab -e
# Add these lines:
# Daily report at 6:00 AM
0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1
# Weekly report at 8:00 AM on Sunday
0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1

Phase 4: Verification

# 1. Run health check
cd /home/poteapp/pote
source venv/bin/activate
python scripts/health_check.py

# Expected output:
# ✅ Database connection: OK
# ✅ Database has X officials, Y trades, Z securities
# ✅ Latest trade: YYYY-MM-DD
# ✅ Price data: X securities with prices

# 2. Test data ingestion (optional)
python scripts/ingest_from_fixtures.py  # Offline test
# OR
python scripts/fetch_congressional_trades.py  # Live API test

# 3. Test email (optional)
python scripts/send_daily_report.py --to test@example.com --test-smtp

# 4. Verify cron jobs
crontab -l | grep POTE

---

📋 Ansible Task Checklist

Your roles/pote/tasks/main.yml should include:

---
# 1. System preparation
- name: Create application user
  user:
    name: "{{ appuser_name }}"
    shell: "{{ appuser_shell }}"
    create_home: yes

- name: Install system dependencies
  apt:
    name:
      - python3.11
      - python3.11-venv
      - python3.11-dev
      - python3-pip
      - postgresql
      - postgresql-contrib
      - libpq-dev
      - build-essential
      - git
      - curl
      - htop
    state: present
    update_cache: yes

# 2. PostgreSQL setup
- name: Create PostgreSQL user
  postgresql_user:
    name: "{{ db_user }}"
    password: "{{ db_password }}"
    state: present
  become_user: postgres

- name: Create PostgreSQL database
  postgresql_db:
    name: "{{ db_name }}"
    owner: "{{ db_user }}"
    state: present
  become_user: postgres

# 3. SSH key setup (for git clone)
- name: Deploy SSH private key for git
  copy:
    content: "{{ pote_git_ssh_key }}"
    dest: "/home/{{ appuser_name }}/.ssh/id_rsa"
    owner: "{{ appuser_name }}"
    mode: '0600'

- name: Add Gitea to known_hosts
  known_hosts:
    name: "{{ pote_git_host }}"
    key: "{{ lookup('pipe', 'ssh-keyscan ' + pote_git_host) }}"
    state: present
  become_user: "{{ appuser_name }}"

# 4. Clone repository
- name: Clone POTE repository
  git:
    repo: "{{ pote_git_repo }}"
    dest: "{{ pote_install_path }}"
    version: "{{ pote_git_branch }}"
    accept_hostkey: yes
  become_user: "{{ appuser_name }}"

# 5. Python environment
- name: Create virtual environment
  command: python3.11 -m venv {{ pote_install_path }}/venv
  args:
    creates: "{{ pote_install_path }}/venv"
  become_user: "{{ appuser_name }}"

- name: Upgrade pip
  pip:
    name: pip
    state: latest
    virtualenv: "{{ pote_install_path }}/venv"
  become_user: "{{ appuser_name }}"

- name: Install POTE application
  pip:
    name: "{{ pote_install_path }}"
    editable: yes
    virtualenv: "{{ pote_install_path }}/venv"
  become_user: "{{ appuser_name }}"

# 6. Configuration
- name: Deploy .env file
  template:
    src: env.j2
    dest: "{{ pote_install_path }}/.env"
    owner: "{{ appuser_name }}"
    mode: '0600'
  become_user: "{{ appuser_name }}"

# 7. Database migrations
- name: Run Alembic migrations
  command: "{{ pote_install_path }}/venv/bin/alembic upgrade head"
  args:
    chdir: "{{ pote_install_path }}"
  become_user: "{{ appuser_name }}"

# 8. Log directory
- name: Create log directory
  file:
    path: "/home/{{ appuser_name }}/logs"
    state: directory
    owner: "{{ appuser_name }}"
    mode: '0755'

# 9. Make scripts executable
- name: Make shell scripts executable
  file:
    path: "{{ pote_install_path }}/scripts/{{ item }}"
    mode: '0755'
  loop:
    - automated_daily_run.sh
    - automated_weekly_run.sh
    - setup_cron.sh
  become_user: "{{ appuser_name }}"

# 10. Cron jobs
- name: Set up daily cron job
  cron:
    name: "POTE Automated Daily Run"
    minute: "{{ pote_daily_report_time.split()[1] }}"
    hour: "{{ pote_daily_report_time.split()[0] }}"
    job: "{{ pote_install_path }}/scripts/automated_daily_run.sh >> /home/{{ appuser_name }}/logs/daily_run.log 2>&1"
    user: "{{ appuser_name }}"
  when: pote_enable_cron and pote_daily_report_enabled

- name: Set up weekly cron job
  cron:
    name: "POTE Automated Weekly Run"
    minute: "{{ pote_weekly_report_time.split()[1] }}"
    hour: "{{ pote_weekly_report_time.split()[0] }}"
    day: "0"  # Sunday
    job: "{{ pote_install_path }}/scripts/automated_weekly_run.sh >> /home/{{ appuser_name }}/logs/weekly_run.log 2>&1"
    user: "{{ appuser_name }}"
  when: pote_enable_cron and pote_weekly_report_enabled

---

🎨 .env Template for Ansible

Create roles/pote/templates/env.j2:

# Database
DATABASE_URL=postgresql://{{ db_user }}:{{ db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }}

# Email
SMTP_HOST={{ smtp_host }}
SMTP_PORT={{ smtp_port }}
SMTP_USER={{ smtp_user }}
SMTP_PASSWORD={{ smtp_password }}
FROM_EMAIL={{ from_email }}
REPORT_RECIPIENTS={{ report_recipients }}

# Monitoring
MARKET_MONITOR_TICKERS={{ market_tickers }}
ALERT_MIN_SEVERITY={{ alert_severity }}

# Logging
LOG_LEVEL={{ log_level }}
LOG_FILE={{ log_file }}

# Optional API Keys
{% if quiverquant_api_key is defined and quiverquant_api_key %}
QUIVERQUANT_API_KEY={{ quiverquant_api_key }}
{% endif %}
{% if fmp_api_key is defined and fmp_api_key %}
FMP_API_KEY={{ fmp_api_key }}
{% endif %}

---

🔄 Update Procedure (Existing Deployment)

# 1. Pull latest code
cd /home/poteapp/pote
git pull origin main

# 2. Activate venv
source venv/bin/activate

# 3. Update dependencies (if pyproject.toml changed)
pip install -e .

# 4. Run migrations (if models changed)
alembic upgrade head

# 5. Restart services (if any)
# (Currently no services - cron jobs will pick up changes)

---

🧪 Testing the Deployment

Quick Health Check

cd /home/poteapp/pote
source venv/bin/activate
python scripts/health_check.py

Exit codes:

• 0 = Healthy
• Non-zero = Issues found

Full Test Sequence

# 1. Test database connection
python -c "from pote.db import get_session; next(get_session()); print('✅ DB OK')"

# 2. Test offline ingestion
python scripts/ingest_from_fixtures.py

# 3. Test enrichment
python scripts/enrich_securities.py

# 4. Test price fetch
python scripts/fetch_sample_prices.py

# 5. Test email (dry run)
python scripts/send_daily_report.py --to test@example.com --test-smtp

# 6. Verify cron
crontab -l | grep POTE

---

📊 Key Variables Reference

Minimal (3 required)

pote_git_repo: "gitea@10.0.30.169:ilia/POTE.git"
pote_git_branch: "main"
vault_smtp_password: "{{ vault_smtp_password }}"

Essential (for production)

# Paths
pote_install_path: "/home/poteapp/pote"
appuser_name: "poteapp"

# Database
db_name: "potedb"
db_user: "poteuser"
db_password: "{{ vault_db_password_prod }}"
db_host: "localhost"
db_port: 5432

# Email
smtp_host: "mail.levkin.ca"
smtp_port: 587
smtp_user: "test@levkin.ca"
smtp_password: "{{ vault_smtp_password }}"
from_email: "test@levkin.ca"
report_recipients: "your@email.com"

# Monitoring
market_tickers: "NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX"
alert_severity: 5

# Automation
pote_enable_cron: true
pote_daily_report_time: "0 6"    # 6:00 AM
pote_weekly_report_time: "0 8"   # 8:00 AM Sunday

# Logging
log_level: "INFO"
log_file: "/home/poteapp/logs/pote.log"

---

🚨 Common Issues & Solutions

Issue: ModuleNotFoundError: No module named 'pote'

Solution: Run pip install -e . from project root

Issue: alembic.util.exc.CommandError: Can't locate revision identified by 'head'

Solution: Database is not initialized. Run alembic upgrade head

Issue: psycopg2.OperationalError: could not connect to server

Solution: Check PostgreSQL is running and DATABASE_URL is correct

Issue: Cron jobs not running

Solution:

• Check crontab: crontab -l
• Check logs: tail -f ~/logs/daily_run.log
• Verify script permissions: ls -la scripts/*.sh

Issue: Email not sending

Solution:

• Test SMTP: python scripts/send_daily_report.py --test-smtp
• Check .env has correct SMTP settings
• Verify SMTP_PASSWORD is correct

---

📞 Support Commands

# Check Python version
python3.11 --version

# Check pip packages
pip list | grep -E "(alembic|sqlalchemy|pydantic)"

# Check database
psql -U poteuser -d potedb -c "\dt"

# Check logs
tail -f ~/logs/daily_run.log
tail -f ~/logs/pote.log

# Check cron
crontab -l
grep CRON /var/log/syslog | tail -20

# Manual test run
cd ~/pote
source venv/bin/activate
./scripts/automated_daily_run.sh

---

Last Updated: December 2025
POTE Version: 0.1.0
Python Required: 3.11+
PostgreSQL Required: 15+