diff --git a/ANSIBLE_TECHNICAL_REFERENCE.md b/ANSIBLE_TECHNICAL_REFERENCE.md new file mode 100644 index 0000000..dd98e3e --- /dev/null +++ b/ANSIBLE_TECHNICAL_REFERENCE.md @@ -0,0 +1,598 @@ +# POTE Ansible Role - Technical Reference + +**Exact commands, paths, and procedures for the Ansible `roles/pote` role.** + +--- + +## ๐Ÿ“ฆ Installation Method + +### โœ… Uses `pyproject.toml` (NOT requirements.txt) + +```bash +# Install in editable mode with all dependencies +pip install -e . + +# For development (includes pytest, ruff, black, mypy) +pip install -e ".[dev]" +``` + +**File:** `pyproject.toml` + +**Key dependencies:** +```toml +dependencies = [ + "sqlalchemy>=2.0", + "alembic>=1.12", + "pydantic>=2.0", + "pydantic-settings>=2.0", + "python-dotenv>=1.0", + "requests>=2.31", + "pandas>=2.0", + "numpy>=1.24", + "yfinance>=0.2", + "psycopg2-binary>=2.9", +] +``` + +**Python version:** `>=3.11` (required) + +--- + +## ๐Ÿ—„๏ธ Database Migrations + +### Command + +```bash +alembic upgrade head +``` + +**Where to run:** From the project root directory (where `alembic.ini` is located) + +**Configuration:** +- Migration scripts: `alembic/versions/` +- Config file: `alembic.ini` (uses `.env` for DATABASE_URL) +- Models: `src/pote/db/models.py` + +**Important:** +- Alembic reads `DATABASE_URL` from `.env` file via `src/pote/config.py` +- Must run AFTER `.env` file is created +- Must run AFTER database and user are created in PostgreSQL + +**Idempotency:** โœ… Safe to re-run (Alembic tracks applied migrations) + +--- + +## ๐Ÿ“ Actual Script Paths + +### All scripts are in `scripts/` directory: + +#### Data Ingestion +```bash +scripts/fetch_congressional_trades.py # Fetch from House Stock Watcher API +scripts/enrich_securities.py # Get company names, sectors from yfinance +scripts/fetch_sample_prices.py # Fetch price data for securities +scripts/ingest_from_fixtures.py # Offline mode - load from fixtures +scripts/add_custom_trades.py # Manual trade entry +scripts/scrape_alternative_sources.py # Import from CSV +scripts/fetch_congress_members.py # Fetch list of current members +``` + +#### Analytics +```bash +scripts/analyze_official.py # Analyze specific official's performance +scripts/calculate_all_returns.py # System-wide return calculations +scripts/generate_trading_report.py # Generate trading activity report +``` + +#### Monitoring (Phase 1-3) +```bash +scripts/monitor_market.py # Real-time market monitoring +scripts/analyze_disclosure_timing.py # Disclosure timing correlation +scripts/generate_pattern_report.py # Pattern detection & rankings +``` + +#### Reporting +```bash +scripts/send_daily_report.py # Send daily email report +scripts/send_weekly_report.py # Send weekly email report +scripts/health_check.py # System health check +``` + +#### Automation (Shell Scripts) +```bash +scripts/automated_daily_run.sh # โญ Daily cron job +scripts/automated_weekly_run.sh # โญ Weekly cron job +scripts/setup_cron.sh # Interactive cron setup +scripts/setup_automation.sh # Alternative automation setup +scripts/daily_fetch.sh # Daily data fetch +scripts/daily_update.sh # Daily update routine +scripts/pre_market_close_update.sh # Pre-market close routine +``` + +#### Deployment +```bash +scripts/proxmox_setup.sh # Bootstrap script for Proxmox LXC +``` + +--- + +## ๐Ÿ”ง Complete Deployment Sequence + +### Phase 1: System Preparation + +```bash +# 1. Create application user +useradd -m -s /bin/bash poteapp + +# 2. Install system dependencies +apt-get update +apt-get install -y \ + python3.11 \ + python3.11-venv \ + python3.11-dev \ + python3-pip \ + postgresql \ + postgresql-contrib \ + libpq-dev \ + build-essential \ + git \ + curl \ + htop + +# 3. Configure PostgreSQL +sudo -u postgres psql << EOF +CREATE USER poteuser WITH PASSWORD 'your_password'; +CREATE DATABASE potedb OWNER poteuser; +GRANT ALL PRIVILEGES ON DATABASE potedb TO poteuser; +EOF + +# 4. Configure PostgreSQL for remote access (if needed) +# Edit /etc/postgresql/15/main/postgresql.conf: +# listen_addresses = '*' +# Edit /etc/postgresql/15/main/pg_hba.conf: +# host all all 0.0.0.0/0 scram-sha-256 +# Restart: systemctl restart postgresql +``` + +### Phase 2: Application Deployment + +```bash +# 1. Switch to application user +su - poteapp + +# 2. Clone repository (requires SSH key setup) +git clone gitea@10.0.30.169:ilia/POTE.git pote +cd pote + +# 3. Checkout appropriate branch +git checkout main # or 'dev', 'qa' depending on environment + +# 4. Create virtual environment +python3.11 -m venv venv + +# 5. Activate virtual environment +source venv/bin/activate + +# 6. Upgrade pip +pip install --upgrade pip + +# 7. Install application +pip install -e . + +# 8. Create .env file (from template or Ansible) +cat > .env << 'EOF' +DATABASE_URL=postgresql://poteuser:your_password@localhost:5432/potedb +SMTP_HOST=mail.levkin.ca +SMTP_PORT=587 +SMTP_USER=test@levkin.ca +SMTP_PASSWORD=your_smtp_password +FROM_EMAIL=test@levkin.ca +REPORT_RECIPIENTS=your@email.com +MARKET_MONITOR_TICKERS=NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX +ALERT_MIN_SEVERITY=5 +LOG_LEVEL=INFO +LOG_FILE=/home/poteapp/logs/pote.log +EOF + +# 9. Secure .env file +chmod 600 .env + +# 10. Run database migrations +alembic upgrade head +``` + +### Phase 3: Automation Setup + +```bash +# 1. Create log directory +mkdir -p /home/poteapp/logs + +# 2. Make scripts executable +chmod +x scripts/*.sh + +# 3. Set up cron jobs +crontab -e +# Add these lines: +# Daily report at 6:00 AM +0 6 * * * /home/poteapp/pote/scripts/automated_daily_run.sh >> /home/poteapp/logs/daily_run.log 2>&1 +# Weekly report at 8:00 AM on Sunday +0 8 * * 0 /home/poteapp/pote/scripts/automated_weekly_run.sh >> /home/poteapp/logs/weekly_run.log 2>&1 +``` + +### Phase 4: Verification + +```bash +# 1. Run health check +cd /home/poteapp/pote +source venv/bin/activate +python scripts/health_check.py + +# Expected output: +# โœ… Database connection: OK +# โœ… Database has X officials, Y trades, Z securities +# โœ… Latest trade: YYYY-MM-DD +# โœ… Price data: X securities with prices + +# 2. Test data ingestion (optional) +python scripts/ingest_from_fixtures.py # Offline test +# OR +python scripts/fetch_congressional_trades.py # Live API test + +# 3. Test email (optional) +python scripts/send_daily_report.py --to test@example.com --test-smtp + +# 4. Verify cron jobs +crontab -l | grep POTE +``` + +--- + +## ๐Ÿ“‹ Ansible Task Checklist + +### Your `roles/pote/tasks/main.yml` should include: + +```yaml +--- +# 1. System preparation +- name: Create application user + user: + name: "{{ appuser_name }}" + shell: "{{ appuser_shell }}" + create_home: yes + +- name: Install system dependencies + apt: + name: + - python3.11 + - python3.11-venv + - python3.11-dev + - python3-pip + - postgresql + - postgresql-contrib + - libpq-dev + - build-essential + - git + - curl + - htop + state: present + update_cache: yes + +# 2. PostgreSQL setup +- name: Create PostgreSQL user + postgresql_user: + name: "{{ db_user }}" + password: "{{ db_password }}" + state: present + become_user: postgres + +- name: Create PostgreSQL database + postgresql_db: + name: "{{ db_name }}" + owner: "{{ db_user }}" + state: present + become_user: postgres + +# 3. SSH key setup (for git clone) +- name: Deploy SSH private key for git + copy: + content: "{{ pote_git_ssh_key }}" + dest: "/home/{{ appuser_name }}/.ssh/id_rsa" + owner: "{{ appuser_name }}" + mode: '0600' + +- name: Add Gitea to known_hosts + known_hosts: + name: "{{ pote_git_host }}" + key: "{{ lookup('pipe', 'ssh-keyscan ' + pote_git_host) }}" + state: present + become_user: "{{ appuser_name }}" + +# 4. Clone repository +- name: Clone POTE repository + git: + repo: "{{ pote_git_repo }}" + dest: "{{ pote_install_path }}" + version: "{{ pote_git_branch }}" + accept_hostkey: yes + become_user: "{{ appuser_name }}" + +# 5. Python environment +- name: Create virtual environment + command: python3.11 -m venv {{ pote_install_path }}/venv + args: + creates: "{{ pote_install_path }}/venv" + become_user: "{{ appuser_name }}" + +- name: Upgrade pip + pip: + name: pip + state: latest + virtualenv: "{{ pote_install_path }}/venv" + become_user: "{{ appuser_name }}" + +- name: Install POTE application + pip: + name: "{{ pote_install_path }}" + editable: yes + virtualenv: "{{ pote_install_path }}/venv" + become_user: "{{ appuser_name }}" + +# 6. Configuration +- name: Deploy .env file + template: + src: env.j2 + dest: "{{ pote_install_path }}/.env" + owner: "{{ appuser_name }}" + mode: '0600' + become_user: "{{ appuser_name }}" + +# 7. Database migrations +- name: Run Alembic migrations + command: "{{ pote_install_path }}/venv/bin/alembic upgrade head" + args: + chdir: "{{ pote_install_path }}" + become_user: "{{ appuser_name }}" + +# 8. Log directory +- name: Create log directory + file: + path: "/home/{{ appuser_name }}/logs" + state: directory + owner: "{{ appuser_name }}" + mode: '0755' + +# 9. Make scripts executable +- name: Make shell scripts executable + file: + path: "{{ pote_install_path }}/scripts/{{ item }}" + mode: '0755' + loop: + - automated_daily_run.sh + - automated_weekly_run.sh + - setup_cron.sh + become_user: "{{ appuser_name }}" + +# 10. Cron jobs +- name: Set up daily cron job + cron: + name: "POTE Automated Daily Run" + minute: "{{ pote_daily_report_time.split()[1] }}" + hour: "{{ pote_daily_report_time.split()[0] }}" + job: "{{ pote_install_path }}/scripts/automated_daily_run.sh >> /home/{{ appuser_name }}/logs/daily_run.log 2>&1" + user: "{{ appuser_name }}" + when: pote_enable_cron and pote_daily_report_enabled + +- name: Set up weekly cron job + cron: + name: "POTE Automated Weekly Run" + minute: "{{ pote_weekly_report_time.split()[1] }}" + hour: "{{ pote_weekly_report_time.split()[0] }}" + day: "0" # Sunday + job: "{{ pote_install_path }}/scripts/automated_weekly_run.sh >> /home/{{ appuser_name }}/logs/weekly_run.log 2>&1" + user: "{{ appuser_name }}" + when: pote_enable_cron and pote_weekly_report_enabled +``` + +--- + +## ๐ŸŽจ `.env` Template for Ansible + +### Create `roles/pote/templates/env.j2`: + +```jinja2 +# Database +DATABASE_URL=postgresql://{{ db_user }}:{{ db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }} + +# Email +SMTP_HOST={{ smtp_host }} +SMTP_PORT={{ smtp_port }} +SMTP_USER={{ smtp_user }} +SMTP_PASSWORD={{ smtp_password }} +FROM_EMAIL={{ from_email }} +REPORT_RECIPIENTS={{ report_recipients }} + +# Monitoring +MARKET_MONITOR_TICKERS={{ market_tickers }} +ALERT_MIN_SEVERITY={{ alert_severity }} + +# Logging +LOG_LEVEL={{ log_level }} +LOG_FILE={{ log_file }} + +# Optional API Keys +{% if quiverquant_api_key is defined and quiverquant_api_key %} +QUIVERQUANT_API_KEY={{ quiverquant_api_key }} +{% endif %} +{% if fmp_api_key is defined and fmp_api_key %} +FMP_API_KEY={{ fmp_api_key }} +{% endif %} +``` + +--- + +## ๐Ÿ”„ Update Procedure (Existing Deployment) + +```bash +# 1. Pull latest code +cd /home/poteapp/pote +git pull origin main + +# 2. Activate venv +source venv/bin/activate + +# 3. Update dependencies (if pyproject.toml changed) +pip install -e . + +# 4. Run migrations (if models changed) +alembic upgrade head + +# 5. Restart services (if any) +# (Currently no services - cron jobs will pick up changes) +``` + +--- + +## ๐Ÿงช Testing the Deployment + +### Quick Health Check + +```bash +cd /home/poteapp/pote +source venv/bin/activate +python scripts/health_check.py +``` + +**Exit codes:** +- `0` = Healthy +- Non-zero = Issues found + +### Full Test Sequence + +```bash +# 1. Test database connection +python -c "from pote.db import get_session; next(get_session()); print('โœ… DB OK')" + +# 2. Test offline ingestion +python scripts/ingest_from_fixtures.py + +# 3. Test enrichment +python scripts/enrich_securities.py + +# 4. Test price fetch +python scripts/fetch_sample_prices.py + +# 5. Test email (dry run) +python scripts/send_daily_report.py --to test@example.com --test-smtp + +# 6. Verify cron +crontab -l | grep POTE +``` + +--- + +## ๐Ÿ“Š Key Variables Reference + +### Minimal (3 required) + +```yaml +pote_git_repo: "gitea@10.0.30.169:ilia/POTE.git" +pote_git_branch: "main" +vault_smtp_password: "{{ vault_smtp_password }}" +``` + +### Essential (for production) + +```yaml +# Paths +pote_install_path: "/home/poteapp/pote" +appuser_name: "poteapp" + +# Database +db_name: "potedb" +db_user: "poteuser" +db_password: "{{ vault_db_password_prod }}" +db_host: "localhost" +db_port: 5432 + +# Email +smtp_host: "mail.levkin.ca" +smtp_port: 587 +smtp_user: "test@levkin.ca" +smtp_password: "{{ vault_smtp_password }}" +from_email: "test@levkin.ca" +report_recipients: "your@email.com" + +# Monitoring +market_tickers: "NVDA,TSLA,AAPL,MSFT,GOOGL,META,AMZN,AMD,INTC,NFLX" +alert_severity: 5 + +# Automation +pote_enable_cron: true +pote_daily_report_time: "0 6" # 6:00 AM +pote_weekly_report_time: "0 8" # 8:00 AM Sunday + +# Logging +log_level: "INFO" +log_file: "/home/poteapp/logs/pote.log" +``` + +--- + +## ๐Ÿšจ Common Issues & Solutions + +### Issue: `ModuleNotFoundError: No module named 'pote'` +**Solution:** Run `pip install -e .` from project root + +### Issue: `alembic.util.exc.CommandError: Can't locate revision identified by 'head'` +**Solution:** Database is not initialized. Run `alembic upgrade head` + +### Issue: `psycopg2.OperationalError: could not connect to server` +**Solution:** Check PostgreSQL is running and DATABASE_URL is correct + +### Issue: Cron jobs not running +**Solution:** +- Check crontab: `crontab -l` +- Check logs: `tail -f ~/logs/daily_run.log` +- Verify script permissions: `ls -la scripts/*.sh` + +### Issue: Email not sending +**Solution:** +- Test SMTP: `python scripts/send_daily_report.py --test-smtp` +- Check `.env` has correct SMTP settings +- Verify SMTP_PASSWORD is correct + +--- + +## ๐Ÿ“ž Support Commands + +```bash +# Check Python version +python3.11 --version + +# Check pip packages +pip list | grep -E "(alembic|sqlalchemy|pydantic)" + +# Check database +psql -U poteuser -d potedb -c "\dt" + +# Check logs +tail -f ~/logs/daily_run.log +tail -f ~/logs/pote.log + +# Check cron +crontab -l +grep CRON /var/log/syslog | tail -20 + +# Manual test run +cd ~/pote +source venv/bin/activate +./scripts/automated_daily_run.sh +``` + +--- + +**Last Updated:** December 2025 +**POTE Version:** 0.1.0 +**Python Required:** 3.11+ +**PostgreSQL Required:** 15+ +