POTE/docs/08_proxmox_deployment.md

Proxmox Deployment Guide

Why Proxmox is Perfect for POTE

✅ Full control - Your hardware, your rules
✅ No monthly costs - Just electricity
✅ Isolated VMs/LXC - Clean environments
✅ Snapshots - Easy rollback if needed
✅ Resource efficient - Run alongside other services

---

Deployment Options on Proxmox

Option 1: LXC Container (Recommended) ⭐

Pros: Lightweight, fast, efficient resource usage
Cons: Linux only (fine for POTE)

Option 2: VM with Docker

Pros: Full isolation, can run any OS
Cons: More resource overhead

Option 3: VM without Docker

Pros: Traditional setup, maximum control
Cons: Manual dependency management

---

Quick Start: LXC Container (Easiest)

1. Create LXC Container

# In Proxmox web UI or via CLI:

# Create Ubuntu 22.04 LXC container
pct create 100 local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
  --hostname pote \
  --memory 2048 \
  --cores 2 \
  --rootfs local-lvm:8 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --unprivileged 1 \
  --features nesting=1

# Start container
pct start 100

# Enter container
pct enter 100

Or via Web UI:

1. Create CT → Ubuntu 22.04
2. Hostname: pote
3. Memory: 2GB
4. Cores: 2
5. Disk: 8GB
6. Network: Bridge, DHCP

2. Install Dependencies

# Inside the container
apt update && apt upgrade -y

# Install Python 3.11, PostgreSQL, Git
apt install -y python3.11 python3.11-venv python3-pip \
  postgresql postgresql-contrib git curl

# Install build tools (for some Python packages)
apt install -y build-essential libpq-dev

3. Setup PostgreSQL

# Switch to postgres user
sudo -u postgres psql

# Create database and user
CREATE DATABASE pote;
CREATE USER poteuser WITH PASSWORD 'your_secure_password';
GRANT ALL PRIVILEGES ON DATABASE pote TO poteuser;
ALTER DATABASE pote OWNER TO poteuser;
\q

4. Clone and Install POTE

# Create app user (optional but recommended)
useradd -m -s /bin/bash poteapp
su - poteapp

# Clone repo
git clone https://github.com/your-username/pote.git
cd pote

# Create virtual environment
python3.11 -m venv venv
source venv/bin/activate

# Install dependencies
pip install --upgrade pip
pip install -e .

5. Configure Environment

# Create .env file
cat > .env << EOF
DATABASE_URL=postgresql://poteuser:your_secure_password@localhost:5432/pote
QUIVERQUANT_API_KEY=
FMP_API_KEY=
LOG_LEVEL=INFO
EOF

chmod 600 .env

6. Run Migrations

source venv/bin/activate
alembic upgrade head

7. Test Ingestion

# Test with fixtures (offline)
python scripts/ingest_from_fixtures.py

# Enrich securities
python scripts/enrich_securities.py

# Test with real data (if internet available)
python scripts/fetch_congressional_trades.py --days 7

8. Setup Cron Jobs

# Edit crontab
crontab -e

# Add these lines:
# Fetch trades daily at 6 AM
0 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/fetch_congressional_trades.py --days 7 >> /home/poteapp/logs/trades.log 2>&1

# Enrich securities daily at 6:15 AM
15 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/enrich_securities.py >> /home/poteapp/logs/enrich.log 2>&1

# Update prices daily at 6:30 AM (when built)
30 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/update_all_prices.py >> /home/poteapp/logs/prices.log 2>&1

9. Setup Logging

# Create logs directory
mkdir -p /home/poteapp/logs

# Rotate logs (optional)
cat > /etc/logrotate.d/pote << EOF
/home/poteapp/logs/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
}
EOF

---

Option 2: VM with Docker (More Isolated)

1. Create VM

Via Proxmox Web UI:

1. Create VM
2. OS: Ubuntu Server 22.04
3. Memory: 4GB
4. Cores: 2
5. Disk: 20GB
6. Network: Bridge

2. Install Docker

# SSH into VM
ssh user@vm-ip

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

# Add user to docker group
sudo usermod -aG docker $USER
newgrp docker

# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

3. Clone and Deploy

git clone https://github.com/your-username/pote.git
cd pote

# Create .env
cat > .env << EOF
POSTGRES_PASSWORD=your_secure_password
DATABASE_URL=postgresql://poteuser:your_secure_password@db:5432/pote
QUIVERQUANT_API_KEY=
FMP_API_KEY=
EOF

# Start services
docker-compose up -d

# Check logs
docker-compose logs -f

# Run migrations
docker-compose exec pote alembic upgrade head

# Test ingestion
docker-compose exec pote python scripts/ingest_from_fixtures.py

4. Setup Auto-start

# Enable Docker service
sudo systemctl enable docker

# Docker Compose auto-start
sudo curl -L https://raw.githubusercontent.com/docker/compose/master/contrib/systemd/docker-compose.service -o /etc/systemd/system/docker-compose@.service

# Enable for your project
sudo systemctl enable docker-compose@pote

---

Proxmox-Specific Tips

1. Backups

# In Proxmox host, backup the container/VM
vzdump 100 --mode snapshot --storage local

# Or via Web UI: Datacenter → Backup → Add
# Schedule: Daily, Keep: 7 days

2. Snapshots

# Before major changes, take snapshot
pct snapshot 100 before-upgrade

# Rollback if needed
pct rollback 100 before-upgrade

# Or via Web UI: Container → Snapshots

3. Resource Monitoring

# Monitor container resources
pct status 100
pct exec 100 -- df -h
pct exec 100 -- free -h

# Check PostgreSQL size
pct exec 100 -- sudo -u postgres psql -c "SELECT pg_size_pretty(pg_database_size('pote'));"

4. Networking

Static IP (Recommended for services):

# Edit container config on Proxmox host
nano /etc/pve/lxc/100.conf

# Change network config
net0: name=eth0,bridge=vmbr0,ip=192.168.1.50/24,gw=192.168.1.1

# Restart container
pct restart 100

Port Forwarding (if needed for API):

# On Proxmox host, forward port 8000 → container
iptables -t nat -A PREROUTING -p tcp --dport 8000 -j DNAT --to 192.168.1.50:8000
iptables -t nat -A POSTROUTING -j MASQUERADE

# Make persistent
apt install iptables-persistent
netfilter-persistent save

5. Security

# Inside container, setup firewall
apt install ufw

# Allow SSH
ufw allow 22/tcp

# Allow PostgreSQL (if remote access needed)
ufw allow from 192.168.1.0/24 to any port 5432

# Enable firewall
ufw enable

6. Performance Tuning

PostgreSQL (for LXC with 2GB RAM):

# Edit postgresql.conf
sudo nano /etc/postgresql/14/main/postgresql.conf

# Optimize for 2GB RAM
shared_buffers = 512MB
effective_cache_size = 1536MB
maintenance_work_mem = 128MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 2621kB
min_wal_size = 1GB
max_wal_size = 4GB

# Restart PostgreSQL
sudo systemctl restart postgresql

---

Resource Requirements

Minimum (Development/Testing)

• Memory: 1GB
• Cores: 1
• Disk: 5GB
• Network: Bridged

Recommended (Production)

• Memory: 2-4GB
• Cores: 2
• Disk: 20GB (with room for logs/backups)
• Network: Bridged with static IP

With Dashboard (Phase 3)

• Memory: 4GB
• Cores: 2-4
• Disk: 20GB

---

Monitoring & Maintenance

1. Check Service Health

# Database connection
pct exec 100 -- sudo -u poteapp bash -c 'cd /home/poteapp/pote && source venv/bin/activate && python -c "from pote.db import SessionLocal; from sqlalchemy import text; s = SessionLocal(); s.execute(text(\"SELECT 1\")); print(\"DB OK\")"'

# Check last ingestion
pct exec 100 -- sudo -u postgres psql pote -c "SELECT COUNT(*), MAX(created_at) FROM trades;"

# Check disk usage
pct exec 100 -- df -h

# Check logs
pct exec 100 -- tail -f /home/poteapp/logs/trades.log

2. Database Maintenance

# Backup database
pct exec 100 -- sudo -u postgres pg_dump pote > pote_backup_$(date +%Y%m%d).sql

# Vacuum (clean up)
pct exec 100 -- sudo -u postgres psql pote -c "VACUUM ANALYZE;"

# Check database size
pct exec 100 -- sudo -u postgres psql -c "SELECT pg_size_pretty(pg_database_size('pote'));"

3. Update POTE

# Enter container
pct enter 100
su - poteapp
cd pote

# Pull latest code
git pull

# Update dependencies
source venv/bin/activate
pip install --upgrade -e .

# Run migrations
alembic upgrade head

# Test
python scripts/ingest_from_fixtures.py

---

Troubleshooting

Container won't start

# Check logs
pct status 100
journalctl -u pve-container@100

# Try start with debug
pct start 100 --debug

PostgreSQL connection issues

# Check if PostgreSQL is running
pct exec 100 -- systemctl status postgresql

# Check connections
pct exec 100 -- sudo -u postgres psql -c "SELECT * FROM pg_stat_activity;"

# Reset password if needed
pct exec 100 -- sudo -u postgres psql -c "ALTER USER poteuser PASSWORD 'new_password';"

Out of disk space

# Check usage
pct exec 100 -- df -h

# Clean logs
pct exec 100 -- find /home/poteapp/logs -name "*.log" -mtime +7 -delete

# Clean apt cache
pct exec 100 -- apt clean

# Resize container disk (on Proxmox host)
lvresize -L +5G /dev/pve/vm-100-disk-0
pct resize 100 rootfs +5G

Python package issues

# Reinstall in venv
pct exec 100 -- sudo -u poteapp bash -c 'cd /home/poteapp/pote && rm -rf venv && python3.11 -m venv venv && source venv/bin/activate && pip install -e .'

---

Cost Analysis

Proxmox LXC (Your Setup)

• Hardware: Already owned
• Power: ~$5-15/mo (depends on your setup)
• Internet: Existing connection
• Total: ~$10/mo (just power)

vs.

• VPS: $10-20/mo
• Cloud: $20-50/mo
• Managed: $50-100/mo

Your Proxmox = 50-90% cost savings!

---

Next Steps

1. ✅ Create LXC container
2. ✅ Install dependencies
3. ✅ Setup PostgreSQL
4. ✅ Deploy POTE
5. ✅ Configure cron jobs
6. ✅ Setup backups
7. ⏭️ Build Phase 2 (Analytics)
8. ⏭️ Add FastAPI dashboard (optional)

---

Example: Complete Setup Script

Save this as proxmox_setup.sh in your container:

#!/bin/bash
set -e

echo "=== POTE Proxmox Setup ==="

# Update system
echo "Updating system..."
apt update && apt upgrade -y

# Install dependencies
echo "Installing dependencies..."
apt install -y python3.11 python3.11-venv python3-pip \
  postgresql postgresql-contrib git curl \
  build-essential libpq-dev

# Setup PostgreSQL
echo "Setting up PostgreSQL..."
sudo -u postgres psql << EOF
CREATE DATABASE pote;
CREATE USER poteuser WITH PASSWORD 'changeme123';
GRANT ALL PRIVILEGES ON DATABASE pote TO poteuser;
ALTER DATABASE pote OWNER TO poteuser;
EOF

# Create app user
echo "Creating app user..."
useradd -m -s /bin/bash poteapp || true

# Clone repo
echo "Cloning POTE..."
sudo -u poteapp git clone https://github.com/your-username/pote.git /home/poteapp/pote || true

# Setup Python environment
echo "Setting up Python environment..."
sudo -u poteapp bash << 'EOF'
cd /home/poteapp/pote
python3.11 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -e .
EOF

# Create .env
echo "Creating .env..."
sudo -u poteapp bash << 'EOF'
cat > /home/poteapp/pote/.env << ENVEOF
DATABASE_URL=postgresql://poteuser:changeme123@localhost:5432/pote
QUIVERQUANT_API_KEY=
FMP_API_KEY=
LOG_LEVEL=INFO
ENVEOF
chmod 600 /home/poteapp/pote/.env
EOF

# Run migrations
echo "Running migrations..."
sudo -u poteapp bash << 'EOF'
cd /home/poteapp/pote
source venv/bin/activate
alembic upgrade head
EOF

# Create logs directory
sudo -u poteapp mkdir -p /home/poteapp/logs

echo ""
echo "✅ Setup complete!"
echo ""
echo "Next steps:"
echo "1. su - poteapp"
echo "2. cd pote && source venv/bin/activate"
echo "3. python scripts/ingest_from_fixtures.py"
echo "4. Setup cron jobs (see docs/08_proxmox_deployment.md)"

Run it:

chmod +x proxmox_setup.sh
./proxmox_setup.sh

---

Your Proxmox setup gives you enterprise-grade infrastructure at hobby costs! 🚀