POTE/docs/08_proxmox_deployment.md

# Proxmox Deployment Guide

## Why Proxmox is Perfect for POTE

✅ **Full control** - Your hardware, your rules  
✅ **No monthly costs** - Just electricity  
✅ **Isolated VMs/LXC** - Clean environments  
✅ **Snapshots** - Easy rollback if needed  
✅ **Resource efficient** - Run alongside other services  

---

## Deployment Options on Proxmox

### Option 1: LXC Container (Recommended) ⭐

**Pros**: Lightweight, fast, efficient resource usage  
**Cons**: Linux only (fine for POTE)

### Option 2: VM with Docker

**Pros**: Full isolation, can run any OS  
**Cons**: More resource overhead

### Option 3: VM without Docker

**Pros**: Traditional setup, maximum control  
**Cons**: Manual dependency management

---

## Quick Start: LXC Container (Easiest)

### 1. Create LXC Container

```bash
# In Proxmox web UI or via CLI:

# Create Ubuntu 22.04 LXC container
pct create 100 local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
  --hostname pote \
  --memory 2048 \
  --cores 2 \
  --rootfs local-lvm:8 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --unprivileged 1 \
  --features nesting=1

# Start container
pct start 100

# Enter container
pct enter 100
```

Or via Web UI:
1. Create CT → Ubuntu 22.04
2. Hostname: `pote`
3. Memory: 2GB
4. Cores: 2
5. Disk: 8GB
6. Network: Bridge, DHCP

### 2. Install Dependencies

```bash
# Inside the container
apt update && apt upgrade -y

# Install Python 3.11, PostgreSQL, Git
apt install -y python3.11 python3.11-venv python3-pip \
  postgresql postgresql-contrib git curl

# Install build tools (for some Python packages)
apt install -y build-essential libpq-dev
```

### 3. Setup PostgreSQL

```bash
# Switch to postgres user
sudo -u postgres psql

# Create database and user
CREATE DATABASE pote;
CREATE USER poteuser WITH PASSWORD 'your_secure_password';
GRANT ALL PRIVILEGES ON DATABASE pote TO poteuser;
ALTER DATABASE pote OWNER TO poteuser;
\q
```

### 4. Clone and Install POTE

```bash
# Create app user (optional but recommended)
useradd -m -s /bin/bash poteapp
su - poteapp

# Clone repo
git clone https://github.com/your-username/pote.git
cd pote

# Create virtual environment
python3.11 -m venv venv
source venv/bin/activate

# Install dependencies
pip install --upgrade pip
pip install -e .
```

### 5. Configure Environment

```bash
# Create .env file
cat > .env << EOF
DATABASE_URL=postgresql://poteuser:your_secure_password@localhost:5432/pote
QUIVERQUANT_API_KEY=
FMP_API_KEY=
LOG_LEVEL=INFO
EOF

chmod 600 .env
```

### 6. Run Migrations

```bash
source venv/bin/activate
alembic upgrade head
```

### 7. Test Ingestion

```bash
# Test with fixtures (offline)
python scripts/ingest_from_fixtures.py

# Enrich securities
python scripts/enrich_securities.py

# Test with real data (if internet available)
python scripts/fetch_congressional_trades.py --days 7
```

### 8. Setup Cron Jobs

```bash
# Edit crontab
crontab -e

# Add these lines:
# Fetch trades daily at 6 AM
0 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/fetch_congressional_trades.py --days 7 >> /home/poteapp/logs/trades.log 2>&1

# Enrich securities daily at 6:15 AM
15 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/enrich_securities.py >> /home/poteapp/logs/enrich.log 2>&1

# Update prices daily at 6:30 AM (when built)
30 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/update_all_prices.py >> /home/poteapp/logs/prices.log 2>&1
```

### 9. Setup Logging

```bash
# Create logs directory
mkdir -p /home/poteapp/logs

# Rotate logs (optional)
cat > /etc/logrotate.d/pote << EOF
/home/poteapp/logs/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
}
EOF
```

---

## Option 2: VM with Docker (More Isolated)

### 1. Create VM

Via Proxmox Web UI:
1. Create VM
2. OS: Ubuntu Server 22.04
3. Memory: 4GB
4. Cores: 2
5. Disk: 20GB
6. Network: Bridge

### 2. Install Docker

```bash
# SSH into VM
ssh user@vm-ip

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

# Add user to docker group
sudo usermod -aG docker $USER
newgrp docker

# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```

### 3. Clone and Deploy

```bash
git clone https://github.com/your-username/pote.git
cd pote

# Create .env
cat > .env << EOF
POSTGRES_PASSWORD=your_secure_password
DATABASE_URL=postgresql://poteuser:your_secure_password@db:5432/pote
QUIVERQUANT_API_KEY=
FMP_API_KEY=
EOF

# Start services
docker-compose up -d

# Check logs
docker-compose logs -f

# Run migrations
docker-compose exec pote alembic upgrade head

# Test ingestion
docker-compose exec pote python scripts/ingest_from_fixtures.py
```

### 4. Setup Auto-start

```bash
# Enable Docker service
sudo systemctl enable docker

# Docker Compose auto-start
sudo curl -L https://raw.githubusercontent.com/docker/compose/master/contrib/systemd/docker-compose.service -o /etc/systemd/system/docker-compose@.service

# Enable for your project
sudo systemctl enable docker-compose@pote
```

---

## Proxmox-Specific Tips

### 1. Backups

```bash
# In Proxmox host, backup the container/VM
vzdump 100 --mode snapshot --storage local

# Or via Web UI: Datacenter → Backup → Add
# Schedule: Daily, Keep: 7 days
```

### 2. Snapshots

```bash
# Before major changes, take snapshot
pct snapshot 100 before-upgrade

# Rollback if needed
pct rollback 100 before-upgrade

# Or via Web UI: Container → Snapshots
```

### 3. Resource Monitoring

```bash
# Monitor container resources
pct status 100
pct exec 100 -- df -h
pct exec 100 -- free -h

# Check PostgreSQL size
pct exec 100 -- sudo -u postgres psql -c "SELECT pg_size_pretty(pg_database_size('pote'));"
```

### 4. Networking

**Static IP (Recommended for services)**:
```bash
# Edit container config on Proxmox host
nano /etc/pve/lxc/100.conf

# Change network config
net0: name=eth0,bridge=vmbr0,ip=192.168.1.50/24,gw=192.168.1.1

# Restart container
pct restart 100
```

**Port Forwarding** (if needed for API):
```bash
# On Proxmox host, forward port 8000 → container
iptables -t nat -A PREROUTING -p tcp --dport 8000 -j DNAT --to 192.168.1.50:8000
iptables -t nat -A POSTROUTING -j MASQUERADE

# Make persistent
apt install iptables-persistent
netfilter-persistent save
```

### 5. Security

```bash
# Inside container, setup firewall
apt install ufw

# Allow SSH
ufw allow 22/tcp

# Allow PostgreSQL (if remote access needed)
ufw allow from 192.168.1.0/24 to any port 5432

# Enable firewall
ufw enable
```

### 6. Performance Tuning

**PostgreSQL** (for LXC with 2GB RAM):
```bash
# Edit postgresql.conf
sudo nano /etc/postgresql/14/main/postgresql.conf

# Optimize for 2GB RAM
shared_buffers = 512MB
effective_cache_size = 1536MB
maintenance_work_mem = 128MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 2621kB
min_wal_size = 1GB
max_wal_size = 4GB

# Restart PostgreSQL
sudo systemctl restart postgresql
```

---

## Resource Requirements

### Minimum (Development/Testing)
- **Memory**: 1GB
- **Cores**: 1
- **Disk**: 5GB
- **Network**: Bridged

### Recommended (Production)
- **Memory**: 2-4GB
- **Cores**: 2
- **Disk**: 20GB (with room for logs/backups)
- **Network**: Bridged with static IP

### With Dashboard (Phase 3)
- **Memory**: 4GB
- **Cores**: 2-4
- **Disk**: 20GB

---

## Monitoring & Maintenance

### 1. Check Service Health

```bash
# Database connection
pct exec 100 -- sudo -u poteapp bash -c 'cd /home/poteapp/pote && source venv/bin/activate && python -c "from pote.db import SessionLocal; from sqlalchemy import text; s = SessionLocal(); s.execute(text(\"SELECT 1\")); print(\"DB OK\")"'

# Check last ingestion
pct exec 100 -- sudo -u postgres psql pote -c "SELECT COUNT(*), MAX(created_at) FROM trades;"

# Check disk usage
pct exec 100 -- df -h

# Check logs
pct exec 100 -- tail -f /home/poteapp/logs/trades.log
```

### 2. Database Maintenance

```bash
# Backup database
pct exec 100 -- sudo -u postgres pg_dump pote > pote_backup_$(date +%Y%m%d).sql

# Vacuum (clean up)
pct exec 100 -- sudo -u postgres psql pote -c "VACUUM ANALYZE;"

# Check database size
pct exec 100 -- sudo -u postgres psql -c "SELECT pg_size_pretty(pg_database_size('pote'));"
```

### 3. Update POTE

```bash
# Enter container
pct enter 100
su - poteapp
cd pote

# Pull latest code
git pull

# Update dependencies
source venv/bin/activate
pip install --upgrade -e .

# Run migrations
alembic upgrade head

# Test
python scripts/ingest_from_fixtures.py
```

---

## Troubleshooting

### Container won't start
```bash
# Check logs
pct status 100
journalctl -u pve-container@100

# Try start with debug
pct start 100 --debug
```

### PostgreSQL connection issues
```bash
# Check if PostgreSQL is running
pct exec 100 -- systemctl status postgresql

# Check connections
pct exec 100 -- sudo -u postgres psql -c "SELECT * FROM pg_stat_activity;"

# Reset password if needed
pct exec 100 -- sudo -u postgres psql -c "ALTER USER poteuser PASSWORD 'new_password';"
```

### Out of disk space
```bash
# Check usage
pct exec 100 -- df -h

# Clean logs
pct exec 100 -- find /home/poteapp/logs -name "*.log" -mtime +7 -delete

# Clean apt cache
pct exec 100 -- apt clean

# Resize container disk (on Proxmox host)
lvresize -L +5G /dev/pve/vm-100-disk-0
pct resize 100 rootfs +5G
```

### Python package issues
```bash
# Reinstall in venv
pct exec 100 -- sudo -u poteapp bash -c 'cd /home/poteapp/pote && rm -rf venv && python3.11 -m venv venv && source venv/bin/activate && pip install -e .'
```

---

## Cost Analysis

### Proxmox LXC (Your Setup)
- **Hardware**: Already owned
- **Power**: ~$5-15/mo (depends on your setup)
- **Internet**: Existing connection
- **Total**: **~$10/mo** (just power)

vs.

- **VPS**: $10-20/mo
- **Cloud**: $20-50/mo
- **Managed**: $50-100/mo

**Your Proxmox = 50-90% cost savings!**

---

## Next Steps

1. ✅ Create LXC container
2. ✅ Install dependencies
3. ✅ Setup PostgreSQL
4. ✅ Deploy POTE
5. ✅ Configure cron jobs
6. ✅ Setup backups
7. ⏭️ Build Phase 2 (Analytics)
8. ⏭️ Add FastAPI dashboard (optional)

---

## Example: Complete Setup Script

Save this as `proxmox_setup.sh` in your container:

```bash
#!/bin/bash
set -e

echo "=== POTE Proxmox Setup ==="

# Update system
echo "Updating system..."
apt update && apt upgrade -y

# Install dependencies
echo "Installing dependencies..."
apt install -y python3.11 python3.11-venv python3-pip \
  postgresql postgresql-contrib git curl \
  build-essential libpq-dev

# Setup PostgreSQL
echo "Setting up PostgreSQL..."
sudo -u postgres psql << EOF
CREATE DATABASE pote;
CREATE USER poteuser WITH PASSWORD 'changeme123';
GRANT ALL PRIVILEGES ON DATABASE pote TO poteuser;
ALTER DATABASE pote OWNER TO poteuser;
EOF

# Create app user
echo "Creating app user..."
useradd -m -s /bin/bash poteapp || true

# Clone repo
echo "Cloning POTE..."
sudo -u poteapp git clone https://github.com/your-username/pote.git /home/poteapp/pote || true

# Setup Python environment
echo "Setting up Python environment..."
sudo -u poteapp bash << 'EOF'
cd /home/poteapp/pote
python3.11 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -e .
EOF

# Create .env
echo "Creating .env..."
sudo -u poteapp bash << 'EOF'
cat > /home/poteapp/pote/.env << ENVEOF
DATABASE_URL=postgresql://poteuser:changeme123@localhost:5432/pote
QUIVERQUANT_API_KEY=
FMP_API_KEY=
LOG_LEVEL=INFO
ENVEOF
chmod 600 /home/poteapp/pote/.env
EOF

# Run migrations
echo "Running migrations..."
sudo -u poteapp bash << 'EOF'
cd /home/poteapp/pote
source venv/bin/activate
alembic upgrade head
EOF

# Create logs directory
sudo -u poteapp mkdir -p /home/poteapp/logs

echo ""
echo "✅ Setup complete!"
echo ""
echo "Next steps:"
echo "1. su - poteapp"
echo "2. cd pote && source venv/bin/activate"
echo "3. python scripts/ingest_from_fixtures.py"
echo "4. Setup cron jobs (see docs/08_proxmox_deployment.md)"
```

Run it:
```bash
chmod +x proxmox_setup.sh
./proxmox_setup.sh
```

---

**Your Proxmox setup gives you enterprise-grade infrastructure at hobby costs!** 🚀