ilia/llm_council
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Major UI/UX improvements and production readiness
CI / backend-test (push) Successful in 4m9s
Details
CI / frontend-test (push) Failing after 3m48s
Details
CI / lint-python (push) Successful in 1m41s
Details
CI / secret-scanning (push) Successful in 1m20s
Details
CI / dependency-scan (push) Successful in 10m50s
Details
CI / workflow-summary (push) Successful in 1m11s
Details

Browse Source 
## Features Added

### Document Reference System
- Implemented numbered document references (@1, @2, etc.) with autocomplete dropdown
- Added fuzzy filename matching for @filename references
- Document filtering now prioritizes numeric refs > filename refs > all documents
- Autocomplete dropdown appears when typing @ with keyboard navigation (Up/Down, Enter/Tab, Escape)
- Document numbers displayed in UI for easy reference

### Conversation Management
- Added conversation rename functionality with inline editing
- Implemented conversation search (by title and content)
- Search box always visible, even when no conversations exist
- Export reports now replace @N references with actual filenames

### UI/UX Improvements
- Removed debug toggle button
- Improved text contrast in dark mode (better visibility)
- Made input textarea expand to full available width
- Fixed file text color for better readability
- Enhanced document display with numbered badges

### Configuration & Timeouts
- Made HTTP client timeouts configurable (connect, write, pool)
- Added .env.example with all configuration options
- Updated timeout documentation

### Developer Experience
- Added `make test-setup` target for automated test conversation creation
- Test setup script supports TEST_MESSAGE and TEST_DOCS env vars
- Improved Makefile with dev and test-setup targets

### Documentation
- Updated ARCHITECTURE.md with all new features
- Created comprehensive deployment documentation
- Added GPU VM setup guides
- Removed unnecessary markdown files (CLAUDE.md, CONTRIBUTING.md, header.jpg)
- Organized documentation in docs/ directory

### GPU VM / Ollama (Stability + GPU Offload)
- Updated GPU VM docs to reflect the working systemd environment for remote Ollama
- Standardized remote Ollama port to 11434 (and added /v1/models verification)
- Documented required env for GPU offload on this VM:
  - `OLLAMA_MODELS=/mnt/data/ollama`, `HOME=/mnt/data/ollama/home`
  - `OLLAMA_LLM_LIBRARY=cuda_v12` (not `cuda`)
  - `LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v12`

## Technical Changes

### Backend
- Enhanced `docs_context.py` with reference parsing (numeric and filename)
- Added `update_conversation_title` to storage.py
- New endpoints: PATCH /api/conversations/{id}/title, GET /api/conversations/search
- Improved report generation with filename substitution

### Frontend
- Removed debugMode state and related code
- Added autocomplete dropdown component
- Implemented search functionality in Sidebar
- Enhanced ChatInterface with autocomplete and improved textarea sizing
- Updated CSS for better contrast and responsive design

## Files Changed
- Backend: config.py, council.py, docs_context.py, main.py, storage.py
- Frontend: App.jsx, ChatInterface.jsx, Sidebar.jsx, and related CSS files
- Documentation: README.md, ARCHITECTURE.md, new docs/ directory
- Configuration: .env.example, Makefile
- Scripts: scripts/test_setup.py

## Breaking Changes
None - all changes are backward compatible

## Testing
- All existing tests pass
- New test-setup script validates conversation creation workflow
- Manual testing of autocomplete, search, and rename features
This commit is contained in:
ira
2025-12-28 18:15:02 -05:00
commit 3546c04348
79 changed files with 13531 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DEPLOYMENT.md
+278
View File
@@ -0,0 +1,278 @@
# Deployment Guide
## Overview
LLM Council can be deployed in several configurations depending on your needs:
- **Local Development**: Everything runs on your local machine
- **Hybrid**: Frontend/Backend local, LLM server on remote GPU VM
- **Full Remote**: Everything on a server/VM
- **Production**: Professional deployment with proper infrastructure
## Architecture Options
### Option 1: Hybrid (Recommended for Development)
**Setup:**
- Frontend + Backend: Run on your local machine
- LLM Server (Ollama): Run on remote GPU VM
**Pros:**
- Easy development and debugging
- GPU resources available remotely
- No need to deploy frontend/backend code
- Fast iteration
**Cons:**
- Requires network connectivity to GPU VM
- Latency for LLM requests
**Configuration:**
```bash
# .env on local machine
USE_LOCAL_OLLAMA=false
OPENAI_COMPAT_BASE_URL=http://your-gpu-vm-ip:11434
```
### Option 2: Full Remote Deployment
**Setup:**
- Everything runs on the GPU VM or dedicated server
**Pros:**
- Centralized deployment
- Can be accessed from multiple machines
- Better for team use
**Cons:**
- More complex setup
- Requires proper security configuration
- Slower development iteration
### Option 3: Production Deployment (Professional)
**Recommended Stack:**
- **Frontend**: Serve static build via nginx/CDN
- **Backend**: Run via systemd/gunicorn/uvicorn with reverse proxy
- **LLM Server**: Separate service on GPU VM
- **Security**: TLS/HTTPS, authentication, rate limiting
## GPU VM Setup
### Prerequisites
1. GPU VM with:
- NVIDIA GPU with CUDA support
- Sufficient VRAM for your models
- Network access from your local machine
2. Ollama installed on GPU VM:
```bash
curl -fsSL https://ollama.ai/install.sh | sh
```
### Step 1: Configure Ollama to Accept Remote Connections
**On GPU VM:**
```bash
# Option A: Environment variable (temporary)
export OLLAMA_HOST=0.0.0.0:11434
# Option B: Systemd service (persistent - recommended)
sudo systemctl edit ollama
```
Add to the override file:
```ini
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_MAX_LOADED_MODELS=3"
```
Then restart:
```bash
sudo systemctl daemon-reload
sudo systemctl restart ollama
```
### Step 2: Configure Firewall
**On GPU VM:**
```bash
# Allow port 11434 from your local network
sudo ufw allow from YOUR_LOCAL_IP to any port 11434
# Or allow from entire subnet (less secure)
sudo ufw allow 11434/tcp
```
### Step 3: Pull Required Models
**On GPU VM:**
```bash
ollama pull qwen2.5:7b
ollama pull llama3.1:8b
ollama pull qwen2.5:14b
ollama pull qwen2:latest
```
### Step 3.5 (GPU VM): Ensure Ollama Uses GPU + Stores Data on /mnt/data
If your VM has a small root disk, keep Ollama's storage and HOME off `/` (common cause of weird failures).
Also note that on this setup, `OLLAMA_LLM_LIBRARY=cuda` caused Ollama to *skip* CUDA libraries; use `cuda_v12`.
**On GPU VM:**
```bash
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf >/dev/null <<'EOF'
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_MODELS=/mnt/data/ollama"
Environment="HOME=/mnt/data/ollama/home"
Environment="OLLAMA_LLM_LIBRARY=cuda_v12"
Environment="LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v12"
EOF
sudo systemctl daemon-reload
sudo systemctl restart ollama
```
**Verify GPU offload (on GPU VM):**
```bash
ollama run qwen2:latest "Write 80 words about GPUs."
ollama ps
```
### Step 4: Verify Remote Access
**From local machine:**
```bash
curl http://YOUR_GPU_VM_IP:11434/api/tags
# Should return list of available models
curl http://YOUR_GPU_VM_IP:11434/v1/models
```
### Step 5: Configure LLM Council
**On local machine `.env`:**
```bash
USE_LOCAL_OLLAMA=false
OPENAI_COMPAT_BASE_URL=http://YOUR_GPU_VM_IP:11434
# Local (small) example:
# COUNCIL_MODELS=llama3.2:1b,qwen2.5:0.5b,gemma2:2b
# CHAIRMAN_MODEL=llama3.2:3b
# GPU (available models):
COUNCIL_MODELS=qwen2.5:7b,llama3.1:8b,qwen2:latest
CHAIRMAN_MODEL=qwen2.5:14b
```
## Security Considerations
### For Development/Internal Use
1. **Network Security:**
- Use VPN or private network
- Restrict firewall to specific IPs
- Consider SSH tunnel for extra security
2. **Ollama Security:**
- Ollama has no built-in authentication
- Only expose on trusted networks
- Consider reverse proxy with auth (nginx + basic auth)
### For Production
1. **Authentication:**
- Add API key authentication to backend
- Use session-based auth for frontend
- Implement rate limiting
2. **Network Security:**
- Use HTTPS/TLS everywhere
- Set up proper firewall rules
- Consider using a reverse proxy (nginx/traefik)
3. **Infrastructure:**
- Use container orchestration (Docker Compose/Kubernetes)
- Set up monitoring and logging
- Implement backup strategy for conversations
## Deployment Scripts
### Quick Start (Local + Remote Ollama)
```bash
# 1. Start Ollama on GPU VM (already running if systemd configured)
# 2. On local machine:
./start.sh
```
### Full Remote Deployment
See `docs/DEPLOYMENT_FULL.md` for complete remote deployment instructions.
## Troubleshooting
### Connection Timeouts
1. Check Ollama is listening on all interfaces:
```bash
# On GPU VM
sudo netstat -tlnp | grep 11434
# Should show 0.0.0.0:11434, not 127.0.0.1:11434
```
2. Check firewall rules:
```bash
# On GPU VM
sudo ufw status
```
3. Test connectivity:
```bash
# From local machine
curl -v http://GPU_VM_IP:11434/api/tags
```
### Model Loading Issues
1. Check available VRAM:
```bash
nvidia-smi
```
2. Adjust `OLLAMA_MAX_LOADED_MODELS` if needed
3. Check model sizes vs available memory
## Performance Tuning
### Ollama Settings
```bash
# On GPU VM, edit systemd override:
Environment="OLLAMA_KEEP_ALIVE=24h" # Keep models loaded
Environment="OLLAMA_MAX_LOADED_MODELS=3" # Max concurrent models
Environment="OLLAMA_NUM_PARALLEL=1" # Parallel requests
```
### LLM Council Timeouts
Adjust in `.env`:
```bash
LLM_TIMEOUT_SECONDS=600.0 # For slow models
CHAIRMAN_TIMEOUT_SECONDS=600.0
OPENAI_COMPAT_TIMEOUT_SECONDS=600.0
OPENAI_COMPAT_CONNECT_TIMEOUT_SECONDS=30.0
OPENAI_COMPAT_WRITE_TIMEOUT_SECONDS=30.0
OPENAI_COMPAT_POOL_TIMEOUT_SECONDS=30.0
```
docs/DEPLOYMENT_RECOMMENDATIONS.md
+178
View File
@@ -0,0 +1,178 @@
# Professional Deployment Recommendations
## Recommended Architecture
### For Development/Personal Use
**Hybrid Approach (Recommended):**
```
┌─────────────────┐ ┌──────────────────┐
│ Local Machine │ │ GPU VM │
│ │ │ │
│ Frontend │ │ Ollama Server │
│ (React/Vite) │ │ (LLM Models) │
│ │◄────────┤ │
│ Backend │ HTTP │ Port 11434 │
│ (FastAPI) │ │ │
└─────────────────┘ └──────────────────┘
```
**Why this is best:**
- ✅ Fast development iteration
- ✅ Easy debugging (logs on local machine)
- ✅ GPU resources available remotely
- ✅ No complex deployment needed
- ✅ Can work offline (if models cached locally)
### For Production/Team Use
**Full Remote Deployment:**
```
┌─────────────────┐
│ Users │
│ (Browsers) │
└────────┬────────┘
│ HTTPS
▼
┌─────────────────┐
│ Reverse Proxy │
│ (nginx/traefik)│
└────────┬────────┘
│
┌────┴────┐
│ │
▼ ▼
┌────────┐ ┌──────────┐
│Frontend│ │ Backend │
│(Static)│ │(FastAPI) │
└────────┘ └────┬─────┘
│ HTTP
▼
┌──────────────┐
│ GPU VM │
│ Ollama │
└──────────────┘
```
## Comparison of Approaches
| Aspect | Hybrid (Local + Remote) | Full Remote | Production |
|--------|------------------------|-------------|------------|
| **Setup Complexity** | Low | Medium | High |
| **Development Speed** | Fast | Medium | Slow |
| **Security** | Medium | Medium | High |
| **Scalability** | Low | Medium | High |
| **Cost** | Low | Medium | High |
| **Best For** | Dev/Personal | Team/Internal | Public/Enterprise |
## Security Best Practices
### Development/Internal Network
1. **Network Isolation:**
- Use private network/VPN
- Restrict firewall to specific IPs
- Consider SSH tunnel for Ollama
2. **Ollama Access:**
```bash
# Only allow from specific IPs
sudo ufw allow from YOUR_IP to any port 11434
```
3. **SSH Tunnel Alternative:**
```bash
# More secure - no direct network exposure
ssh -L 11434:localhost:11434 user@gpu-vm
# Then use localhost:11434 in .env
```
### Production Deployment
1. **Authentication:**
- Add API keys to backend
- Implement user sessions
- Use OAuth2/JWT for API
2. **Network Security:**
- HTTPS/TLS everywhere
- WAF (Web Application Firewall)
- Rate limiting
- DDoS protection
3. **Infrastructure:**
- Container orchestration (Docker/K8s)
- Service mesh for internal communication
- Monitoring and alerting
- Automated backups
## Deployment Checklist
### Hybrid Setup (Recommended for Dev)
- [ ] GPU VM has Ollama installed
- [ ] Ollama configured to listen on 0.0.0.0:11434
- [ ] Firewall allows connections from local machine
- [ ] Models pulled on GPU VM
- [ ] Local `.env` configured with GPU VM IP
- [ ] Test connection: `curl http://GPU_VM_IP:11434/api/tags`
### Full Remote Setup
- [ ] Server/VM provisioned
- [ ] Frontend built and served (nginx/static host)
- [ ] Backend running as service (systemd/supervisor)
- [ ] Reverse proxy configured (nginx/traefik)
- [ ] SSL certificates installed
- [ ] Authentication implemented
- [ ] Monitoring set up
- [ ] Backup strategy in place
### Production Setup
- [ ] All of Full Remote checklist
- [ ] Load balancing configured
- [ ] Database for conversations (optional upgrade)
- [ ] Logging and monitoring (Prometheus/Grafana)
- [ ] CI/CD pipeline
- [ ] Security audit
- [ ] Documentation for ops team
- [ ] Disaster recovery plan
## Cost Considerations
### Hybrid (Local + Remote GPU VM)
- **Cost**: GPU VM only (~$0.50-2/hour depending on GPU)
- **Best for**: Development, personal projects, small teams
### Full Remote
- **Cost**: GPU VM + Application Server (~$1-3/hour)
- **Best for**: Teams, internal tools
### Production
- **Cost**: $100-1000+/month depending on scale
- **Best for**: Public services, enterprise
## Migration Path
1. **Start**: Hybrid (local dev, remote GPU)
2. **Grow**: Full remote (when team needs it)
3. **Scale**: Production (when going public/enterprise)
## Recommendation
**For your use case (development/personal):**
Use the **Hybrid approach**:
- Run frontend + backend locally
- Connect to Ollama on GPU VM
- Use SSH tunnel for extra security if needed
- Simple, fast, cost-effective
This gives you:
- Fast development iteration
- Easy debugging
- GPU resources when needed
- Minimal infrastructure complexity
- Low cost
docs/GPU_VM_SETUP.md
+93
View File
@@ -0,0 +1,93 @@
# GPU VM Setup - Quick Reference
## Quick Setup Steps
### 1. On GPU VM: Configure Ollama to Accept Remote Connections
```bash
# Create systemd override
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf > /dev/null <<EOF
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=24h"
# Keep Ollama storage off root disk (recommended) and ensure the service user
# has a writable HOME (runners/keys/cache).
Environment="OLLAMA_MODELS=/mnt/data/ollama"
Environment="HOME=/mnt/data/ollama/home"
# IMPORTANT (GPU): on this VM, `OLLAMA_LLM_LIBRARY=cuda` caused Ollama to SKIP CUDA.
# Use the libdir selector instead.
Environment="OLLAMA_LLM_LIBRARY=cuda_v12"
# Ensure the dynamic linker can resolve Ollama's bundled CUDA + ggml libs.
Environment="LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v12"
EOF
# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart ollama
# Verify
curl http://0.0.0.0:11434/api/tags
```
**Verify GPU is actually used (on GPU VM):**
```bash
ollama run qwen2:latest "Write 80 words about GPUs."
ollama ps
watch -n 0.2 nvidia-smi
```
### 2. On GPU VM: Configure Firewall
```bash
# Allow port 11434 (adjust IP/subnet as needed)
sudo ufw allow from YOUR_LOCAL_IP to any port 11434
# Or allow from entire subnet (less secure)
sudo ufw allow 11434/tcp
```
### 3. On GPU VM: Pull Required Models
```bash
ollama pull qwen2.5:7b
ollama pull llama3.1:8b
ollama pull qwen2.5:14b
ollama pull qwen2:latest
```
### 4. On Local Machine: Configure .env
```bash
USE_LOCAL_OLLAMA=false
OPENAI_COMPAT_BASE_URL=http://YOUR_GPU_VM_IP:11434
# Local (small) example:
# COUNCIL_MODELS=llama3.2:1b,qwen2.5:0.5b,gemma2:2b
# CHAIRMAN_MODEL=llama3.2:3b
# GPU (available models):
COUNCIL_MODELS=qwen2.5:7b,llama3.1:8b,qwen2:latest
CHAIRMAN_MODEL=qwen2.5:14b
```
### 5. Test Connection
```bash
# From local machine
curl http://YOUR_GPU_VM_IP:11434/api/tags
curl http://YOUR_GPU_VM_IP:11434/v1/models
```
## Troubleshooting
- **Connection timeout**: Check Ollama is listening on `0.0.0.0:11434` (not `127.0.0.1`)
- **Firewall blocking**: Check `sudo ufw status` and allow port 11434
- **CPU instead of GPU**: Run `ollama ps` and confirm it doesn't say `100% CPU`. If it does:
- Ensure `OLLAMA_LLM_LIBRARY=cuda_v12` (not `cuda`)
- Ensure `LD_LIBRARY_PATH` includes `/usr/local/lib/ollama` and `/usr/local/lib/ollama/cuda_v12`
- Ensure `OLLAMA_MODELS` and `HOME` are on a disk with free space (root disk full can break runner/cache)
See [DEPLOYMENT.md](DEPLOYMENT.md) for detailed instructions.
docs/README.md
+11
View File
@@ -0,0 +1,11 @@
# Documentation
## Getting Started
- **[README](../README.md)** - Main project documentation
- **[Architecture](../ARCHITECTURE.md)** - System architecture overview
## Deployment
- **[Deployment Guide](DEPLOYMENT.md)** - Complete deployment instructions with testing steps
- **[Deployment Recommendations](DEPLOYMENT_RECOMMENDATIONS.md)** - Professional deployment options and best practices
- **[GPU VM Setup](GPU_VM_SETUP.md)** - Quick reference for GPU VM configuration