# LLM Council


The idea of this repo is that instead of asking a question to a single LLM, you can group multiple LLMs into your "LLM Council". This repo is a simple, local web app that essentially looks like ChatGPT except it sends your query to multiple LLMs via an OpenAI-compatible API (Ollama, vLLM, TGI, etc.), it then asks them to review and rank each other's work, and finally a Chairman LLM produces the final response.

In a bit more detail, here is what happens when you submit a query:

1. **Stage 1: First opinions**. The user query is given to all LLMs individually, and the responses are collected. The individual responses are shown in a "tab view", so that the user can inspect them all one by one.
2. **Stage 2: Review**. Each individual LLM is given the responses of the other LLMs. Under the hood, the LLM identities are anonymized so that the LLM can't play favorites when judging their outputs. The LLM is asked to rank them in accuracy and insight.
3. **Stage 3: Final response**. The designated Chairman of the LLM Council takes all of the model's responses and compiles them into a single final answer that is presented to the user.

## Vibe Code Alert

This project was 99% vibe coded as a fun Saturday hack because I wanted to explore and evaluate a number of LLMs side by side in the process of [reading books together with LLMs](https://x.com/karpathy/status/1990577951671509438). It's nice and useful to see multiple responses side by side, and also the cross-opinions of all LLMs on each other's outputs. You're not going to support it in any way, it's provided here as is for other people's inspiration and you don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like.

## Setup

### 1. Install Dependencies

The project uses [uv](https://docs.astral.sh/uv/) for project management.

**Backend:**
```bash
uv sync
```

**Frontend:**
```bash
cd frontend
npm install
cd ..
```

### 2. Configure Ollama Server

LLM Council requires an OpenAI-compatible API server. The easiest way to get started is with Ollama running locally or on a remote server.

**For local Ollama:**
1. Install and start Ollama: https://ollama.ai
2. Pull some models:
```bash
ollama pull llama3.2:3b
ollama pull qwen2.5:3b
ollama pull gemma2:2b
```

### 3. Configure Environment

Create a `.env` file in the project root with your configuration:

**For local Ollama:**
```bash
USE_LOCAL_OLLAMA=true
COUNCIL_MODELS=llama3.2:3b,qwen2.5:3b,gemma2:2b
CHAIRMAN_MODEL=llama3.2:3b
MAX_TOKENS=1024
LLM_MAX_CONCURRENCY=1
```

**For remote Ollama or other OpenAI-compatible server:**
```bash
OPENAI_COMPAT_BASE_URL=http://your-server:11434
COUNCIL_MODELS=llama3.2:3b,qwen2.5:3b,gemma2:2b
CHAIRMAN_MODEL=llama3.2:3b
MAX_TOKENS=2048
LLM_MAX_CONCURRENCY=1
```

**Optional timeout configuration:**
```bash
LLM_TIMEOUT_SECONDS=120.0          # Default timeout for LLM queries
CHAIRMAN_TIMEOUT_SECONDS=180.0     # Timeout for chairman synthesis
TITLE_GENERATION_TIMEOUT_SECONDS=120.0  # Timeout for title generation
OPENAI_COMPAT_TIMEOUT_SECONDS=300.0    # Timeout for OpenAI-compatible server
OPENAI_COMPAT_CONNECT_TIMEOUT_SECONDS=10.0  # HTTP connection timeout
OPENAI_COMPAT_WRITE_TIMEOUT_SECONDS=10.0    # HTTP write timeout
OPENAI_COMPAT_POOL_TIMEOUT_SECONDS=10.0    # HTTP pool timeout
```

See `.env.example` for all available configuration options. Alternatively, you can edit `backend/config.py` directly to set defaults.

## Running the Application

**Option 1: Use the start script**
```bash
./start.sh
```

**Option 2: Use Makefile**
```bash
make dev
```

**Option 3: Run manually**

Terminal 1 (Backend):
```bash
uv run python -m backend.main
```

Terminal 2 (Frontend):
```bash
cd frontend
npm run dev
```

Then open http://localhost:5173 in your browser.

**Option 4: Test setup with pre-configured conversation**
```bash
# Set in .env:
# TEST_MESSAGE="Your message"
# TEST_DOCS="doc1.md,doc2.md"
make test-setup
```

This creates a new conversation with today's date/time, uploads documents, and **pre-fills** the message in the UI (it does **not** auto-send).

### Frontend theme default (optional)

By default, the UI theme is persisted in `localStorage`. If there is no saved theme yet, you can set a default theme via a Vite env var:

```bash
# Example (starts in dark mode if there's no localStorage value yet)
VITE_DEFAULT_THEME=dark make dev
```

## Using Ollama on a Remote Server

If you have Ollama running on a remote server or VM:

1. In your project `.env`, set:

```bash
OPENAI_COMPAT_BASE_URL=http://your-server-ip:11434
COUNCIL_MODELS=llama3.2:3b,qwen2.5:3b,gemma2:2b
CHAIRMAN_MODEL=llama3.2:3b
MAX_TOKENS=2048
LLM_MAX_CONCURRENCY=1
```

2. Verify connectivity from your machine:

```bash
curl http://your-server-ip:11434/api/tags
```

## Using Other OpenAI-Compatible Servers (vLLM, TGI, etc.)

If you're running vLLM, TGI, or another OpenAI-compatible server:

1. Ensure your server exposes:
   - `POST /v1/chat/completions`
   - `GET /v1/models`

2. In your project `.env`, set:

```bash
OPENAI_COMPAT_BASE_URL=http://your-server:port
COUNCIL_MODELS=your-model-1,your-model-2,your-model-3
CHAIRMAN_MODEL=your-model-1
MAX_TOKENS=2048
LLM_MAX_CONCURRENCY=1

# (optional) if your server requires auth:
# OPENAI_COMPAT_API_KEY=...
```

3. Verify connectivity:

```bash
curl http://your-server:port/v1/models
```

## Documentation

- **[Architecture](ARCHITECTURE.md)** - System architecture and design
- **[Deployment Guide](docs/DEPLOYMENT.md)** - How to deploy with remote GPU VM
- **[Deployment Recommendations](docs/DEPLOYMENT_RECOMMENDATIONS.md)** - Professional deployment options

## Tech Stack

- **Backend:** FastAPI (Python 3.10+), async httpx, OpenAI-compatible API
- **Frontend:** React + Vite, react-markdown for rendering
- **Storage:** JSON files in `data/conversations/`
- **Package Management:** uv for Python, npm for JavaScript
- **LLM Backend:** Ollama, vLLM, TGI, or any OpenAI-compatible server