homelab-notes/proxmox1/searxng-lxc-notes.md

# SearXNG LXC Notes

Self-hosted SearXNG instance running in a Debian LXC, fronted by external Caddy VM, using Docker for SearXNG + Valkey/Redis.

---

## Design

- SearXNG runs in a Debian LXC (10.0.10.70) with Docker; only SearXNG and Valkey live in this container.
- Caddy runs on a separate VM and is the single external reverse proxy for the homelab, including `search.levkin.ca`.
- The SearXNG Docker setup uses "bring your own reverse proxy" mode from the official searxng-docker repo, so internal Caddy in the stack is removed.

---

## Network and URLs

- **LXC IP**: 10.0.10.70 (Proxmox container)
- **SearXNG container**: Listens on `0.0.0.0:8080` on the LXC, published via Docker ports: `- "8080:8080"`
- **Public URL**: `https://search.levkin.ca` via Caddy VM reverse proxy to `10.0.10.70:8080`

---

## Initial Setup

### 1. Create and Configure LXC Container

In Proxmox:

```bash
# Create Debian LXC container
# - Template: debian-12-standard
# - Storage: local-lvm
# - Network: Bridge vmbr0, IP 10.0.10.70/24
# - Resources: 2GB RAM, 1 CPU core minimum
```

### 2. Install Docker in LXC

SSH into the LXC container:

```bash
# Update system
apt update && apt upgrade -y

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

# Add current user to docker group (if needed)
usermod -aG docker $USER

# Verify installation
docker --version
docker compose version
```

### 3. Clone SearXNG Docker Repository

```bash
# Create directory
mkdir -p /opt/searxng-docker
cd /opt/searxng-docker

# Clone the official repository
git clone https://github.com/searxng/searxng-docker.git .

# Or download and extract if git is not available
```

---

## Docker Compose Configuration

**Location**: `/opt/searxng-docker/docker-compose.yaml` inside the LXC.

### Key Points

- Only two services: `redis` (Valkey) and `searxng`
- Internal Caddy service and volumes from the upstream searxng-docker example are removed
- SearXNG base URL is set via env so generated links use the public domain

### Complete docker-compose.yaml

```yaml
services:
  redis:
    container_name: redis
    image: docker.io/valkey/valkey:8-alpine
    command: valkey-server --save 30 1 --loglevel warning
    restart: unless-stopped
    networks:
      - searxng
    volumes:
      - valkey-data2:/data
    healthcheck:
      test: ["CMD", "valkey-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 3

  searxng:
    container_name: searxng
    image: docker.io/searxng/searxng:latest
    restart: unless-stopped
    networks:
      - searxng
    ports:
      - "8080:8080"   # Must NOT be bound to 127.0.0.1, Caddy VM needs LAN access
    volumes:
      - ./searxng:/etc/searxng:rw
      - searxng-data:/var/cache/searxng:rw
    environment:
      - SEARXNG_BASE_URL=https://search.levkin.ca/
    depends_on:
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080/"]
      interval: 30s
      timeout: 10s
      retries: 3

networks:
  searxng:
    driver: bridge

volumes:
  valkey-data2:
  searxng-data:
```

### Important Notes

- **Port binding**: Binding to `127.0.0.1:8080:8080` broke access from the Caddy VM; use `8080:8080` so it listens on `10.0.10.70`
- **Volumes**: Match the official documentation: config under `/etc/searxng`, cache under `/var/cache/searxng`
- **Health checks**: Added to ensure services are ready before accepting traffic

---

## SearXNG Configuration

### Settings File

Location: `/opt/searxng-docker/searxng/settings.yml`

Key configuration options:

```yaml
server:
  secret_key: "CHANGE_ME_TO_SOMETHING_RANDOM"  # Generate with: openssl rand -hex 32
  base_url: "https://search.levkin.ca/"
  bind_address: "0.0.0.0"
  port: 8080

search:
  formats:
    - html
    - json
    - csv
    - rss

engines:
  # Enable/disable specific engines as needed
  - name: google
  - name: bing
  - name: duckduckgo
  - name: wikipedia
  # Add more engines as desired

general:
  instance_name: "SearXNG"
  enable_metrics: false  # Set to true if you want metrics endpoint

ui:
  theme: simple  # Options: simple, oscar, courgette
  infinite_scroll: true
  results_on_new_tab: false

outgoing:
  max_request_timeout: 3.0
  pool_connections: 100
  useragent_suffix: "SearXNG"

redis:
  url: "redis://redis:6379/0"
```

### Generate Secret Key

```bash
openssl rand -hex 32
```

Update the `secret_key` in `settings.yml` with the generated value.

---

## Deployment Steps

1. **SSH to LXC** and navigate to the directory:

   ```bash
   cd /opt/searxng-docker
   ```

2. **Edit docker-compose.yaml** to:
   - Remove the caddy service and related volumes
   - Ensure searxng has `ports: - "8080:8080"` and the correct `SEARXNG_BASE_URL`

3. **Configure settings.yml** (if customizing):

   ```bash
   # Copy default settings if not present
   cp searxng/settings.yml.example searxng/settings.yml

   # Edit settings.yml
   nano searxng/settings.yml
   ```

4. **Start/restart stack**:

   ```bash
   docker compose up -d
   docker compose ps  # Confirm searxng -> 0.0.0.0:8080->8080
   ```

5. **Check logs**:

   ```bash
   docker compose logs -f searxng
   docker compose logs -f redis
   ```

6. **Quick local test** from another host on LAN:

   ```bash
   curl -v http://10.0.10.70:8080/ | head
   ```

Expected: HTML for SearXNG front page, status 200 from server granian.

---

## Caddy VM Configuration

Caddy runs on a separate VM and handles HTTPS + public DNS.

### Site Block

```caddy
search.levkin.ca {
    encode gzip
    reverse_proxy 10.0.10.70:8080 {
        header_up Host {host}
        header_up X-Real-IP {remote}
        header_up X-Forwarded-For {remote}
        header_up X-Forwarded-Proto {scheme}
    }
}
```

### Notes

- Caddy handles TLS certificates and HTTPS for `search.levkin.ca`
- Upstream is plain HTTP on LAN; change IP/port here if the LXC moves
- This follows the "bring your own reverse proxy" approach recommended by the SearXNG Docker docs
- Headers are forwarded to preserve client information

---

## Firewall Configuration

### LXC Container

Ensure port 8080 is accessible from the Caddy VM:

```bash
# If using ufw
ufw allow from 10.0.10.0/24 to any port 8080

# Or if using iptables directly
iptables -A INPUT -s 10.0.10.0/24 -p tcp --dport 8080 -j ACCEPT
```

### Proxmox Host

Ensure the Proxmox firewall allows traffic between VMs/LXCs on the same network.

---

## Security Considerations

1. **Secret Key**: Always use a strong, randomly generated secret key in `settings.yml`
2. **Network Isolation**: SearXNG only needs to be accessible from the Caddy VM, not the entire network
3. **Updates**: Regularly update Docker images:

   ```bash
   cd /opt/searxng-docker
   docker compose pull
   docker compose up -d
   ```

4. **Logging**: Consider disabling or limiting logging for privacy:

   ```yaml
   # In settings.yml
   general:
     enable_metrics: false
   ```

5. **Rate Limiting**: Configure rate limiting in Caddy if needed to prevent abuse

---

## Backup and Restore

### Backup

```bash
# Backup configuration
tar -czf searxng-backup-$(date +%Y%m%d).tar.gz \
  /opt/searxng-docker/searxng \
  /opt/searxng-docker/docker-compose.yaml

# Backup volumes (optional, for cache)
docker run --rm -v searxng-docker_searxng-data:/data -v $(pwd):/backup \
  alpine tar czf /backup/searxng-data-$(date +%Y%m%d).tar.gz /data
```

### Restore

```bash
# Extract configuration
tar -xzf searxng-backup-YYYYMMDD.tar.gz -C /

# Restore volumes (if needed)
docker run --rm -v searxng-docker_searxng-data:/data -v $(pwd):/backup \
  alpine tar xzf /backup/searxng-data-YYYYMMDD.tar.gz -C /
```

---

## Monitoring and Maintenance

### Check Container Status

```bash
docker compose ps
docker compose logs --tail=50 searxng
```

### View Resource Usage

```bash
docker stats searxng redis
```

### Update Containers

```bash
cd /opt/searxng-docker
docker compose pull
docker compose up -d
docker compose image prune -f  # Remove old images
```

### Clear Cache

```bash
docker compose exec searxng rm -rf /var/cache/searxng/*
docker compose restart searxng
```

---

## Troubleshooting

### Container Won't Start

```bash
# Check logs
docker compose logs searxng

# Verify configuration
docker compose config

# Test connectivity
docker compose exec searxng ping -c 3 redis
```

### Can't Access from Caddy VM

1. Verify port binding: `docker compose ps` should show `0.0.0.0:8080->8080/tcp`
2. Test from Caddy VM: `curl http://10.0.10.70:8080/`
3. Check firewall rules on LXC
4. Verify network connectivity: `ping 10.0.10.70` from Caddy VM

### SearXNG Returns Errors

1. Check Redis connectivity: `docker compose exec searxng ping redis`
2. Verify settings.yml syntax: `docker compose exec searxng cat /etc/searxng/settings.yml`
3. Check logs for specific errors: `docker compose logs searxng | grep -i error`

### Performance Issues

1. Monitor resource usage: `docker stats`
2. Check Redis performance: `docker compose exec redis valkey-cli info stats`
3. Consider increasing LXC resources if needed
4. Review search engine timeouts in settings.yml

---

## Privacy and Usage Notes

- **SearXNG is a metasearch engine**: It queries multiple search providers and aggregates results, instead of you talking to each provider directly
- **Self-hosting means**:
  - No profiling or tracking by the instance; logs can be disabled or tightly controlled
  - You no longer have to trust a random public SearXNG operator with your search queries
- **Intended use in this homelab**:
  - Default search engine in browsers via `https://search.levkin.ca`
  - Potential backend for LLM tools or scripts that need web search via SearXNG's APIs

---

## API Usage

SearXNG provides a JSON API for programmatic access:

```bash
# Example search query
curl "https://search.levkin.ca/search?q=example&format=json" | jq

# RSS feed
curl "https://search.levkin.ca/search?q=example&format=rss"
```

Useful for automation, LLM integrations, or custom search tools.

---

## Resource Limits

Recommended LXC container resources:

- **CPU**: 1-2 cores
- **RAM**: 2-4 GB
- **Disk**: 10-20 GB (for cache and logs)
- **Network**: Standard bridge connection

Adjust based on usage patterns and concurrent users.