Add SearXNG LXC notes and LLM prompt for service documentation (#3)
Reviewed-on: #3
This commit is contained in:
parent
e6ab067641
commit
f913e5995b
19
README.md
19
README.md
@ -21,3 +21,22 @@ Personal notes for my homelab: build logs, configs, and operational runbooks for
|
||||
## Status
|
||||
|
||||
This is an actively evolving notebook. Some sections may be incomplete or “work in progress”.
|
||||
|
||||
### Using an LLM to draft new service notes
|
||||
|
||||
When adding a new container/VM/service, use this prompt with an LLM to generate a first draft of notes (similar style to the rest of this repo):
|
||||
|
||||
> You are helping me maintain my homelab-notes repo.
|
||||
> I am adding or modifying ONE service/container/VM.
|
||||
> Read the snippets I paste from my terminal (docker-compose, configs, IPs, curl outputs, etc.) and then produce a single Markdown file in the same style as other *-notes.md in this repo:
|
||||
>
|
||||
> - Start with a short description of what the service is and where it runs.
|
||||
> - Capture design/architecture decisions (networking, reverse proxy, storage, security).
|
||||
> - Document IPs, hostnames, ports, container names, and paths that are specific to my environment.
|
||||
> - Include trimmed config examples (docker-compose, Caddy/nginx, app config) with comments where helpful.
|
||||
> - Add a small "Deployment steps" section summarizing the exact commands I used (cd, docker compose up, systemctl, etc.).
|
||||
> - Add brief "Troubleshooting notes" if we fixed any issues (e.g., wrong port binding, DNS, reverse proxy issues).
|
||||
>
|
||||
> Do NOT invent services or values that I did not show you.
|
||||
> Only use the information I provide plus obvious inferences (e.g., 0.0.0.0:8080 means accessible on LAN).
|
||||
> Output ONLY the Markdown for the new note file, no explanation around it.
|
||||
|
||||
450
proxmox1/searxng-lxc-notes.md
Normal file
450
proxmox1/searxng-lxc-notes.md
Normal file
@ -0,0 +1,450 @@
|
||||
# SearXNG LXC Notes
|
||||
|
||||
Self-hosted SearXNG instance running in a Debian LXC, fronted by external Caddy VM, using Docker for SearXNG + Valkey/Redis.
|
||||
|
||||
---
|
||||
|
||||
## Design
|
||||
|
||||
- SearXNG runs in a Debian LXC (10.0.10.70) with Docker; only SearXNG and Valkey live in this container.
|
||||
- Caddy runs on a separate VM and is the single external reverse proxy for the homelab, including `search.levkin.ca`.
|
||||
- The SearXNG Docker setup uses "bring your own reverse proxy" mode from the official searxng-docker repo, so internal Caddy in the stack is removed.
|
||||
|
||||
---
|
||||
|
||||
## Network and URLs
|
||||
|
||||
- **LXC IP**: 10.0.10.70 (Proxmox container)
|
||||
- **SearXNG container**: Listens on `0.0.0.0:8080` on the LXC, published via Docker ports: `- "8080:8080"`
|
||||
- **Public URL**: `https://search.levkin.ca` via Caddy VM reverse proxy to `10.0.10.70:8080`
|
||||
|
||||
---
|
||||
|
||||
## Initial Setup
|
||||
|
||||
### 1. Create and Configure LXC Container
|
||||
|
||||
In Proxmox:
|
||||
|
||||
```bash
|
||||
# Create Debian LXC container
|
||||
# - Template: debian-12-standard
|
||||
# - Storage: local-lvm
|
||||
# - Network: Bridge vmbr0, IP 10.0.10.70/24
|
||||
# - Resources: 2GB RAM, 1 CPU core minimum
|
||||
```
|
||||
|
||||
### 2. Install Docker in LXC
|
||||
|
||||
SSH into the LXC container:
|
||||
|
||||
```bash
|
||||
# Update system
|
||||
apt update && apt upgrade -y
|
||||
|
||||
# Install Docker
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sh get-docker.sh
|
||||
|
||||
# Add current user to docker group (if needed)
|
||||
usermod -aG docker $USER
|
||||
|
||||
# Verify installation
|
||||
docker --version
|
||||
docker compose version
|
||||
```
|
||||
|
||||
### 3. Clone SearXNG Docker Repository
|
||||
|
||||
```bash
|
||||
# Create directory
|
||||
mkdir -p /opt/searxng-docker
|
||||
cd /opt/searxng-docker
|
||||
|
||||
# Clone the official repository
|
||||
git clone https://github.com/searxng/searxng-docker.git .
|
||||
|
||||
# Or download and extract if git is not available
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker Compose Configuration
|
||||
|
||||
**Location**: `/opt/searxng-docker/docker-compose.yaml` inside the LXC.
|
||||
|
||||
### Key Points
|
||||
|
||||
- Only two services: `redis` (Valkey) and `searxng`
|
||||
- Internal Caddy service and volumes from the upstream searxng-docker example are removed
|
||||
- SearXNG base URL is set via env so generated links use the public domain
|
||||
|
||||
### Complete docker-compose.yaml
|
||||
|
||||
```yaml
|
||||
services:
|
||||
redis:
|
||||
container_name: redis
|
||||
image: docker.io/valkey/valkey:8-alpine
|
||||
command: valkey-server --save 30 1 --loglevel warning
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- searxng
|
||||
volumes:
|
||||
- valkey-data2:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "valkey-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
|
||||
searxng:
|
||||
container_name: searxng
|
||||
image: docker.io/searxng/searxng:latest
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- searxng
|
||||
ports:
|
||||
- "8080:8080" # Must NOT be bound to 127.0.0.1, Caddy VM needs LAN access
|
||||
volumes:
|
||||
- ./searxng:/etc/searxng:rw
|
||||
- searxng-data:/var/cache/searxng:rw
|
||||
environment:
|
||||
- SEARXNG_BASE_URL=https://search.levkin.ca/
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080/"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
networks:
|
||||
searxng:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
valkey-data2:
|
||||
searxng-data:
|
||||
```
|
||||
|
||||
### Important Notes
|
||||
|
||||
- **Port binding**: Binding to `127.0.0.1:8080:8080` broke access from the Caddy VM; use `8080:8080` so it listens on `10.0.10.70`
|
||||
- **Volumes**: Match the official documentation: config under `/etc/searxng`, cache under `/var/cache/searxng`
|
||||
- **Health checks**: Added to ensure services are ready before accepting traffic
|
||||
|
||||
---
|
||||
|
||||
## SearXNG Configuration
|
||||
|
||||
### Settings File
|
||||
|
||||
Location: `/opt/searxng-docker/searxng/settings.yml`
|
||||
|
||||
Key configuration options:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
secret_key: "CHANGE_ME_TO_SOMETHING_RANDOM" # Generate with: openssl rand -hex 32
|
||||
base_url: "https://search.levkin.ca/"
|
||||
bind_address: "0.0.0.0"
|
||||
port: 8080
|
||||
|
||||
search:
|
||||
formats:
|
||||
- html
|
||||
- json
|
||||
- csv
|
||||
- rss
|
||||
|
||||
engines:
|
||||
# Enable/disable specific engines as needed
|
||||
- name: google
|
||||
- name: bing
|
||||
- name: duckduckgo
|
||||
- name: wikipedia
|
||||
# Add more engines as desired
|
||||
|
||||
general:
|
||||
instance_name: "SearXNG"
|
||||
enable_metrics: false # Set to true if you want metrics endpoint
|
||||
|
||||
ui:
|
||||
theme: simple # Options: simple, oscar, courgette
|
||||
infinite_scroll: true
|
||||
results_on_new_tab: false
|
||||
|
||||
outgoing:
|
||||
max_request_timeout: 3.0
|
||||
pool_connections: 100
|
||||
useragent_suffix: "SearXNG"
|
||||
|
||||
redis:
|
||||
url: "redis://redis:6379/0"
|
||||
```
|
||||
|
||||
### Generate Secret Key
|
||||
|
||||
```bash
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
Update the `secret_key` in `settings.yml` with the generated value.
|
||||
|
||||
---
|
||||
|
||||
## Deployment Steps
|
||||
|
||||
1. **SSH to LXC** and navigate to the directory:
|
||||
|
||||
```bash
|
||||
cd /opt/searxng-docker
|
||||
```
|
||||
|
||||
2. **Edit docker-compose.yaml** to:
|
||||
- Remove the caddy service and related volumes
|
||||
- Ensure searxng has `ports: - "8080:8080"` and the correct `SEARXNG_BASE_URL`
|
||||
|
||||
3. **Configure settings.yml** (if customizing):
|
||||
|
||||
```bash
|
||||
# Copy default settings if not present
|
||||
cp searxng/settings.yml.example searxng/settings.yml
|
||||
|
||||
# Edit settings.yml
|
||||
nano searxng/settings.yml
|
||||
```
|
||||
|
||||
4. **Start/restart stack**:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose ps # Confirm searxng -> 0.0.0.0:8080->8080
|
||||
```
|
||||
|
||||
5. **Check logs**:
|
||||
|
||||
```bash
|
||||
docker compose logs -f searxng
|
||||
docker compose logs -f redis
|
||||
```
|
||||
|
||||
6. **Quick local test** from another host on LAN:
|
||||
|
||||
```bash
|
||||
curl -v http://10.0.10.70:8080/ | head
|
||||
```
|
||||
|
||||
Expected: HTML for SearXNG front page, status 200 from server granian.
|
||||
|
||||
---
|
||||
|
||||
## Caddy VM Configuration
|
||||
|
||||
Caddy runs on a separate VM and handles HTTPS + public DNS.
|
||||
|
||||
### Site Block
|
||||
|
||||
```caddy
|
||||
search.levkin.ca {
|
||||
encode gzip
|
||||
reverse_proxy 10.0.10.70:8080 {
|
||||
header_up Host {host}
|
||||
header_up X-Real-IP {remote}
|
||||
header_up X-Forwarded-For {remote}
|
||||
header_up X-Forwarded-Proto {scheme}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Caddy handles TLS certificates and HTTPS for `search.levkin.ca`
|
||||
- Upstream is plain HTTP on LAN; change IP/port here if the LXC moves
|
||||
- This follows the "bring your own reverse proxy" approach recommended by the SearXNG Docker docs
|
||||
- Headers are forwarded to preserve client information
|
||||
|
||||
---
|
||||
|
||||
## Firewall Configuration
|
||||
|
||||
### LXC Container
|
||||
|
||||
Ensure port 8080 is accessible from the Caddy VM:
|
||||
|
||||
```bash
|
||||
# If using ufw
|
||||
ufw allow from 10.0.10.0/24 to any port 8080
|
||||
|
||||
# Or if using iptables directly
|
||||
iptables -A INPUT -s 10.0.10.0/24 -p tcp --dport 8080 -j ACCEPT
|
||||
```
|
||||
|
||||
### Proxmox Host
|
||||
|
||||
Ensure the Proxmox firewall allows traffic between VMs/LXCs on the same network.
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Secret Key**: Always use a strong, randomly generated secret key in `settings.yml`
|
||||
2. **Network Isolation**: SearXNG only needs to be accessible from the Caddy VM, not the entire network
|
||||
3. **Updates**: Regularly update Docker images:
|
||||
|
||||
```bash
|
||||
cd /opt/searxng-docker
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
4. **Logging**: Consider disabling or limiting logging for privacy:
|
||||
|
||||
```yaml
|
||||
# In settings.yml
|
||||
general:
|
||||
enable_metrics: false
|
||||
```
|
||||
|
||||
5. **Rate Limiting**: Configure rate limiting in Caddy if needed to prevent abuse
|
||||
|
||||
---
|
||||
|
||||
## Backup and Restore
|
||||
|
||||
### Backup
|
||||
|
||||
```bash
|
||||
# Backup configuration
|
||||
tar -czf searxng-backup-$(date +%Y%m%d).tar.gz \
|
||||
/opt/searxng-docker/searxng \
|
||||
/opt/searxng-docker/docker-compose.yaml
|
||||
|
||||
# Backup volumes (optional, for cache)
|
||||
docker run --rm -v searxng-docker_searxng-data:/data -v $(pwd):/backup \
|
||||
alpine tar czf /backup/searxng-data-$(date +%Y%m%d).tar.gz /data
|
||||
```
|
||||
|
||||
### Restore
|
||||
|
||||
```bash
|
||||
# Extract configuration
|
||||
tar -xzf searxng-backup-YYYYMMDD.tar.gz -C /
|
||||
|
||||
# Restore volumes (if needed)
|
||||
docker run --rm -v searxng-docker_searxng-data:/data -v $(pwd):/backup \
|
||||
alpine tar xzf /backup/searxng-data-YYYYMMDD.tar.gz -C /
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### Check Container Status
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
docker compose logs --tail=50 searxng
|
||||
```
|
||||
|
||||
### View Resource Usage
|
||||
|
||||
```bash
|
||||
docker stats searxng redis
|
||||
```
|
||||
|
||||
### Update Containers
|
||||
|
||||
```bash
|
||||
cd /opt/searxng-docker
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
docker compose image prune -f # Remove old images
|
||||
```
|
||||
|
||||
### Clear Cache
|
||||
|
||||
```bash
|
||||
docker compose exec searxng rm -rf /var/cache/searxng/*
|
||||
docker compose restart searxng
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container Won't Start
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
docker compose logs searxng
|
||||
|
||||
# Verify configuration
|
||||
docker compose config
|
||||
|
||||
# Test connectivity
|
||||
docker compose exec searxng ping -c 3 redis
|
||||
```
|
||||
|
||||
### Can't Access from Caddy VM
|
||||
|
||||
1. Verify port binding: `docker compose ps` should show `0.0.0.0:8080->8080/tcp`
|
||||
2. Test from Caddy VM: `curl http://10.0.10.70:8080/`
|
||||
3. Check firewall rules on LXC
|
||||
4. Verify network connectivity: `ping 10.0.10.70` from Caddy VM
|
||||
|
||||
### SearXNG Returns Errors
|
||||
|
||||
1. Check Redis connectivity: `docker compose exec searxng ping redis`
|
||||
2. Verify settings.yml syntax: `docker compose exec searxng cat /etc/searxng/settings.yml`
|
||||
3. Check logs for specific errors: `docker compose logs searxng | grep -i error`
|
||||
|
||||
### Performance Issues
|
||||
|
||||
1. Monitor resource usage: `docker stats`
|
||||
2. Check Redis performance: `docker compose exec redis valkey-cli info stats`
|
||||
3. Consider increasing LXC resources if needed
|
||||
4. Review search engine timeouts in settings.yml
|
||||
|
||||
---
|
||||
|
||||
## Privacy and Usage Notes
|
||||
|
||||
- **SearXNG is a metasearch engine**: It queries multiple search providers and aggregates results, instead of you talking to each provider directly
|
||||
- **Self-hosting means**:
|
||||
- No profiling or tracking by the instance; logs can be disabled or tightly controlled
|
||||
- You no longer have to trust a random public SearXNG operator with your search queries
|
||||
- **Intended use in this homelab**:
|
||||
- Default search engine in browsers via `https://search.levkin.ca`
|
||||
- Potential backend for LLM tools or scripts that need web search via SearXNG's APIs
|
||||
|
||||
---
|
||||
|
||||
## API Usage
|
||||
|
||||
SearXNG provides a JSON API for programmatic access:
|
||||
|
||||
```bash
|
||||
# Example search query
|
||||
curl "https://search.levkin.ca/search?q=example&format=json" | jq
|
||||
|
||||
# RSS feed
|
||||
curl "https://search.levkin.ca/search?q=example&format=rss"
|
||||
```
|
||||
|
||||
Useful for automation, LLM integrations, or custom search tools.
|
||||
|
||||
---
|
||||
|
||||
## Resource Limits
|
||||
|
||||
Recommended LXC container resources:
|
||||
|
||||
- **CPU**: 1-2 cores
|
||||
- **RAM**: 2-4 GB
|
||||
- **Disk**: 10-20 GB (for cache and logs)
|
||||
- **Network**: Standard bridge connection
|
||||
|
||||
Adjust based on usage patterns and concurrent users.
|
||||
Loading…
x
Reference in New Issue
Block a user