nanobot/SETUP.md

Nanobot Setup Guide

This guide will help you set up nanobot on a fresh system, pulling from the repository and configuring it to use Ollama and AirLLM with Llama models.

Prerequisites

• Python 3.10 or higher
• Git
• (Optional) CUDA-capable GPU for AirLLM (recommended for better performance)

Step 1: Clone the Repository

git clone <repository-url>
cd nanobot

If you're using a specific branch (e.g., the cleanup branch):

git checkout feature/cleanup-providers-llama-only

Step 2: Create Virtual Environment

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Step 3: Install Dependencies

pip install --upgrade pip
pip install -e .

If you plan to use AirLLM, also install:

pip install airllm bitsandbytes

Step 4: Choose Your Provider Setup

You have two main options:

Option A: Use Ollama (Easiest, No Tokens Needed)

1. Install Ollama (if not already installed):

   # Linux/Mac
   curl -fsSL https://ollama.ai/install.sh | sh
   
   # Or download from: https://ollama.ai
   

2. Pull a Llama model:

   ollama pull llama3.2:latest
   

3. Configure nanobot:

   mkdir -p ~/.nanobot
   cat > ~/.nanobot/config.json << 'EOF'
   {
     "providers": {
       "ollama": {
         "apiKey": "dummy",
         "apiBase": "http://localhost:11434/v1"
       }
     },
     "agents": {
       "defaults": {
         "model": "llama3.2:latest"
       }
     }
   }
   EOF
   chmod 600 ~/.nanobot/config.json
   

Option B: Use AirLLM (Direct Local Inference, No HTTP Server)

1. Get Hugging Face Token (one-time, for downloading gated models):

   • Go to: https://huggingface.co/settings/tokens
   • Create a new token with "Read" permission
   • Copy the token (starts with hf_)

2. Accept Llama License:

   • Go to: https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct
   • Click "Agree and access repository"
   • Accept the license terms

3. Download Llama Model (one-time):

   # Install huggingface_hub if needed
   pip install huggingface_hub
   
   # Download model to local directory
   huggingface-cli download meta-llama/Llama-3.2-3B-Instruct \
     --local-dir ~/.local/models/llama3.2-3b-instruct \
     --token YOUR_HF_TOKEN_HERE
   

4. Configure nanobot:

   mkdir -p ~/.nanobot
   cat > ~/.nanobot/config.json << 'EOF'
   {
     "providers": {
       "airllm": {
         "apiKey": "/home/YOUR_USERNAME/.local/models/llama3.2-3b-instruct",
         "apiBase": null,
         "extraHeaders": {}
       }
     },
     "agents": {
       "defaults": {
         "model": "/home/YOUR_USERNAME/.local/models/llama3.2-3b-instruct"
       }
     }
   }
   EOF
   chmod 600 ~/.nanobot/config.json
   

   Important: Replace YOUR_USERNAME with your actual username, or use ~/.local/models/llama3.2-3b-instruct (the ~ will be expanded).

Step 5: Test the Setup

nanobot agent -m "Hello, what is 2+5?"

You should see a response from the model. If you get errors, see the Troubleshooting section below.

Step 6: (Optional) Use Setup Script

Instead of manual configuration, you can use the provided setup script:

python3 setup_llama_airllm.py

This script will:

• Guide you through model selection
• Help you configure the Hugging Face token
• Set up the config file automatically

Configuration File Location

• Path: ~/.nanobot/config.json
• Permissions: Should be 600 (read/write for owner only)
• Backup: Always backup before editing!

Available Providers

After setup, nanobot supports:

• Ollama: Local OpenAI-compatible server (no tokens needed)
• AirLLM: Direct local model inference (no HTTP server, no tokens after download)
• vLLM: Local OpenAI-compatible server (for advanced users)
• DeepSeek: API or local models (for future use)

Recommended Models

For Ollama:

• llama3.2:latest - Fast, minimal memory (recommended)
• llama3.1:8b - Good balance
• llama3.1:70b - Best quality (needs more GPU)

For AirLLM:

• meta-llama/Llama-3.2-3B-Instruct - Fast, minimal memory (recommended)
• meta-llama/Llama-3.1-8B-Instruct - Good balance
• Local path: ~/.local/models/llama3.2-3b-instruct (after download)

Troubleshooting

"Model not found" error (AirLLM)

• Make sure you've accepted the Llama license on Hugging Face
• Verify your HF token has read permissions
• Check that the model path in config is correct
• Ensure the model files are downloaded (check ~/.local/models/llama3.2-3b-instruct/)

"Connection refused" error (Ollama)

• Make sure Ollama is running: ollama serve
• Check that Ollama is listening on port 11434: curl http://localhost:11434/api/tags
• Verify the model is pulled: ollama list

"Out of memory" error (AirLLM)

• Try a smaller model (Llama-3.2-3B-Instruct instead of 8B)
• Use compression: set apiBase to "4bit" or "8bit" in the airllm config
• Close other GPU-intensive applications

"No API key configured" error

• For Ollama: Use "dummy" as apiKey (it's not actually used)
• For AirLLM: No API key needed for local paths, but you need the model files downloaded

Import errors

• Make sure virtual environment is activated
• Reinstall dependencies: pip install -e .
• For AirLLM: pip install airllm bitsandbytes

Using Local Model Paths (No Tokens After Download)

Once you've downloaded a model locally with AirLLM, you can use it forever without any tokens:

{
  "providers": {
    "airllm": {
      "apiKey": "/path/to/your/local/model"
    }
  },
  "agents": {
    "defaults": {
      "model": "/path/to/your/local/model"
    }
  }
}

The model path should point to a directory containing:

• config.json
• tokenizer.json (or tokenizer_config.json)
• Model weights (model.safetensors or pytorch_model.bin)

Next Steps

• Read the main README.md for usage examples
• Check nanobot --help for available commands
• Explore the workspace features: nanobot workspace create myproject

Getting Help

• Check the repository issues
• Review the code comments
• Test with a simple query first: nanobot agent -m "Hello"