Build Your Private AI Router: Smart Model Routing with LiteLLM + Ollama, 100% Offline

Once you've spent some time running local-first AI on your own hardware, you'll inevitably end up with a growing collection of LLMs, each suited to a different job: a lightweight model like Llama 3.2 (3B) for quick chat responses, a deep-reasoning workhorse like DeepSeek-R1 for complex problem-solving, or Llama 3.2 Vision for reading images and PDFs.

The problem? You're constantly swapping model names across your apps—Obsidian, VS Code, Open WebUI, Python scripts—every time the task changes. It's tedious, error-prone, and wastes cycles.

In this guide, we'll solve that by building a Private AI Router on your own machine using LiteLLM alongside Ollama. The result: a single unified endpoint that intelligently routes each request to the best local model—automatically, securely, and 100% offline.

---

### What Is a Private AI Router, and Why Does Local AI Need One?

A Private AI Router acts as an AI gateway (or proxy) that intercepts every request from your applications and decides which local model should handle it.

Think of it as a personal dispatcher:

Quick summaries / short-form chat: routed to a lightweight model that responds in seconds.
Deep reasoning / code generation: routed to a reasoning model that thinks step by step.
Image and document analysis: routed to a vision model automatically.

The key benefit: every app on your machine connects to the same endpoint (http://localhost:4000/v1). No more hunting through settings panels to change model names.

---

### Prerequisites

1. Ollama: installed and running locally (http://localhost:11434).

2. Models pulled into Ollama: download the models you want to route to, e.g.:

```bash

ollama pull llama3.2:3b

ollama pull deepseek-r1:14b

ollama pull llama3.2-vision:11b

```

3. Python (3.10+): required to install and run the LiteLLM package.

---

### Step 1: Install the LiteLLM Proxy

Open a terminal and install LiteLLM with its proxy server:


pip install "litellm[proxy]"

LiteLLM is an open-source library that normalizes the APIs of dozens of model providers into a single OpenAI-compatible interface. In practice, this means it plugs into virtually every AI-aware tool in the ecosystem.

---

### Step 2: Write the Router Config (config.yaml)

Create a file called config.yaml in your working directory. This is where you define model aliases and the routing rules that point to Ollama:


model_list:
  # 1. Fast tier — lightweight model for everyday tasks
  - model_name: fast-model
    litellm_params:
      model: ollama/llama3.2:3b
      api_base: http://localhost:11434

  # 2. Reasoning tier — deep-thinking model for complex work
  - model_name: reasoning-model
    litellm_params:
      model: ollama/deepseek-r1:14b
      api_base: http://localhost:11434

  # 3. Vision tier — image and document understanding
  - model_name: vision-model
    litellm_params:
      model: ollama/llama3.2-vision:11b
      api_base: http://localhost:11434

  # 4. Default fallback — maps a legacy OpenAI name to a local model
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: ollama/llama3.2:3b
      api_base: http://localhost:11434

router_settings:
  routing_strategy: usage-based-routing-v2
  num_retries: 2
  request_timeout: 120

> Pro tip: Aliasing gpt-3.5-turbo to ollama/llama3.2:3b means any legacy app that hard-codes an OpenAI model name will "just work" against your local stack—zero code changes required.

---

### Step 3: Start Your Private AI Router

Launch the proxy server on port 4000:


litellm --config config.yaml --port 4000

Once it's up, your personal AI gateway is live at: http://localhost:4000/v1

---

### Step 4: Test It and Connect Your Apps

#### Quick test with curl

Send a request to the Reasoning tier:


curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "reasoning-model",
    "messages": [{"role": "user", "content": "Explain how a vector database works in two sentences."}]
  }'

#### Connect your local apps (Obsidian / Open WebUI / VS Code)

Update the AI settings in each app to point at your router:

OpenAI API Base URL: http://localhost:4000/v1
API Key: sk-local-anything (any non-empty string works)
Model Name: pick fast-model, reasoning-model, or vision-model—whichever you defined in config.yaml

---

### 3 Advanced Techniques to Level Up Your Router

1. Automatic Fallback Strategy: If your VRAM is saturated while a large model is running, configure config.yaml to gracefully fall back to a smaller model. This keeps your system responsive instead of crashing under memory pressure.

2. Context Window Management: Set max_tokens and context-size limits in LiteLLM to prevent any single model from consuming more RAM than your hardware can handle.

3. Zero Telemetry (True Privacy): Force offline mode and disable all outbound analytics by setting the environment variable DISABLE_TELEMETRY=True. No stats, no pings, no exceptions.

---

### The Bottom Line

Building a complete Private Brain OS isn't just about having AI models on your machine—it's about architecting a system that routes work intelligently, runs smoothly, and stays energy-efficient.

Pairing LiteLLM with Ollama as a Private AI Router lets you pull the right model for the right task without ever touching a settings panel again. And most importantly, every byte of data stays on your hardware—100% local, zero cloud, zero leaks. Your brain, your machine, your rules.

🛠️ Hardware for Running Local AI at Home