Running Local LLMs with Ollama and Open WebUI
“Why rent compute when you can reign over it?”
— AI, after setting up Ollama on my own machine.
Introduction
Large Language Models (LLMs) have become the backbone of modern AI applications. But relying on cloud APIs can be costly, slow, and privacy-compromising. Enter Ollama and Open WebUI—a dynamic duo that lets you run powerful models locally with a sleek interface.
Whether you’re building a chatbot, testing prompts, or just want to flex your AI muscles offline, this guide will walk you through setting up Ollama and Open WebUI on your own machine.
Prerequisites
- Operating System: macOS, Linux, or Windows (WSL recommended)
- Docker: Installed and running
- Git: For cloning repos
- Basic CLI skills:
Step 1: Install Ollama
Ollama is a lightweight runtime for local LLMs. It supports models like LLaMA2, Mistral, and more.
Installation
1
curl -fsSL https://ollama.com/install.sh | sh
Then start the service:
1
ollama run llama2
This will download and run the LLaMA2 model locally. You can swap llama2 with other models like mistral, codellama, or gemma.
Step 2: Set Up Open WebUI
Open WebUI is a beautiful front-end for interacting with Ollama. Think ChatGPT, but local and customizable.
Clone and Run
1
2
3
git clone https://github.com/open-webui/open-webui.git
cd open-webui
docker compose up -d
This spins up the WebUI and connects it to your local Ollama instance.
Access the UI
Open your browser and go to:
1
http://localhost:3000
You’ll be greeted with a slick interface where you can chat with your local models.
Bonus: Load Custom Models
Want to run a custom GGUF model?
1
2
ollama create mymodel -f ./Modelfile
ollama run mymodel
You can define your own Modelfile with parameters and base models. Ollama handles the rest.
Why This Setup Rocks
- Privacy: No data leaves your machine
- Speed: No latency from cloud APIs
- Cost: Free and open-source
- Customization: Tweak models, prompts, and UI to your heart’s content
Final Thoughts
Running LLMs locally used to be a pain. Now, with Ollama and Open WebUI, it’s plug-and-play. Whether you’re building a lore-rich chatbot or testing prompt engineering strategies, this setup gives you full control.