Deploying LLMs with Ollama & Docker

Running LLMs locally ensures complete data privacy, offline capabilities, and zero API costs.

Ollama is a lightweight local inference framework that runs models efficiently on consumer hardware. We containerize it with Docker to enable cloud deployment.

Step 1: Docker Compose Configuration

Create a `docker-compose.yml` file supporting GPU access (CUDA for Nvidia cards):

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_storage:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama_storage:

---

Step 2: Ingesting Models

Start the containers and pull your model (e.g. Llama 3 8B):

docker compose up -d
docker exec -it ollama ollama run llama3
`/bash
``

---

## Step 3: Accessing via API

Once deployed, Ollama exposes a standard HTTP API compatible with client libraries:

const response = await fetch("http://localhost:11434/api/generate", {

method: "POST",

body: JSON.stringify({

model: "llama3",

prompt: "Why is the sky blue?",

stream: false

})

});

const data = await response.json();

console.log(data.response);