Ollama + Lite LLM + CrewAI

.env

DOMAIN=mydomain.com
STORAGE_ROOT=/mnt/data/ai

config.yaml

litellm_settings:
  drop_params: True
model_list:
  - model_name: llama3.1 # name of the model hitting the /chat/completions endpoint
    litellm_params:
      model: ollama/llama3.1
      api_base: http://host.docker.internal:11434
  - model_name: mxbai-embed-large
    litellm_params:
      model: ollama/mxbai-embed-large
      api_base: http://host.docker.internal:11434

crew.py

from crewai import Agent, Task, Crew
from crewai.process import Process
from langchain_community.chat_models import ChatOllama


# we can just import and create directly (no litellm required)
llama31 = ChatOllama(temperature=0, model="llama3.1", base_url="http://localhost:11434")

...

info_agent = Agent(
    role="Information Agent",
    goal="Give compelling information about a certain topic",
    backstory=dedent(
        """
        You love to know information. People love and hate you for it. You win most 
        of the quizzes at your local pub.
    """
    ),
    tools=[search_tool],
    max_iter=5,
    llm=llama31,
    cache=True,
)

...

# Now we use litellm as an 'openai' provider but with a different api_base for the embeddings
crew = Crew(
    agents=[info_agent],
    tasks=[task1],
    verbose=True,
    process=Process.hierarchical,
    manager_llm=llama31,
    memory=True,
    embedder=dict(
        provider="openai",
        config=dict(
            model="mxbai-embed-large",
            api_key="ollama",
            api_base="http://localhost:1234/v1"
        )
    )
)

result = crew.kickoff()

docker-compose.yaml

services:

  litellm:
    profiles:
      - chat
    image: ghcr.io/berriai/litellm:main-latest
    container_name: openai
    hostname: openai.${DOMAIN:-localhost}
    ports:
      - "1234:4000"
    env_file: .env
    volumes:
      # Configs
      - "${STORAGE_ROOT:-.}/services/litellm/config.yaml:/app/config.yaml:ro"
    command: >
      --config /app/config.yaml --detailed_debug
    extra_hosts:
      - host.docker.internal:host-gateway

overview.md

Ollama + Lite LLM + CrewAI

A quick summary of how to integrate Ollama (model serving), Lite LLM (OpenAI compatible proxy), and Crew AI (agents).