Using ollama 1.24 OpenAI API with DSPY - WORKAROUND

dspy_ollama_1-24.py

# install DSPy: pip install dspy
import dspy

# Ollam 1.24 is now compatible with OpenAI APIs
# But DSPy has hard coded some logic around the names of the OpenAI models
# This is a workaround for now.
# You have to create a custom Ollama Model and use the name 'gpt-3.5-turbo' or at least 'gpt-3.5'
# 
# Here is the DSPy code that is causing the issue:
# 
# dspy/dsp/modules/gpt3.py
#
# default_model_type = (
#             "chat"
#             if ("gpt-3.5" in model or "turbo" in model or "gpt-4" in model)
#             and ("instruct" not in model)
#             else "text"
#         )
#
# So this appears to be a hard coded check 'chat' or 'text' based on the model name.
#
# The trick is to name your custom ollama model 'gpt-3.5-turbo` (insert laughing emoji here) 
#
# The results aren't great but I haven't tried too many models yet. 
#
# Here is the model file I am using: 
#
# FROM mistral:7b-instruct-v0.2-q6_K
# PARAMETER temperature .1
# PARAMETER num_ctx 4096
# 
# And you create the ollama model with: `ollama create -n gpt-3.5-turbo -f <path to modfile>`
#

ollama_model = dspy.OpenAI(api_base='http://localhost:11434/v1/', api_key='ollama', model='gpt-3.5-turbo', stop='\n\n')

# This sets the language model for DSPy.
dspy.settings.configure(lm=ollama_model)

# This is not required but it helps to understand what is happening
my_example = {
    "question": "What system was Super Mario Brothers made for?",
    "answer": "NES",
}

# This is the signature for the predictor. It is a simple question and answer model.
class BasicQA(dspy.Signature):
    """Answer questions about classic video games."""

    question = dspy.InputField(desc="a question about classic video games")
    answer = dspy.OutputField(desc="often between 1 and 5 words")

# Define the predictor.
generate_answer = dspy.Predict(BasicQA)

# Call the predictor on a particular input.
pred = generate_answer(question=my_example['question'])

# Print the answer...profit :)
print(pred.answer)

I'm a big fan of mistral q8, but regardless you will do better with the proper chat template. (probably why you need the \n\n) here's my modelfile. I don't use system-prompt because it's better for instruct tasks without... but I still leave the <s></s> because it expects to see those.

TEMPLATE """
<s>{{ if .System  }}[INST]{{ .System }}[/INST]{{ end }}</s>[INST] {{ .Prompt }} [/INST]
"""
SYSTEM """"""
PARAMETER num_ctx 8000
PARAMETER num_gpu -1
PARAMETER num_predict 4000