Function calling for RAG Query Rewriting: GPT 3.5 vs Llama 3.1:8B

comparison.md

This is a comparison of calls made to either OpenAI gpt-35-turbo or Ollama Llama3.1 with a function calling request. The geneal goal of the function is to take a user query and turn it into a search query, along with optional price filter or brand filter. See the file below this one for the tools definition.

User question: "Best shoe for hiking"

gpt-35: 😊

{"search_query":"Best shoe for hiking"}

llama3.1: 😊

{"brand_filter":null,"price_filter":null,"search_query":"best shoe for hiking"}'

User question: "Climbing gear cheaper than $30?"

gpt-35: 😊 Includes a valid price_filter

{"search_query":"Climbing gear","price_filter":{"comparison_operator":"<","value":30}}'

llama3.1: 😕 Completely makes up the structure of the price_filter object-typed argument!

{"brand_filter":"{}","price_filter":"{\'max\': 30}","search_query":""}

User question: "Shoes by WildRunner?"

gpt-35: 😊 Includes a valid brand_filter

{"search_query":"Shoes WildRunner","brand_filter":{"comparison_operator":"=","value":"WildRunner"}}

llama3.1: 😕 Doesn't include a brand filter, plus sets them to "None" instead of "null", confusing my parsing.

{"brand_filter":"None","price_filter":"None","search_query":"shoes by wildrunner"}

User question: "Shoes by the brand WildRunner?"

gpt-35: 😊 Includes a valid brand_filter

{"search_query":"WildRunner","brand_filter":{"comparison_operator":"=","value":"WildRunner"}}

llama3.1: 😕 Completely makes up the structure of the object-typed argument!

{"brand_filter":"{\'brand\': \'Wildrunner\'}","price_filter":"{}","search_query":""}

tools.json

[
        {
            "type": "function",
            "function": {
                "name": "search_database",
                "description": "Search PostgreSQL database for relevant products based on user query",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "search_query": {
                            "type": "string",
                            "description": "Query string to use for full text search, e.g. 'red shoes'",
                        },
                        "price_filter": {
                            "type": "object",
                            "description": "Filter search results based on price of the product",
                            "properties": {
                                "comparison_operator": {
                                    "type": "string",
                                    "description": "Operator to compare the column value, either '>', '<', '>=', '<=', '='",  # noqa
                                },
                                "value": {
                                    "type": "number",
                                    "description": "Value to compare against, e.g. 30",
                                },
                            },
                        },
                        "brand_filter": {
                            "type": "object",
                            "description": "Filter search results based on brand of the product",
                            "properties": {
                                "comparison_operator": {
                                    "type": "string",
                                    "description": "Operator to compare the column value, either '=' or '!='",
                                },
                                "value": {
                                    "type": "string",
                                    "description": "Value to compare against, e.g. AirStrider",
                                },
                            },
                        },
                    },
                    "required": ["search_query"],
                },
            },
        }
    ]