OpenAI Token Counting.ipynb

Great!! How did you find out FUNCTION_OVERHEAD? I wonder where it says publicly using this prompts for the function calling

See In [58]. The model returned it.

Thanks for great help.

Great work

Neat!

Thank you very much!

---

a few updates:

...
# `json.dumps(o, ensure_ascii=False)` to support Japanese and Chinese.
def format_enum(schema, indent):
    return " | ".join(json.dumps(o, ensure_ascii=False) for o in schema["enum"])
...

# Up to December 16, 2023, after multiple tests, the FUNCTION_OVERHEAD should be 12.
FUNCTION_OVERHEAD = 12
...

I applied your ensure_ascii=False change, but I disagree about your FUNCTION_OVERHEAD change. Instead, OpenAI appears to now be formatting integers as "1" instead of "1.0".

    def format_default(schema):
        v = schema["default"]
        if schema["type"] == "number":
            return f"{v:.0f}" if float(v).is_integer() else str(v)
        else:
            return str(v)

The gist is updated with these changes (tests still pass).

Amazing work! The latest gpt-3.5-turbo and gpt-4-turbo add support for parallel tool calls by injecting an extra tool. Here is the namespace and description I obtained from the OpenAI's API:
(continuation from the normal functions namespace)

// namespace functions

## multi_tool_use

// This tool serves as a wrapper for utilizing multiple tools. Each tool that can be used must be specified in the tool sections. Only tools in the functions namespace are permitted.
// Ensure that the parameters provided to each tool are valid according to that tool's specification.
namespace multi_tool_use {

// Use this function to run multiple tools simultaneously, but only if they can operate in parallel. Do this even if the prompt suggests using the tools sequentially.
type parallel = (_: {
// The tools to be executed in parallel. NOTE: only functions tools are permitted
tool_uses: {
// The name of the tool to use. The format should either be just the name of the tool, or in the format namespace.function_name for plugin and function tools.
recipient_name: string,
// The parameters to pass to the tool. Ensure these are valid according to the tool's own specifications.
parameters: object,
}[],
}) => any;

I've updated the notebook to use the new tool calling interface and support the parallel tool calling option. Notable changes:

• Validations are now exposed to the model (minimum, maximum, pattern).
• Enums are no longer exposed to the model (note: it's still possible that OpenAI supports them through controlled generation, but untested)
• Type titles are now exposed to the model. If you are autogenerating the schema title from the field name, this is wasting tokens.

One interesting note is that the overhead of the parallel tool calls doesn't seem to be reflected in the prompt usage value.

https://openai.com/index/introducing-structured-outputs-in-the-api/

Any changes from this?

I've updated the notebook to use the new tool calling interface and support the parallel tool calling option. Notable changes:

• Validations are now exposed to the model (minimum, maximum, pattern).
• Enums are no longer exposed to the model (note: it's still possible that OpenAI supports them through controlled generation, but untested)
• Type titles are now exposed to the model. If you are autogenerating the schema title from the field name, this is wasting tokens.

One interesting note is that the overhead of the parallel tool calls doesn't seem to be reflected in the prompt usage value.

Thanks for this wonderful work and details regarding the tool calling interface.

May you share some insights why OpenAI does not expose Enum to model? is it because the performance is not desirable when providing Enum items?
I noticed that "features" and "content" in send_message_tool are not exposed to the model even they are required. Any insights regarding that?
Thank you!

I don't work at OpenAI; I just documented observed behavior.

You're right about "features" and "content". I guess because they are "object" types, and they only allow a single flat object to be provided. Note: behavior may have changed since the notebook was last updated, I would advise verifying before implementing in your stack.