Skip to content

Instantly share code, notes, and snippets.

View Attunewise's full-sized avatar

Attunewise

View GitHub Profile
@Attunewise
Attunewise / countTokens.js
Last active August 5, 2024 03:32
Counts ChatGPT tokens
const { GPT4Tokenizer } = require('gpt4-tokenizer')
const json = require('./conversations.json')
const tokenizer = new GPT4Tokenizer({type:'gpt-3'})
const nodes = []
for (const row of json) {
let id = row.current_node
const { mapping } = row
while (id) {
const node = mapping[id]
if (node.message) {
@Attunewise
Attunewise / ConvertFunctions.py
Last active April 24, 2024 01:44
Convert OpenAI functions to system prompt that works with other LLMs
# Converts OpenAI compatible function calling JSON schema to a prompt that instructs the LLM to return
# a JSON object that is a choice of a function call conforming to one of the functions or a message reply
def convert_schema_to_typescript(schema):
if not schema:
return 'any'
if '$ref' in schema:
return schema['$ref'].replace('#/definitions/', '')