Joshua Lochner xenova

Bringing the power of machine learning to the web. Currently working on Transformers.js

xenova / export-florence-to-onnx.py

Last active February 17, 2025 23:13

	# !pip install --upgrade onnx==1.17.0 onnxruntime==1.20.1 onnxslim==0.1.48 optimum==1.24.0 transformers==4.48.3

	import torch
	import torch.nn as nn
	from transformers import AutoProcessor, AutoModelForCausalLM
	import os
	import onnxslim
	from optimum.onnx.graph_transformations import merge_decoders, check_and_save_model

	model = AutoModelForCausalLM.from_pretrained(

xenova / tiktoken-to-hf.ipynb

Last active December 19, 2024 12:39

Convert tiktoken tokenizers to the Hugging Face tokenizers format

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.