Hamel Husain hamelsmu

Why `torchserve` is great in the age of LLMs

I previously wrote about TorchServe as nice way to serve models.

There are a plethora of ways to optimize LLM models for inference, such as quantization, Paged Attention, kernel fusion, and other compilation techniques. The list of these keeps expanding as the demand for serving OSS LLMs increases. There is a groundswell of demand for serving OSS LLMs, because many organizations have tried OpenAI/Anthropic/Palm, but want a a solution they can control

This is why TorchServe is nice, you can use any framework you want in the custom handler, such as GPTQ, transformers, vLLM, etc (as long as there is a python client). With TorchServe, there is enough f

	> Entering new SmartLLMChain chain...
	Prompt after formatting:
	I have a 12 liter jug and a 6 liter jug.I want to measure 6 liters. How do I do it?
	Idea 1:
	1. Fill the 12 liter jug completely.
	2. Pour the contents of the 12 liter jug into the 6 liter jug. This will leave you with 6 liters in the 12 liter jug.
	3. Empty the 6 liter jug.
	4. Pour the remaining 6 liters from the 12 liter jug into the now empty 6 liter jug.
	5. You now have 6 liters in the 6 liter jug.
	Idea 2:

	# # Fast inference with vLLM (Llama 2 13B)
	#
	# In this example, we show how to run basic inference, using [`vLLM`](https://github.com/vllm-project/vllm)
	# to take advantage of PagedAttention, which speeds up sequential inferences with optimized key-value caching.
	#
	# `vLLM` also supports a use case as a FastAPI server which we will explore in a future guide. This example
	# walks through setting up an environment that works with `vLLM ` for basic inference.
	#
	# We are running the Llama 2 13B model here, and you can expect 30 second cold starts and well over 100 tokens/second.
	# The larger the batch of prompts, the higher the throughput. For example, with the 60 prompts below,

Hamel Husain hamelsmu

Why torchserve is great in the age of LLMs

Why `torchserve` is great in the age of LLMs