richardliaw

vLLM Batch

A high-performance batch inference library for Large Language Models (LLMs) powered by vLLM and Ray.

Overview

vLLM-Batch enables efficient, large-scale batch processing of LLM inference workloads with:

• High Throughput: Optimized performance using vLLM's paged attention and continuous batching
• Distributed Processing: Scale across multiple GPUs and machines using Ray Data

richardliaw

vLLM Batch

High-throughput batch inference for LLMs using vLLM.

Quick Start

from vllm_batch import BatchProcessor, LLMConfig, PromptConfig

# Basic usage

richardliaw

RayLLM-Batch

Engine Configurations

Here are the full supported engine configurations:

model_id: <HF model ID or local model path>
llm_engine: vllm
accelerator_type: <GPU type>

richardliaw

Ray Serve LLM Primitives

Deploy LLMs with full OpenAI API compatibility using Ray Serve primitives.

Features

• 🚀 Automatic scaling and load balancing
• 🔄 Tensor parallelism for large models
• 🎯 JSON mode and function calling
• 📦 Multi-LoRA adapter support

richardliaw

RayLLM-Batch

Engine Configurations

Here are the full supported engine configurations:

model_id: <HF model ID or local model path>
llm_engine: vllm
accelerator_type: <GPU type>

richardliaw

Ray Serve LLM Primitives

Deploy LLMs with full OpenAI API compatibility using Ray Serve primitives.

Features

• 🚀 Automatic scaling and load balancing
• 🔄 Tensor parallelism for large models
• 🎯 JSON mode and function calling
• 📦 Multi-LoRA adapter support

richardliaw

Ray Serve LLM Primitives

Deploy LLMs with full OpenAI API compatibility using Ray Serve primitives.

Features

• 🚀 Automatic scaling and load balancing
• 🔄 Tensor parallelism for large models
• 🎯 JSON mode and function calling
• 📦 Multi-LoRA adapter support

richardliaw

Ray Serve LLM Primitives

Deploy LLMs with full OpenAI API compatibility using Ray Serve primitives.

Features

• 🚀 Automatic scaling and load balancing
• 🔄 Tensor parallelism for large models
• 🎯 JSON mode and function calling
• 📦 LoRA adapter support

richardliaw

from time import perf_counter
from time import sleep
from contextlib import contextmanager

@contextmanager
def catchtime() -> Callable[[], float]:
    t1 = t2 = perf_counter() 
    yield lambda: t2 - t1
    t2 = perf_counter() 

richardliaw

import torch
import ray.data 

class DataGenerator:
    def __init__(self, permute_config):
        device = torch.device("cuda")
        self.model = Model().to(device)
        self.config = permute_config

    def __call__(self, input):