Richard Liaw richardliaw

Ray Data LLM

High-performance batch inference for large language models, powered by Ray Data.

Ray Data LLM provides an efficient, scalable solution for batch processing LLM inference workloads with:

High Throughput: Optimized performance using vLLM's paged attention and continuous batching
Distributed Processing: Scale across multiple GPUs and machines using Ray Data

Here are the full supported engine configurations:

model_id: <HF model ID or local model path>
llm_engine: vllm
accelerator_type: <GPU type>

High-performance batch inference for large language models, powered by Ray Data.

Ray Data LLM provides an efficient, scalable solution for batch processing LLM inference workloads with:

High Throughput: Optimized performance using vLLM's paged attention and continuous batching
Distributed Processing: Scale across multiple GPUs and machines using Ray Data

A high-performance batch inference library for Large Language Models (LLMs) powered by vLLM and Ray.

vLLM-Batch enables efficient, large-scale batch processing of LLM inference workloads with:

High Throughput: Optimized performance using vLLM's paged attention and continuous batching
Distributed Processing: Scale across multiple GPUs and machines using Ray Data

High-throughput batch inference for LLMs using vLLM.

from vllm_batch import BatchProcessor, LLMConfig, PromptConfig

# Basic usage

	#!/usr/bin/env python3
	"""
	vLLM multi-node deployment script.
	Automatically handles Ray cluster setup and vLLM server launch.

	This script simplifies vLLM multi-node deployment by automatically handling the Ray cluster
	setup and vLLM server launch based on the current node's role, eliminating the need for
	multiple terminals and manual Ray cluster management.

	Usage Examples:

	import argparse
	from datetime import datetime, timedelta
	from typing import Dict

	import numpy as np
	import pandas as pd
	# from benchmark import Benchmark

	import ray

	#!/usr/bin/env python3
	"""
	dataloader_benchmark.py - A script to benchmark Ray Data performance on image datasets.
	"""
	import ray
	import argparse
	import os
	import time
	import torch
	import torch.utils.data

	#!/usr/bin/env python3
	"""
	dataloader_benchmark.py - A script to benchmark PyTorch DataLoader performance on image datasets.
	"""

	import argparse
	import os
	import time
	import torch
	import torch.utils.data

	#!/usr/bin/env python3
	"""
	dataloader_benchmark.py - A script to benchmark PyTorch DataLoader performance on image datasets.
	"""

	import argparse
	import os
	import time
	import torch
	import torch.utils.data