Ankur Dhuriya ankurdhuriya

The Ultimate Guide to LLM Inference Optimization: Speeding up the Generation Process

Large Language Models (LLMs) have revolutionized the AI landscape, but serving them efficiently at scale remains a massive computational challenge. The autoregressive nature of text generation—where tokens are generated one by one—places an immense burden on both computation (FLOPs) and memory bandwidth. To make real-world deployment practical, researchers have developed an array of brilliant optimization techniques.

In this blog post, we will break down the core optimizations that power modern LLM inference, exploring them in this specific order: KV Cache, PagedAttention, FlashAttention, Continuous Batching, Quantization, and Speculative Decoding.

Twitter Unlike Bot Script

This script automates the process of unliking tweets on Twitter using browser JavaScript console. It repeatedly finds and clicks the "unlike" button on tweets until no more are found, up to a maximum count.

Features

Dynamically scrolls to load more tweets if none are immediately available for unliking.
Uses adjustable wait times to balance speed and avoid Twitter's rate limits.
Logs progress to the console for monitoring.
Stops gracefully when no more unlikes are found or maximum count is reached.

	"""
	In this example, we're configuring Celery to use heartbeats.
	We've set the worker_heartbeat option to 120 seconds, which means
	that each worker will send a heartbeat message to the broker every 2 minutes.
	This can help to ensure that tasks are not lost if a worker goes offline or crashes.
	"""
	from celery import Celery

	app = Celery('tasks', broker='pyamqp://guest@localhost//')

	"""
	In this example, we're configuring Celery to prefetch 10 tasks at a time.
	This means that each worker will load up to 10 tasks into its memory before
	they are executed. This can help to reduce the time it takes to fetch new
	tasks from the broker, and can improve overall performance.
	"""
	from celery import Celery

	app = Celery('tasks', broker='pyamqp://guest@localhost//')

	"""
	In this example, we're configuring Celery to use threads for concurrency.
	We've set the task_concurrency option to 4, which means that each worker
	can handle up to 4 tasks simultaneously. We've also set the worker_prefetch_multiplier
	option to 1, which means that each worker will only prefetch one task at a time.
	"""

	from celery import Celery

	app = Celery('tasks', broker='pyamqp://guest@localhost//')

	# https://www.redblobgames.com/grids/hexagons/ "double-height" horizontal layout
	import random
	import math

	class Cell():
	""" Hexagonal cell """
	def __init__(self, x, y, walls=[0,1,2,3,4,5]):
	""" x,y int
	walls : list of number 0..5
	"""

	import argparse
	import librosa
	import numpy as np
	import os
	import scipy
	import scipy.io.wavfile
	import sys

	from glob import glob
	from tqdm import tqdm