hamelsmu

Why torchserve is great in the age of LLMs

I previously wrote about TorchServe as nice way to serve models.

There are a plethora of ways to optimize LLM models for inference, such as quantization, Paged Attention, kernel fusion, and other compilation techniques. The list of these keeps expanding as the demand for serving OSS LLMs increases. There is a groundswell of demand for serving OSS LLMs, because many organizations have tried OpenAI/Anthropic/Palm, but want a a solution they can control

This is why TorchServe is nice, you can use any framework you want in the custom handler, such as GPTQ, transformers, vLLM, etc (as long as there is a python client). With TorchServe, there is enough f

hamelsmu

hamelsmu

hamelsmu

hamelsmu

hamelsmu

hamelsmu

hamelsmu

hamelsmu

hamelsmu

import wandb
import pandas as pd
from fastcore.all import Path

def val_pred_table(run_id, entity='wandb_course', proj='lemon-project'):
    "Get prediction table on the validation set for the lemon project."
    api = wandb.Api()
    path = api.artifact(f'{entity}/{proj}/run-{run_id}-predictions_table:v0').download()
    preds = (Path(path)/'predictions_table.table.json').read_json()
    return pd.DataFrame([{'pred':p, 'label':t}  for _,p,_,t in preds['data']])