This is a quick guide that demonstrates how to build a VLLM container locally.
First clone the VLLM branch/fork you would like to build:
git clone https://github.com/theobjectivedad/vllm.git src
cd src
git checkout outlines_subschemas
#!/bin/bash | |
# Copyright (c) 2025, The Objective Dad | |
# | |
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and | |
# associated documentation files (the “Software”), to deal in the Software without restriction, | |
# including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, | |
# and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, | |
# subject to the following conditions: | |
# | |
# The above copyright notice and this permission notice shall be included in all copies or substantial |
from typing import List, Optional | |
from mcp.types import ( | |
EmbeddedResource, | |
ImageContent, | |
TextContent, | |
) | |
from mcp_agent import PromptMessageMultipart, RequestParams, TextContent | |
from mcp_agent.agents.workflow.chain_agent import ChainAgent | |
from mcp_agent.core.prompt import Prompt |
This is an ugly and minimally tested workaround for folks who would like to use PydanticAI response model formatting with VLLM prior to 13002 being merged into VLLM. I've tested this avainst PydanticAI 0.0.46:
First apply this monkeypatch to PydanticAI. Basically it allows agent runs to specify named tools on model settings:
# 2025-03-30 Patched 0.0.46
# pylint: disable=protected-access
async def patched_completions_create(
This is a monkeypatch to workaround AutoGPTQ issue 459:
import os
import time
from logging import getLogger
import torch
import torch.nn as nn
import transformers
Quick instrustions on how to run miqu in the aphrodite-engine.
STEP 1: Convert GGUF to PyTorch format per documentation. I used the aphrodite-engine container to avoid setting up dependencies.
docker run --gpus=all -it --rm -v /models:/models -v `pwd`:/workspace alignmentlabai/aphrodite-engine:latest bash
This is the docker image I am using for Axolotl R&D. You may need to tweak TORCH_CUDA_ARCH_LIST
to include your GPU architecture. There are several improvments over the official axolotl image including:
what is the nature of reality?
The nature of reality is a complex and deeply debated philosophical question. Generally speaking, it refers to the nature of fundamental concepts such as existence, time, space, consciousness and so on. Different schools of thought have different answers, but the most widely accepted view is that reality is composed of physical, mental, and spiritual aspects that interact and shape our experience of the world.
List the top 10 business capabilities for sustainability materials audit. Include a detailed description for each and how each capabilities adds business value.
What are the 10 top service providers that offer sustainability certification validation services via an API. For example I want to validate that a material supplier's products are indeed 100% certified organic cotton. For each provider include distinguishing features, website URL, advantages/disadvantages, and industry standards they adhere to.