Big Boss 0xBigBoss

Run vLLM in Distributed Mode with Ray

Prerequisites

A docker image with the vLLM server installed.

export DOCKER_IMAGE=docker.io/fxnlabs/vllm-openai
# or use the following if you want to use the latest version

Demo a Simple LLM

A simple LLM and tokenizor for demonstrating KV Cache and inference using the transformer architecture(generating text).

Need PyTorch installed.

$ python ./demo_llm.py
Demonstrating KV cache

VS Code Extensions

Frontend

naumovs.color-highlight
bierner.color-info
pranaygp.vscode-css-peek
dbaeumer.vscode-eslint
ms-playwright.playwright
Orta.vscode-twoslash-queries

Unified Activity Table

A more generic table that can be used to capture all events, onchain and offchain, for the user. For example, there may be a user referral tracked inside the database that should show up in the activity table. The table should be generic enough to be used for any event that is emitted by the send account contract or to represent any other event that a user may want to track and see in their activity feed.

The view will be adapted so that it can be added to the activity table.

create table activity (
    id serial primary key,
    event_name text not null, -- the name of the event usually the integration or source table name

	async function callDebugRpcMethod(method: string, params: any[]) {
	const response = await fetch("http://localhost:3030/rpc", {
	headers: {
	"content-type": "application/json",
	},
	referrer: "http://localhost:3000/",
	body: JSON.stringify({
	jsonrpc: "2.0",
	id: 1,
	method: method,

	name: pytorch
	channels:
	- https://repo.anaconda.com/pkgs/main
	- https://repo.anaconda.com/pkgs/r
	dependencies:
	- python=3.12.8
	- cmake
	- ninja
	- libjpeg-turbo
	- libpng

	macos-option-as-alt = true
	keybind = cmd+right=text:\x05
	keybind = cmd+left=text:\x01
	keybind = alt+left=esc:b
	keybind = alt+right=esc:f

	#!/bin/bash

	set -eo pipefail

	user_input="$*"

	if [[ -z "$user_input" ]]; then
	user_input="beginner shell commands"
	echo "Using default input: $user_input"
	fi


	def Attention(X, W_q, W_k, W_v):
	Q = X @ W_q
	K = X @ W_k
	V = X @ W_v
	# Q, K, V: [b, n, d]
	s = 1 / sqrt(d)
	A = Q @ K.transpose(−1,−2) ∗ s
	return softmax(A) @ V

	#!/bin/bash

	# create a prefix based on the parent git repo directory name
	prefix=$(basename "$(git rev-parse --show-toplevel)")

	# Create a directory to store the flattened files
	outdir="/tmp/flattened"
	mkdir -p "$outdir"

	# Find c++ files, rename them, and copy to flattened directory