Venkat2811

RADAR

Things that i'm looking to or in active backlog for background agents.

Fabrics

High performance distributed networking fabrics

UCCL

Venkat2811

Key OSS Contribution Highlights

---

LLM Inference Systems

All things LLM Inference

SGLang

• Websocket response api support in sglang router, following openai's wss response api, 15-55% improvement latency in multi-turn chat, tool use and agentic workloads

Venkat2811

Intro

The goal was to achieve Speed-of-Light SW efficiency by building ultra low-latency primitives.

6 months of 14 hr work days with AI assisted R&D. 2 months of unsuccessful fundraising attempt. Attempted to build a venture scale company around these ideas. Work will be cleaned up and OSS'd:

---

Myelon - ZERO COPY SHM Based IPC Library – Inference Engine <> GPU worker process

• Rust shared-memory IPC (1M msgs/sec @ p99.9 ≈7µs, 1kb) that speeds up my vllm.rs prefill by 30% (not vllm).
• 80% Speed-of-Light (CPU cache electrical signal) in AMD Ryzen 7 5800X. Works on all CPUs & UNIX systems. This is CO-aware latency measurement, not raw throughput. More Details

Venkat2811

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

Venkat2811

INTRO

Concurrency is a domain I have wanted to explore for a long time because the locks and the race conditions have always intimidated me. I recall somebody suggesting concurrency patterns in golang because they said "you share the data and not the variables".

Amused by that, I searched for "concurrency in golang" and bumped into this awesome slide by Rob Pike: https://talks.golang.org/2012/waza.slide#1 which does a great job of explaining channels, concurrency patterns and a mini-architecture of load-balancer (also explains the above one-liner).

Let's dig in:

Goroutines

Venkat2811

Youtube channels:

1. Gaurav Sen
2. Rachit Jain
3. Akshay Saini
4. Tushar Roy
5. mycodeschool
6. Saurabh School
7. Keerthi Purswani
8. Nick White
9. Back To Back SWE

Venkat2811

Common Options

-#, --progress-bar Make curl display a simple progress bar instead of the more informational standard meter.

-b, --cookie <name=data> Supply cookie with request. If no =, then specifies the cookie file to use (see -c).

-c, --cookie-jar <file name> File to save response cookies to.

Venkat2811

#!/bin/bash

i="1"
echo "Running Script to kill all $1 processes"

while [[ $i -lt 100 ]]; do
	PID=`ps -ef | grep $1 | grep -v 'grep' | awk '{print $2}'`
	if [[ "" !=  "$PID" ]]; then
		echo "killing $PID"	
		kill -9 $PID

Venkat2811

# coding=utf-8
from StringIO import StringIO
from datetime import timedelta

from rx import Observer
from rx.concurrency import TwistedScheduler
from rx.disposables import Disposable
from rx.subjects import Subject
from twisted.internet.protocol import Factory, Protocol, connectionDone
from twisted.internet.endpoints import TCP4ServerEndpoint