parajain

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

neubig

# NOTE:
# You can find an updated, more robust and feature-rich implementation
# in Zeno Build
# - Zeno Build: https://github.com/zeno-ml/zeno-build/
# - Implementation: https://github.com/zeno-ml/zeno-build/blob/main/zeno_build/models/providers/openai_utils.py

import openai
import asyncio
from typing import Any

parajain

import numpy as np

def log_softmax(x):
    e_x = np.exp(x - np.max(x))
    return np.log(e_x / e_x.sum())

def lognormalize(x):
    a = np.logaddexp.reduce(x)
    return np.exp(x - a)
    

ejmg

\usepackage{multirow,array}
\usepackage{floatrow}
% in combination with floatrow for properly adjusted figures, placeins ensures said figures appear
% where they are declared within the latex code, i.e. in subsection Part I of this paper. For proper
% usage with subsections, MUST use \FloatBarrier command after done with a figure
\usepackage[section]{placeins}
% ensures captions are properly centered under their respective figures
\usepackage[justification=centering]{caption}
% special command for table captions
\newfloatcommand{capbtabbox}{table}[][\FBwidth]

jboner

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD