Nick-Harvey

Introducing the Lambda Inference API: Lowest-Cost Inference Anywhere

Nick Harvey
December 10, 2024 • 19 min read
Think Big, Spend Small - Introducing the Lambda Inference API
Today, we’re excited to announce the GA release of the Lambda Inference API, the lowest-cost inference anywhere. For just a fraction of a cent, you can access the latest LLMs through a serverless API.

Generate your own API key and see it for yourself:

  curl https://api.lambdalabs.com/v1/completions \

Nick-Harvey

curl https://api.lambdalabs.com/v1/completions \
  -H "Authorization: Bearer $LAMBDA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1-70b-instruct-fp8",
    "prompt": "Create a concise relevant reply to the following message from a customer: I cant log into the app, Im getting a 500 error",
    "temperature": 0
  }' | jq .

Nick-Harvey

curl https://api.lambdalabs.com/v1/completions \
  -H "Authorization: Bearer $LAMBDA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(curl -sL https://tinyurl.com/3cpzh7c6 | jq -Rs --arg prompt "Summarize this text:" \
    '{
      model: "hermes3-405b",
      prompt: ($prompt + "\n\n" + .),
      temperature: 0
    }')" | jq .

Nick-Harvey

THE ODYSSEY
BOOK I
THE GODS IN COUNCIL—MINERVA’S VISIT TO ITHACA—THE CHALLENGE FROM TELEMACHUS TO THE SUITORS.

Tell me, O Muse, of that ingenious hero who travelled far and wide after he had sacked the famous town of Troy. Many cities did he visit, and many were the nations with whose manners and customs he was acquainted; moreover he suffered much by sea while trying to save his own life and bring his men safely home; but do what he might he could not save his men, for they perished through their own sheer folly in eating the cattle of the Sun-god Hyperion; so the god prevented them from ever reaching home. Tell me, too, about all these things, oh daughter of Jove, from whatsoever source you may know them.

So now all who escaped death in battle or by shipwreck had got safely home except Ulysses, and he, though he was longing to return to his wife and country, was detained by the goddess Calypso, who had got him into a large cave and wanted to marry him. But as years went by, there came a time when the gods

Nick-Harvey

Using the Lambda Inference API#
The Lambda Inference API enables you to use the Llama 3.1 405B Instruct large language model (LLM), and fine-tuned versions such as Nous Research's Hermes 3 and Liquid AI's LFM 40.3B MoE (Mixture of Experts), without the need to set up your own vLLM API server on an on-demand instance or 1-Click Cluster (1CC).

Tip

Try Lambda Chat!

Also try Companion, powered by the Lambda Inference API.

Contact us to learn more about our:

Nick-Harvey

import board
import neopixel
import time
import random
import argparse

num_pixels = 60
iterations = 2

pixels = neopixel.NeoPixel(board.D18, num_pixels)

Nick-Harvey

I0404 09:25:16.189599 123145420140544 video.py:50] Thread <datasources.video.Video object at 0x1571cced0>: starting
04/04 09:25 INFO Thread <datasources.video.Video object at 0x1571cced0>: starting
[[[ 23  21  26]
  [ 22  20  25]
  [ 22  20  25]
  ...
  [  9  12  12]
  [  9  12  12]
  [  9  12  12]]

Nick-Harvey

#!/usr/bin/env python3

import os
import json

import kfp
import kfp.dsl
import kfp.components
from kubernetes.client.models import V1EnvVar

Nick-Harvey

default	18:35:13.047064 -0400	loginwindow	-[ApplicationManager checkInAppContext:refCon:eventData:] | enter
default	18:35:13.048349 -0400	CommCenter	#I handleLSNotitifcation_sync: Application launched: <private>
default	18:35:13.050550 -0400	loginwindow	-[PersistentAppsSupport applicationLaunched:] | enter. clearing logout in progress and last logout time
default	18:35:13.050587 -0400	loginwindow	-[PersistentAppsSupport applicationReady:] | App: Brave Browser Beta, ready, updating active tracking timer
default	18:35:13.050609 -0400	loginwindow	-[PersistentAppsSupport(Internal) updateActiveTrackingTimer] | Creating timer
default	18:35:13.051061 -0400	loginwindow	-[ApplicationManager checkInAppContext:refCon:eventData:] |      checked in app : Brave Browser Beta
default	18:35:13.067393 -0400	mds	directQueryFetchResultsReply 40962 1795162112 35
default	18:35:13.120021 -0400	Brave Browser Beta	[0411/183513.119136:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
default	18:35:13.120296 -0400	Brave Browser

Nick-Harvey

Process:               Brave Browser Beta [16872]
Path:                  /Applications/Brave Browser Beta.app/Contents/MacOS/Brave Browser Beta
Identifier:            com.brave.Browser.beta
Version:               74.0.63.35 (63.35)
Code Type:             X86-64 (Native)
Parent Process:        ??? [1]
Responsible:           Brave Browser Beta [16872]
User ID:               501

Date/Time:             2019-04-11 18:30:58.363 -0400