carsonfarmer

Agent Evaluation Guide for Vercel AI SDK with Langfuse

This guide demonstrates how to evaluate LLM agents built with the Vercel AI SDK using Langfuse's evaluation framework. We'll walk through a 3-phase evaluation approach, moving from manual inspection to automated testing at scale.

What is an LLM Agent?

Agents are systems operating in continuous loops where the LLM:

1. Receives input
2. Decides on an action (like calling external tools)
3. Receives feedback from the environment

carsonfarmer

graph TD;
  3 --> 1((1))
  3 --> 2((2))
  6 --> 4((4))
  6 --> 5((5))
  10 --> 8((8))
  10 --> 9((9))
  13 --> 11((11))
 13 --> 12((12))

carsonfarmer

[package]
name = "rust-sqlite-test"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
anyhow = "1.0.72"
byteorder = "1.4.3"

carsonfarmer

import resolve from '@rollup/plugin-node-resolve';

export default {
  input: 'index.js',
  output: {
    dir: 'output',
    format: 'cjs',
  },
  onwarn: (error) => {
    console.log(error);

carsonfarmer

# Initialize npm module
npm init

# Install eslint
npm i -D eslint

# Initialize eslint
npx eslint --init
# To check syntax, find problems, and enforce code style
# JavaScript modules (import/export)

carsonfarmer

Dynamic Data on IPFS: An Introduction to Textile's Buckets and Threads

This document: https://tinyurl.com/ybk38mfh
Slides from intro presentation: https://tinyurl.com/y8hxgrq2

Description

In this workshop we’re going to cover tools for building engaging apps, for real users, dealing with real data. We’ll focus on JavaScript app building and we’ll show you simple and fast ways to add IPFS, IPNS, and even potentially Filecoin so you can create interoperable & unstoppable data for your users.

What we’ll cover:

We'll cover in introduction to Textile’s Threads, Buckets, and Hub tools. We’ll also cover some command-line (CLI) actions, and Textile’s JavaScript Client(s). To prepare, we encourage you to install a few things before we get started:

carsonfarmer

carsonfarmer

Keybase proof

I hereby claim:

• I am carsonfarmer on github.
• I am carsonfarmer (https://keybase.io/carsonfarmer) on keybase.
• I have a public key ASDg9OIITd2PRxa3AilDjevMta2t0QVv2d1ofwbC7_zwuwo

To claim this, I am signing this object:

carsonfarmer

#!/usr/bin/env node

import { cac } from 'cac'
import textile, { Block, FeedItem } from '@textile/js-http-client'
import readline from 'readline'
import chalk from 'chalk'
import { emojify } from 'node-emoji'

const cli = cac('txtl')
const { log } = console

carsonfarmer

diff --git a/index.js b/index.js
index 963d96f..6e7bfc1 100644
--- a/index.js
+++ b/index.js
@@ -3,6 +3,8 @@
 const cli = require('cac')('txtl')
 const textile = require('@textile/js-http-client').default
 var readline = require('readline')
+const chalk = require('chalk')
+const { emojify } = require('node-emoji')