albinekb

Picking a cheap model to drive an agent loop

I benchmarked every model in OpenRouter's Discounted Models collection — plus a full-price frontier model and a local 27B — to answer one question: which of these should drive an agent that calls tools and verifies its work, rather than guessing?

Short answer: the model with the smallest discount in the collection won, and a $15/M frontier model couldn't beat it by more than noise.

Scope, stated up front. Everything here is deterministic-verifier work — did it call get_contacts, did the cron job land in place, is the arithmetic right. That excludes everything where the answer is a judgment call: prose quality, voice, taste. A low score here means "don't put it in an agent loop." It says nothing about whether a model writes well. See What this doesn't measure.

Harness. BenchLocal 0.3.0, temperature 0, 1 run per test, driven through its Agent API. Three valid

albinekb

import * as React from "react";
import { useMousePosition } from "~/hooks/useMousePosition";

/** Component to cover the area between the mouse cursor and the sub-menu, to allow moving cursor to lower parts of sub-menu without the sub-menu disappearing. */
export function MouseSafeArea(props: { parentRef: React.RefObject<HTMLDivElement> }) {
  const { x = 0, y = 0, height: h = 0, width: w = 0 } = props.parentRef.current?.getBoundingClientRect() || {};
  const [mouseX, mouseY] = useMousePosition();
  const positions = { x, y, h, w, mouseX, mouseY };
  return (
    <div

albinekb

function download(csv) {
  if (csv == null) return

  const filename = `export-${new Date()}.csv`

  if (!csv.match(/^data:text\/csv/i)) {
    csv = 'data:text/csv;charset=utf-8,' + csv
  }
  const data = encodeURI(csv)

albinekb

type Method = 'get' | 'post' | 'put' | 'delete'
type Handler = (req: Request) => Promise<any>

function route (method: Method, path: string, handler: Handler) {
  app[method](path, (req, res, next) => {
    handler(req)
      .then(result => result ? res.json(result) : res.status(204).end(''))
      .catch(next)
  })
}

albinekb

{
    "editor.fontFamily": "FiraCode-Light",
    "editor.fontLigatures": true,
    "window.zoomLevel": 4,
    "prettier.printWidth": 140,
    "prettier.semi": false,
    "prettier.singleQuote": true,
    "prettier.trailingComma": "all",
    "editor.formatOnSave": true,
    "workbench.iconTheme": "seti",

albinekb

import * as React from 'react'

import Table from '@material-ui/core/Table'
import TableBody from '@material-ui/core/TableBody'
import TableHead from '@material-ui/core/TableHead'
import TableRow from '@material-ui/core/TableRow'
import TableCell from '@material-ui/core/TableCell'

const ArrayOf = ({ type }) => (
  <React.Fragment>

albinekb

const http = require('http')
const httpProxy = require('http-proxy')

const PORT = 3005
const TARGET = 'http://example.com:80'
const USER = ''
const PASSWORD = ''

const options = {
  auth: `${USER}:${PASSWORD}`,

albinekb

vlc \
  screen:// \
  -I rc \
  --screen-left=0 --screen-top=0 --screen-width=800 --screen-height=600 \
  --sout "#transcode{vcodec=h264,vb=800,fps=5,scale=1,acodec=none}:duplicate{dst=std{access=file,mux=mp4,dst='./screen.mp4'}}"

albinekb

FROM node:7

RUN apt-get update &&\
    apt-get install -y libgtk2.0-0 libgconf-2-4 \
    libasound2 libxtst6 libxss1 libnss3 xvfb

# Exposing port 5000 (micro)
EXPOSE 5000

WORKDIR /app

albinekb

#!/usr/bin/env bash
set -e

# Start Xvfb
Xvfb -ac -screen scrn 1280x800x24 :9.0 &
export DISPLAY=:9.0

exec "$@"