Ivan Charapanau av

Synthetic Agentic Benchmark Generation — Ontology

Concept map for notes/viktor/synthetic-agentic-benchmarks/. Built 2026-07-13.

Survey scaffold for the whole space: [[llm-agent-eval-benchmarking-survey]] (taxonomy of eval objectives, interaction modes, difficulty levers, and named benchmarks — read it first to place any single note).

The core problem

Two failure modes recur when you auto-generate agentic benchmarks:

what makes a harness

an agentic harness is surprisingly simple. it's a loop that calls an llm, checks if it wants to use tools, executes them, feeds results back, and repeats. here's how each part works.

tools

the agent needs to affect the outside world. tools are just functions that take structured args and return a string. three tools is enough for a general-purpose coding agent:

const tools = {

	<!DOCTYPE html>
	<html lang="en">
	<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>tracequest — session</title>
	<style>

	:root {
	--bg: #111113;

	<!DOCTYPE html>
	<html lang="en">
	<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>tracequest — 019efdd4</title>
	<style>

	:root {
	--bg: #111113;

	<!DOCTYPE html>
	<html lang="en">
	<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>tracequest — 019efdd4</title>
	<style>

	:root {
	--bg: #111113;

	<!DOCTYPE html>
	<html lang="en">
	<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>tracequest — 019ef8af</title>
	<style>

	:root {
	--bg: #111113;

	<!doctype html>
	<html lang="en">
	<head>
	<meta charset="UTF-8" />
	<meta name="viewport" content="width=1920, height=1080" />
	<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
	<style>
	@import url("https://fonts.googleapis.com/css2?family=Inter:wght@400;700&family=JetBrains+Mono:wght@100;200;300;400;500;600;700;800&display=swap");

	* {

	lint-content:
	@command -v rg >/dev/null 2>&1 \|\| { echo >&2 "Error: rg (ripgrep) is missing. Please install it."; exit 1; }
	@rg --multiline --color=always -n -i -g '!.staged.md' -g '!.bak' \
	-e '—' \
	-e ';' \
	-e '\bdelve\b' \
	-e '\btapestry\b' \
	-e '\bleverage\b' \
	-e '\butilize\b' \
	-e '\bseamless(ly)?\b' \

	Read:
	https://github.com/agentskills/agentskills/blob/main/docs/specification.mdx
	https://github.com/agentskills/agentskills/blob/main/docs/integrate-skills.mdx
	Architect an implementation of agent skills for this codebase.

	I can think of a few requirements right away:

	Dedicated use-skill module
	A module instance per skill, i.e. each specific skill has is its own use-skill module instance in the workflow
	Only remote skills: from github, from URL, from storages