sam-saffron-jarvis

Classify Guardian shell-1000 live evaluation

Date: 2026-09-17 (Australia/Sydney)

Conditions

Command: term-llm guardian eval --concurrency 4 --json --report-only --raw-output …
Provider/model: TypeSafe jev-latest, resolved by responses to jev-1.13.0
Policy SHA-256: 87d7e081c968201d117d1ddb32208b890cfe663cc3428acb9d437f81856e74c9
Corpus SHA-256: 05ae29529036f0fa3f77a5b41e4e418e7dd00f3058eea80a0e4d56adfc30d483

Full-tutorial model acceptance eval — research snapshot

This runs the actual term-llm tutorial in its isolated browser Linux VM, while inference runs in native local Ollama rather than WebGPU.

Read RESULTS.md first. It reports measured configurations, representative answers, failure cases, grader corrections and caveats. It is not a leaderboard or a claim of universally reliable model behavior. Predicates only catch some errors; final acceptance requires reading actual outputs. Three seeds are not a robust statistical sample. Harness checks evolved during investigation; earlier raw results were preserved and corrections documented, rather than silently reclassified.

Reproduce

Clone https://github.com/SamSaffron/term-llm-tutorial and use tested lesson revision 1d57647 (the later security-only merge 9c0b2d8 preserves those lessons).
Put tutorial-eval.mjs and tutorial-eval-checks.mjs from this gist under scripts/, and tutorial-eval.test.mjs under `tes

Ruby 3.4.9 → 3.4.10 review

Ruby 3.4.10 is a much more substantial patch release than its terse announcement suggests.

Released: 30 June 2026
Exact diff: v3_4_9…v3_4_10
Size: 72 commits, 114 files, 2,554 insertions, 458 deletions
Nature: bug fixes and security hardening; no new language features
Recommendation: upgrade. The net-imap CVEs alone justify it, and there are several independent VM memory-safety and correctness fixes.

Iodine vs Agoo as a Ruby micro web server dependency

Assessment date: 23 July 2026

Conclusion

Agoo is the better-shaped dependency, but I would not ship either current release. If forced to choose between the projects, I would use iodine's released 0.7 line with the current load regression patched and additional parser-focused tests.

Agoo is materially smaller, has better test coverage signals, and is more focused. Unfortunately, its current release has a remotely triggerable request-parsing crash that I reproduced locally. Iodine has broader contributor history and active development, but its current release is broken on Ruby versions below 4.0 and its upcoming 0.8 line is a very large, API-breaking rewrite.

	{
	"model": "jev-1.13.0",
	"runs": 5,
	"reviews": 25,
	"config": {
	"min_confidence": 0.15,
	"classify_all_shell": true
	},
	"all": {
	"n": 25,

	<!doctype html>
	<html lang="en">
	<head>
	<meta charset="utf-8" />
	<meta name="viewport" content="width=device-width, initial-scale=1" />
	<title>Discourse checklist persistence studies</title>
	<style>
	:root {
	--primary: #222;
	--secondary: #fff;