GroktoCrawl v0.12.0 transforms a self-hosted web scraper into an agent-native research platform. Every new capability ships with both a REST API and a groktocrawl CLI command. No .env changes needed — upgrade and go.
Upgrade:
cd ~/docker-compose/groktocrawl
git pull origin main
docker compose pull
docker compose up -d
I needed a capable local coding model that would fit on a single consumer GPU with meaningful context. The RTX 5070 Ti has 16 GB of VRAM — not enough for most frontier models, but more than enough for interesting things if you pick the right architecture.
Ornith 1.0-9B is that architecture. It's a self-improving open-source coding model from DeepReinforce, MIT licensed, scoring 69.4 on SWE-Bench Verified and 43.1 on Terminal-Bench 2.1. Those numbers match Gemma 4-31B, a model with 3.4x more parameters. And thanks to its hybrid attention design, it runs at 400K context on a widely-available consumer GPU.
This guide shows you how to set it up, what to expect, and three exercises to see for yourself what it can do.
GroktoCrawl v0.9.0 - What's New in the CLI
100 commits · 127 files changed · 34,000+ lines added · 6 days of work
GroktoCrawl v0.9.0 is the biggest release yet. It introduces a full Firecrawl-compatible site crawling engine, AI-powered research features, change monitoring, document parsing, semantic similarity search, and major infrastructure hardening.
This gist walks through the CLI commands you'll use most, with real output from a live instance.
groktocrawl scrape https://www.gutenberg.org/ebooks/11→ full Alice in Wonderland as chapter-structured markdown.
Public-domain books are the largest open corpus of high-quality text on the internet. But getting them into a format an LLM, RAG system, or agent can use means: