GroktoCrawl v0.9.0 - What's New in the CLI
100 commits · 127 files changed · 34,000+ lines added · 6 days of work
GroktoCrawl v0.9.0 is the biggest release yet. It introduces a full Firecrawl-compatible site crawling engine, AI-powered research features, change monitoring, document parsing, semantic similarity search, and major infrastructure hardening.
This gist walks through the CLI commands you'll use most, with real output from a live instance.
groktocrawl scrape https://www.gutenberg.org/ebooks/11→ full Alice in Wonderland as chapter-structured markdown.
Public-domain books are the largest open corpus of high-quality text on the internet. But getting them into a format an LLM, RAG system, or agent can use means:
What this demonstrates: How the GroktoCrawl CLI combined with its skill document creates a seamless experience for both human CLI users and AI agents. The CLI handles 20+ different site types through a transparent adapter system — one scrape command, radically different structured output depending on what you feed it. The skill tells agents exactly which command to reach for and when.
The adapter system dispatches to the right handler automatically. You don't pick an adapter; you just scrape <url>.
Deployed live on bishop.local:8090. 15 new security engine adapters across two PRs.
| Engine | Type | Description | Demo Status |
|---|---|---|---|
| CRT.sh | API | Certificate Transparency log search | Timeout from network (API reachable, 20s response) |
| URLhaus | API (POST) | Malware URL/payload tracking | API now requires auth (changed) |
| FIRST EPSS | API | CVE exploit probability (0-1) | Working |