changelog.md

TestDriver 7.9.3 → 7.11.21 — Upgrade Changelog

Span: 2026-03-28 → 2026-07-16 · 555 non-merge commits · 198 SDK files changed

This covers everything between the v7.9.3 and v7.11.21 tags. It leads with the changes that alter test behavior on upgrade, because several defaults flipped in ways that are invisible until a test starts failing.

---

⚠️ Read this first: element finding got two safeguards turned off

If tests that passed on 7.9.3 stopped locating elements after upgrading, it is almost certainly one of these two. Both flipped in the same commit (6ed074cbb), four days after 7.9.3 shipped.

zoom now defaults to off (was on)

// 7.9.3
let zoom = true;                 // default enabled
zoom = options.zoom !== false;

// 7.11.21
let zoom = false;                // default disabled
zoom = options.zoom === true;

With zoom on, find() ran a two-phase locate: a rough pass proposes up to 4 candidate regions, each is cropped to 30% of the screen at full resolution and re-located in parallel, and the highest-confidence result wins.

With zoom off, there is a single pass against the screenshot downscaled to 1000px wide. At the default 1366×768 that discards ~27% of the linear resolution. Small icons, dense toolbars, and visually similar neighbors are what degrade first.

Restore per call:

await testdriver.find("Submit button", { zoom: true });

There is no constructor-level equivalent for zoom — it is per-call only.

AI verification now defaults to off (was always on)

7.9.3's SDK had no verify concept, so it never sent the field and the API defaulted skipVerify: false — meaning every non-cached find was verified. 7.11.21 sends skipVerify: !verify with verify defaulting to false, and the API default flipped to skipVerify: true to match.

What that removes is not just a check. On rejection, the old path retried the locate with the verifier's reasoning injected as a correction hint, against a stronger model. That self-correction loop is gone by default.

Symptom profile: find() succeeds and returns coordinates, but the click lands on the wrong element. Zoom does not help here — the miss is never reported.

Restore globally (added in 1623c30f0, PR #303):

const testdriver = new TestDriver({ verify: true });     // applies to every find()
await testdriver.find("Submit", { verify: false });      // per-call override

Both restored together

const testdriver = new TestDriver({ verify: true });
await testdriver.find("Submit button", { zoom: true });

Expect both to cost latency. That is exactly why they were disabled.

---

Other behavior changes in the find path

Vision models were swapped

cf47066ab (PR #218) split the single GEMINI_FLASH_MODEL constant into three purpose-specific models; e3dee7e5d ("update to hopefully faster models") then bumped them again. Net effect:

Pipeline stage	7.9.3	7.11.21
Full-image locate, zoom-verify	gemini-3-flash-preview	gemini-3.5-flash
Zoom rough-candidate pass	gemini-3-flash-preview	gemini-3.1-flash-lite
Element verification	gemini-3-flash-preview	gemini-3.1-flash-lite
Creative tasks (assert, check, generate, summarize)	gemini-3-flash-preview	gemini-3.1-pro-preview

A different vision model grounds coordinates differently on the same screen. Note that candidate selection and verification both moved to a lite model, so even tests that explicitly pass zoom: true get a weaker first phase than they had on 7.9.3.

Out-of-bounds coordinates are clamped, not rejected

a5cb398db (PR #226). Previously a hallucinated coordinate outside the image returned found: false, so polling retried and the test eventually failed honestly. Now it is clamped into bounds and returned as found: true.

Symptom: a click at the extreme edge of the screen instead of a clean not-found error.

Selector cache is unaffected by the channel switch

7.11.21-canary resolves to https://api-canary.testdriver.ai, while a clean 7.9.3 resolves to https://api.testdriver.ai. Canary and stable point at the same MongoDB, so cached selectors carry over — a cold cache is not part of this regression. But be aware that moving to a canary build also moves you to a different API deployment, not just different SDK code.

---

New features

Provisioning API extracted and expanded — sdk/lib/provision.js

749 lines, new module. Application launch (chrome, chromeExtension, vscode, installer, electron) and dashcam initialization now live behind a Proxy that automatically skips provisioning in reconnect mode — previously a reconnect could re-run launch steps against an already-running app.

Cross-client installer — sdk/lib/install-clients.js

470 lines, new module. testdriverai init can now wire the MCP server, the test-creator agent, and the skills into supported AI coding clients, each of which stores config in a different place and format. All writers are merge-safe and idempotent. Web-only clients (Lovable, Replit, v0) return setup guidance rather than writing files.

Reconnect became explicit and addressable

New public options:

sandboxId?: string        // reconnect to a specific sandbox; implies newSandbox: false
requireSandbox?: boolean  // throw if that sandbox is gone rather than silently
                          // provisioning a blank replacement (default: false)

requireSandbox closes a real trap: provision methods only run at session start and are never re-applied on reconnect, so a silent replacement sandbox comes up blank. Callers that depend on existing sandbox state should set this and handle the error by starting a fresh session explicitly.

getLastSandboxId() now reads from the live SDK instance when connected and falls back to .testdriver/last-sandbox. Its return shape changed — see Breaking changes below.

MCP server rewritten

sdk/mcp-server/src/server.ts dropped from ~1690 lines to a thin shell over a new shared core/actions.ts (1202 lines), now exported as public subpath entries so other tooling can import the same action layer:

"./mcp-core", "./mcp-core/provision-types", "./mcp-core/env-utils"

Also: TD_OS environment variable support (PR #245), a testdriver/init tool (PR #247), end-to-end MCP tests (PR #246), a setup wizard (PR #276), and a Claude Code plugin that ships the MCP server and agent together (PR #251).

Unified distributed tracing

PR #188. A single root trace per session spanning SDK → API → Runner → Image Worker, so a slow or failed find can be followed end to end in one timeline.

Sandbox slot polling

The SDK now polls for a free concurrency slot instead of failing on first denial, with a configurable ceiling (0 disables polling and restores fail-fast), a 10-minute provisioning timeout, and event-driven completion that falls back to authenticate-polling only when events don't arrive.

---

Breaking changes

Runner no longer auto-upgrades on a minor bump (7986097c2). A separate minor-version bump to 7.11.0-test was needed to force runner reinstall at sandbox spawn (9cd9c4110).

No public SDK methods were removed. ai(), matchImage(), and the rest of the command surface are all still present — some example files and doc pages were deleted, which is not the same thing.

---

Fixes

Screenshots

• Retry system screenshots on failure, up to 3 attempts with exponential backoff
• Fixed all-black captures: stopped sending the super key while trying to heal the display, because on Linux desktops it opens the Activities overview and steals focus — which made Chrome appear to "go missing" right after a flaky capture. The xset display-wake calls are what actually recover the screen.
• Fixed window minimize during screenshot capture
• Runner returns a raw PNG buffer instead of round-tripping through base64, cutting peak memory and ~10–30ms per capture

Connection stability

• Plugged Ably reconnection and publish leaks in session cleanup (PR #225)
• Fixed Ably disconnects and message spam
• keepAlive fixes (PR #302)
• Replaced repeated Redis connections with an API endpoint

Caching and performance

• OpenCV template matching now runs all cache entries in parallel rather than sequentially — total latency drops from N × T_match to one worker round-trip (PR #208)
• Removed an unused S3 presign per cache hit that existed only to populate a debug log line
• Auth user/team cached in Redis, profile sync gated to daily (PR #300)
• Sentry sampling reduced to 1%, screenshot attachment uploads removed from the locate path entirely (PR #280) — these were uploading every input screenshot on every find

Other

• Storage migrated to Tigris (PRs #209, #229); TIGRIS_BUCKET_IMAGE_TRANSFER now takes precedence over AWS_BUCKET_IMAGE_TRANSFER
• Per-attempt error isolation in the test retry dropdown (PR #236)
• testFile filter now queries TdTestCase instead of TdTestRun (PR #266)
• Dashcam state directory setup hardened for global npm layouts (PR #272)
• Dashcam now recognizes the dev API environment alongside test/canary
• Vitest plugin: per-attempt error message and stack now recorded
• Fail fast on invalid production Redis config in image-worker (PR #268)
• Extra help for Chrome extension icon clicks (PR #232)
• Windows AMI: fixed chrome.exe 0xc0000906, hanging Packer builds, and screen-resolution install retries (PRs #242, #243)

---

Documentation

PR #212 audited the v7 docs against the actual implementation. Worth knowing: the screenshots option was documented as default: true but the code has always used options.autoScreenshots === true — i.e. false. The docs were corrected; the behavior never changed.

Also: extract() documented (225 lines, new page), scroll docs rewritten, find docs updated for zoom/verify, quickstart substantially expanded, skills republished under the testdriver:* namespace, and the marketing site migrated to Astro (PR #258).

Note that resolution is now documented as Enterprise-only for custom values. The default (1366x768) is unchanged and no API-side enforcement was added in this range — this is a docs/policy statement, not a code change.

---

Upgrade checklist

1. Add verify: true to your SDK constructor if you rely on finds being correct rather than fast.
2. Add zoom: true to finds that target small or crowded UI. There's no global switch — it's per-call.
3. Audit any use of getLastSandboxId() for the removed ami, instanceType, and timestamp fields.
4. Set requireSandbox: true on reconnects that depend on sandbox state, so you get an error instead of a blank replacement.