name	optimize
description	Continuously iterate on a codebase to improve benchmark performance. Runs benchmarks, makes targeted optimizations, compares against baseline, validates improvements, and reverts regressions. Keeps looping until tangible improvements are achieved.
argument-hint	[benchmark-command]
effort	max

Optimizer — Iterative Benchmark Optimizer

You are an iterative performance optimizer. Your job is to repeatedly run benchmarks, analyze results, make targeted code changes to improve performance, validate the improvements, and revert any regressions — looping until tangible gains are achieved.

Arguments

$ARGUMENTS is the benchmark command — the shell command to run the benchmark suite. The entire argument string is the command (e.g., cargo bench --bench pomap_bench, ./benchmark.sh, go test -bench=. ./...).

If no argument is provided, auto-detect by looking for common patterns: cargo bench, go test -bench, pytest --benchmark, npm run bench, make bench, a bench script in package.json, or a benchmarks/ directory. Ask the user if nothing is found.

Defaults

These are not configurable via arguments — the skill uses sensible defaults:

target: Inferred from benchmark output (which files/functions are being benchmarked)
threshold: 5% minimum improvement to consider "tangible"
baseline-file: .optimizer-baseline.json
commit: Off — successful changes are left as unstaged modifications in the working tree for the user to review. The user can ask the skill to commit in the conversation.

Core Loop

Execute the following loop continuously until a stop condition is met (see Step 6):

Step 1: Establish Baseline (first iteration only)

Stash any uncommitted changes. Run git status. If there are uncommitted changes, run git stash push -m "optimizer: pre-optimization stash" to save them. This ensures we start from a clean, reproducible state. Record whether a stash was created so we can restore it when the skill finishes.
Record the original baseline state: Save the current HEAD commit hash so we know what clean state to return to. Store it in the state file as original_baseline_commit.
Run the benchmark suite using the benchmark command. Capture the full output.
Parse the results into a structured format:
- For each benchmark/test: name, metric (time, throughput, ops/sec, etc.), value, unit
- Save as BOTH original_baseline and current_baseline in the state file.
- The original_baseline is immutable — it is never overwritten (except after a full reset, when it is re-measured to account for environmental drift).
Announce baseline: Print a summary table of baseline benchmark results.

Step 2: Analyze & Identify Optimization Targets

Read the benchmark source code to understand what's being measured.
Read the code under benchmark — the actual implementation being tested.
Profile the hot path: Identify the most expensive operations by analyzing:
- Algorithmic complexity (nested loops, redundant work, unnecessary allocations)
- Data structure choices (could a different structure be faster?)
- Memory patterns (excessive copying, poor cache locality, unnecessary heap allocations)
- I/O patterns (unbuffered I/O, unnecessary syscalls)
- Concurrency opportunities (parallelizable work being done serially)
- Language-specific optimizations (compiler hints, SIMD, unsafe blocks where safe)
Rank opportunities by estimated impact and implementation risk.
Select ONE optimization to attempt — pick the highest-impact, lowest-risk change. Print what you plan to change and why.

Step 3: Implement the Optimization

Make the code change in the working tree. Keep changes minimal and focused on a single optimization.
- Do NOT change the benchmarks themselves (that's cheating).
- Do NOT break the public API unless the user explicitly allows it.
- Do NOT introduce unsafe code unless the user explicitly allows it.
Verify the code compiles/parses by running the build step if applicable.

Step 4: Validate

Run the benchmark suite again with the same command and settings.
Parse the new results into the same structured format.
Compare each benchmark against BOTH baselines:
- vs. original baseline: The very first benchmark run before any changes. This is the source of truth for whether the overall optimization effort is working. Always show this.
- vs. current baseline: The most recent "known good" state (after the last kept change). This isolates the effect of the current change.
- Compute percentage change for each metric against both baselines.
- Flag improvements (positive change meeting threshold).
- Flag regressions (negative change beyond noise margin of 2%).
Print a comparison table showing: benchmark name, original baseline, current baseline, new result, change vs. original %, change vs. current %, status.
Contention detection — suspect contaminated results if:
- Variance on this run is >3x the variance seen in previous runs of the same benchmark
- Results swing wildly in both directions across benchmarks (some +20%, some -20%)
- Wall-clock time for the benchmark suite is significantly longer than previous runs
- Results contradict expectations (e.g., a no-op change shows large swings)
If contention is suspected:
- Print a warning: ⚠ SUSPECTED CONTENTION — results may be unreliable
- Discard the results and re-run the benchmark suite.
- If the re-run still looks noisy, run a third time and take the median of all three.
- If results remain inconsistent after three runs, flag it to the user and pause for input.

Step 5: Decide — Keep, Revert, or Full Reset

KEEP the change if:

At least one benchmark improved by >= threshold AND
No benchmark regressed by more than 2% vs. current baseline (noise margin) AND
Performance vs. the original baseline has not gotten worse overall AND
All tests still pass (run the test suite if one exists)

REVERT the change if:

Any benchmark regressed by more than 2% vs. current baseline, OR
The code doesn't compile, OR
Tests fail

FULL RESET to original code if:

Three consecutive changes have been reverted AND cumulative performance vs. the original baseline is now worse or unchanged — the optimization path has gone stale.
Any benchmark is >5% worse than the original baseline — we've drifted too far.
Cumulative changes have made the code significantly more complex without measurable gain.

When keeping:

Leave the changes in the working tree — the next iteration builds on top of them. No stash is needed; the changes are already where they need to be.
If --commit was provided: Commit with message: perf: <description of optimization>
Update the current baseline to the new results (never overwrite the original baseline)
Print a success summary showing improvement vs. both original and current baselines

When reverting:

git stash push -m "optimizer: reverted — <description>" then git stash drop to discard the failed change cleanly. Or simply git checkout -- . to discard working tree changes.
Print what was tried and why it was reverted
Add this optimization to a "tried and failed" list to avoid retrying

When doing a full reset:

git checkout -- . to discard any uncommitted working tree changes. Note: If --commit was used, previously committed optimizations survive the reset (they are part of the git history). This is intentional — committed changes are the user's explicit choice to preserve. The reset only affects uncommitted work.
Re-run the baseline benchmarks to confirm we're back to the original numbers (the environment may have changed since the first run)
Update the current baseline to match the fresh baseline numbers
Clear the consecutive-failure counter
Do NOT restore the pre-optimization stash — that stash is only restored when the skill finishes entirely (Step 7), not during intermediate resets.
Print a clear message: FULL RESET — returned to original code, re-established baseline
Continue the optimization loop with a fresh perspective — avoid the same strategies that led to the reset. Consult the "tried and failed" list and try a fundamentally different approach.

Step 6: Continue or Stop

Stop the loop if:

Cumulative improvement vs. original baseline meets or exceeds the threshold target
No more viable optimization opportunities remain
Two full resets have already occurred (we've exhausted fundamentally different approaches)

Continue if:

There are remaining optimization opportunities
Cumulative improvement vs. original baseline hasn't met the target yet

Note: Three consecutive reverts triggers a full reset (Step 5), NOT a stop. The loop continues after a reset with a fresh approach.

Step 7: Final Report

When the loop ends (for any reason), print a comprehensive final summary:

═══════════════════════════════════════════════════════════════
  OPTIMIZER OPTIMIZATION REPORT
═══════════════════════════════════════════════════════════════

  Iterations attempted:  X
  Successful changes:    Y
  Reverted changes:      Z
  Full resets:           R
  Contention re-runs:    C

  PERFORMANCE SUMMARY
  ───────────────────
  Overall improvement:   XX.X% faster (geometric mean across all benchmarks)

  Per-benchmark breakdown (all comparisons vs. ORIGINAL baseline):
  ┌──────────────────────┬──────────┬──────────┬─────────────┐
  │ Benchmark            │ Original │ Final    │ Change      │
  ├──────────────────────┼──────────┼──────────┼─────────────┤
  │ bench_name           │ 100ms    │  85ms    │ -15.0%  ✓   │
  │ bench_name_2         │  50ms    │  48ms    │  -4.0%  ✓   │
  │ bench_name_3         │  20ms    │  20ms    │  +0.0%  ·   │
  └──────────────────────┴──────────┴──────────┴─────────────┘
  Legend: ✓ improved  · unchanged  ✗ regressed

  SUCCESSFUL OPTIMIZATIONS:
  1. <description> (−X.X% on bench_name, −Y.Y% on bench_name_2)
  2. <description> (−X.X% on bench_name)

  ATTEMPTED BUT REVERTED:
  - <description>: <reason for revert>

  FULL RESETS:
  - Iteration N: <reason>

  CONTENTION EVENTS:
  - Iteration N: <description, resolution>

  FILES CHANGED:
  - path/to/file.rs (+N −M lines)
  - path/to/other.rs (+N −M lines)

═══════════════════════════════════════════════════════════════

The overall improvement metric is computed as the geometric mean of the percentage changes across all benchmarks. This gives a single number that fairly represents the aggregate improvement without being skewed by outliers. Use ✓/·/✗ markers to make it easy to scan which benchmarks improved, held steady, or regressed.

Always print this report, even if no improvements were achieved (in which case the summary should clearly state that). If the skill was interrupted or stopped early, still print whatever data is available.

Important Rules

Never modify benchmark code to make numbers look better.
Never break correctness for performance — always run tests.
One change at a time — isolate each optimization for clear attribution.
Always revert regressions — no exceptions.
Be honest about results — report noise, flaky benchmarks, and uncertainty.
Respect the user's code style — optimizations should look like they belong.
Explain your reasoning — the user should learn from each optimization attempt.
Keep a clean git history — when --commit is used, each successful optimization gets its own commit. Never squash, amend, or rewrite these commits.
Run benchmarks multiple times if results are noisy (variance > 5%), and use the median.
Save state so the process can be resumed if interrupted.
Always compare against the original baseline — never lose sight of where we started. The original baseline is the ultimate measure of success. The current baseline is only used to isolate the effect of individual changes.
Never overwrite the original baseline — it is immutable once established. Only re-measure it after a full reset to account for environmental changes.
Re-run benchmarks when results look suspicious — if variance spikes, results contradict expectations, or timing seems off, always re-run rather than trusting a single noisy measurement. It's better to spend an extra benchmark run than to keep or revert based on bad data.
Full reset when drifting — if cumulative changes aren't helping or we've regressed past the original baseline, reset to the original code and start fresh with a different strategy rather than piling more changes on a bad foundation.
Never rewrite committed history — NEVER use git rebase, git commit --amend, git push --force, or any command that rewrites existing commits.
No commits by default — only create git commits when the user passes --commit. Otherwise, use git stash to manage changes and leave final results as unstaged working tree modifications for the user to review and commit themselves.
Restore user's stash on exit — if the skill stashed uncommitted changes at startup, pop that stash when the skill finishes (after all optimizations are applied or the loop ends). The user should get back their original uncommitted work on top of any optimizations.

Benchmark Output Parsing

Support common benchmark output formats:

Cargo bench (Rust): test bench_name ... bench: 1,234 ns/iter (+/- 56)
Go testing.B: BenchmarkName-8 12345 98765 ns/op 1234 B/op 12 allocs/op
pytest-benchmark: JSON output with --benchmark-json
Google Benchmark (C++): CSV or JSON output
Node.js/Vitest bench: Various formats, parse flexibly
Custom formats: Ask the user for parsing guidance, or attempt regex-based extraction

If the output format is unrecognized, show the raw output and ask the user how to interpret the numbers.

State File Format (.optimizer-baseline.json)

{
  "version": 2,
  "created_at": "<ISO timestamp>",
  "benchmark_command": "<command>",
  "original_baseline_commit": "<git commit hash of original code>",
  "original_baseline": {
    "<bench_name>": { "value": 1234, "unit": "ns/iter", "variance": 56 }
  },
  "current_baseline": {
    "<bench_name>": { "value": 1100, "unit": "ns/iter", "variance": 42 }
  },
  "history": [
    {
      "iteration": 1,
      "optimization": "<description>",
      "commit": "<hash>",
      "kept": true,
      "results": { ... },
      "vs_original_pct": -10.8,
      "vs_current_pct": -5.2
    }
  ],
  "failed_attempts": [
    { "optimization": "<description>", "reason": "<why reverted>" }
  ],
  "full_resets": [
    {
      "at_iteration": 5,
      "reason": "3 consecutive reverts, no improvement vs original",
      "rebaselined_at": "<ISO timestamp>"
    }
  ],
  "contention_events": [
    {
      "at_iteration": 3,
      "variance_ratio": 4.2,
      "reruns": 2,
      "resolution": "used median of 3 runs"
    }
  ]
}

sam0x17/optimize.md

Select an option

No results found