Skip to content

Instantly share code, notes, and snippets.

@CreedsCode

CreedsCode/SKILL.md

Created March 24, 2026 14:34
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save CreedsCode/4a4be70bd034e931ff9fd2332c35c9be to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save CreedsCode/4a4be70bd034e931ff9fd2332c35c9be to your computer and use it in GitHub Desktop.
Download ZIP
A/B Test Significance Calculator
Raw
SKILL.md
name ab-test-calculator
description Render an interactive A/B statistical significance calculator widget whenever the user wants to know if a difference between two variants is real or random noise. Use this skill across ANY testing context: email campaigns (open rate, click rate, reply rate), paid ads (CTR, ROAS, CPA), newsletters (open rate, unsubscribe rate), SaaS products (feature adoption, activation rate, conversion rate, retention, churn), landing pages (signup rate, bounce rate), onboarding flows (completion rate, drop-off), pricing page tests, and any before/after experiment. Trigger when the user mentions "A/B test", "split test", "experiment", "significance", "is this difference real?", "which version is better?", "did this change work?", "feature flag", "rollout result", "test vs control", or pastes two sets of numbers from any platform — Apollo, Mailchimp, HubSpot, Google Ads, Meta Ads, Mixpanel, Amplitude, LaunchDarkly, PostHog, or similar. Always pre-fill the widget with numbers found in the conversation.

A/B Test Significance Calculator

Render an interactive statistical significance calculator widget. Works across any domain where two variants are compared. Always extract and pre-fill numbers from the conversation — never ask the user to re-enter data they already shared.

Step 1 — Detect context and map fields

Before rendering, identify the testing domain from the conversation. This determines labels and the interpretation note after the widget.

Domain Exposure field label Conversion field label Default metric tab
Email (cold outreach, newsletters) Delivered Opens / Clicks / Replies Open rate or Click rate
Paid ads (Google, Meta, LinkedIn) Impressions Clicks or Conversions CTR or Conv. rate
SaaS feature / flag Users exposed Users who adopted / activated Adoption rate
Landing page / signup Visitors Signups / Trials Conv. rate
Onboarding / activation Users started Users completed Completion rate
Pricing / upgrade Users shown Upgrades / purchases Upgrade rate
Newsletter Subscribers Opens / Clicks / Unsubs Open rate
Generic / unclear Total Conversions Conv. rate

Extract:

  • n_a / n_b — exposure count per variant (delivered, impressions, visitors, users, etc.)
  • c_a / c_b — conversion count per variant (opens, clicks, signups, adoptions, etc.)
  • Variant names — use actual names if given (e.g. "Control", "With feature X"), else "Variant A / B"
  • Metric label — pick the closest from the domain table above

If numbers are ambiguous or missing, default to n=100, c=0 and note that the user should adjust.

Step 2 — Render the widget

Call visualize:read_me with ["interactive"] first, then render with visualize:show_widget.

Substitute the extracted values. Key substitutions:

  • N_A_VALUE, C_A_VALUE, N_B_VALUE, C_B_VALUE — pre-filled numbers
  • EXPOSURE_LABEL — e.g. "Delivered", "Impressions", "Visitors", "Users exposed"
  • CONVERSION_LABEL — e.g. "Opens", "Clicks", "Signups", "Adoptions"
  • VARIANT_A_NAME, VARIANT_B_NAME — e.g. "Control", "Variant A"
  • METRIC_TABS — see tab templates below; pick the set matching the domain
<style>
  .ab-wrap { padding: 1rem 0; font-family: var(--font-sans); }
  .ab-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; margin-bottom: 1.5rem; }
  .ab-card { background: var(--color-background-primary); border: 0.5px solid var(--color-border-tertiary); border-radius: var(--border-radius-lg); padding: 1rem 1.25rem; }
  .ab-card-label { font-size: 12px; font-weight: 500; text-transform: uppercase; letter-spacing: 0.05em; margin-bottom: 12px; }
  .ab-card.variant-a .ab-card-label { color: #185FA5; }
  .ab-card.variant-b .ab-card-label { color: #0F6E56; }
  .input-row { display: flex; flex-direction: column; gap: 4px; margin-bottom: 10px; }
  .input-row label { font-size: 13px; color: var(--color-text-secondary); }
  .input-row input { width: 100%; box-sizing: border-box; }
  .metric-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 10px; margin-bottom: 1.5rem; }
  .metric { background: var(--color-background-secondary); border-radius: var(--border-radius-md); padding: 12px; }
  .metric-label { font-size: 12px; color: var(--color-text-secondary); margin-bottom: 4px; }
  .metric-value { font-size: 22px; font-weight: 500; color: var(--color-text-primary); }
  .result-bar { height: 6px; border-radius: 99px; background: var(--color-border-tertiary); margin: 1rem 0; overflow: hidden; }
  .result-bar-fill { height: 100%; border-radius: 99px; transition: width 0.4s ease; }
  .result-box { border-radius: var(--border-radius-lg); padding: 1rem 1.25rem; border: 0.5px solid; margin-bottom: 1rem; }
  .result-box.sig { background: #E1F5EE; border-color: #5DCAA5; }
  .result-box.not-sig { background: #FAEEDA; border-color: #EF9F27; }
  .result-box.insufficient { background: var(--color-background-secondary); border-color: var(--color-border-tertiary); }
  .result-title { font-size: 15px; font-weight: 500; margin-bottom: 4px; }
  .result-title.sig { color: #085041; }
  .result-title.not-sig { color: #633806; }
  .result-title.insufficient { color: var(--color-text-primary); }
  .result-body { font-size: 13px; line-height: 1.6; }
  .result-body.sig { color: #0F6E56; }
  .result-body.not-sig { color: #854F0B; }
  .result-body.insufficient { color: var(--color-text-secondary); }
  .conf-row { display: flex; gap: 8px; margin-bottom: 1rem; }
  .conf-btn { flex: 1; padding: 6px; font-size: 13px; border-radius: var(--border-radius-md); border: 0.5px solid var(--color-border-secondary); background: transparent; color: var(--color-text-secondary); cursor: pointer; }
  .conf-btn.active { background: var(--color-background-info); border-color: var(--color-border-info); color: var(--color-text-info); font-weight: 500; }
  .stats-row { display: grid; grid-template-columns: 1fr 1fr 1fr; gap: 10px; margin-top: 1rem; }
  .stat-item { font-size: 12px; color: var(--color-text-secondary); }
  .stat-val { font-size: 14px; font-weight: 500; color: var(--color-text-primary); margin-top: 2px; }
  .section-label { font-size: 12px; font-weight: 500; color: var(--color-text-secondary); text-transform: uppercase; letter-spacing: 0.05em; margin-bottom: 8px; }
</style>

<div class="ab-wrap">
  <div class="section-label">Variant names</div>
  <div style="display:flex;gap:8px;margin-bottom:1.5rem">
    <input style="flex:1;font-size:13px" type="text" id="name_a" value="VARIANT_A_NAME" placeholder="Variant A" oninput="updateNames()">
    <input style="flex:1;font-size:13px" type="text" id="name_b" value="VARIANT_B_NAME" placeholder="Variant B" oninput="updateNames()">
  </div>

  <div class="section-label">Metric</div>
  <div class="conf-row" style="margin-bottom:1.5rem" id="metric-tabs">
    METRIC_TABS
  </div>

  <div class="ab-grid">
    <div class="ab-card variant-a">
      <div class="ab-card-label" id="label-a">VARIANT_A_NAME</div>
      <div class="input-row"><label>EXPOSURE_LABEL</label><input type="number" id="n_a" value="N_A_VALUE" min="1" oninput="calc()"></div>
      <div class="input-row"><label id="conv-label-a">CONVERSION_LABEL</label><input type="number" id="c_a" value="C_A_VALUE" min="0" oninput="calc()"></div>
    </div>
    <div class="ab-card variant-b">
      <div class="ab-card-label" id="label-b">VARIANT_B_NAME</div>
      <div class="input-row"><label>EXPOSURE_LABEL</label><input type="number" id="n_b" value="N_B_VALUE" min="1" oninput="calc()"></div>
      <div class="input-row"><label id="conv-label-b">CONVERSION_LABEL</label><input type="number" id="c_b" value="C_B_VALUE" min="0" oninput="calc()"></div>
    </div>
  </div>

  <div class="section-label">Confidence level</div>
  <div class="conf-row" id="conf-btns">
    <button class="conf-btn" onclick="setConf(0.90,this)">90%</button>
    <button class="conf-btn active" onclick="setConf(0.95,this)">95%</button>
    <button class="conf-btn" onclick="setConf(0.99,this)">99%</button>
  </div>

  <div class="metric-grid">
    <div class="metric"><div class="metric-label" id="rate-label-a">Rate A</div><div class="metric-value" id="cr_a">—</div></div>
    <div class="metric"><div class="metric-label" id="rate-label-b">Rate B</div><div class="metric-value" id="cr_b">—</div></div>
    <div class="metric"><div class="metric-label">Uplift</div><div class="metric-value" id="uplift">—</div></div>
  </div>

  <div id="result-box" class="result-box insufficient">
    <div class="result-title insufficient" id="result-title">Enter data above</div>
    <div class="result-body insufficient" id="result-body">Fill in your variant numbers to see significance.</div>
  </div>

  <div class="result-bar"><div class="result-bar-fill" id="bar-fill" style="width:0%;background:#5DCAA5"></div></div>
  <div style="display:flex;justify-content:space-between;font-size:11px;color:var(--color-text-tertiary);margin-top:4px">
    <span>0%</span><span id="conf-label">95% confidence threshold</span><span>100%</span>
  </div>

  <div class="stats-row">
    <div class="stat-item"><div>Z-score</div><div class="stat-val" id="zscore">—</div></div>
    <div class="stat-item"><div>p-value</div><div class="stat-val" id="pval">—</div></div>
    <div class="stat-item"><div>Power est.</div><div class="stat-val" id="power">—</div></div>
  </div>
  <div style="margin-top:1rem;font-size:12px;color:var(--color-text-tertiary)">One-sided z-test. Small samples are directional only — aim for 200+ per variant for reliable results.</div>
</div>

<script>
let confLevel = 0.95;

function updateNames() {
  const a = document.getElementById('name_a').value || 'Variant A';
  const b = document.getElementById('name_b').value || 'Variant B';
  document.getElementById('label-a').textContent = a;
  document.getElementById('label-b').textContent = b;
  document.getElementById('rate-label-a').textContent = 'Rate — ' + a;
  document.getElementById('rate-label-b').textContent = 'Rate — ' + b;
}

function setMetric(convLabel, btn) {
  document.getElementById('metric-tabs').querySelectorAll('.conf-btn').forEach(b => b.classList.remove('active'));
  btn.classList.add('active');
  document.getElementById('conv-label-a').textContent = convLabel;
  document.getElementById('conv-label-b').textContent = convLabel;
  calc();
}

function setConf(c, btn) {
  confLevel = c;
  document.getElementById('conf-btns').querySelectorAll('.conf-btn').forEach(b => b.classList.remove('active'));
  btn.classList.add('active');
  document.getElementById('conf-label').textContent = Math.round(c * 100) + '% confidence threshold';
  calc();
}

function normCDF(z) {
  const a1=0.254829592,a2=-0.284496736,a3=1.421413741,a4=-1.453152027,a5=1.061405429,p=0.3275911;
  const sign = z < 0 ? -1 : 1;
  z = Math.abs(z) / Math.sqrt(2);
  const t = 1/(1+p*z);
  const y = 1-(((((a5*t+a4)*t)+a3)*t+a2)*t+a1)*t*Math.exp(-z*z);
  return 0.5*(1+sign*y);
}

function sampleNeeded(pA, pB, conf) {
  const z = conf===0.90?1.282:conf===0.95?1.645:2.326;
  const pBar = (pA+pB)/2;
  const se = Math.sqrt(2*pBar*(1-pBar));
  const delta = Math.abs(pB-pA);
  if (!delta) return null;
  return Math.ceil(Math.pow((z*se)/delta, 2));
}

function calc() {
  const nA=parseFloat(document.getElementById('n_a').value)||0;
  const cA=parseFloat(document.getElementById('c_a').value)||0;
  const nB=parseFloat(document.getElementById('n_b').value)||0;
  const cB=parseFloat(document.getElementById('c_b').value)||0;
  if (!nA||!nB) return;
  const pA=cA/nA, pB=cB/nB;
  document.getElementById('cr_a').textContent=(pA*100).toFixed(1)+'%';
  document.getElementById('cr_b').textContent=(pB*100).toFixed(1)+'%';
  if (pA===0&&pB===0) { setResult('insufficient','No conversions yet','Both variants have 0 — nothing to compare yet.'); return; }
  if (pA===0) { setResult('insufficient','No conversions in A','Cannot compute uplift — A has 0 conversions.'); return; }
  const uplift=((pB-pA)/pA)*100;
  document.getElementById('uplift').textContent=(uplift>=0?'+':'')+uplift.toFixed(1)+'%';
  const seA=Math.sqrt(pA*(1-pA)/nA), seB=Math.sqrt(pB*(1-pB)/nB);
  const seDiff=Math.sqrt(seA*seA+seB*seB);
  if (!seDiff) { setResult('insufficient','Cannot compute','Standard error is zero.'); return; }
  const z=(pB-pA)/seDiff;
  const pval=1-normCDF(z);
  const zThresh=confLevel===0.90?1.282:confLevel===0.95?1.645:2.326;
  const power=normCDF(Math.abs(z)-zThresh)*100;
  document.getElementById('zscore').textContent=z.toFixed(3);
  document.getElementById('pval').textContent=pval.toFixed(4);
  document.getElementById('power').textContent=power>0?Math.round(power)+'%':'<1%';
  const barPct=Math.min(100,Math.max(0,(1-pval)*100));
  document.getElementById('bar-fill').style.width=barPct.toFixed(1)+'%';
  document.getElementById('bar-fill').style.background=pval<(1-confLevel)?'#1D9E75':'#EF9F27';
  const nameA=document.getElementById('name_a').value||'Variant A';
  const nameB=document.getElementById('name_b').value||'Variant B';
  if (nA+nB<20) {
    setResult('insufficient','Sample too small',`Only ${Math.round(nA+nB)} total — directional only. Need 200+ per variant for reliable significance.`);
  } else if (pval<(1-confLevel)) {
    const winner=pB>pA?nameB:nameA;
    setResult('sig',`Significant — ${winner} wins`,`${Math.round((1-pval)*100)}% confident this ${uplift>=0?'+':''}${uplift.toFixed(1)}% difference is real, not random noise.`);
  } else {
    const needed=sampleNeeded(pA,pB,confLevel);
    const msg=needed
      ?`${Math.round((1-pval)*100)}% confidence so far. At these rates, you'd need ~${needed.toLocaleString()} per variant to reach ${Math.round(confLevel*100)}%.`
      :`${Math.round((1-pval)*100)}% confidence — keep collecting data.`;
    setResult('not-sig','Not yet significant',msg);
  }
}

function setResult(type,title,body) {
  const box=document.getElementById('result-box'),t=document.getElementById('result-title'),b=document.getElementById('result-body');
  box.className='result-box '+type;
  t.className='result-title '+type; t.textContent=title;
  b.className='result-body '+type; b.textContent=body;
}

calc();
</script>

Metric tab templates — paste the right one as METRIC_TABS

Email outreach / cold email:

<button class="conf-btn active" onclick="setMetric('Opens',this)">Open rate</button>
<button class="conf-btn" onclick="setMetric('Clicks',this)">Click rate</button>
<button class="conf-btn" onclick="setMetric('Replies',this)">Reply rate</button>
<button class="conf-btn" onclick="setMetric('Conversions',this)">Custom</button>

Newsletter:

<button class="conf-btn active" onclick="setMetric('Opens',this)">Open rate</button>
<button class="conf-btn" onclick="setMetric('Clicks',this)">Click rate</button>
<button class="conf-btn" onclick="setMetric('Unsubscribes',this)">Unsub rate</button>
<button class="conf-btn" onclick="setMetric('Conversions',this)">Custom</button>

Paid ads (Google, Meta, LinkedIn):

<button class="conf-btn active" onclick="setMetric('Clicks',this)">CTR</button>
<button class="conf-btn" onclick="setMetric('Conversions',this)">Conv. rate</button>
<button class="conf-btn" onclick="setMetric('Installs',this)">Install rate</button>
<button class="conf-btn" onclick="setMetric('Custom',this)">Custom</button>

SaaS feature / flag / rollout:

<button class="conf-btn active" onclick="setMetric('Adoptions',this)">Adoption</button>
<button class="conf-btn" onclick="setMetric('Activations',this)">Activation</button>
<button class="conf-btn" onclick="setMetric('Retained',this)">Retention</button>
<button class="conf-btn" onclick="setMetric('Custom',this)">Custom</button>

Landing page / signup / pricing:

<button class="conf-btn active" onclick="setMetric('Signups',this)">Signup rate</button>
<button class="conf-btn" onclick="setMetric('Trials',this)">Trial rate</button>
<button class="conf-btn" onclick="setMetric('Upgrades',this)">Upgrade rate</button>
<button class="conf-btn" onclick="setMetric('Custom',this)">Custom</button>

Onboarding / activation flow:

<button class="conf-btn active" onclick="setMetric('Completed',this)">Completion</button>
<button class="conf-btn" onclick="setMetric('Reached step',this)">Step reached</button>
<button class="conf-btn" onclick="setMetric('Dropped off',this)">Drop-off</button>
<button class="conf-btn" onclick="setMetric('Custom',this)">Custom</button>

Step 3 — Post-render interpretation

Write 2–4 sentences after the widget. Do not explain the controls. Cover: what the result means, what to do next, any domain-specific caveats.

Domain caveats to include:

  • Email outreach: Small batches are normal — treat as directional until 100+ delivered per variant. Kill the loser and scale the winner once significant.
  • Paid ads: Rate significance doesn't capture ROAS or CPA. Recommend a 7-day minimum window and checking cost-per-conversion alongside the rate.
  • Newsletter: Unsub rates are typically <0.5% — need very large lists to reach significance. Open rate is the more actionable metric.
  • SaaS feature flag: Distinguish adoption (did they use it) from activation (did it move the outcome). Run through a full retention window (7 or 30 days) before deciding.
  • Landing page: Low base conversion rates mean long test duration. A lower-converting page at higher ACV can still win — flag that revenue impact matters more than rate alone.
  • Onboarding: Completion rate gains compound at scale. Note whether the change also affects downstream retention, not just the funnel step being tested.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment