Skip to content
We use cookies to improve the site and measure traffic. See our Cookie Policy. You can accept or reject non-essential cookies.
ToolYour Logo
  • Free
  • No signup
  • Fast
  • Privacy-friendly

Back to Category

Marketing tools

A/B Test Significance Calculator

After an A/B test, you need a two-proportion z-test on visitors and conversions for A vs B.

Campaign workspace

A/B Test Significance Calculator

Structured inputs and readable results — copy URLs, scan limits, and review findings without digging through JSON.

Two-proportion z-test on visitors and conversions for A vs B.

Results

Waiting

Results appear here

Fill the form and run the tool — outputs are structured cards, not a raw JSON dump.

Free · API · MCPToolYour platform

Use A/B Test Significance Calculator three ways

The free A/B Test Significance Calculator on this page works in your browser. The same workflow is available via REST API for apps and via MCP for Cursor, Claude, and other agents — one API key, one plan, one quota. This tool is part of the marketing-apis module under Marketing Tools.

  • 1. Get an API keySign in and open Dashboard → API Keys (works for REST and MCP).
  • 2. Call or connectUse the REST API from your server, or paste the MCP URL into your agent config.
  • 3. Same qualityOutput matches the free web tool across all three surfaces.
Manage API keysMCP setupAPI referenceCreate free account

What is A/B Test Significance Calculator?

After an A/B test, you need a two-proportion z-test on visitors and conversions for A vs B. ToolYour A/B Test Significance Calculator returns rates, lift, z-score, p-value, and a 95% significance flag. It does not adjust for peeking, multiple comparisons, or CUPED. Same route via REST and MCP (POST /api/v1/marketing-apis/ab-test-significance-calculator).

What are common questions about A/B Test Significance Calculator?

What inputs?

visitorsA, conversionsA, visitorsB, conversionsB (aliases n1/c1/n2/c2). Conversions cannot exceed visitors.

Does p < 0.05 mean ship B?

It means the difference is statistically significant at 95% under this test’s assumptions — still weigh practical impact, risk, and experiment quality.

Can I check significance mid-test, before it's fully run?

You can, but repeated peeking without correction inflates false positives. Prefer checking once at your pre-planned sample size, or use sequential-testing methodology if you need to peek.

What are the key features of A/B Test Significance Calculator?

z-test outputs

Absolute/relative lift, z, p-value, significant95.

How do you use A/B Test Significance Calculator?

Enter A/B visitor and conversion counts

From your experiment platform export.

Interpret with care

Confirm you did not peek early or underpower the test.

Why this tool exists

After an A/B test finishes, someone needs to know whether the difference between A and B is statistically real or just noise — and the two-proportion z-test that answers this is easy to get wrong by hand under time pressure. A/B Test Significance Calculator returns rates, lift, z-score, p-value, and a 95% significance flag from raw visitor/conversion counts.

Problems it solves

'Variant B converted better' isn't the same question as 'is this difference statistically significant', and conflating the two leads to shipping changes based on noise. A consistent, formula-transparent significance check keeps that distinction explicit in every result.

Common use cases

  • In-house CRO team: checking whether a completed landing page test's result is significant before rolling out the winner.
  • Agency test reporting: giving a client a clear significance read alongside raw lift numbers in a results deck.
  • Agent/MCP: an agent computing significance as one step in an automated experiment-reporting playbook.
  • QA before spend: confirming a test wasn't called early on a false-positive p-value before scaling the winning variant.

Honest scope

A p-value under 0.05 means the difference is statistically significant at 95% under this test's assumptions — it does not adjust for peeking mid-test, multiple comparisons across several metrics, or CUPED-style variance reduction. Weigh practical impact, business risk, and experiment quality alongside the number, not instead of it.

Where do you browse more marketing tools tools?

Related tools

Popular in Marketing Tools

Which toolkits include A/B Test Significance Calculator?