Null hypothesis

What is the null hypothesis?

The null hypothesis (mathematically denoted as H0) is the foundational, default scientific and statistical proposition in experimental hypothesis testing which posits that there is no true difference, no measurable effect, no causal relationship, and no experimental variation between the specific populations, treatments, or variables being evaluated. In digital marketing experimentation, conversion rate optimization (CRO), web analytics, and product feature testing, the null hypothesis serves as the rigorous mathematical benchmark of skepticism.

When a growth marketing team launches a controlled A/B test—such as testing a new checkout button design against the current production baseline—the null hypothesis formally states: "The new checkout button has zero genuine effect on conversion rates; any observed difference in performance between the test variant and the control variant is the product of pure random sampling variance." Under classical frequentist statistics, an experimentation team cannot set out to "prove" their new idea; they must gather sufficient empirical data to reject the null hypothesis beyond a reasonable threshold of statistical doubt.

The legal analogy: Presumption of innocence in statistics

To understand the philosophical and operational necessity of the null hypothesis, experimenters frequently utilize the legal analogy of a criminal trial: the presumption of innocence. In a court of law, a defendant is presumed innocent until proven guilty beyond a reasonable doubt. The prosecution cannot demand that the defendant prove their innocence; the burden of proof rests entirely upon the prosecution to present overwhelming evidence of guilt.

In statistical experimentation, the null hypothesis represents the presumption of innocence:

• Presumption: The experimental marketing variation is completely ineffective (H0 is assumed true).
• Burden of Proof: The growth marketing team must collect sufficient conversion data to prove that the observed lift cannot reasonably be explained by random chance.
• Alternative Hypothesis (H1 or Ha): Represents the claim of guilt (effectiveness)—the proposition that a genuine, measurable difference exists between the variants.
• The Verdict: If the empirical evidence is strong enough, the team "rejects the null hypothesis" and declares a statistically significant winner. If the evidence is insufficient, the team "fails to reject the null hypothesis." Statisticians never say they "accept the null hypothesis," just as a jury delivers a verdict of "not guilty" rather than declaring innocence.

P-values and statistical significance thresholds

The mathematical mechanism used to evaluate the null hypothesis is the p-value. The p-value is the probability of observing experimental results at least as extreme as the actual observed test data, assuming that the null hypothesis is completely true.

To determine whether a result is strong enough to reject the null hypothesis, experimenters establish an a priori statistical significance threshold known as alpha (typically set at 0.05):

• P-Value Below Alpha (p < 0.05): The observed conversion lift is so rare under the assumption of the null hypothesis that it would occur by random chance less than five percent of the time. The team rejects the null hypothesis and declares statistical significance at a 95% confidence level.
• P-Value Above Alpha (p >= 0.05): The observed performance difference is well within the realm of random sampling fluctuation. The team fails to reject the null hypothesis, concluding that there is insufficient statistical evidence to support adopting the new design.

A critical misunderstanding among marketing practitioners is assuming that a p-value of 0.03 means there is a 97% probability that the variant is superior. The p-value tells you nothing about the probability of the hypothesis itself; it measures only the probability of the observed data given that the null hypothesis is true.

Type I and Type II errors in commercial experimentation

Executing statistical hypothesis tests on commercial websites inevitably exposes organizations to two fundamental categories of decision errors:

• Type I Error (False Positive, Alpha): Occurs when an experimenter rejects the null hypothesis when it was actually true. The team falsely declares that a new landing page design generated a ten percent conversion lift when, in reality, the variation was functionally identical to the control. The business wastes engineering resources permanently deploying a variation that provides zero commercial benefit. Alpha (usually 0.05) sets the maximum acceptable probability of committing a Type I error.
• Type II Error (False Negative, Beta): Occurs when an experimenter fails to reject the null hypothesis when it was actually false. A genuine, valuable eight percent conversion lift truly existed, but because the experiment was stopped prematurely with inadequate sample size, the test lacked the statistical resolution to confirm the lift. The business discards a profitable product enhancement. Statistical Power (1 - Beta, usually set at 0.80 or 0.90) represents the probability of correctly rejecting a false null hypothesis.

Balancing Type I and Type II errors requires conducting rigorous pre-test power calculations to guarantee adequate sample sizes before deploying experiments.

Common pitfalls in null hypothesis interpretation

Even experienced digital growth teams frequently stumble into severe analytical fallacies when interpreting null hypothesis tests. Foremost among these is the "Absence of Evidence Fallacy." When an A/B test concludes with p = 0.24, marketing teams frequently declare: "We have proven that changing the headline makes no difference to our customers." This is completely false. Failing to reject the null hypothesis does not prove that no difference exists; it proves only that the experiment lacked the sample size or statistical power to detect whatever subtle effect may have been present.

Another pervasive failure mode is Continuous Peeking Bias. In automated testing platforms, analysts frequently check test dashboards daily. If the dashboard flashes a green "Significant Lift (p < 0.05)" on day four, eager marketers stop the test immediately and declare victory. Repeatedly checking p-values as data accumulates inflates the true Type I error rate from five percent to over thirty percent, turning random noise into false winners. Tests must run for their full planned duration and pre-calculated sample size regardless of early dashboard fluctuations.

Strategic framework for rigorous hypothesis governance

Institutionalizing statistical integrity across an experimentation program requires a standardized operational methodology that protects commercial decision-making from false discoveries. High-performing experimentation teams execute a five-stage operational framework:

• Step 1: Formal Null and Alternative Pre-Registration. Before writing a line of test code, explicitly document both H0 (no effect) and H1 (expected lift direction and primary metric) in an immutable experiment charter.
• Step 2: A Priori Power and Duration Calculations. Execute two-tailed statistical power calculations to establish required sample sizes per variant based on historical baseline conversion rates, minimum detectable effects (MDE), and acceptable error rates (alpha = 0.05, power = 0.80).
• Step 3: Fixed-Horizon Sample Enforcement. Lock test runtimes to pre-calculated sample sizes, strictly forbidding team members from terminating experiments early based on intermediate p-value peeking.
• Step 4: Multi-Testing and Multiplicity Adjustments. Apply automated statistical corrections (Bonferroni, Holm-Bonferroni, or Benjamini-Hochberg) whenever multiple variations or secondary metrics are tested simultaneously.
• Step 5: Post-Test Business Decision Protocols. If the null hypothesis is rejected, permanently deploy the winning code to production; if the team fails to reject H0, archive the qualitative learnings and iterate with a bolder design hypothesis.

Related terms

• Alternative hypothesis: The experimental proposition stating that a genuine, measurable difference or effect exists between variables.
• P-value: The probability of observing results at least as extreme as the actual data assuming the null hypothesis is true.
• Statistical significance: A quantitative standard determining that observed test variations are unlikely due to chance.
• Statistical power: The mathematical probability that a test correctly rejects a false null hypothesis when an effect exists.
• A/B testing: Experimental method comparing two versions to determine which generates superior performance.