SEO A/B Testing: How to Run Split Tests on Page Templates

An SEO split test changes a group of pages and compares their search performance with a similar group left unchanged. It is not the same as the A/B testing used in conversion optimization, where each visitor is randomly shown version A or B of the same page. Search engines see one version of each URL, so SEO tests split pages, not people: some product pages get the new title format, comparable product pages keep the old one, and the question is whether the changed group’s organic traffic moved differently from the control group’s. Done carefully, this gives evidence for template-level decisions. Done carelessly, it measures seasonality, algorithm updates or noise and calls it a result.

Two different kinds of test

Google’s guidance on website testing addresses the visitor-level kind. It says not to show one set of URLs to Googlebot and another to humans, because “this is called cloaking, and is against our spam policies, whether you’re running a test or not”; to use 302 redirects, not 301, when a test redirects users to variant URLs; to use rel="canonical" on alternate URLs; and to run experiments “only as long as necessary” (Google on website testing). It adds that Googlebot generally does not support cookies, which matters for cookie-based test assignment.

Designing an SEO split test

1. Hypothesis

State the change and the expected effect in one sentence: “Adding the product’s key specification to title elements on product pages will increase organic clicks to those pages.”

2. Population and cohorts

• Choose a population of pages that share a template and a search intent (for example, 2,000 product pages in one category tree).
• Split them into a test group and a control group. Random assignment is a starting point; stratify so both groups have similar traffic levels, seasonality and categories.
• Exclude pages with too little traffic to contribute, pages under other changes, and pages likely to be affected by external events (a product launch, a promotion).

3. Pre-period

Collect at least several weeks of data for both groups before the change. The pre-period shows how closely the groups move together without intervention. If they diverge already, the comparison will not be reliable.

4. Change

Apply the change to test pages only. Do not change control pages during the test. Log the deployment date and confirm with URL Inspection on a sample that Google has recrawled the changed pages.

5. Measurement period

Allow enough time for recrawling and for any effect to show. Avoid starting or running a test across a Google core update or spam update window if possible; check the Search Status Dashboard.

6. Analysis

Compare the test group’s actual performance with what the control group predicts it would have been. One established method is a Bayesian structural time-series model, such as Google’s open-source CausalImpact package, which estimates “the causal effect of a designed intervention on a time series” (CausalImpact documentation). Its documentation states the conditions: “valid conclusions require strong assumptions”, including that the control series “were themselves not affected by the intervention” and that the relationship between control and test series established in the pre-period “remains stable throughout the post-period”.

Report the estimated effect with its uncertainty interval, not only a point estimate, the same discipline a forecast’s scenario ranges require.

A hypothetical test design

Say a hardware retailer is testing title changes on product pages.

• Hypothesis: adding the main dimension (for example, “18-inch”) to product title elements increases organic clicks.
• Population: 1,600 product pages in the “power tools” tree with at least 30 organic clicks per month.
• Cohorts: 800 test, 800 control, stratified by sub-category and pre-period clicks.
• Pre-period: 8 weeks; test and control daily clicks correlate closely.
• Change: template update for test pages; deployment confirmed; sample recrawl confirmed after 10 days.
• Measurement: 6 weeks after recrawl.
• Result (illustrative): test pages receive an estimated 6 percent more clicks than predicted, with an interval from 1 to 11 percent.

Valid interpretation: under the model’s assumptions, the title change is associated with a modest increase in clicks for this page population during this period.

Limitations:

• Google may rewrite title links, so the change may not appear in every result.
• A competitor’s change or a SERP feature appearing only for some product types could affect one group more than the other.
• The result applies to this template and population; it does not prove the change will work elsewhere.
• An interval that includes zero would mean the test could not distinguish an effect from noise.

Threats to validity

• Too few pages or too little traffic: noise dominates.
• Unbalanced groups: the test group contains seasonal products the control lacks.
• Contamination: internal links from test pages change control pages’ performance.
• Simultaneous changes: another release during the test.
• Algorithm updates: affect groups differently.
• Stopping early: checking daily and stopping when the result looks good inflates false positives.

When not to run an SEO split test

• The site has too few comparable pages.
• The change is required anyway (a bug fix, an accessibility requirement).
• The change affects the whole site at once (a domain migration), so no control group exists; use a before-and-after analysis with explicit caveats instead.

Cleaning up

Google’s testing guidance asks site owners to remove all elements of a test as soon as it concludes, such as alternate URLs or testing scripts, and warns that running an experiment for an unnecessarily long time may be interpreted as an attempt to deceive search engines. For SEO split tests that change page templates directly, the cleanup is a decision: roll the change out to all pages, or revert the test group.

An SEO test produces evidence, not certainty. A clear hypothesis, balanced cohorts, a stable pre-period and an honest uncertainty interval are what turn a template change into a decision someone can defend.