SEO split testing: how to prove a change actually worked when AI Overviews keep moving your traffic

A before-and-after chart no longer proves that an SEO change worked. In 2026, core updates, seasonality, AI Overviews and AI Mode move organic traffic even when a brand changes nothing on its site. The way to isolate the effect of a change from everything else is a page-based split test, with a control group and a decision rule set before the test begins.

That is the recommendation of an analysis published by Search Engine Journal on November 9, written from an agency's point of view. According to the piece, the right question is no longer whether traffic went up. What matters is whether it went up more than it would have without the change, an estimate known as the counterfactual.

Why you split pages, not users

A CRO test splits visitors between two versions of a page, and that does not work for SEO. Google ranks only one version of each URL at a time. Showing Googlebot a different page from the one people see is cloaking, which Google's guidance on testing sites treats as a violation of its spam policies, even during a test.

So an SEO test takes a set of pages built on the same template, such as product, category or location pages, and splits them into two groups. The variant pages get the change and the control pages stay as they are. Pages competing for the same queries are kept together, the groups are balanced by traffic level and category, assignment is random, and both groups run over the same dates.

The control group is used to predict what the variants would have done without the change, and the gap between that prediction and the actual result is the effect. Before launch, the analysis suggests testing the model on fake dates in the past: if it finds an effect where nothing happened, it will not be able to measure the change. CausalImpact, Google's open source package, is the best known model for this kind of prediction.

The brief comes before the test

The piece recommends a one-page brief, shared before any change goes live:

• the hypothesis: what changes and why it should help;
• the primary metric: for most on-page changes, non-branded organic clicks to the tested pages, tracked alongside total clicks, because Search Console drops anonymized queries when you filter by query;
• the guardrail metrics, such as conversion rate, which must not fall past an agreed threshold;
• the smallest gain that justifies acting;
• the analysis date, in full weeks;
• the decision rule: what counts as rolling out, rolling back or logging the result as inconclusive.

When it is time to present, a range is more persuasive than a single number. Saying clicks rose 14% invites debate; saying the model puts the gain between 8% and 19% is harder to dispute. If the interval includes zero, the result shows no positive effect, and a 90% probability of a positive effect still leaves a one in ten chance that the change did nothing or made things worse.

AI visibility belongs in the math

According to Google, Search Console's generative AI performance reports reached all sites on August 31, 2026, except those with few AI impressions. They show, page by page, how often content appeared in AI Overviews and AI Mode, which makes it possible to compare variants against the control group. There is no click data yet, so the number works as a supporting signal. And, as Google's John Mueller confirmed, every link inside an AI Overview is assigned the position of the AI Overview itself.

For brand teams, the takeaway is to measure presence in AI answers alongside clicks: if a change lifts one and drags down the other, it is worth investigating before rolling it out. Smaller sites that cannot support a full test can keep a set of pages untouched, use Search Console's branded queries filter and annotate the dashboard with the day the change went live. Tests that did not work out should be reported too, because they are what make the wins credible.