In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. You find a thing that, in theory, makes the website better. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side. But is it going to move you up in a competitive ranking environment? But I hope you enjoyed this journey on surprising results from SEO A/B tests.
You can make all the tweaks and changes in the world, but how do you know they’re the best choice for the site you’re working on? Without data to support your hypotheses, it’s hard to say. In this week’s edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he’s seen that prove you can’t always trust your instinct in our industry.
Click on the whiteboard image above to open a high-resolution version in a new tab!
Video Transcription
Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I’m the founder and CEO at Distilled. At Distilled, one of the things that we’ve been working on recently is building an SEO A/B testing platform. It’s called the ODN, the Optimization Delivery Network. We’re now deployed on a bunch of big sites, and we’ve been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we’ve seen.
What is SEO A/B testing?
We’re going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.
Then you make the change that you’re hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn’t made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.
This is data that we’ve never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let’s get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you’ve ever seen.
Result #1: Targeting higher-volume keywords can actually result in traffic drops
I’ve stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there’s one particular way of searching for whatever it is that you offer that has more search volume than the way that you’re talking about it on your website right now, so higher search volume for a particular way of phrasing?
You make the recommendation, “Let’s talk about this stuff on our website the way that people are searching for it. Let’s put this kind of phrasing in our title and elsewhere on our pages.” I’ve made those recommendations. You’ve probably made those recommendations. They don’t always work. We’ve seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.
We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so…
COMMENTS