SEO & Statistics: How Bayes’ Theorem Can Make Our Job Easier

Nate Silver, of 538 fame, is a big fan of Bayesian statistics. In his book The Signal and The Noise, he applies it to weather prediction, chess, poker, and politics, the financial collapse of 2008, stock trading, and terrorism. Basically, a bunch of subjects you thought you understood until he talks about them (he’s super smart). If it can apply to this wide range of industries, can we apply it to SEO and inbound marketing?

First, there’s a good chance you don’t know what Bayesian statistics is. You’re in luck, because its really cool and super simple at the same time. Basically, it’s a way of determining the probability of something based on another event. The book gives the specific definition of “it tells us the probability that a theory or hypothesis is true if some event has happened.”

It’s kind of hard to visualize without an example, so let’s jump into one. The example that Nate gives in the book is really good: let’s say you’re a woman who’s in a relationship with a man and you come home and find a strange pair of women’s underwear in your drawer. Can you determine what the chances are that he’s cheating on you given that this event has happened? Let’s use Bayes’s Theorem to do just that.

Here’s the equation:

xy/(xy+z(1-x))

That’s it. A fraction with 3 x’s, 2 y’s, 1 z, and a 1 in it. Here’s what the variables are:

X, or the “prior probability”

This is the probability that the hypothesis was true before the event happened. In this case, it’s what you thought the chances of him cheating on you were before you found the underwear. So what would that be? Let’s say you had an average level of trust in your relationship, and we’re calling “average” what the U.S. News & World Report found from a poll in 2008. There’s apparently, on average, a tiny bit of doubt in every relationship, which makes sense. They found that the average amount of un-trust in a normal American relationship is about 4%, so let’s go with that.

Y, or the “if true” variable

This is the probability that the event happened and the hypothesis is true. In this case, it’s what the chances would be of the underwear appearing if he is cheating on you. So what would that be? Chances are pretty high that he’s cheating, so you’d think that this would be 100%, but Nate brings up a good point. If he is cheating, he will probably make some kind of effort to be secretive about that fact, so leaving a pair of underwear around would actually not be that common. He puts the number at 50%.

Z, or the “if false” variable

This is the probability that the event happened and the hypothesis is false. In this case, it’s what the chances would be of the underwear appearing if he is not cheating on you. So what would that be? This one is a little funny because you’d think it would be 0%, but he again brings up good points. Maybe it’s a gift for you for later? Maybe it’s his sisters pair and somehow got mixed up in his laundry? It could happen, even if it’s unlikely. He puts the number at 5%.

So there we go. The final equation is just this:

(.04)(.5)/((.04)(.5)+(.05)(1-(.04)))

And the answer ends up being quite low: only 29%. That may seem a bit off, but it’s heavily dependent on the “x” variable. After all, your thought of the chances of him cheating on you before the event occurred were very low. Using this system, any single event cannot drastically change the probability of theory being true or not. That is one of its most attractive attributes. It builds stability into any system to which you apply it. For example, you wouldn’t assume that a restaurant is serving unsanitary food all the time because you found one hair in your burger. Chance events occur, so it’s important to keep our “prior probabilities” in mind when evaluating a situation.

It gets interesting when you start to use the theorem repeatedly. Let’s say, a week later, you found another pair of underwear. If we just change our x variable to .29, the final answer jumps quite a bit, to 80%. If, instead, there was an event that reassured you he wasn’t cheating, it may lower the probability back closer to 4%. The important takeaway is that it is an ever-changing, scientific process that is never blinded by biases or snap judgements.

Anyways, that’s the theorem. He goes on to give a few more examples and references the idea throughout the entire book. Really fascinating stuff. In all the examples he mentions, the reason he gives as to why the Bayes’s Theorem is so powerful is because it makes us think in probabilities when it comes to causes and effects. It’s easy to see an event and think of its cause and effect as a black-and-white issue, when in reality, thinking in terms of “chances” can keep us objective.

So how can we apply this to search marketing?

There are many parts of our job that rely on prediction, testing, and verification. It’s the reason SEO has a healthy dose of “art” to it in addition to science–it’s hard to tell exactly what causes what because Google’s algorithm is a black box that no one can decipher. Maybe we could use Bayes’ theorem to make our job easier?

For example, let’s say you change one of the most basic SEO elements: the title tag. Traditionally, you’d have some kind of prediction in mind. For example, let’s say you’re starting with a brand new site that only has the brand name in the title tags. Your guess is that adding a keyword to it will help the page rank better. You make the change, you see what happens, and you decide if your prediction was accurate or not based on what happened.

The problem with this standard of testing is that it relies on a single, unscientific judgement of the effect of the change. It doesn’t weigh things probabalistically. Let’s say you made the change and the rankings for that page went up slightly. The result may tempt you to assume that the title tag change worked as expected, and the rest of the site should mirror the pattern. But remember: one event should never greatly impact a hypothesis. Let’s see what this would look like if we used Bayesian statistics.

Bayes’ Theorem and Title Tag Changes

Let’s say we’re testing the hypothesis that adding a keyword to the title tag will increase a page’s ranking for that keyword, and our first test has shown a slight increase. The “x” variable would be how sure we were that the change was going to increase rankings. Let’s just assume we’re being conservative and our guess is about a 90% chance–we’re not convinced so let’s test it out.

The “y” variable is our probability that the rankings went up if our prediction was true. We’d have to consider all the external factors–is there a lot of churn in the SERPs lately? Has there been a change in any competitor websites? Has anyone made other changes on the page or site that could affect the rankings? I’d guess that there are many external factors, so we can’t assume it was our doing–let’s say it’s 60%.

The “z” variable is our probability that the rankings went up if our prediction was not true. This is the interesting one. Could there be so many external factors that, even though the title tag change was not helpful, our rankings went up anyways? We can’t really rule it out, especially if other people have been making changes to the site, so let’s call it 15%.

(.9)(.6)/((.9)(.6)+(.15)(1-(.9)))

We end up with a 97% change that our prediction was correct. That’s pretty convincing, mostly because our prior probability was quite high. If we were really unsure of our prediction (say the prior probability is 60%), we only end up with an 85% final answer. Or, if we were 90% sure and the rankings actually went down (which just means we’d switch our y and z variables), the final answer would be a 69% probability.

The point is this: instead of a one-time assumption of the results of an experiment, we can continue to test the theory. Each test result would be the prior probability for the next one. That way, if the test proved successful over and over again, we’d arrive at a scientifically proven conclusion that the hypothesis was correct. If the tests flip back and forth from successful to not, we may need to keep testing until we can come to a conclusion. If it never trends toward one side or the other, we can decide the test was a wash and can move on to the next one.

The title tag example is an easy one, but this applies to a wide range of problems. Changes in copy affecting conversion rate, making the logo bigger affecting time on site. I could even see it applied to predict how an algorithm change (like Panda) may affect a website based on certain criteria. By adding more scientific rigor to our work, we should be able to make more meaningful conclusions and limit uncertainty. I’m a fan of making SEO and inbound marketing more and more scientific. My guess is that it could make the job more quantifiable, and therefore would hold more clout with executives that need hard numbers. I could see it being especially helpful in areas like social media where results are infamously hard to track. Let’s leave the art to the artists.

What do you think?