How Google’s Search Engine Really Works (A Peek Under The Hood)

Source: Neil Patel

Google’s search engine is technically complex.

There are hundreds (some say thousands) of different factors taken into account so that the search engine can figure out what should go where.

It’s like a mysterious black box, and very few people know exactly what’s inside.

However, the good news is that search engines are actually pretty easy to understand.

We may not know every single factor (out of a hundred or thousand), but we also don’t need to.

I’ll bring it down to the basics with a simple method to please Google, rank higher, and bring in more website traffic.

I’m also going to introduce you to some of the latest developments, like RankBrain, that help Google guess what you’re actually looking for (even if you don’t type it in).

But first, I’m going to walk you through exactly how Google’s search engine really works so that you can see that it’s not as difficult to understand as you might think.

How do search engines crawl the web?

Google’s first job is to ‘crawl’ the web with ‘spiders.’

These are little-automated programs or bots that scour the ‘net for any and all new information.

The spiders will take notes on your website, from the titles you use to the text on each page to learn more about who you are, what you do, and who might be interested in finding you.

That may sound simplistic on the face of it.

But that’s no small feat considering that there are anywhere from 300-500 new web pages created every single minute of the day.

So the first massive challenge is to locate new data, record what it’s about, and then store that information (with some accuracy) in a database.

Google’s next job is to figure out how to best match and display the information in its database when someone types in a search query. Scaling becomes a problem again, though.

Google now processes over two trillion searches in a single year. That’s up from only one billion a year in 1999.

That’s roughly a 199,900% volume increase in the last seventeen years!

So the information in its database needs to be categorized correctly, rearranged, and displayed in less than a second after someone expects it.

And time is of the essence here. Speed wins, according to Marissa Mayer back when she worked for Google over a decade ago.

She reported that when they were able to speed up the time it took the Google Maps home page to load (by cutting down on its size), traffic leaped 10% within seven days and 25% just a few weeks later.

Google won the search engine race, then, because it’s able to:

1. Find and record more information
2. Deliver more accurate results
3. And do both of those two tasks faster than any other engine

It’s gotten incredibly good at shuttling information back-and-forth across its “pipeline,” which connects users to its database of information.

One of the reasons Google jumped out to an early head start on all of this stuff came down to the accuracy of its results.

The information it displayed was simply a lot better.

Think about it this way.

When you type something into Google, you’re expecting something. It might be a simple answer, like the weather in your city, or maybe a little more complex, like “how does Google’s search engine really work?”

Google’s results, compared to other alternatives at the time, answered those queries better. The information was the best of the best.

And this breakthrough came from an initial theory Google’s co-founders actually worked on in college.

Why do links matter?

Google’s co-founders were still back in Stanford in 1998 when they released a paper entitled “The PageRank Citation Ranking: Bringing Order to the Web.”

Check it out — you can even read the whole thing right here!

The PageRank breakthrough was simple.

Academic papers were often ‘ranked’ by the number of citations a paper received. The more they received, the more authoritative they were considered on that topic.

Google co-founders, Larry Page and Sergey Brin, wanted to apply the same ‘grading’ system to the web’s information. They used backlinks as a proxy for votes. The more links a page received, the more authoritative it was perceived on that particular topic.

Of course, they didn’t just look at the number of links. They also factored in quality by considering who was doing the linking.

If you received two links, for example, from two different websites, the one with the more ‘authority’ on a topic would be worth more.

They also considered relevance to better gauge the ‘quality’ of a link.

For example, if your website talks about “dog food,” links from other pages or sites that talk about things related to “dogs” or “dog food” would be worth more than one talking about “truck tires.”

Now, before we go any further, please understand that we’re talking about concepts that are over a decade old.

PageRank may have mattered years ago, but it’s evolved tremendously since then. So don’t worry about it explicitly today.

One of the reasons is because of new algorithm developments like RankBrain.

What is ‘RankBrain’ and how does it work?

RankBrain was first acknowledged only a few years ago by Google engineer Greg Corrado:

RankBrain has become the third-most important signal contributing to the result of a search query.

Google’s been working on this technology for the past five years to help the search engine handle the massive increases in volume without losing accuracy.

The RankBrain secret sauce is that it uses artificial intelligence to continually learn how to improve.

So the more it processes new information or new search queries for users, it actually gets better and more accurate at returning this information.

For example, Google’s algorithm “might have up to 10,000 variations or sub-signals,” according to Search Engine Land. That’s a lot!

As you can imagine, somehow managing all of those on the fly would be incredibly difficult (if not impossible).

That’s where RankBrain comes in to help manage the workload.

Generally, the two most important ranking factors are:

1. Links (and citations)
2. Words (content and queries)

RankBrain…

Click here to read more