Back in 1996, 2 Stanford PhDs theorized a new kind of search engine. Instead of ranking results based on how many times a keyword appeared on a webpage, Larry Page and Sergey Brin figured it would be better to rank results based on the relationships between pages. They called their idea "BackRub" because it ranked search results based on backlinks.

That's pretty different compared to how search engines work today. Folio and Brin's search engine, Google, receives five.5 billion searches a mean solar day. Or 63,000 searches per second. For every ane of those queries, the search engine trawls through more than than 130 trillion private pages internet-wide and selects results in less than a second.

Backside those results lies a lot of groundwork. While Google — and other search engines — is notoriously secretive about the mechanisms backside search results, marketers benefit from knowing how search engines work. Understanding how search engines find, organize, and select results means you can better optimize your web pages to rank.

How Search Engines Work: The Basics

A "search engine" is several interlinked mechanisms that work together to identify pieces of web content — images, videos, website pages, etc. — based on the words yous type into a search bar. Site owners use Search Engine Optimization to improve the chances that content on their site will show up in search results.

Search engines utilise three basic mechanisms:

  • Web crawlers: Bots that continually browse the web for new pages. Crawlers collect the information needed to alphabetize a page correctly and use hyperlinks to hop to other pages and alphabetize them too.
  • Search index: A tape of all web pages online, organized in a way that allows association between keyword terms and page content. Search engines too have ways of grading the quality of content in their indexes.
  • Search algorithms: Calculations that grade the quality of web pages, figure out how relevant that folio is to a search term, and decide how the results are ranked based on quality and popularity.

Search engines try to deliver the most useful results for each user to keep large numbers of users coming back time and once more. This makes business concern sense, every bit most search engines brand money through advertizing. Google made an impressive $116B in 2018, for instance.

How Search Engines Clamber, Index, and Rank Content

Search engines await simple from the outside. You blazon in a keyword, and you get a list of relevant pages. But that deceptively easy interchange requires a lot of computational heavy lifting backstage.

The hard work starts way before you make a search. Search engines work round-the-clock, gathering information from the world's websites and organizing that data, so it's easy to detect. This is a three-step process of beginning crawling web pages, indexing them, then ranking them with search algorithms.

Itch

Search engines rely on crawlers — automated scripts — to scour the web for information. Crawlers start out with a listing of websites. Algorithms — sets of computational rules — automatically decide which of these sites to clamber. The algorithms also dictate how many pages to crawl and how frequently.

Crawlers visit each site on the list systematically, following links through tags like HREF and SRC to leap to internal or external pages. Over fourth dimension, the crawlers build an e'er-expanding map of interlinked pages.

Takeaway for Marketers

Make certain your site is easily accessible to crawlers. If bots tin't crawl it, they can't index it, and that means your site won't announced in search results. You can help guarantee crawler accessibility by implementing the following:

  • Logical site hierarchy: Define a logical site architecture that flows from domain to category to subcategory. This lets crawlers motion through your site more quickly, allowing the site to stay within its crawl budget.
  • Links: Use internal on every page. Crawlers need links to move between pages. Pages without any links are un-crawlable and therefore united nations-indexable.
  • XML sitemap: Brand a listing of all your website'due south pages, including blog posts. This list acts as an instruction transmission for crawlers, telling them which pages to clamber. There are plugins and tools—similar Yoast and Google XML Sitemaps—that volition generate and update your sitemap when you publish new content.

If you're non sure whether your site is accessible to crawlers, check out our Site Audit tool. The tool catches accessibility issues and gives advice on how to fix them. It likewise ship a fresh technical SEO written report for your site every two weeks, so you can stay on top of your site visibility for crawlers.

improve crawler accessibility

Alexa's Site Audit tool identifies pages without links, assuasive you lot to improve crawler accessibility.

Indexing

After finding a page, a bot fetches (or renders) it similar to the way your browser does. That means the bot should "come across" what yous see, including images, videos, or other types of dynamic folio content.

The bot organizes this content into categories, including images, CSS and HTML, text and keywords, etc. This procedure allows the crawler to "understand" what's on the page, a necessary precursor to deciding for which keyword searches the folio is relevant.

Search engines and then shop this information in an index, a behemothic database with a itemize entry for every word seen on every webpage indexed. Google's alphabetize, the Caffeine Index, takes upwardly around 100,000,000 gigabytes and fills "server farms," thousands of computers that never become turned off, around the world.

Takeaway for Marketers

Brand sure crawlers "see" your site how you lot want them to; control which parts of the site you permit them to alphabetize.

  • URL Inspection Tool: If you want to know what crawlers see when they land on your site, utilise the URL Inspection Tool. You can likewise use the tool to observe out why crawlers aren't indexing the page or request that Google crawl information technology.
  • Robots.txt: You won't want crawlers to prove every page of your site in SERPs; writer pages or pagination pages, for instance, can exist excluded from indexes. Utilise a robots.txt file to command access past telling bots which pages they tin crawl.

Blocking crawlers from sure work-a-24-hour interval areas of your site won't affect your search rankings. Rather, it'll help crawlers focus clamber budget on the most of import pages.

Ranking

In the terminal step, search engines sort through indexed information and return the right results for each query. They do that with search algorithms, rules that clarify what a searcher is looking for and which results best answer the query.

Algorithms use numerous factors to define the quality of the pages in their index. Google is leverages a whole serial of algorithms to rank relevant results. Many of the ranking factors used in these algorithms clarify the general popularity of a piece of content and even the qualitative experience users take when they land on the page. These factors include:

  • Backlink quality
  • Mobile-friendliness
  • "Freshness," or how recently content was updated
  • Engagement
  • Page speed
  • To brand sure that the algorithms are doing their job properly, Google uses homo Search Quality Raters to exam and refine the algorithm. This is one of the few times when humans, not programs, are involved in how search engines work.

Takeaway for Marketers

Search engines want to bear witness the almost relevant, usable results. This keeps searchers happy and advertising revenue rolling in. That's why about search engines' ranking factors are actually the same factors that human being searchers estimate content by such as page speed, freshness, and links to other helpful content.

When designing and refreshing websites, optimize folio speed, readability, and keyword density to ship positive ranking signals to search engines. Working to improve engagement metrics like time-on-page and bounciness rate tin also help boost rankings.

When designing and refreshing websites, optimize page speed, readability, and keyword density to transport positive ranking signals to search engines. Click To Tweet

Detect out more near how to rank on Google.

What Happens When a Search Is Performed?

Now nosotros know almost the three-step process search engines use to render relevant results. Crawling, indexing, and ranking allow search engines to find and organize information. But how does that help them respond your search query?

Let's walk through how search engines answer queries step-by-step, from the moment you lot type a term in the search bar.

Footstep i: Search Engines Parse Intent

To render relevant results, search engines accept to "sympathise" the search intent behind a term. They use sophisticated language models to do that, breaking downwardly your query into chunks of keywords and parsing meaning.

For instance, Google's synonym organisation allows the search engine to recognize when groups of words hateful the same affair. And then when you blazon in "dark colored dresses," search engines volition render results for black dresses as well as dark tones. The engine understands that dark is often synonymous with black.

understanding the search intent with Google's synonym system

Search results of "dark colored wearing apparel" pull up synonymous results as well.

Search engines also use keywords to empathize wide "categories" of search intent. In the "dark colored dress" instance, the term "buy" signals to search engines that information technology should pull upward product pages to friction match a shopping searcher's intent.

Find out how to optimize for semantic search.

Search results too use "freshness" algorithms to empathize searcher intent. These algorithms identify trending keywords and return newer pages. You lot'll come across this for terms such as "ballot results," which return radically different SERP results during election time and non-election time.

Step two: Search Engines Match Pages to Query Intent

Once the search engine understands what kind of result you want to see, it needs to discover matching pages. A serial of factors help the search engine determine which pages are best, including:

  • Title/content relevance
  • Types of content
  • Content quality
  • Site quality and freshness
  • Folio popularity
  • Language of query

Then if you search "best places to eat sushi," search engines volition friction match list pages with "sushi" or synonyms (eastward.g., "japanese nutrient") in the championship and body content. They'll order these results based on popularity, freshness, and quality factors.

Depending on the search intent, search engines may besides prove enriched results such as the knowledge graph or image carousel.

Step 3: Search Engines Use 'Localized' Factors

A number of private factors come into play when search engines make up one's mind which results you come across. You may meet different results for "all-time frozen cheese pizza" than a friend who lives in another state thanks to a combination of private factors.

  • Location: Some searches, similar "restaurants near me," are obviously location-dependent. Just Google will rank results for local factors fifty-fifty in non-location-specific searches. A search for "football" volition probable show pages about the Steelers to someone in Pittsburgh and pages about the 49ers to someone in San Francisco.
  • Search settings: Search settings are also an important indicator of which results y'all're likely to find useful, such as if y'all set a preferred language or opted into SafeSearch (a tool that helps filter out explicit results).
  • Search history: A user's search history too influences the results they see. For instance, search for the term "hemingway," and you lot'll see results for both the author and the editing app. Click on a few results about the writer, and run a search for "hemingway" again. This fourth dimension, you lot'll see a greater number of results about the writer than the app.

Takeaway for Marketers

Search results are highly specific and dynamic. It's impossible to predict when and how your site will appear to each individual searcher. The best approach is to send potent relevance signals to search engines through keyword research, technical SEO, and content strategy. That way, you'll show up in SERPs that are truly relevant to your content.

Related: How to Get Canonical as a Google News Source

Utilize This Noesis To Boost Results

Once you know how search engines work, it's easier to create websites that are crawlable and indexable. Sending the right signals to search engines guarantees that your pages appear in results pages relevant to your business. Serving up to searchers, and search engines, the content they desire is a step along the path to a successful online business.

Sign up for a free trial of Alexa'south Advanced Program to get the Site Audit tools you need to make sure your content is in skilful condition for crawlers. Plus, you get access to comprehensive reports that place technical and on-page optimization opportunities you lot may have missed.

Google and the Google logo are registered trademarks of Google LLC, used with permission.