A crawler is a program used by search engines that traverses the internet in order to gather and index data.

A crawler will go to a website through a hyperlink. The crawler will then read the website’s content and embedded links before following the links away from the website. The crawler will continue this until it has visited and indexed data from every single site that has a link to another website. It basically crawls the internet, hence the name.

Why are crawlers important to SEO?

SEO is influenced by crawlers in many ways.

Sites that are easier to crawl will be favoured to those that aren’t. If a website is easy to visit and navigate, it will not only make your site more readable for crawlers, but for users too. Crawlers also use sitemaps to help them identify the important content on a website.

Secondly, crawlers follow the links to and from a website, as well as the internal links on a website. A crawlable internal link structure is necessary for a crawler to index all of the pages on a site, and good quality links to and from your website (external links) are vital as they show that your website is credible and has valuable content.

Thirdly, crawlers index the content on the website that hasn’t been tagged nofollow. It looks for keywords, so the search engine will know what words and queries the page will rank for. It also looks for duplicate content, copied from other websites or other pages from the website, ensuring the site content is unique.

More SEO Guides


301 Redirects

A 301 redirect is an instruction that tells a browser: “The requested page is no longer available at the URL you have, you’ll find it at this new new address”.


What Is Link Juice?

A site has an overall Domain Rank. This is the total amount of positive link metrics associated with a website as a whole. The Domain Rank is determined by the combined PageRank of all of the individual pages on a website.

Got a project in mind?

Let’s talk