What is the difference between crawler and indexer?

Understand the difference between crawler and indexer.

Many terms used in the SEO world seem to be synonymous. A perfect example of terms that are mistakenly used as synonyms is that of “Crawl” and “ Indexing ”. In many SEO articles, the author leads the reader to believe that the two words mean the same thing. This is certainly not the case. So… what exactly is the difference between crawling and indexing?

What is Crawl?

Crawler is a term used when a search engine (Google, for example) sends its bot to a web page or web article and “reads” the page. In French we could translate this term by “exploration”. Do not confuse the crawl with the indexing of a page. The crawl is the first step for a search engine to recognize your page and display it in the search results. The fact that your page is browsed by a crawler does not necessarily mean that it has been indexed and that it will be found during the search of Internet users. Pages are crawled for a variety of reasons… the most common are:

• have a sitemap that Google reads that points to your new page or
• receive a backlink from another site, itself crawled by the Google robot

Related: Check the indexing of your URLs in GoogleSheet

How to force the Google crawl? 

For a search engine to crawl a page, the fastest ways are:

• have an XML sitemap (sitemap) uploaded to Google Search Console (formerly Google Webmaster Tools), giving Google the roadmap for all your new content.
• publish your url on Google+ (and add a +1), now only possible with a GSuite account,
• submit URL in Google Search Console
• use a crawl tool like  Colinkri: this tool forces Google to crawl the web pages.

If Google just skimmed your page and crawled it. This means that Google has reviewed the page, and if Google thinks the content is "new" or has something to "offer the internet", they can schedule it to be indexed. Additionally, when Google crawls a page, it examines its links (internal and external) and schedules their crawl. The fact that your page is crawled does not mean that it has been indexed: it will not be found in a Google search .

What is indexing?

Getting your page indexed by Google is the next step in the crawl. Under no circumstances are all crawled (crawled) sites indexed, but all indexed sites must have been crawled. During its crawl, if Google deems your new page worthy of being present in its search results, it will index it. Once your page is indexed, Google determines how your page should be found in its search. Google then decides the keywords and ranking of your page in related searches.

Related: Top 20 Reasons People Misinterpret Marketing Data and Reports

How to force Google to index a page? 

To force the indexing of a page, you can use tools external to the search engine:

• bulkaddURL  : a high-performance tool for near real-time indexing, but a bit expensive (€0.02 per URL).
• Omega Indexer  : a well-thought-out tool that offers URL uploading directly via a csv file. Its advantage: the credits do not expire.
• Index Inject  : Monthly subscriptions allow this indexer to be inexpensive. Index Inject works fine. Unfortunately, unused credits are lost from month to month.

Good to know

All the links present on the indexed page are now programmed to be crawled by the Google robot. A priori, Google explores at a depth level of 5. That is to say that the Google robot will search up to five sites downstream. This means that if a page links to a page, which links to a page, which links to a page which links to your page (which has just been indexed), then they will all be crawled. This is why the  external links  that come to your site are so important. The higher the quality of the page that links to your site, the higher you will rank in the all-powerful Google search.