Robots.txt file in SEO

What is Robots.txt?

The Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. it is a group of web standards that regulate web robot behavior and search engine indexing. The original REP from 1994, extended 1997, defining crawler directives for robots.txt. Some search engines support extensions like URI patterns (wild cards).Its extension from 1996 defining indexer directives (REP tags) for use in the robots meta element, also known as "robots meta tag." Meanwhile, search engines support additional REP tags with an X-Robots-Tag. Webmasters can apply REP tags in the HTTP header of non-HTML resources like PDF documents or images.           

Cheat Sheet

Block all web crawlers from all content

User-agent: *

Disallow: /

Block a specific web crawler from a specific folder

User-agent: Googlebot

Disallow: /no-google/

Block a specific web crawler from a specific web page

User-agent: Googlebot

Disallow: /no-google/blocked-page.html

Sitemap Parameter

User-agent: *

Disallow:

Sitemap: http://www.example.com/none-standard-location/sitemap.xml

Important Rules

·         In most cases, meta robots with parameters "noindex, follow" should be employed as a way to to restrict crawling or indexation.

·         It is important to note that malicious crawlers are likely to completely ignore robots.txt and as such, this protocol does not make a good security mechanism.

·         Only one "Disallow:" line is allowed for each URL.

·         Each subdomain on a root domain uses separate robots.txt files.

·         Google and Bing accept two specific regular expression characters for pattern exclusion The filename of robots.txt is case sensitive. Use "robots.txt", not "Robots.TXT."

·         Spacing is not an accepted way to separate query parameters.

    Author:

WDC is a Leading Web Solution providing and Ecommerce Website Development Company which is headquartered in Bangalore, India. Our Company has 5+ years experience in Internet Marketing and providing web solutions. Our team has 50+ Professional Experts who all are having 5+ years of experience in their Specialization.