How to Set Up the Perfect Robots.Txt File for Your SEO Campaign?

Starboard Technology  web design, development and digital marketing comp
1.Place your robots.txt file in the top-level directory of your website code to simplify crawling and indexing.
2.Structure your robots.txt properly, like this: User-agent → Disallow → Allow → Host → Sitemap. This way, search engine spiders access categories and web pages in the appropriate order.
3.Make sure that every URL you want to “Allow:” or “Disallow:” is placed on an individual line. If several URLs appear on one single line, crawlers will have a problem accessing them.
4.Use lowercase to name your robots.txt. Having “robots.txt” is always better than “Robots.TXT.” Also, file names are case sensitive.
5.Don’t separate query parameters with spacing. For instance, a line query like this “/cars/ /audi/” would cause mistakes in the robots.txt file.
6.Don’t use any special characters except * and $. Other characters aren’t recognized.
7.Create separate robots.txt files for different subdomains. For example, “hubspot.com” and “blog.hubspot.com” have individual files with directory- and page-specific directives.
8.Use # to leave comments in your robots.txt file. Crawlers don’t honor lines with the # character.
9.Don’t rely on robots.txt for security purposes. Use passwords and other security mechanisms to protect your site from hacking, scraping, and data fraud.
Dec 21st 2018 05:36   

Joaquin Velazquez  Marketing
It depends! First of all how are you working your web page with a CMS, a framework or just html.

Depending on this you must make the configuration of your robots file, restricting access to folders that you do not want to be indexed, pointing to the sitemap.xml and caching the necessary files to save loading time.
Dec 21st 2018 12:13 1 Likes  

Shriya World  IT
Hello,
If you are using wordpress then below is the reference sample sitemap

User-agent: *
Disallow: /wp-admin/
Disallow: /xmlrpc.php
Disallow: /trackback/
Disallow: */trackback
Disallow: /cgi-bin/
Disallow: /wp-login/
Disallow: /wp-register/
Allow: /goto/
Allow: /wp-content/uploads/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://***/sitemap_index.xml
Sitemap: https://***/post-sitemap.xml


1. User-agent: * Represent allowing all search engine bots to crawl your website.

if you want only respective bot to crawl we can mention as below & allow / disallow is used to allow or block the bot to crawl your website

User-agent: Googlebot
Allow: /

User-agent: MJ12bot
Disallow: /

2. Disallow: represent the path / directory should not be crawled

3 .Allow means the search bots will crawl the files/ paths

4. Sitemap: This is intimate the sitemap url available with your website & very easy to intimate about it to search bots.

Thanks!!
Dec 21st 2018 22:54 1 Likes  

Rob Stephen   getaprogrammer
User-agent: *
Allow: /
Dec 21st 2018 23:08   

JONATHAN PAUL  Jonathan Paul working at PHPProgrammers, a leading
user-agent:* Allow: /
Dec 21st 2018 23:13   

Ahmad Quershi  Ahmad Quershi
it depends on what cms are you using and then you need to disallow such content you need to block like
Disallow: //.pdf
Dec 21st 2018 23:49   

Enterslice ITES Pvt. Ltd.   Start and Manage Business
Syntex for Robots.txt

User-agent: *
Disallow: /wp-admin/
Allow: /goto/
Sitemap: https://***/sitemap_index.xml
Sitemap: https://***/post-sitemap.xml
Dec 22nd 2018 04:40   

Ganesh Kulariya  Content Writer, SEO, SMO Expert
For best SEO I recommendation for robots.txt to make a perfect SEO.
If you want every crawler crawled your website then set
User-agent: *
Disallow: /admin
Dec 23rd 2018 23:04   

Murtza Abbas  Sr. Digital Marketing | SEO | SMO
User-agent: *
Allow: /
Dec 23rd 2018 23:41   

Think Tribe  Think..Solve...Execute
Take a .txt file and decide which page do you want to get follow and indexed by search engine and which page do not want.
And write the below :

User-agent: *
Allow: / which page you want to get follow and index by search engine.
Disallow: / which page do not want to get follow and index.

Leave Allow: / It will follow and index all page.

Leave Disallow: / It will not follow and index any page.

And upload this robots.txt file in index page on your web server.

Thank you
Dec 24th 2018 04:47    Edited in Dec 24th 2018 04:50

Wonders Mind  Web Designing Company
Hi,

Robots file will help you in block certen pages from search engines.
Dec 24th 2018 05:18   

Jack Dolson  SEO
Disallow all the unwanted bots in robots.
Dec 24th 2018 09:11   

Parahombre USA  Business
It depends! First of all how are you working your web page with a CMS, a framework or just html.

Depending on this you must make the configuration of your robots file, restricting access to folders that you do not want to be indexed, pointing to the sitemap.xml and caching the necessary files to save loading time.
Dec 25th 2018 00:01   

Mobiloitte Technologies  Marketing Manager
Use * i.e for all bots and hide the pages which you don't want to let the crawler to crawl
Dec 26th 2018 01:30   

Modernday Music  Modern Day Music School
User-agent: *
Allow: /
Dec 26th 2018 03:45   

John A.  Owner at Organic Beds in Toronto
It totally depends on the type of cms you are using! Use * i.e for all bots and hide the pages which you don't want to let the crawler to crawl Also additionally you can put the robots.txt file on the root directory of the domain.
Dec 26th 2018 06:42   

Nitish Chandra Sharma  Digital Marketing Expert
robot file basic code, save with robots.txt and upload it on the root directory of the domain -

User-agent: *
Allow: /
Jan 5th 2019 04:31