Why robots.txt file is so important?
7 people like it
Advertise on APSense
This advertising space is available.
Post Your Ad Here
Post Your Ad Here
Comments (23)
Aaron Habel3
Writer
It's the first file of the website from where the crawlers enter the website.
Gaurav P.6
Internet Marketing Expert
robots.txt file contains some url that you don't want to indexed its about admin folder or other.You can also make a folder not to crawl by this
John Mathew2
Financial Market Expert
It is the roadmap for crawlers to read website pages as per command.