Mentors in Motion Search Engine Optimization...
Designing a Web Crawler Friendly Web Site of Your Own
by Mick R. Webmaster & Entrepreneur
Mick R. 
Webmaster & Entrepreneur
Get Paid to Submit Article like "Designing a Web Crawler Friendly Web Site of Your Own"
The most successful online businesses all have one thing in common. They all knew how to make search engine optimization work for them.
Search engine optimization is the art and science of making websites attractive to the internet's search engines. The first step in successfully achieving stellar search engine optimization is to lure search engine's web crawlers to your website. Web crawlers are computer programs that the search engines use gather data and index information from the websites. The information the web crawlers gather is used to determine the ranking of a webpage.
One of the fastest ways to hamper a web crawler is to construct a website that has frames. Most search engines have crawlers that can't penetrate the frames, if they can't get into a webpage to read it then that webpage remains unindexed and unranked. Two search engines, Google and Inktome, have web crawlers that are capable of penetrating frames. Before submitting your website to a search engine do some research and find out if they have a crawler that is incapable of penetrating any frames.
If you have written frames into your URL it will probably be worth your effort to go back and rewrite your URL's. Once you have rewritten your URLs you might be surprised to find that the new addresses are easier on humans as well as web crawlers, the frameless URLs are easier to type in documents as links and references.
Once you have rewritten your URL's it is time to start submitting your website to search engines. Some webmasters like to use an automated search engine submission service. If you decide to go with the submission service you should be aware that there will be a fee involved, the minimum fee is typically fifty-nine US dollars. This price should keep a few URLs on the search engines for a year. Other webmasters like to avoid big fees by submitting their website to individual search engine on their own.
Once your webpage is submitted to a search engine you need to sit down and design a crawler page. A crawler page is a webpage that contains nothing else expect links to every single page of your website, Use the title of each page as the as the link text. This will also give you some extra keywords that will help improve the ranking the crawlers assign to your website. Think of the crawler page as a site map to the rest of your website.
Typically, the crawler page won't appear in the search results. This happens because the page doesn't have enough text for the crawlers to give that individual page a high ranking, after all its nothing more then a portal to the rest of your site and your human users won't need to use it. Don't panic if it crawlers don't instantly appear to index your website. There are a lot of websites available on the internet that need to be crawled, indexed, and then ranked. It can sometimes take up to three months for a web crawler to get to yours.
Mick
Seo Elite: New Seo Software!
MAKE MONEY on your PROFILE - ProfileDough.com
Nov 7th 2007 13:08
Webmaster & EntrepreneurGet Paid to Submit Article like "Designing a Web Crawler Friendly Web Site of Your Own"
The most successful online businesses all have one thing in common. They all knew how to make search engine optimization work for them.
Search engine optimization is the art and science of making websites attractive to the internet's search engines. The first step in successfully achieving stellar search engine optimization is to lure search engine's web crawlers to your website. Web crawlers are computer programs that the search engines use gather data and index information from the websites. The information the web crawlers gather is used to determine the ranking of a webpage.
One of the fastest ways to hamper a web crawler is to construct a website that has frames. Most search engines have crawlers that can't penetrate the frames, if they can't get into a webpage to read it then that webpage remains unindexed and unranked. Two search engines, Google and Inktome, have web crawlers that are capable of penetrating frames. Before submitting your website to a search engine do some research and find out if they have a crawler that is incapable of penetrating any frames.
If you have written frames into your URL it will probably be worth your effort to go back and rewrite your URL's. Once you have rewritten your URLs you might be surprised to find that the new addresses are easier on humans as well as web crawlers, the frameless URLs are easier to type in documents as links and references.
Once you have rewritten your URL's it is time to start submitting your website to search engines. Some webmasters like to use an automated search engine submission service. If you decide to go with the submission service you should be aware that there will be a fee involved, the minimum fee is typically fifty-nine US dollars. This price should keep a few URLs on the search engines for a year. Other webmasters like to avoid big fees by submitting their website to individual search engine on their own.
Once your webpage is submitted to a search engine you need to sit down and design a crawler page. A crawler page is a webpage that contains nothing else expect links to every single page of your website, Use the title of each page as the as the link text. This will also give you some extra keywords that will help improve the ranking the crawlers assign to your website. Think of the crawler page as a site map to the rest of your website.
Typically, the crawler page won't appear in the search results. This happens because the page doesn't have enough text for the crawlers to give that individual page a high ranking, after all its nothing more then a portal to the rest of your site and your human users won't need to use it. Don't panic if it crawlers don't instantly appear to index your website. There are a lot of websites available on the internet that need to be crawled, indexed, and then ranked. It can sometimes take up to three months for a web crawler to get to yours.
Mick
Seo Elite: New Seo Software!
MAKE MONEY on your PROFILE - ProfileDough.com
Nov 7th 2007 13:08
Sponsor Ads
Comments
|
Brad Parent
 Regarding the comment by MKWeb - do you know if Google Webmaster Tools creates both of these files when it creates a sitemap? I have used that for my site although I had no clue exactly what it was generating. Google Webmaster Tools seemed to get my site indexed fast. Nov 8th 2007 09:16 |
|
Mark Hultgren
 Wordpress SpecialistHi Brad, I don't believe that Google will generate the ror file. The reason being it would be a conflict of interest. The ROR is an XML file that almost all search engines can read, whereas Google only creates a file that THEIR robots can read. I have found a site that WILL crawl your url and generate it online for free (it has a limit of the number of pages it can do online), and they have a desktop version that I would recommend for large sites. You can find them both at: http://www.rorweb.com/rormap.htm Build the files, upload to your server in the public_html or www folder and then submit those files to the SE's. I always wait until my site has been crawled by the SE BEFORE i submit the files to them though, that way it seems to help let them know that you are not trying to boost your rating fraudulently. (Submitting the sitemaps and ROR's was a 'grey hat' method of boosting your site's validity for authority sites a few months ago) Nov 8th 2007 09:58 |
|
|
Brad Parent
Thanks for the helpful reply. I hope the rest of the group will read this. Nov 17th 2007 08:41 |
|
Hi Mick,
As far as the 'Crawler' page goes, there are two files that should be on everyone's server that has a website.
A Siteindex.html or php page (as well as XML for Google - you can generate this page from inside a Google account)
and the second should be the ROR file. these two files enable the robots and crawlers the ability to index every page listed in the files in seconds. A couple of my domains have hundreds of pages on them and if the SE's had to crawl each page every time they visited, it would blow my bandwidth out of the water in the first week.
These two files (Wordpress creates one of them automatically if you have the Sitemap plugin installed) not only lists the pages, but also how often they are updated and when they were first created.
Nov 7th 2007 13:18