SEO: Robots.txt
Warning: Shameless plug. If you want to know more about the search engine optimization services offered by ev3 studios, browse through our services.
A robots.txt file will not improve your search engine ranking but it will tell the search engine robots what to do once they arrive at your front door.
Some of the options you have are below. Insert whatever type of robots file you prefer.
Process:
- Right click in any windows folder and create a new text file.
- Open the file, insert the text of your choice (examples below) into it and save it as robots.txt.
- Once you upload this file into your main public_html, www, or initial viewing directory, the robots will gleefully scour your site and place all of your pages in their databases, rather than just the initial page.
Examples:
- Spider my entire site, all its directories and sub-directories. You've selected the easy option here, simply create a text file and upload this to the main directory of your web page.
User-Agent: *
Disallow:
- Disallow specific directories Simply enter the specific directories under each specific "Disallow:" command. Give each directory it's own Disallow header.
User-agent: *
Disallow: /images/
Disallow: /stats/Note: these directories are ignored by most robots; however, some robots do not pay attention to your disallow commands. If you're worried about some information getting spidered and offered to the searchers of the world, password protecting it is a good option.
Cheers,
the ev3 crew
