What is a robots.txt file? How do I make one?

The robots.txt file is a file which instructs various spiders/web crawlers on what area of the site they are allowed to visit, and what sections they are not allowed to visit.

A popular page that details much of how the robots.txt file works is located here: http://www.robotstxt.org

You can also use the Robots META tags from inside your html pages to instruct some spiders/web crawlers on whether to index the page in the search engine by using the following:

(brackets on HTML MARKUP have been removed as it will not display in our KB articles)

html
head
meta name="robots" content="noindex,nofollow"
meta name="description" content="This page ...."
title.../title
/head
body
...

A handy utility that we found on the web was a place that will make your robots.txt file for you.

http://www.1-hit.com/all-in-one/tool-robots.txt-generator.htm

Using their step by step guide, it will walk you through creating a robots.txt file for your site. Once the file is created, just upload it to your site in the /public_html folder and the search engines will find it.

The robots.txt file is not supported by all search engines.



Reviewed: 11/15/2016

  • Email, SSL
  • 21 Users Found This Useful
Was this answer helpful?

Related Articles

What if I need more hard drive space or bandwidth? - delete

Submitted by user: I need more HD space.. There are a few other companies out there that are...

What language can I program in? Where can I download scripts?

For the most part PHP is the most common scripting on the web. Many other scripts and programs in...

Help - my site's been infected by a Virus

The viruses that attach themselves to web pages are unfortunately common, and though they are...

Fix for FrontPage hit counter on subdomain

Problem: You're using MS Front Page with web bots. The hitcounter works fine on your main domain,...

Can you tranfser files from my old host to you?

If your current web hosting provider has cPanel all you would need to do is create a BACKUP of...