Robots.txt

[BWelford]

Even an empty robots.txt is worth adding since the search engine robots tend to look for that as a sign of a complete website.

More importantly you can block robots from spidering some parts of your website that you do not wish to have indexed. That could include parts that are under development. Not all robots observe the instructions in a robots.txt file but the more important ones do.

[Logan]

Its a protocol that search engine robots use to identify what should be excluded from being indexed. Reading through this site will quickly make you an expert. I agree that an empty one is ok, but I've never experienced a search engine not indexing because a robots.txt wasn't in place. I rarely run into a need for one.

[caseyturton]

As you are developing a site, even if you use a development folder, make sure you include a robots.txt file to exclude your files in development or the pages will get indexed. They'll show up on SERP's and you'll be sorry.

Learned this the hard way.

[abuyouts]

robot text prevents spiders to crawl on pages that you don't want to be indexed, its a good idea to add it up on your site.

[vickie21]

how can you get one?robot.txt...

[andrasan]

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

Here's a basic "robots.txt":

User-agent: *
Disallow: /

With the above declared, all robots (indicated by "*") are instructed to not index any of your pages (indicated by "/"). Most likely not what you want, but you get the idea

[torka]

Originally Posted by vickie21

how can you get one?robot.txt...

You don't get one, you make one.

(Then you upload it to your web server.)

Follow the link Logan pointed out (up above) and you'll find a site that explains all about how to create a robots.txt file.

HTH!

--Torka

[OLCDSarah]

Be REAL CAREFUL... one wrong statement in your robot.txt file will stop all crawlers... Sign up your site for Google webmaster tools - it will creat one for you to upload to your site

[Danielle77]

So even if I create an empty robot.txt file and upload my sitewill have better place on search engines? That's really interesting! I thought that a robot.txt was very difficult to be done and work well.
Thanks for sharing

[AirForce1]

Thanks for the nice sharing on robots.txt and I think sometimes you might really need this file to be under your site root because search engines like Bing would ask for this for a successful sitmap inclusion.

Thanks,