Problems with robots.txt

1. What is robots.txt used for?

To restrict or direct access on your site for web spiders and bots.

2. How is it configured?

There is a standards document for the robots.txt file. While all bots and spiders are not 100% in compliance with the document it remains the standard. We have copied it for you here: norobots.html