The FreeFind spider is an automatic web crawling robot. The spider reads pages from a website in order to build an index of that website. This website index is used to perform searches for pages which are part of the website.

A description of the FreeFind service can be found at the FreeFind home page.

The complete FreeFind FAQ can be found here.

The Robots Exclusion Standard (aka robots.txt)

The FreeFind spider is a polite robot which always honors your robots.txt file.

If the FreeFind spider is crawling parts of your website that you do not wish to be indexed, simply use the robots exclusion standard to control indexing.

Using this standard FreeFind can be prevented from indexing all or part of your website.

Our user-agent name for use in your robots.txt file is FreeFind.

Examples of using robots.txt with FreeFind can be found here in the FreeFind FAQ.

If you are not familiar with the robots exclusion standard you can find it at robotstxt.org.

Server Problems

If the FreeFind spider is causing server problems which cannot be corrected using the robots exclusion standard (see examples in our FAQ), please contact us with a description of the server problem you are seeing.