freefind
menu   Login

home features pricing search faq library site map policies contact
Indexing (Spidering)
Library > Reference > FAQ > Indexing  
Here are some quick answers to the most commonly asked questions about indexing (spidering) a site. If you don't find what you are looking for here, try the FAQs: Too Few Pages Indexed, Too Many Pages Indexed, and Host/ISP-specific Issues.



 


Q. Why is my site no longer spidered?
A.
This typically happens because your hosting company has moved your site. Amazingly, this happens a lot! Double-check the current location of your site and make the necessary corrections.
[top]

Q. I changed my site, how do I re-index?
A.
Log in to the Control Center then go to the build index page and use the index now link.
[top]

Q. I update my site continuously. Can it be automatically re-indexed?
A.
Yes. Log in to the Control Center then go to the build index page and use the schedule re-indexing link. Please don't have our system spider your site more than actually needed!
[top]

Q. How does the spider (indexer) find my pages?
A.
The search engine spider starts with the web address (URL) you entered when you signed-up. It reads all the links on that page. Those links which start with the same URL are then read by the spider. It then looks for links on those pages that start with the same URL as the one you signed-up with, and reads them, and so forth. Once all pages have been read an index is built and an email is sent to you informing you of the results.

Here we see a simplified example, showing a site with six pages, and a link to someone else's web site. The spider would start by reading your home page and following the links to other pages on your site (blue lines). It will not follow links to sites that are not yours (red line). In this example pages in white will be indexed, and gray ones ignored.
[top]

Q. Can I prevent a page from being indexed?
A.
Yes, there are lots of ways of doing this. For more information, read How to Exclude Pages from Search.
[top]

Q. Can I prevent part of a page from being indexed?
A.
Yes. For instructions, see How to Exclude Pages from Search.
[top]

Q. Can my forum be indexed?
A.
Yes. For more information, read Searching Forums.
[top]

Q. Does the spider honor "robots.txt" files?
A.
Always. Our user-agent name is "FreeFind".
For more information, read How to Use Robots.txt.
[top]

Q. What parts of each page get indexed?
A.
By default words in the following parts of each web page are included in the index:

  • the title,
  • the keywords meta tag,
  • the description meta tag,
  • the body of the page.

The spider does not index words in javascript nor in the image tag "alt" attribute.
[top]

 

login home features pricing search faq library sitemap policies contact
FreeFind and FreeFind.com are trademarks of FreeFind.com.
Copyright 1998 - 2016