[Xitami] Xitami has slowed to a crawl

[Thomas J. Hruska]

At 10:27 AM 1/12/2005 -0600, Johannes Prost writeth:
>Hi Ron
>
>I use a txt file as well. Have a look below
>
>Am 12 Jan 2005 um 10:30 hat Ron Ackerman geschrieben:
>
>> Seems Googlebot is ignoring the robots.txt file, I can see other bots hit
>> the file and obey it but the Googlebot just keeps on hitting the server.
>> 
>> Here is what I have in my robots.txt file
>> ----------------------------------
>> User-agent: *
>> Disallow: /comment.php
>> Disallow: /submit.php
>> Disallow: /forum/createtopic.php
>> Disallow: /calendar.php

I forgot to mention:  Some bots intentionally follow disallowed pages.  So,
make sure only pages that are actually referenced on the main site are
included.  You can test for which bots try this by including a bogus,
non-existent page in the robots.txt file (requests for the bogus page will
generate 404's - should be easy to spot in the logs and in log analysis
tools).


Thomas J. Hruska
Shining Light Productions
http://www.slproweb.com/