[Xitami] Xitami has slowed to a crawl

[Thomas J. Hruska]

At 10:30 AM 1/12/2005 -0500, Ron Ackerman writeth:
>Seems Googlebot is ignoring the robots.txt file, I can see other bots hit
>the file and obey it but the Googlebot just keeps on hitting the server.

Yeah, Google will ignore that since it never existed before, Google tracks
whether or not you have ever had a robots.txt file and won't look for it
for a while.  What you have to do in the interim period (Google will
EVENTUALLY look for a robots.txt file) is to set up a
"no-follow,no-cache,etc" META tag in those four global PHP pages to keep
them from getting indexed (Google DOES pay immediate attention to METAs).
Between METAs and your robots.txt file, you should be able to stop bots
from accessing those pages.


Thomas J. Hruska
Shining Light Productions
http://www.slproweb.com/