[Xitami] Xitami has slowed to a crawl

[Ron Ackerman]

Good info I'll give that a shot. I would rather not exclude Google from the
server. I want to be found on the Search Engines.

-----Original Message-----
From: xitami-admin@lists.xitami.org [mailto:xitami-admin@lists.xitami.org]On
Behalf Of Thomas J. Hruska
Sent: Wednesday, January 12, 2005 11:18 AM
To: xitami@lists.xitami.org
Subject: RE: [Xitami] Xitami has slowed to a crawl


At 10:30 AM 1/12/2005 -0500, Ron Ackerman writeth:
>Seems Googlebot is ignoring the robots.txt file, I can see other bots hit
>the file and obey it but the Googlebot just keeps on hitting the server.

Yeah, Google will ignore that since it never existed before, Google tracks
whether or not you have ever had a robots.txt file and won't look for it
for a while.  What you have to do in the interim period (Google will
EVENTUALLY look for a robots.txt file) is to set up a
"no-follow,no-cache,etc" META tag in those four global PHP pages to keep
them from getting indexed (Google DOES pay immediate attention to METAs).
Between METAs and your robots.txt file, you should be able to stop bots
from accessing those pages.


Thomas J. Hruska
Shining Light Productions
http://www.slproweb.com/


--
Xitami Users Mailing List -- For Xitami support
To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
--
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005

--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005