[Xitami] Xitami has slowed to a crawl

[Ron Ackerman]

Thomas and Johannes,

Thanks for the info, my slow down problems seem to have gone away with the
exclusion of MSNbot. MSNbot obeyed the robots.txt file immediately (at least
it looks like it). As far as Google goes, It hasn't yet obeyed the
robots.txt file and seems to be ignoring the meta tag. I will work with the
meta tag a little more. One problem is that I use one header for every page
(php include). So the meta tag for robots can't go in the <head>.

Ron

-----Original Message-----
From: xitami-admin@lists.xitami.org [mailto:xitami-admin@lists.xitami.org]On
Behalf Of Thomas J. Hruska
Sent: Wednesday, January 12, 2005 11:33 AM
To: xitami@lists.xitami.org
Subject: RE: [Xitami] Xitami has slowed to a crawl


At 10:27 AM 1/12/2005 -0600, Johannes Prost writeth:
>Hi Ron
>
>I use a txt file as well. Have a look below
>
>Am 12 Jan 2005 um 10:30 hat Ron Ackerman geschrieben:
>
>> Seems Googlebot is ignoring the robots.txt file, I can see other bots hit
>> the file and obey it but the Googlebot just keeps on hitting the server.
>>
>> Here is what I have in my robots.txt file
>> ----------------------------------
>> User-agent: *
>> Disallow: /comment.php
>> Disallow: /submit.php
>> Disallow: /forum/createtopic.php
>> Disallow: /calendar.php

I forgot to mention:  Some bots intentionally follow disallowed pages.  So,
make sure only pages that are actually referenced on the main site are
included.  You can test for which bots try this by including a bogus,
non-existent page in the robots.txt file (requests for the bogus page will
generate 404's - should be easy to spot in the logs and in log analysis
tools).


Thomas J. Hruska
Shining Light Productions
http://www.slproweb.com/


--
Xitami Users Mailing List -- For Xitami support
To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
--
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005

--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005