[Xitami] Xitami has slowed to a crawl

Wed, 12 Jan 2005 10:30:08 -0500

Seems Googlebot is ignoring the robots.txt file, I can see other bots hit
the file and obey it but the Googlebot just keeps on hitting the server.

Here is what I have in my robots.txt file
----------------------------------
User-agent: *
Disallow: /comment.php
Disallow: /submit.php
Disallow: /forum/createtopic.php
Disallow: /calendar.php

User-agent: msnbot
Disallow: /
------------------------------
Yet this is in my log

66.249.66.51 - - [12/Jan/2005:10:21:11 -0500] "GET
/submit.php?type=event&mode=&month=04&day=23&year=2004&hour=8 HTTP/1.1" 403
0

------------------------------
My next step is to exclude it in the Webmask= area, I am using a text file
called webmask.txt. My question is the format of the ip? Should it look like
this

!66.249.66.51,

or like this

'!66.249.66.51'

Does the ballbat (!) exclude or include.
Do I need the ballbat?  (I know it's an exclaimation point but it's easier
to type ball bat).
How about the single quotes do I need them?
The info in the xitami.cfg isn't clear to me.

Thanks,
Ron

-----Original Message-----
From: xitami-admin@lists.xitami.org
[mailto:xitami-admin@lists.xitami.org]On Behalf Of William R. Palaia
Sent: Tuesday, January 11, 2005 3:41 PM
To: xitami@lists.xitami.org
Subject: Re: [Xitami] Xitami has slowed to a crawl

Go here:

http://www.robotstxt.org/wc/norobots.html

>
> My system is Pentium 4, 1 gig DDRAM2.0, 2 dual 250Gb data drives
(mirrored)
> with a 40Gb IOS drive that has 10Gb set aside as virtual memory.  I am
> running Win2K Advanced Server and Xitami ver2.49d.  MySQL, PHP, Perl and
am
> running about 18 sites at the moment.  My server is a work horse but by
the
> sounds of the MSNbot it won't matter b/c it will be killing the bandwidth
> more so than the system.
>
> Good research and great info about the msnbot - I read the whole thread
and
> got more than I was expecting.
>
> Question: I am surprised but I have never even heard of the robots.txt
file,
> could you send me a copy of yours so I could see the proper format?  Where
> does it go?  I def. want to restrict the msnbot from my server until I can
> properly understand everything and be able to determine the best time to
> allow it back on later.
>
> Thanks Ron.
>
>
> Russel P. Olinger
> (310) 466-3399
> rolinger1@hotmail.com
>
>
>
>
>> From: "Ron Ackerman" <ron@miplanet.net>
>> Reply-To: xitami@lists.xitami.org
>> To: <xitami@lists.xitami.org>
>> Subject: RE: [Xitami] Xitami has slowed to a crawl
>> Date: Tue, 11 Jan 2005 14:22:42 -0500
>>
>> UPDATE:
>>
>> I have disallowed MSNbot for now based on information from this forum
>>
http://my.jimworld.com/apps/searchengine.forums/forum::other-search-engines/
>> thread::1085675955/action::thread/
>>
>> -----Original Message-----
>> From: xitami-admin@lists.xitami.org
>> [mailto:xitami-admin@lists.xitami.org]On Behalf Of Ron Ackerman
>> Sent: Tuesday, January 11, 2005 2:10 PM
>> To: xitami@lists.xitami.org
>> Subject: RE: [Xitami] Xitami has slowed to a crawl
>>
>>
>> Just read this on a forum, it seems others are having problems with
MSNbot.
>> Ok most don't want to be EXCLUDED from MSN but like me they don't want
>> MSNbot to eat up our bandwidth. They had several solutions.
>>
>> ======================================================
>> HELP! MSN Bot Eating Up ALL my bandwidth!!!
>>
>>
>> MSN Bot Statistics for the month of May (up to May 15th -- only 15 days
of
>> stats!!!)
>>
>> msnbot64047.search.msn.com:
>> bandwidth: 7625158 Kbytes (7.6GB)
>> files: 166806
>> hits: 167027
>>
>> This is 29.4% of my sites bandwidth this month just from MSN Bot!
Yesterday
>> alone it used 1.3 GB bandwidth!
>>
>> By comparison, Googles bot(s) have only used about 1.4GB total bandwidth
>> this month.
>>
>> How can I stop MSN from killing me! At this rate itwill use 15 GB this
>> month! Just from being indexed
>>
============================================================================
>> ========
>>
>>
>>
>> -----Original Message-----
>> From: xitami-admin@lists.xitami.org
>> [mailto:xitami-admin@lists.xitami.org]On Behalf Of R O
>> Sent: Tuesday, January 11, 2005 1:31 PM
>> To: xitami@lists.xitami.org
>> Subject: RE: [Xitami] Xitami has slowed to a crawl
>>
>>
>>
>> Ron,
>>
>> I am curious, what ver of Xitami are you running, IOS, and what kind of
>> system are you running it on.  You mention Xitami slowing down b/c of the
>> crawlers, but it is Xitami or is it system performance trying to service
>> the
>> bots?  Just trying to get some clarity on where the actual problem lies
so
>> I
>> can A) know what to look for and B) try to correct or prevent the same
>> issue
>> from happening to my system.
>>
>> -Russel
>>
>>
>> Russel P. Olinger
>> (310) 466-3399
>> rolinger1@hotmail.com
>>
>>
>>
>>
>>> From: "Ron Ackerman" <ron@miplanet.net>
>>> Reply-To: xitami@lists.xitami.org
>>> To: <xitami@lists.xitami.org>
>>> Subject: RE: [Xitami] Xitami has slowed to a crawl
>>> Date: Tue, 11 Jan 2005 12:44:05 -0500
>>>
>>> That will exclude all robots, I'm not sure I want to do that because I
>>> would
>>> like to be included in the Search Engines. Is there a time setting that
>> can
>>> be applied. Like Excluding the robots between 7am and 11pm?
>>>
>>> -----Original Message-----
>>> From: xitami-admin@lists.xitami.org
>>> [mailto:xitami-admin@lists.xitami.org]On
>>> Behalf Of Jeff Hamblin
>>> Sent: Tuesday, January 11, 2005 12:39 PM
>>> To: xitami@lists.xitami.org
>>> Subject: Re: [Xitami] Xitami has slowed to a crawl
>>>
>>>
>>>> I think I'm finding the source of my slow downs. Crawlers and the worst
>>>> right now is msn bot. But I'm also seeing Yahoo Slurp. Which normally
>> I
>>>> woudn't mind them indexing my websites but when it is affecting the
>>> server
>>>> then it's a problem.
>>>
>>>
>>> Try putting a "robots.txt" file in the root of your website with the
>>> following:
>>>
>>> # robots.txt file to stop indexing of site
>>> User-agent: *
>>> Disallow: /
>>>
>>>
>>> --
>>> Xitami Users Mailing List -- For Xitami support
>>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>>> --
>>> No virus found in this incoming message.
>>> Checked by AVG Anti-Virus.
>>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>>
>>> --
>>> No virus found in this outgoing message.
>>> Checked by AVG Anti-Virus.
>>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>>
>>>
>>> --
>>> Xitami Users Mailing List -- For Xitami support
>>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>>
>>
>>
>> --
>> Xitami Users Mailing List -- For Xitami support
>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>> --
>> No virus found in this incoming message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>> --
>> No virus found in this outgoing message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>>
>> --
>> Xitami Users Mailing List -- For Xitami support
>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>> --
>> No virus found in this incoming message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>> --
>> No virus found in this outgoing message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>>
>> --
>> Xitami Users Mailing List -- For Xitami support
>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>> --
>> No virus found in this incoming message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>> --
>> No virus found in this outgoing message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>>
>> --
>> Xitami Users Mailing List -- For Xitami support
>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>
>
>
> --
> Xitami Users Mailing List -- For Xitami support
> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>

--
Xitami Users Mailing List -- For Xitami support
To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
--
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005

--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005