[Xitami] Xitami has slowed to a crawl
William R. Palaia
xitami@lists.xitami.org
Tue, 11 Jan 2005 14:40:59 -0600
Go here:
http://www.robotstxt.org/wc/norobots.html
>
> My system is Pentium 4, 1 gig DDRAM2.0, 2 dual 250Gb data drives (mirrored)
> with a 40Gb IOS drive that has 10Gb set aside as virtual memory. I am
> running Win2K Advanced Server and Xitami ver2.49d. MySQL, PHP, Perl and am
> running about 18 sites at the moment. My server is a work horse but by the
> sounds of the MSNbot it won't matter b/c it will be killing the bandwidth
> more so than the system.
>
> Good research and great info about the msnbot - I read the whole thread and
> got more than I was expecting.
>
> Question: I am surprised but I have never even heard of the robots.txt file,
> could you send me a copy of yours so I could see the proper format? Where
> does it go? I def. want to restrict the msnbot from my server until I can
> properly understand everything and be able to determine the best time to
> allow it back on later.
>
> Thanks Ron.
>
>
> Russel P. Olinger
> (310) 466-3399
> rolinger1@hotmail.com
>
>
>
>
>> From: "Ron Ackerman" <ron@miplanet.net>
>> Reply-To: xitami@lists.xitami.org
>> To: <xitami@lists.xitami.org>
>> Subject: RE: [Xitami] Xitami has slowed to a crawl
>> Date: Tue, 11 Jan 2005 14:22:42 -0500
>>
>> UPDATE:
>>
>> I have disallowed MSNbot for now based on information from this forum
>> http://my.jimworld.com/apps/searchengine.forums/forum::other-search-engines/
>> thread::1085675955/action::thread/
>>
>> -----Original Message-----
>> From: xitami-admin@lists.xitami.org
>> [mailto:xitami-admin@lists.xitami.org]On Behalf Of Ron Ackerman
>> Sent: Tuesday, January 11, 2005 2:10 PM
>> To: xitami@lists.xitami.org
>> Subject: RE: [Xitami] Xitami has slowed to a crawl
>>
>>
>> Just read this on a forum, it seems others are having problems with MSNbot.
>> Ok most don't want to be EXCLUDED from MSN but like me they don't want
>> MSNbot to eat up our bandwidth. They had several solutions.
>>
>> ======================================================
>> HELP! MSN Bot Eating Up ALL my bandwidth!!!
>>
>>
>> MSN Bot Statistics for the month of May (up to May 15th -- only 15 days of
>> stats!!!)
>>
>> msnbot64047.search.msn.com:
>> bandwidth: 7625158 Kbytes (7.6GB)
>> files: 166806
>> hits: 167027
>>
>> This is 29.4% of my sites bandwidth this month just from MSN Bot! Yesterday
>> alone it used 1.3 GB bandwidth!
>>
>> By comparison, Googles bot(s) have only used about 1.4GB total bandwidth
>> this month.
>>
>> How can I stop MSN from killing me! At this rate itwill use 15 GB this
>> month! Just from being indexed
>> ============================================================================
>> ========
>>
>>
>>
>> -----Original Message-----
>> From: xitami-admin@lists.xitami.org
>> [mailto:xitami-admin@lists.xitami.org]On Behalf Of R O
>> Sent: Tuesday, January 11, 2005 1:31 PM
>> To: xitami@lists.xitami.org
>> Subject: RE: [Xitami] Xitami has slowed to a crawl
>>
>>
>>
>> Ron,
>>
>> I am curious, what ver of Xitami are you running, IOS, and what kind of
>> system are you running it on. You mention Xitami slowing down b/c of the
>> crawlers, but it is Xitami or is it system performance trying to service
>> the
>> bots? Just trying to get some clarity on where the actual problem lies so
>> I
>> can A) know what to look for and B) try to correct or prevent the same
>> issue
>> from happening to my system.
>>
>> -Russel
>>
>>
>> Russel P. Olinger
>> (310) 466-3399
>> rolinger1@hotmail.com
>>
>>
>>
>>
>>> From: "Ron Ackerman" <ron@miplanet.net>
>>> Reply-To: xitami@lists.xitami.org
>>> To: <xitami@lists.xitami.org>
>>> Subject: RE: [Xitami] Xitami has slowed to a crawl
>>> Date: Tue, 11 Jan 2005 12:44:05 -0500
>>>
>>> That will exclude all robots, I'm not sure I want to do that because I
>>> would
>>> like to be included in the Search Engines. Is there a time setting that
>> can
>>> be applied. Like Excluding the robots between 7am and 11pm?
>>>
>>> -----Original Message-----
>>> From: xitami-admin@lists.xitami.org
>>> [mailto:xitami-admin@lists.xitami.org]On
>>> Behalf Of Jeff Hamblin
>>> Sent: Tuesday, January 11, 2005 12:39 PM
>>> To: xitami@lists.xitami.org
>>> Subject: Re: [Xitami] Xitami has slowed to a crawl
>>>
>>>
>>>> I think I'm finding the source of my slow downs. Crawlers and the worst
>>>> right now is msn bot. But I'm also seeing Yahoo Slurp. Which normally
>> I
>>>> woudn't mind them indexing my websites but when it is affecting the
>>> server
>>>> then it's a problem.
>>>
>>>
>>> Try putting a "robots.txt" file in the root of your website with the
>>> following:
>>>
>>> # robots.txt file to stop indexing of site
>>> User-agent: *
>>> Disallow: /
>>>
>>>
>>> --
>>> Xitami Users Mailing List -- For Xitami support
>>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>>> --
>>> No virus found in this incoming message.
>>> Checked by AVG Anti-Virus.
>>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>>
>>> --
>>> No virus found in this outgoing message.
>>> Checked by AVG Anti-Virus.
>>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>>
>>>
>>> --
>>> Xitami Users Mailing List -- For Xitami support
>>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>>
>>
>>
>> --
>> Xitami Users Mailing List -- For Xitami support
>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>> --
>> No virus found in this incoming message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>> --
>> No virus found in this outgoing message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>>
>> --
>> Xitami Users Mailing List -- For Xitami support
>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>> --
>> No virus found in this incoming message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>> --
>> No virus found in this outgoing message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>>
>> --
>> Xitami Users Mailing List -- For Xitami support
>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>> --
>> No virus found in this incoming message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>> --
>> No virus found in this outgoing message.
>> Checked by AVG Anti-Virus.
>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>>
>>
>> --
>> Xitami Users Mailing List -- For Xitami support
>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>
>
>
> --
> Xitami Users Mailing List -- For Xitami support
> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>