[Xitami] Xitami has slowed to a crawl

[Ron Ackerman]

Just read this on a forum, it seems others are having problems with MSNbot.
Ok most don't want to be EXCLUDED from MSN but like me they don't want
MSNbot to eat up our bandwidth. They had several solutions.

======================================================
HELP! MSN Bot Eating Up ALL my bandwidth!!!


MSN Bot Statistics for the month of May (up to May 15th -- only 15 days of
stats!!!)

msnbot64047.search.msn.com:
bandwidth: 7625158 Kbytes (7.6GB)
files: 166806
hits: 167027

This is 29.4% of my sites bandwidth this month just from MSN Bot! Yesterday
alone it used 1.3 GB bandwidth!

By comparison, Googles bot(s) have only used about 1.4GB total bandwidth
this month.

How can I stop MSN from killing me! At this rate itwill use 15 GB this
month! Just from being indexed
============================================================================
========



-----Original Message-----
From: xitami-admin@lists.xitami.org
[mailto:xitami-admin@lists.xitami.org]On Behalf Of R O
Sent: Tuesday, January 11, 2005 1:31 PM
To: xitami@lists.xitami.org
Subject: RE: [Xitami] Xitami has slowed to a crawl



Ron,

I am curious, what ver of Xitami are you running, IOS, and what kind of
system are you running it on.  You mention Xitami slowing down b/c of the
crawlers, but it is Xitami or is it system performance trying to service the
bots?  Just trying to get some clarity on where the actual problem lies so I
can A) know what to look for and B) try to correct or prevent the same issue
from happening to my system.

-Russel


Russel P. Olinger
(310) 466-3399
rolinger1@hotmail.com




>From: "Ron Ackerman" <ron@miplanet.net>
>Reply-To: xitami@lists.xitami.org
>To: <xitami@lists.xitami.org>
>Subject: RE: [Xitami] Xitami has slowed to a crawl
>Date: Tue, 11 Jan 2005 12:44:05 -0500
>
>That will exclude all robots, I'm not sure I want to do that because I
>would
>like to be included in the Search Engines. Is there a time setting that can
>be applied. Like Excluding the robots between 7am and 11pm?
>
>-----Original Message-----
>From: xitami-admin@lists.xitami.org
>[mailto:xitami-admin@lists.xitami.org]On
>Behalf Of Jeff Hamblin
>Sent: Tuesday, January 11, 2005 12:39 PM
>To: xitami@lists.xitami.org
>Subject: Re: [Xitami] Xitami has slowed to a crawl
>
>
> >I think I'm finding the source of my slow downs. Crawlers and the worst
> > right now is msn bot. But I'm also seeing Yahoo Slurp. Which normally I
> > woudn't mind them indexing my websites but when it is affecting the
>server
> > then it's a problem.
>
>
>Try putting a "robots.txt" file in the root of your website with the
>following:
>
># robots.txt file to stop indexing of site
>User-agent: *
>Disallow: /
>
>
>--
>Xitami Users Mailing List -- For Xitami support
>To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>--
>No virus found in this incoming message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>--
>No virus found in this outgoing message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>
>--
>Xitami Users Mailing List -- For Xitami support
>To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami



--
Xitami Users Mailing List -- For Xitami support
To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
--
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005

--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005


--
Xitami Users Mailing List -- For Xitami support
To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
--
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005

--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005