[Xitami] Xitami has slowed to a crawl

Tue, 11 Jan 2005 12:10:04 -0800

My system is Pentium 4, 1 gig DDRAM2.0, 2 dual 250Gb data drives (mirrored) 
with a 40Gb IOS drive that has 10Gb set aside as virtual memory.  I am 
running Win2K Advanced Server and Xitami ver2.49d.  MySQL, PHP, Perl and am 
running about 18 sites at the moment.  My server is a work horse but by the 
sounds of the MSNbot it won't matter b/c it will be killing the bandwidth 
more so than the system.

Good research and great info about the msnbot - I read the whole thread and 
got more than I was expecting.

Question: I am surprised but I have never even heard of the robots.txt file, 
could you send me a copy of yours so I could see the proper format?  Where 
does it go?  I def. want to restrict the msnbot from my server until I can 
properly understand everything and be able to determine the best time to 
allow it back on later.

Thanks Ron.

Russel P. Olinger
(310) 466-3399
rolinger1@hotmail.com

>From: "Ron Ackerman" <ron@miplanet.net>
>Reply-To: xitami@lists.xitami.org
>To: <xitami@lists.xitami.org>
>Subject: RE: [Xitami] Xitami has slowed to a crawl
>Date: Tue, 11 Jan 2005 14:22:42 -0500
>
>UPDATE:
>
>I have disallowed MSNbot for now based on information from this forum
>http://my.jimworld.com/apps/searchengine.forums/forum::other-search-engines/
>thread::1085675955/action::thread/
>
>-----Original Message-----
>From: xitami-admin@lists.xitami.org
>[mailto:xitami-admin@lists.xitami.org]On Behalf Of Ron Ackerman
>Sent: Tuesday, January 11, 2005 2:10 PM
>To: xitami@lists.xitami.org
>Subject: RE: [Xitami] Xitami has slowed to a crawl
>
>
>Just read this on a forum, it seems others are having problems with MSNbot.
>Ok most don't want to be EXCLUDED from MSN but like me they don't want
>MSNbot to eat up our bandwidth. They had several solutions.
>
>======================================================
>HELP! MSN Bot Eating Up ALL my bandwidth!!!
>
>
>MSN Bot Statistics for the month of May (up to May 15th -- only 15 days of
>stats!!!)
>
>msnbot64047.search.msn.com:
>bandwidth: 7625158 Kbytes (7.6GB)
>files: 166806
>hits: 167027
>
>This is 29.4% of my sites bandwidth this month just from MSN Bot! Yesterday
>alone it used 1.3 GB bandwidth!
>
>By comparison, Googles bot(s) have only used about 1.4GB total bandwidth
>this month.
>
>How can I stop MSN from killing me! At this rate itwill use 15 GB this
>month! Just from being indexed
>============================================================================
>========
>
>
>
>-----Original Message-----
>From: xitami-admin@lists.xitami.org
>[mailto:xitami-admin@lists.xitami.org]On Behalf Of R O
>Sent: Tuesday, January 11, 2005 1:31 PM
>To: xitami@lists.xitami.org
>Subject: RE: [Xitami] Xitami has slowed to a crawl
>
>
>
>Ron,
>
>I am curious, what ver of Xitami are you running, IOS, and what kind of
>system are you running it on.  You mention Xitami slowing down b/c of the
>crawlers, but it is Xitami or is it system performance trying to service 
>the
>bots?  Just trying to get some clarity on where the actual problem lies so 
>I
>can A) know what to look for and B) try to correct or prevent the same 
>issue
>from happening to my system.
>
>-Russel
>
>
>Russel P. Olinger
>(310) 466-3399
>rolinger1@hotmail.com
>
>
>
>
> >From: "Ron Ackerman" <ron@miplanet.net>
> >Reply-To: xitami@lists.xitami.org
> >To: <xitami@lists.xitami.org>
> >Subject: RE: [Xitami] Xitami has slowed to a crawl
> >Date: Tue, 11 Jan 2005 12:44:05 -0500
> >
> >That will exclude all robots, I'm not sure I want to do that because I
> >would
> >like to be included in the Search Engines. Is there a time setting that 
>can
> >be applied. Like Excluding the robots between 7am and 11pm?
> >
> >-----Original Message-----
> >From: xitami-admin@lists.xitami.org
> >[mailto:xitami-admin@lists.xitami.org]On
> >Behalf Of Jeff Hamblin
> >Sent: Tuesday, January 11, 2005 12:39 PM
> >To: xitami@lists.xitami.org
> >Subject: Re: [Xitami] Xitami has slowed to a crawl
> >
> >
> > >I think I'm finding the source of my slow downs. Crawlers and the worst
> > > right now is msn bot. But I'm also seeing Yahoo Slurp. Which normally 
>I
> > > woudn't mind them indexing my websites but when it is affecting the
> >server
> > > then it's a problem.
> >
> >
> >Try putting a "robots.txt" file in the root of your website with the
> >following:
> >
> ># robots.txt file to stop indexing of site
> >User-agent: *
> >Disallow: /
> >
> >
> >--
> >Xitami Users Mailing List -- For Xitami support
> >To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
> >--
> >No virus found in this incoming message.
> >Checked by AVG Anti-Virus.
> >Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >
> >--
> >No virus found in this outgoing message.
> >Checked by AVG Anti-Virus.
> >Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >
> >
> >--
> >Xitami Users Mailing List -- For Xitami support
> >To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>
>
>
>--
>Xitami Users Mailing List -- For Xitami support
>To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>--
>No virus found in this incoming message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>--
>No virus found in this outgoing message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>
>--
>Xitami Users Mailing List -- For Xitami support
>To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>--
>No virus found in this incoming message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>--
>No virus found in this outgoing message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>
>--
>Xitami Users Mailing List -- For Xitami support
>To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>--
>No virus found in this incoming message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>--
>No virus found in this outgoing message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>
>--
>Xitami Users Mailing List -- For Xitami support
>To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami