[Xitami] Xitami has slowed to a crawl

Ron Ackerman xitami@lists.xitami.org
Tue, 11 Jan 2005 13:52:30 -0500 

Sure,

I am running an AMD 850mhz processor with 1.04gb of memmory on a Windows
2000 Pro with all updates and service packs. My Xitami ver is v2.5b5 running
12 websites and I have done all required updates. I have Perl, PHP (latest
update), MySQL, and Cold Fusion installed.

Just a note, It appears to be the MSN bot that causes the problem. I can
reboot and be fine until the msnbot starts back up. Once the bot was
finished things seem to go back to normal. I may disallow the msnbot for a
week or so to see if it resolves the problem. I don't see that I can limit
the time when the bots hit the server. So I may have to do a batch file that
changes robots.txt twice a day.

Here is the log from when it pretty much died.

65.54.188.101 - - [11/Jan/2005:11:58:22 -0500] "GET
/advt/detail.php?cat=10&de=680 HTTP/1.0" 503 10684 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
192.168.1.15 - - [11/Jan/2005:11:59:23 -0500] "GET
/basic%20site/bkground.jpg HTTP/1.1" 404 -1
"http://www.causewaylighting.com/advt/order.php" "Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)"
65.54.188.101 - - [11/Jan/2005:11:59:27 -0500] "GET
/advt/detail.php?cat=10&de=680 HTTP/1.0" 503 10684 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
207.46.98.59 - - [11/Jan/2005:12:02:05 -0500] "GET
/links.php?category=Healthcare HTTP/1.0" 503 12878 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
207.46.98.59 - - [11/Jan/2005:12:03:09 -0500] "GET
/links.php?category=Healthcare HTTP/1.0" 503 12878 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
65.54.188.101 - - [11/Jan/2005:12:03:49 -0500] "GET
/advt/detail.php?cat=10&de=678 HTTP/1.0" 503 10684 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
65.54.188.101 - - [11/Jan/2005:12:04:46 -0500] "GET
/advt/detail.php?cat=10&de=678 HTTP/1.0" 503 10684 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
207.46.98.59 - - [11/Jan/2005:12:05:50 -0500] "GET
/links.php?category=Restauraunt HTTP/1.0" 503 12878 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
65.54.188.81 - - [11/Jan/2005:12:06:50 -0500] "GET
/staticpages/index.php?page=main HTTP/1.0" 503 10033 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
207.46.98.59 - - [11/Jan/2005:12:06:50 -0500] "GET
/links.php?category=Restauraunt HTTP/1.0" 503 12878 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
207.150.160.200 - - [11/Jan/2005:12:08:02 -0500] "HEAD / HTTP/1.0" 503 -1 ""
"WebRescuer v0.2.4"
65.54.188.101 - - [11/Jan/2005:12:08:34 -0500] "GET
/advt/detail.php?cat=15&de=669 HTTP/1.0" 503 10684 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
65.54.188.81 - - [11/Jan/2005:12:09:26 -0500] "GET
/staticpages/index.php?page=main HTTP/1.0" 503 10033 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
207.46.98.54 - - [11/Jan/2005:12:10:25 -0500] "GET /robots.txt HTTP/1.0"
404 -1 "" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"
65.54.188.101 - - [11/Jan/2005:12:10:25 -0500] "GET
/advt/detail.php?cat=15&de=669 HTTP/1.0" 503 10684 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
65.54.188.57 - - [11/Jan/2005:12:12:21 -0500] "GET
/profiles.php?sid=20030715225415843&what=emailstory HTTP/1.0" 503 17658 ""
"msnbot/0.3 (+http://search.msn.com/msnbot.htm)"
207.46.98.59 - - [11/Jan/2005:12:12:49 -0500] "GET
/links.php?category=Churches HTTP/1.0" 503 12878 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
65.54.188.81 - - [11/Jan/2005:12:13:21 -0500] "GET /index.php?topic=outdoor_
HTTP/1.0" 503 10019 "" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"
65.54.188.101 - - [11/Jan/2005:12:14:38 -0500] "GET /advt/list.php?cat=24
HTTP/1.0" 503 14652 "" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"
65.54.188.81 - - [11/Jan/2005:12:15:09 -0500] "GET
/advt/detail.php?cat=16&de=57 HTTP/1.0" 503 10684 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
65.54.188.101 - - [11/Jan/2005:12:16:02 -0500] "GET /advt/list.php?cat=24
HTTP/1.0" 503 14652 "" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"
65.54.188.81 - - [11/Jan/2005:12:17:22 -0500] "GET
/advt/detail.php?cat=16&de=57 HTTP/1.0" 503 10684 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
207.46.98.59 - - [11/Jan/2005:12:17:30 -0500] "GET
/links.php?category=Family HTTP/1.0" 503 12878 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
65.54.188.57 - - [11/Jan/2005:12:18:44 -0500] "GET /index.php?topic=GeekLog
HTTP/1.0" 503 10019 "" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"
66.196.90.48 - - [11/Jan/2005:12:19:03 -0500] "GET /robots.txt HTTP/1.0"
404 -1 "" "Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)"
207.46.98.59 - - [11/Jan/2005:12:19:11 -0500] "GET
/links.php?category=Family HTTP/1.0" 503 12878 "" "msnbot/0.3
(+http://search.msn.com/msnbot.htm)"
65.54.188.57 - - [11/Jan/2005:12:19:54 -0500] "GET /index.php?topic=GeekLog
HTTP/1.0" 503 10019 "" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"
207.46.98.59 - - [11/Jan/2005:12:21:29 -0500] "GET /index.php?topic=SiteNews
HTTP/1.0" 503 10019 "" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"
207.46.98.59 - - [11/Jan/2005:12:23:22 -0500] "GET /index.php?topic=SiteNews
HTTP/1.0" 503 10019 "" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"





-----Original Message-----
From: xitami-admin@lists.xitami.org
[mailto:xitami-admin@lists.xitami.org]On Behalf Of R O
Sent: Tuesday, January 11, 2005 1:31 PM
To: xitami@lists.xitami.org
Subject: RE: [Xitami] Xitami has slowed to a crawl



Ron,

I am curious, what ver of Xitami are you running, IOS, and what kind of
system are you running it on.  You mention Xitami slowing down b/c of the
crawlers, but it is Xitami or is it system performance trying to service the
bots?  Just trying to get some clarity on where the actual problem lies so I
can A) know what to look for and B) try to correct or prevent the same issue
from happening to my system.

-Russel


Russel P. Olinger
(310) 466-3399
rolinger1@hotmail.com




>From: "Ron Ackerman" <ron@miplanet.net>
>Reply-To: xitami@lists.xitami.org
>To: <xitami@lists.xitami.org>
>Subject: RE: [Xitami] Xitami has slowed to a crawl
>Date: Tue, 11 Jan 2005 12:44:05 -0500
>
>That will exclude all robots, I'm not sure I want to do that because I
>would
>like to be included in the Search Engines. Is there a time setting that can
>be applied. Like Excluding the robots between 7am and 11pm?
>
>-----Original Message-----
>From: xitami-admin@lists.xitami.org
>[mailto:xitami-admin@lists.xitami.org]On
>Behalf Of Jeff Hamblin
>Sent: Tuesday, January 11, 2005 12:39 PM
>To: xitami@lists.xitami.org
>Subject: Re: [Xitami] Xitami has slowed to a crawl
>
>
> >I think I'm finding the source of my slow downs. Crawlers and the worst
> > right now is msn bot. But I'm also seeing Yahoo Slurp. Which normally I
> > woudn't mind them indexing my websites but when it is affecting the
>server
> > then it's a problem.
>
>
>Try putting a "robots.txt" file in the root of your website with the
>following:
>
># robots.txt file to stop indexing of site
>User-agent: *
>Disallow: /
>
>
>--
>Xitami Users Mailing List -- For Xitami support
>To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
>--
>No virus found in this incoming message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>--
>No virus found in this outgoing message.
>Checked by AVG Anti-Virus.
>Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>
>--
>Xitami Users Mailing List -- For Xitami support
>To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami



--
Xitami Users Mailing List -- For Xitami support
To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
--
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005

--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005