[Xitami] Xitami has slowed to a crawl
Johannes Prost
xitami@lists.xitami.org
Wed, 12 Jan 2005 10:58:41 -0600
Hi Ron
I use a txt file as well. Have a look below
Am 12 Jan 2005 um 10:30 hat Ron Ackerman geschrieben:
> Seems Googlebot is ignoring the robots.txt file, I can see other bots hit
> the file and obey it but the Googlebot just keeps on hitting the server.
>
> Here is what I have in my robots.txt file
> ----------------------------------
> User-agent: *
> Disallow: /comment.php
> Disallow: /submit.php
> Disallow: /forum/createtopic.php
> Disallow: /calendar.php
>
> User-agent: msnbot
> Disallow: /
> ------------------------------
> Yet this is in my log
>
> 66.249.66.51 - - [12/Jan/2005:10:21:11 -0500] "GET
> /submit.php?type=event&mode=&month=04&day=23&year=2004&hour=8 HTTP/1.1" 403
> 0
>
> ------------------------------
> My next step is to exclude it in the Webmask= area, I am using a text file
> called webmask.txt. My question is the format of the ip? Should it look like
> this
>
> !66.249.66.51,
>
> or like this
>
> '!66.249.66.51'
>
> Does the ballbat (!) exclude or include.
> Do I need the ballbat? (I know it's an exclaimation point but it's easier
> to type ball bat).
> How about the single quotes do I need them?
> The info in the xitami.cfg isn't clear to me.
>
here some lines of my txt file which is working here at my place
webmask= !211.*
webmask= !216.239.244.98
webmask= !216.136.85.161
webmask= !217.236.57.66
webmask= !218.150.163.30
webmask= !221.232.128.2
webmask= *
# 195.92.95.61 netcraft
# 209.237.238.* ia archiver
even the comment lines are okay in there. Just make sure that the last entry of all lines
"webmask" is set to webmask= * like in my sample. Xitami reads the file from top to bottom, each
line separate, and to ba able to allow all other addresses except the ones with ! the entry has to
be there. In other words: first all disallowed IP's and then free entry.
Hope this helps
Johannes
PS. I found that the ia archiver is very pesty and is NOT obeying the robots.txt. I had to
occasions were thex explecitly (hopefully correct spelled, not my native language) downloaded
picture files from directories which were disallowed correctly.
> Thanks,
> Ron
>
>
>
>
> -----Original Message-----
> From: xitami-admin@lists.xitami.org
> [mailto:xitami-admin@lists.xitami.org]On Behalf Of William R. Palaia
> Sent: Tuesday, January 11, 2005 3:41 PM
> To: xitami@lists.xitami.org
> Subject: Re: [Xitami] Xitami has slowed to a crawl
>
>
> Go here:
>
> http://www.robotstxt.org/wc/norobots.html
>
>
> >
> > My system is Pentium 4, 1 gig DDRAM2.0, 2 dual 250Gb data drives
> (mirrored)
> > with a 40Gb IOS drive that has 10Gb set aside as virtual memory. I am
> > running Win2K Advanced Server and Xitami ver2.49d. MySQL, PHP, Perl and
> am
> > running about 18 sites at the moment. My server is a work horse but by
> the
> > sounds of the MSNbot it won't matter b/c it will be killing the bandwidth
> > more so than the system.
> >
> > Good research and great info about the msnbot - I read the whole thread
> and
> > got more than I was expecting.
> >
> > Question: I am surprised but I have never even heard of the robots.txt
> file,
> > could you send me a copy of yours so I could see the proper format? Where
> > does it go? I def. want to restrict the msnbot from my server until I can
> > properly understand everything and be able to determine the best time to
> > allow it back on later.
> >
> > Thanks Ron.
> >
> >
> > Russel P. Olinger
> > (310) 466-3399
> > rolinger1@hotmail.com
> >
> >
> >
> >
> >> From: "Ron Ackerman" <ron@miplanet.net>
> >> Reply-To: xitami@lists.xitami.org
> >> To: <xitami@lists.xitami.org>
> >> Subject: RE: [Xitami] Xitami has slowed to a crawl
> >> Date: Tue, 11 Jan 2005 14:22:42 -0500
> >>
> >> UPDATE:
> >>
> >> I have disallowed MSNbot for now based on information from this forum
> >>
> http://my.jimworld.com/apps/searchengine.forums/forum::other-search-engines/
> >> thread::1085675955/action::thread/
> >>
> >> -----Original Message-----
> >> From: xitami-admin@lists.xitami.org
> >> [mailto:xitami-admin@lists.xitami.org]On Behalf Of Ron Ackerman
> >> Sent: Tuesday, January 11, 2005 2:10 PM
> >> To: xitami@lists.xitami.org
> >> Subject: RE: [Xitami] Xitami has slowed to a crawl
> >>
> >>
> >> Just read this on a forum, it seems others are having problems with
> MSNbot.
> >> Ok most don't want to be EXCLUDED from MSN but like me they don't want
> >> MSNbot to eat up our bandwidth. They had several solutions.
> >>
> >> ======================================================
> >> HELP! MSN Bot Eating Up ALL my bandwidth!!!
> >>
> >>
> >> MSN Bot Statistics for the month of May (up to May 15th -- only 15 days
> of
> >> stats!!!)
> >>
> >> msnbot64047.search.msn.com:
> >> bandwidth: 7625158 Kbytes (7.6GB)
> >> files: 166806
> >> hits: 167027
> >>
> >> This is 29.4% of my sites bandwidth this month just from MSN Bot!
> Yesterday
> >> alone it used 1.3 GB bandwidth!
> >>
> >> By comparison, Googles bot(s) have only used about 1.4GB total bandwidth
> >> this month.
> >>
> >> How can I stop MSN from killing me! At this rate itwill use 15 GB this
> >> month! Just from being indexed
> >>
> ============================================================================
> >> ========
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: xitami-admin@lists.xitami.org
> >> [mailto:xitami-admin@lists.xitami.org]On Behalf Of R O
> >> Sent: Tuesday, January 11, 2005 1:31 PM
> >> To: xitami@lists.xitami.org
> >> Subject: RE: [Xitami] Xitami has slowed to a crawl
> >>
> >>
> >>
> >> Ron,
> >>
> >> I am curious, what ver of Xitami are you running, IOS, and what kind of
> >> system are you running it on. You mention Xitami slowing down b/c of the
> >> crawlers, but it is Xitami or is it system performance trying to service
> >> the
> >> bots? Just trying to get some clarity on where the actual problem lies
> so
> >> I
> >> can A) know what to look for and B) try to correct or prevent the same
> >> issue
> >> from happening to my system.
> >>
> >> -Russel
> >>
> >>
> >> Russel P. Olinger
> >> (310) 466-3399
> >> rolinger1@hotmail.com
> >>
> >>
> >>
> >>
> >>> From: "Ron Ackerman" <ron@miplanet.net>
> >>> Reply-To: xitami@lists.xitami.org
> >>> To: <xitami@lists.xitami.org>
> >>> Subject: RE: [Xitami] Xitami has slowed to a crawl
> >>> Date: Tue, 11 Jan 2005 12:44:05 -0500
> >>>
> >>> That will exclude all robots, I'm not sure I want to do that because I
> >>> would
> >>> like to be included in the Search Engines. Is there a time setting that
> >> can
> >>> be applied. Like Excluding the robots between 7am and 11pm?
> >>>
> >>> -----Original Message-----
> >>> From: xitami-admin@lists.xitami.org
> >>> [mailto:xitami-admin@lists.xitami.org]On
> >>> Behalf Of Jeff Hamblin
> >>> Sent: Tuesday, January 11, 2005 12:39 PM
> >>> To: xitami@lists.xitami.org
> >>> Subject: Re: [Xitami] Xitami has slowed to a crawl
> >>>
> >>>
> >>>> I think I'm finding the source of my slow downs. Crawlers and the worst
> >>>> right now is msn bot. But I'm also seeing Yahoo Slurp. Which normally
> >> I
> >>>> woudn't mind them indexing my websites but when it is affecting the
> >>> server
> >>>> then it's a problem.
> >>>
> >>>
> >>> Try putting a "robots.txt" file in the root of your website with the
> >>> following:
> >>>
> >>> # robots.txt file to stop indexing of site
> >>> User-agent: *
> >>> Disallow: /
> >>>
> >>>
> >>> --
> >>> Xitami Users Mailing List -- For Xitami support
> >>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
> >>> --
> >>> No virus found in this incoming message.
> >>> Checked by AVG Anti-Virus.
> >>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >>>
> >>> --
> >>> No virus found in this outgoing message.
> >>> Checked by AVG Anti-Virus.
> >>> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >>>
> >>>
> >>> --
> >>> Xitami Users Mailing List -- For Xitami support
> >>> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
> >>
> >>
> >>
> >> --
> >> Xitami Users Mailing List -- For Xitami support
> >> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
> >> --
> >> No virus found in this incoming message.
> >> Checked by AVG Anti-Virus.
> >> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >>
> >> --
> >> No virus found in this outgoing message.
> >> Checked by AVG Anti-Virus.
> >> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >>
> >>
> >> --
> >> Xitami Users Mailing List -- For Xitami support
> >> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
> >> --
> >> No virus found in this incoming message.
> >> Checked by AVG Anti-Virus.
> >> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >>
> >> --
> >> No virus found in this outgoing message.
> >> Checked by AVG Anti-Virus.
> >> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >>
> >>
> >> --
> >> Xitami Users Mailing List -- For Xitami support
> >> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
> >> --
> >> No virus found in this incoming message.
> >> Checked by AVG Anti-Virus.
> >> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >>
> >> --
> >> No virus found in this outgoing message.
> >> Checked by AVG Anti-Virus.
> >> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
> >>
> >>
> >> --
> >> Xitami Users Mailing List -- For Xitami support
> >> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
> >
> >
> >
> > --
> > Xitami Users Mailing List -- For Xitami support
> > To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
> >
>
>
> --
> Xitami Users Mailing List -- For Xitami support
> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami
> --
> No virus found in this incoming message.
> Checked by AVG Anti-Virus.
> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
> --
> No virus found in this outgoing message.
> Checked by AVG Anti-Virus.
> Version: 7.0.300 / Virus Database: 265.6.10 - Release Date: 01/10/2005
>
>
> --
> Xitami Users Mailing List -- For Xitami support
> To unsubscribe: http://lists.xitami.org/mailman/listinfo/xitami