Search


-->

Comments

« Name that Gallery • Home • $2.99 Domain Registration »

My Robots.txt

• Working On My Website

Mostly I'm protecting myself from bandwidth drain. Not as big an issue as it used to be. But in some cases the bots worked the scripts so heavily that the CPU drain on the server was pretty bad.

First the usual stuff:

User-agent: *
Disallow: /cgi-bin
Disallow: /mt
Disallow: /common

PicSearch's psbot was consuming much bandwidth grabbing images. Mostly from affliate pages. Near as I can tell Become.com is a fancy scraper site.

User-agent: psbot
User-agent: BecomeBot
Disallow: /

Some times Google just goes berserk grabbing images. Again, mostly affilate program stuff. Never got any good out of it.

User-agent: Googlebot-Image
Disallow: /*.gif$
User-agent: Googlebot-Image
Disallow: /*.jpg$
User-agent: Googlebot-Image
Disallow: /

Jeeves will just gobble and gobble but send you very few hits. And sometimes slurp is just too hungry.

User-agent: Teoma
Crawl-delay: 60
User-agent: Slurp
Crawl-delay: 30

Richard
May 28, 2006 •
BlinkList • del.icio.us • Digg it • Furl • ma.gnolia • Netvouz
RawSugar • Shadows • Simpy • Spurl • Yahoo MyWeb
Top↑

Comments

Thank you for posting that. Google has been grabbing stuff from some of my sites like a maniac but not sending traffic so i t does me no good at all … !

Posted by: Worried Webmaster | June 8, 2006 07:25 PM

How do you feel?

Feel free to share your feelings about My Robots.txt. Please stick to the theme of the entry. Disagreement is fine. Homophobia, racism, and kindred expressions of hatred will be deleted. This site is one of my hobbies. I genuinely enjoy hearing from people and hate moderating or killing comments. Forthright disagreement is fine as long as it is civil.
My thanks,
Richard


Monthly

June 2006, May 2006, April 2006, March 2006, December 2005, November 2005, October 2005, September 2005, August 2005, May 2005, April 2005, September 2004, August 2004, July 2004, June 2004, May 2004, March 2004, February 2004, January 2004, December 2003, November 2003, October 2003, September 2003, August 2003, July 2003, June 2003, May 2003, April 2003, March 2003, February 2003, January 2003, December 2002, November 2002, October 2002, April 2002, March 2002,

--!>

The Register - Music and Media Biting the hand that feeds IT

  • Indies celebrate Independence Day Finding the sharing button

    Interview Did anyone, I wonder, ever buy just one Motown single? Or just one 2-Tone single? And while you're pondering... can you even remember what major label your favourite artist is on? Unigram, perhaps. Or Polycorpse.?


  • Virgin warns 800 punters for file-sharing 'Important. If you don't read this, your broadband could be disconnected'

    The BPI has written to 800 Virgin Media customers warning them to stop sharing music files or risk losing their broadband connection.?


  • Italian authorities raid alleged music filesharing forum Megaupload and Rapidshare link site targeted

    Italy's Guardia di Finanza has shut down forum site Downrevolution.net for offering links to pirated music, video and software hosted on popular filesharing sites.?


  • Family Guy creator's sellout to Google almost complete Two minute clips served with your ads

    Google is revealing new details on the recruitment of Family Guy creator Seth MacFarlane to attract prime-time television advertising dollars to web-only video content.?


  • Real researchers discover 'Music Without Limits' Otherwise known as DRM-less MP3s

    The DRM deathwatch continues. Today, Real Networks announced that its Rhapsody digital music service is selling DRM-free MP3s from all four major records labels, following in the footsteps of Amazon and Napster.?



Newsfeed display by CaRP