This article on the Google Answers site about only allowing the Google crawler to index the site contains a nice example (from webmasterworld.com) of a spider trap to block badly behaved spiders and site copiers from pulling your sire down too quickly.
It looks as if the spider trap will only work on Linux type systems so if you have a Windows server you will need to find some other method of protecting your bandwidth.
Friday, September 30, 2005