I continue to get a fair number of web server “404 Not Found” errors on my site (caused by users requesting documents that aren't found). I strive to minimize such errors when making changes to the site, so normally I read through the server logs to find anything that needs to be fixed.
To make this process easier, I recently installed a custom 404 error handler in PHP based on Harold Bakker's Mail on 404 script. This script will send you an e-mail message anytime a 404 error is generated on the server. I modified it slightly to also send the missing URL in the e-mail subject to make it easier to scan.
As expected, I get a fair amount of 404 errors from scripts looking for known Windows security holes. Even though my provider runs Apache on Linux, this is kind of annoying. I've modified the script to search for any URL containing .asp, .dll, or .exe and save the requesting IP address in a special “ban” file. Now I just need to wire that into my site-wide .htaccess file and I can permanently ban such users. Maybe this is too drastic?
On the positive side I was able to see that users trying to reach my PDA-friendly web site list (http://cantoni.mobi) often type in variations of the URL. I simply added those to the local .htaccess file to help these folks out by redirecting them to the page they're looking for.