Looking for some apache config help to block evil spiders
Tony Nelson
tonynelson at georgeanelson.com
Sat Oct 10 21:16:30 UTC 2009
On 09-10-10 14:37:29, Steven W. Orr wrote:
> ... Here's what I added to my httpd.conf:
>
> RewriteLog logs/rewrite_log
> RewriteLogLevel 1
>
> RewriteCond %{HTTP_USER_AGENT} ^Baiduspider.* [OR]
> RewriteCond %{HTTP_USER_AGENT} ^msnbot.* [OR]
> RewriteCond %{HTTP_USER_AGENT} ^NaverBot.* [OR]
> RewriteCond %{HTTP_USER_AGENT} ^Sogou-Test-Spider.*
> RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4.0.*
> RewriteCond %{HTTP_USER_AGENT} ^T-Mobile Dash.*
> RewriteRule .* - [F,L]
>
> and inside each of the virtual domains, I added:
>
> RewriteEngine On
> RewriteOptions Inherit
>
> Here's the problem. What I want to see is the rewrite_log telling me
> what it has redirected or failed. Instead, I'm getting a line telling
> me every link that it does NOT rewrite. For example:
>
> 72.30.65.61 - - [10/Oct/2009:14:28:24 --0400] \
> [vdom.syslang.net/sid#b7298ed0][rid#b6b488e8/initial] (1) pass
> through /d1/fn
>
> I have googled my brains out and it seems like others have had the
> same questions. I see no answers. If anyone has any idea I love to
> hear it.
WAG: The RewriteRule doesn't actually rewrite anything. Perhaps
something would be logged if it did. You'd probably still have the
other log lines as well.
--
____________________________________________________________________
TonyN.:' <mailto:tonynelson at georgeanelson.com>
' <http://www.georgeanelson.com/>
More information about the fedora-list
mailing list