|
I have made some security modifications to my main .htaccess file. By doing so I am thinking I am blocking the msn crawler that indexes my site. My links in msn search are almost non-existent when I search msn with site:mysite.com it returns like 3-5 results. Google returns hundreds, yahoo over a hundred.
So I am thinking something in my .htaccess might be preventing it from crawling. I am concerned about security so I don't really want to remove anything that I don't have to.
Here is a partial that I believe is causing the problem somewhere if someone could help:
Options -MultiViews Options -Indexes <IfModule mod_rewrite.c> RewriteEngine on RewriteBase / RewriteCond %{HTTP_REFERER} ^(.*)(<|>|'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR] RewriteCond %{QUERY_STRING} ^http [OR] RewriteCond %{QUERY_STRING} ^.+www\. [OR] RewriteCond %{QUERY_STRING} ^.+https [OR] RewriteCond %{QUERY_STRING} ^.+\.txt [OR] RewriteCond %{QUERY_STRING} ^.+ftp RewriteCond %{QUERY_STRING} ^.*(;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|cast|set|declare|drop|update|md5|benchmark).* [NC,OR] RewriteRule .* - [L,F] RewriteCond %{HTTP_USER_AGENT} ^libwww [OR] RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|scan).* [NC,OR] RewriteCond %{HTTP_USER_AGENT} ^Wget RewriteCond %{HTTP_USER_AGENT} !^.*msnbot.* [OR,NC] RewriteRule .* - [F]
I know this part of the code blocks a lot of stuff not sure if it will the msn bot too: RewriteCond %{QUERY_STRING} ^http [OR] RewriteCond %{QUERY_STRING} ^.+www\. [OR]
But google and yahoo bots seem to come and crawl my sites with the same code which is odd. Some of the other part ot the code Rewrite conditions I'm not 100% sure about.
Thanks.
|