htaccess Elite

.htaccess tutorial


All times are UTC [ DST ]





Post new topic Reply to topic  [ 1 post ] 
Author Message
 Post subject: Allow the MSN Crawler
PostPosted: 31 Oct 2008 10:59 
Offline

Joined: 31 Oct 2008 10:48
Posts: 1
Location: USA
I have made some security modifications to my main .htaccess file. By doing so I am thinking I am blocking the msn crawler that indexes my site. My links in msn search are almost non-existent when I search msn with site:mysite.com it returns like 3-5 results. Google returns hundreds, yahoo over a hundred.

So I am thinking something in my .htaccess might be preventing it from crawling. I am concerned about security so I don't really want to remove anything that I don't have to.

Here is a partial that I believe is causing the problem somewhere if someone could help:

Options -MultiViews
Options -Indexes
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_REFERER} ^(.*)(<|>|'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR]
RewriteCond %{QUERY_STRING} ^http [OR]
RewriteCond %{QUERY_STRING} ^.+www\. [OR]
RewriteCond %{QUERY_STRING} ^.+https [OR]
RewriteCond %{QUERY_STRING} ^.+\.txt [OR]
RewriteCond %{QUERY_STRING} ^.+ftp
RewriteCond %{QUERY_STRING} ^.*(;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|cast|set|declare|drop|update|md5|benchmark).* [NC,OR]
RewriteRule .* - [L,F]
RewriteCond %{HTTP_USER_AGENT} ^libwww [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|scan).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget
RewriteCond %{HTTP_USER_AGENT} !^.*msnbot.* [OR,NC]
RewriteRule .* - [F]


I know this part of the code blocks a lot of stuff not sure if it will the msn bot too:
RewriteCond %{QUERY_STRING} ^http [OR]
RewriteCond %{QUERY_STRING} ^.+www\. [OR]

But google and yahoo bots seem to come and crawl my sites with the same code which is odd. Some of the other part ot the code Rewrite conditions I'm not 100% sure about.

Thanks.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 1 post ] 

All times are UTC [ DST ]


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Powered by phpBB