Block Google and bots using htaccess and robots.txt

It is as important as including your pages in SERP to prevent some pages from listing from SERPs. Specially when you are a web development company and want to demo your clients with site status on your demo server.

Solution 1 : Password protection

Protecting site with htaccess password is the best way to block anyone else accessing the site. But that is not possible all the time when you have demo audience test.

Solution 2 : Robots.txt

Another Solution Google is providing is to use Robots.txt file to tell Bots not to crawl or list pages in results. But that’s not always a solution. Google’s Matt Cuts has confirmed that Google may include pages from such sites if Google think is relevant.

User-agent: *
Disallow: /

Solution 3 : Using .htaccess RewriteCond

So the solution is to block Google and other similar bots from accessing your site. For that, put following code in your htaccess.

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} AltaVista [OR]
RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
RewriteCond %{HTTP_USER_AGENT} msnbot [OR]
RewriteCond %{HTTP_USER_AGENT} Slurp
RewriteRule ^.*$ "http\:\/\/htmlremix\.com" [R=301,L]

Change URL in last line to your main site so that your site gets SEO ranking if someone linked in to your blocked site.
Download this htaccess block other common bot access