Block Google and bots using htaccess and robots.txt

It is as important as including your pages in SERP to prevent some pages from listing from SERPs. Specially when you are a web development company and want to demo your clients with site status on your demo server.

Block Google

Solution 1 : Password protection

Protecting site with htaccess password is the best way to block anyone else accessing the site. But that is not possible all the time when you have demo audience test.

Solution 2 : Robots.txt

Another Solution Google is providing is to use Robots.txt file to tell Bots not to crawl or list pages in results. But that’s not always a solution. Google’s Matt Cuts has confirmed that Google may include pages from such sites if Google think is relevant.

User-agent: *
Disallow: /

Solution 3 : Using .htaccess RewriteCond

So the solution is to block Google and other similar bots from accessing your site. For that, put following code in your htaccess.

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} AltaVista [OR]
RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
RewriteCond %{HTTP_USER_AGENT} msnbot [OR]
RewriteCond %{HTTP_USER_AGENT} Slurp
RewriteRule ^.*$ "http\:\/\/htmlremix\.com" [R=301,L]

Change URL in last line to your main site so that your site gets SEO ranking if someone linked in to your blocked site.
Download this htaccess block other common bot access

Remiz

Remixed version of unstable human emotions and thirst of mankind actions. UX designer, UI developer and HE of WebCastle Media Pvt LTD

You may also like...

8 Responses

  1. Jacob James says:

    Great effort and collection which are very useful to many people , thaks for sharing the info with us

  2. $userAgent = $_SERVER[‘HTTP_USER_AGENT’];
    if(stristr(strtolower($userAgent), ‘googlebot’)

  3. Hiren says:

    Ravi, I think we can keept the htaccess in the directory which we do not want to be indexed. Right? I do not know that this will be the right solution or not. Its just a thought….

  4. ravi says:

    Good post

    how can we block google from indexing a specific directory using .htaccess?

    Can you please help

  5. Ariq Naufal says:

    Thanks bro..
    i will try with this step.. 🙂

  1. July 10, 2013

    […] access to your information. Things like using alternatives to Google Analytics, prevent G from indexing your site, stop using Chrome (and Firefox, as they are funded by Google) and start using Bing search […]

  2. November 4, 2013

    […] your main site so that your site gets SEO ranking if someone linked in to your blocked site. from http://www.htmlremix.com/seo/block-g…and-robots-txt Reply With […]

  3. May 19, 2022

    […] – Disallowing htaccess and robots.txt. […]

Talk your view

Read previous post:
Changing view in progress

My YouTube channel VISIONRemix's logo release.

Close