How to block access to a website and keep it from being indexed

This guide will show you how to block visitors – or just bots – from accessing your website. There’s a few reasons you might want to do this:

• Your site is under development and you don’t want anyone to see it until it’s done
• You wish to keep an old copy of a website available, but only accessible to you
• You don’t want bots like Googlebot to index a temp or development copy of your site

Tip: if you’re looking to set up a development environment, take a read over our guide here to learn how to do so then come on back here.

Why is it important that non-live sites, like development sites or staging sites, not be indexed? Google and other search engines are generally not the biggest fans of duplicate content, and so having your development site indexed could result in a drop in rankings. Additionally, it’s not great for the user experience if they end up on a dev site where there may be slightly older content or problems with functionality as you’re working on things.

There’s a number of ways to accomplish this and it’s best to only use one at a time. The first two will block access to everyone, the second pair are purely about preventing bots from indexing the site.

1. Use Plesk to restrict access at the web server level by IP address or Password (Best for site performance)
2. Use a WordPress plugin to restrict access at the web app / PHP level
3. Use WordPress to tell it to block bots via your site’s HTML header
4. Use a robots.txt file to tell it to block bots

Read on to learn how to use each of these options!

How to block everyone from accessing your site

The following Plesk-based options will apply settings at the web server level, so web apps like WordPress will only be available to those who have been provided access. This makes them the most secure options.

Use IP Restrictions in Plesk to block access

You can lock down your development site by IP address. This is the most secure method available, but it can be annoying if you intend to access the development site regularly from various locations without using a VPN.

Tip: if you use a VPN and connect to the same location each time then your IP should remain static. If you don’t already have access to a VPN service, we recommend Private Internet Access.

Here’s how to block access to your site by IP address:

• Begin by logging in to Plesk
• If you see a list of domains, click on your development domain or subdomain to edit its config. If you see domains listed with many buttons underneath, bring your attention to the buttons that match up with your development domain or subdomain.
• Press the button called “Apache & nginx Settings”. Tip: if you don’t see Apache & nginx Settings, click the “Hosting & DNS” tab and you’ll find the button under that tab.
• Under Common Apache settings you’ll find Deny access to the site. Select “Enter custom value” and enter an asterisk (*) in the top box. Under the second box called “Excluding” enter the IPs for which you wish to provide access to the site, one per line. (See tip about localhost below)
• Scroll down and press the OK button

That’s it! Only those using the IPs you entered will have access to the site.

Tips:

• Even though the title of this section includes “Apache” most (if not all) items here will apply to nginx as well unless it is disabled.
• It is recommended to add the localhost IP (127.0.0.1) to the list as well to ensure Installatron / 1-click web apps and other local utilities (like WordPress Site Health) can access the site as well.

Use web server passwords in Plesk to block access

You could also opt to password protect your site from anyone trying to access it. This is the second most secure option and is best for people with changing IPs or accessing the site from various locations without a VPN. Here’s how to password protect any folder with Plesk.

Use a plugin for your website

The third most secure method of accomplishing this is to use a maintenance plugin that blocks access to the site to any user not logged in to your website admin. This prevents access to the entire site unless the visitor is logged in. Here’s how to do that with common web applications:

• WordPress: check out our dedicated guide to enabling maintenance mode with WordPress
• Modx: this community post covers maintenance mode for modx nicely

How to block just bots from accessing your website

The following options will not block general visitors, rather they ask bots to not crawl the site. Note that it’s not an actual block – it’s a request – so the bot must obey the request for it to not crawl the site. Common search engines are expected to obey the request.

Use WordPress’s Search Engine Visibility Setting

This option uses an HTML tag to ask robots not to index your site. You’ll find it in the WordPress admin under Settings > Reading.

You can read more about how it works in the WordPress.org documentation here.

Use a manually created robots.txt file

If not using WordPress and your web app doesn’t provide the option to block search engines either built-in or with a plugin/module/extension, you can do it manually! Here’s how:

1. Access the files for your website via FTP or the Plesk File Manager and navigate to your web root folder (the linked guide will show you how to do this)
2. Create a file called ‘robots.txt’ (without the quotes) and place the following within it:

User-agent: *
Disallow: /

More info on how robots.txt works here.

By employing one of these options, you’ll ensure that your development environment does not get picked up as a live site by search engines, which is beneficial for site security, usability, and SEO.

Important: If you’re using a web app / WordPress-based solution, when you sync your development site over to live, the block will copy over to live so do not forget to undo the block on live! If you forget to do this, your site will not be found by Google and if it was previously on Google, it will eventually become de-indexed.