How to keep your development sites from being indexed

This guide will show you how to configure your development site to ensure it is not indexed by Google. If you’re looking to set up a development environment, take a read over our guide here to learn how to do so then come on back here.

Why is it important that your development sites not be indexed? Google and other search engines are generally not the biggest fans of duplicate content, and so having your development site indexed could result in a drop in rankings. Additionally, it’s not great for the user experience if they end up on a dev site where there may be bugs, such as if they can’t actually contact you or place an order.

For those reasons, it’s considered best practice to ensure your development environments are not found. There’s a number of ways to accomplish this and it’s best to only choose one of the following options, though multiple can be used if you wish.

Both Plesk-based options will apply settings at the web server level, so web apps like WordPress will only be available to those who have been provided access. This makes them the most secure options available to you as even if there’s a WordPress zero day vulnerability (for example), they cannot be used if the attacker can’t access WordPress’s files at all.

Plesk: IP Restrictions

You can lock down your development site by IP address. This is the most secure method available, but it can be annoying if you intend to access the development site regularly from various locations without using a VPN.

Tip: if you use a VPN and connect to the same location each time then your IP should remain static. If you don’t already have access to a VPN service, we recommend Private Internet Access.

Here’s how to block access to your site by IP address:

• Begin by logging in to Plesk
• If you see a list of domains, click on your development domain or subdomain to edit its config. If you see domains listed with many buttons underneath, bring your attention to the buttons that match up with your development domain or subdomain.
• Press the button called “Apache & nginx Settings”. Tip: if you don’t see Apache & nginx Settings, click the “Hosting & DNS” tab and you’ll find the button under that tab.
• Under Common Apache settings you’ll find Deny access to the site. Select “Enter custom value” and enter an asterisk (*) in the top box. Under the second box called “Excluding” enter the IPs for which you wish to provide access to the site, one per line. (See tip about localhost below)
• Scroll down and press the OK button

That’s it! Only those using the IPs you entered will have access to the site.

Tips:

• Even though the title of this section includes “Apache” most (if not all) items here will apply to nginx as well unless it is disabled.
• It is recommended to add the localhost IP (127.0.0.1) to the list as well to ensure Installatron / 1-click web apps and other local utilities (like WordPress Site Health) can access the site as well.

Plesk: HTTP Authentication

You could also opt to password protect your site from anyone trying to access it. This is the second most secure option and is best for people with changing IPs or accessing the site from various locations without a VPN. Here’s how to password protect any folder with Plesk.

Web App: Maintenance Plugins

The third most secure method of accomplishing this is to use an under construction plugin that blocks access to the site to any user not logged in to your website admin. This prevents access to all pages of the site (posts, and custom post types included) unless the visitor is logged in. By default most of these plugins require the admin role, though that can be changed in the settings.

We’ve used a number of different plugins for this in the past, and honestly just about any one of them will do the trick. The two most recent ones we use for WordPress are:

• Under Construction by WebFactory Ltd. There’s a free and paid version, though the free version should work well for most people.
• Beaver Lodge Maintenance. Since we love BeaverBuilder around here, this plugin allows us to build the coming soon page using BeaverBuilder!

Other web apps will probably either have a similar function built in, or plugins just like these ones available to use. Check their documentation and plugin/extension repository for more info!

WordPress: Search Engine Visibility

While this option will not block access to the site if someone were to guess the URL, it will successfully prevent indexing by search engines, effectively hiding the dev site. It does this by setting the robots meta value to noindex as well as specifying a blanket disallow in the dynamically generated robots.txt file

Manual: robots.txt

If your web app doesn’t provide the option to disallow access to the admin or block search engines from within its settings or a plugin, you can do it manually! Here’s how:

1. Access the files for your website via FTP or the Plesk File Manager and navigate to your web root folder (the linked guide will show you how to do this)
2. Create a file called ‘robots.txt’ (without the quotes) and place the following within it:

User-agent: *
Disallow: /

More info on how robots.txt works here.

By employing one of these options, you’ll ensure that your development environment does not get picked up as a live site by search engines, which is beneficial for site security, usability, and SEO.

Important: If you’re using any of the above options that are not configured per-domain in Plesk, when you sync your development site over to live, these changes will likely copy over so do not forget to revert them! If you forget to do this, your site will not be found by Google and if it was previously on Google, it will eventually become de-indexed.