How to keep your development sites from being indexed

This guide will show you how to configure your development site to ensure it is not indexed by Google. If you’re looking to set up a development environment, take a read over our guide here to learn how to do so then come on back here.

Why is it important that your development sites not be indexed? Google and other search engines are generally not the biggest fans of duplicate content, and so having your development site indexed could result in a drop in rankings. Additionally, it’s not great for the user experience if they end up on a dev site where there may be bugs, such as if they can’t actually contact you or place an order.

For those reasons, it’s considered best practice to ensure your development environments are not found. There’s four ways to accomplish this, the first two of which are available only by WordPress. It’s best to only choose one of the following options, though multiple can be used if you wish.

Web App: Maintenance Plugins

The most secure method of accomplishing this is to use an under construction plugin that blocks access to the site to any user not logged in to your website admin. This prevents access to all pages of the site (posts, and custom post types included) unless the visitor is logged in. By default most of these plugins require the admin role, though that can be changed in the settings.

We’ve used a number of different plugins for this in the past, and honestly just about any one of them will do the trick. The two most recent ones we use for WordPress are:

Other web apps will probably either have a similar function built in, or plugins just like these ones available to use. Check their documentation and plugin/extension repository for more info!

WordPress: Search Engine Visibility

While this option will not block access to the site if someone were to guess the URL, it will successfully prevent indexing by search engines, effectively hiding the dev site. It does this by setting the robots meta value to noindex as well as specifying a blanket disallow in the dynamically generated robots.txt file

Manual: robots.txt

If your web app doesn’t provide the option to disallow access to the admin or block search engines from within its settings or a plugin, you can do it manually! Here’s how:

  1. Access the files for your website via FTP or the Plesk File Manager and navigate to your web root folder (the linked guide will show you how to do this)
  2. Create a file called ‘robots.txt’ (without the quotes) and place the following within it:
User-agent: *
Disallow: /

More info on how robots.txt works here.

Plesk: HTTP Authentication

You could also opt to password protect your site from anyone trying to access it. Here’s how to password protect any folder with Plesk.

By employing one of these options, you’ll ensure that your development environment does not get picked up as a live site by search engines, which is beneficial for site security, usability, and SEO.

Important: When you make your development site live, do not forget to undo whichever of the above options you’ve chosen to hide your site! If you do forget to do this, your site will not be found by Google and if it was previously on Google, it will eventually become de-indexed.

About Jordan Schelew

Jordan has been working with computers, security, and network systems since the 90s and is a managing partner at Websavers Inc. As a founder of the company, he's been in the web tech space for over 15 years.

Leave a Comment