This guide will show you how to configure your development site to ensure it is not indexed by Google. If you’re looking to set up a development environment, take a read over our guide here to learn how to do so then come on back here.
Why is it important that your development sites not be indexed? Google and other search engines are generally not the biggest fans of duplicate content, and so having your development site indexed could result in a drop in rankings. Additionally, it’s not great for the user experience if they end up on a dev site where there may be bugs, such as if they can’t actually contact you or place an order.
For those reasons, it’s considered best practice to ensure your development environments are not found. There’s four ways to accomplish this, the first two of which are available only by WordPress. It’s best to only choose one of the following options, though multiple can be used if you wish.
Web App: Maintenance Plugins
The most secure method of accomplishing this is to use an under construction plugin that blocks access to the site to any user not logged in to your website admin. This prevents access to all pages of the site (posts, and custom post types included) unless the visitor is logged in. By default most of these plugins require the admin role, though that can be changed in the settings.
We’ve used a number of different plugins for this in the past, and honestly just about any one of them will do the trick. The two most recent ones we use for WordPress are:
- Under Construction by WebFactory Ltd. There’s a free and paid version, though the free version should work well for most people.
- Beaver Lodge Maintenance. Since we love BeaverBuilder around here, this plugin allows us to build the coming soon page using BeaverBuilder!
Other web apps will probably either have a similar function built in, or plugins just like these ones available to use. Check their documentation and plugin/extension repository for more info!
WordPress: Search Engine Visibility
While this option will not block access to the site if someone were to guess the URL, it will successfully prevent indexing by search engines, effectively hiding the dev site. It does this by setting the robots meta value to noindex as well as specifying a blanket disallow in the dynamically generated robots.txt file
If your web app doesn’t provide the option to disallow access to the admin or block search engines from within its settings or a plugin, you can do it manually! Here’s how:
- Access the files for your website via FTP or the Plesk File Manager and navigate to your web root folder (the linked guide will show you how to do this)
- Create a file called ‘robots.txt’ (without the quotes) and place the following within it:
User-agent: * Disallow: /
More info on how robots.txt works here.
Plesk: HTTP Authentication
You could also opt to password protect your site from anyone trying to access it. Here’s how to password protect any folder with Plesk.
By employing one of these options, you’ll ensure that your development environment does not get picked up as a live site by search engines, which is beneficial for site security, usability, and SEO.