How to Stop Search Engines From Crawling Your WordPress Site
When someone builds a website, usually they create it with the intent of drawing in new users to help create a following. I mean, that’s why we build websites right? However, certain situations may arise when we actually want to keep our websites private or hidden and stop search engines from crawling your WordPress site.
Maybe you’re wanting to build a website for just a small group of friends and don’t want it getting indexed in Google. More often than not, users who build their first website with WordPress need some time to get the hang of things and learn the ropes.
Instead of building their site on a local development environment like their localhost, they often work on a live site instead. Having a half completed website go live for the entire world to see can be undesirable for new users. Whatever your reason for wanting to keep your site underwraps, it doesn’t matter. The point is you do, and we’re going to show you how.
Search Engine Bots
Usually when we hear the word bot, our mind conjures up images of large machines that go around in \ carrying out complex tasks. In the realm of search engines and the world wide web, they mean something a little different. Bots, spiders, web bots, or web crawlers are all programs that scour the internet and indexes web pages.
Major search engines like Google, Bing, and Yahoo all have search engine bots that they send out to crawl the web and index the pages of every website. It’s these bots that find your website and display them as search engine results.
How to Block Bots
When it comes to blocking search engine bots from being able to index your WordPress site, you have a few options. For the purpose of this article, we’re going to the most popular method for accomplishing this.
The quickest and easiest method to stop search engines from indexing your site is using the built in option, which is using the “search engine visibility” box.
Simply login to the administrative backend of your website. Once there, use the navigation bar on the left and make your way to the settings menu item. Hover over settings menu item and select the sub menu item labeled “reading” as shown in the image above.
Once you’re there, you’ll see an option towards the bottom of the page called “Search Engine Visibility” with a box. You can use tick this box on or off to enable or disable search engines from indexing your website. It’s just that easy!
Doing this causes WordPress to specify in your robots.txt file for search bots to ignore your site and not crawl any pages. It’s a really simple and easy way to tell bots not to index your site. Using this method generally works really well, however as it says, it’s up to search engines to honor this request or not.
There are tons of other methods that this can be accomplished, but this is by far the quickest and easiest way. So if in the future you need to hide your site from search engines, never fear, WordPress makes this easy for you.
