This brief tutorial shows students and new users how to quickly stop or prevent search engines and web crawlers from crawling and indexing WordPress websites.
There are many reasons why one might want to stop search engines from crawling or indexing WordPress websites.
One might be to prevent the indexing of website content while the site is still under development. You probably don’t want Google to index your under-construction or maintenance mode page while the site is still under construction.
Another reason might be to prevent a search engine from indexing or listing sensitive information that should only be hosted on your sites.
Whatever the case, it’s easy to configure WordPress sites to instruct search engines not to crawl or index content from a site you don’t want to list online.
When you’re ready to prevent search engines from crawling WordPress, follow the steps below:
Discourage Search Engine from Indexing the Site
WordPress comes with a built-in feature that allows this to be done quickly. It instructs all search engines not to index the site and all pages produced.
Although the instruction to these crawlers is not to index or list your content, it’s up to the crawlers and bots to respect and honor the site’s request. Some bots online will still crawl and list your content regardless of your no-index policy.
To instruct all bots not to index your site, go to do is visit Settings » Reading and check the box next to the Search Engine Visibility option as shown in the image below:
Click the Save Changes button, and you’re done.
When you enable the settings above, WordPress will add the line below to your website’s header file.
<meta name='robots' content='noindex,follow' />
WordPress also modifies your site’s robots.txt file and adds these lines to it:
User-agent: * Disallow: /
These lines ask robots and web crawlers not to index your pages.
Again, it is up to search engines to accept or ignore this request. Most search engines respect will respect your request, and this may not stop some of your pages from getting indexed. It’s just how the web works.
Password Protection
A surefire way to ensure search engines don’t index or crawl your website is to password-protect the entire site at the server level. Password protecting your sites makes it so that anyone who wants to view the content must enter the correct password.
We have shown you how to password-protect directories and content using Apache2 or Nginx HTTP server.
You may also install the Password Protected plugin to do that for you.
To view our posts on protecting Apache2 or Nginx directory, click the links below:
Password protect directories using Apache2
Password protect directories using Nginx
These steps may help protect your sites from search engines, bots, and other crawlers from listing or indexing your content.
Hope this helps
You may also like the post below:
Leave a Reply