Solved! “Your home page cannot be indexed” Yoast SEO Plugin problem caused for Wordfence Blocking Fake Google Bots

google-penalty

This website was acting VERY penalized

We recently had a website losing rankings day after day after day. Which made me think we had a penalty. I had the year-over-year graphs to prove it was so. However it didn’t add up. The short story is Wordfence had been blocking Google bots at random for months.

Wordfence blocking normal Google Crawlers

Wordfence blocking normal Google crawlers

FULL List of steps to correct / prevent this:

Why is it hard to diagnose?

The reason the bots are block intermittently is that be default Wordfence blocks an IP for a set period of time. For example 2 hours. This can cause intermittent problems, not persistent. So for example if you were to go to WMT and Fetch your site and it fetches fine you might thing “Oh I’m good, it fetched”. Yes, for now it did, but tomorrow or next week it might not.

What’s another scenario that could be causing you problems?

If you’ve been hacked in the past and your website had gobs of urls injected into it and those urls were indexed by Google, it’s very likely that you might have a Google crawlers coming to your site VERY often to look for those urls. Assuming you’ve properly cleaned up your site, those URLs no longer exist. This means Google crawlers will be hitting 404s on those pages. Yet another Wordfence setting could be causing you to block Google.

  • Go to Wordfence > Options and find Firewall settings:
  • Locate this setting: “If a crawler’s pages not found (404s) exceed:”
    Set it to at least “16 per second (throttle it)”
Change Wordfence Firewall settings for Crawler 404s to throttled

Change Wordfence Firewall settings for Crawler 404s to throttled

This will keep Wordfence from blocking Google and will only throttle it instead.

Do you have or will you soon have this problem?

Are you:

  • Using Wordfence
  • Using Wordfence Firewall
  • Have “Block fake google crawlers immediately” checked
  • Site or pages on your site not indexed, deindexed or sporadic indexing
  • New pages not indexing easily or sporadic indexing

NOTE: We also had the Yoast SEO plugin saying “Google can’t index your site” this is related to this Wordfence issue as Onpage.org uses bots that Wordfence blocks as fake Google Bots.

Wrap up

Hopefully this helps you solve or prevent a potentially disastrous problem with your site deindexing, pages deindexing or rankings dropping significantly.

Please share this post on your social media channels, tweet it, FB share it and more!

The following two tabs change content below.

ssadmin

2 thoughts on “Solved! “Your home page cannot be indexed” Yoast SEO Plugin problem caused for Wordfence Blocking Fake Google Bots

  1. Jesse,

    I’ve been using WordFence as well, and when I saw your email come in on this, I read it to find out what you discovered. I also checked a newer clients site that has WordFence installed, but I did not have checked “Immediately block fake Google crawlers” under Firewall. Also verified GWT’s to make sure everything looked good and it did.

    This client did have some previous “injected” pages when I came on-board to re-design his site and they have been cleaned up. So, Googlebot is seeing some 404’s for those. I set the “404” option to throttle and we’ll see if that helps it.

    I also use Yoast SEO and I did verify indexability with the OnPage button and I received the message “Your homepage can be indexed by search engines.”. So, should be good with that.

    This is why it’s important to keep an eye on your rankings, indexability and overall status of clients and your own sites, especially, when using plugins that could cause issues with indexing and crawlability.

    Thanks for catching this and sharing it with us. Keep up the good work.

  2. Totally Patrick. This stuff can blindside anyone. It’s not always easy to diagnose either. It took us a lot of hunting. Since Google searching the problem didn’t turn up much I’m glad I took time to do this write up and seeing people take action. If it saves one person their rankings its worth it. Cheers and high five for quick action!

Leave a Comment