Solved! “Your home page cannot be indexed” Yoast SEO Plugin problem caused for Wordfence Blocking Fake Google Bots

by
This website was acting VERY penalized

We recently had a website losing rankings day after day after day. Which made me think we had a penalty. I had the year-over-year graphs to prove it was so. However it didn’t add up. The short story is Wordfence had been blocking Google bots at random for months.

Wordfence blocking normal Google crawlers

FULL List of steps to correct / prevent this:

(contnet unavailable0

Why is it hard to diagnose?

The reason the bots are block intermittently is that be default Wordfence blocks an IP for a set period of time. For example 2 hours. This can cause intermittent problems, not persistent. So for example if you were to go to WMT and Fetch your site and it fetches fine you might thing “Oh I’m good, it fetched”. Yes, for now it did, but tomorrow or next week it might not.

What’s another scenario that could be causing you problems?

If you’ve been hacked in the past and your website had gobs of urls injected into it and those urls were indexed by Google, it’s very likely that you might have a Google crawlers coming to your site VERY often to look for those urls. Assuming you’ve properly cleaned up your site, those URLs no longer exist. This means Google crawlers will be hitting 404s on those pages. Yet another Wordfence setting could be causing you to block Google.

  • Go to Wordfence > Options and find Firewall settings:
  • Locate this setting: “If a crawler’s pages not found (404s) exceed:”
    Set it to at least “16 per second (throttle it)”
Change Wordfence Firewall settings for Crawler 404s to throttled

This will keep Wordfence from blocking Google and will only throttle it instead.

Do you have or will you soon have this problem?

Are you:

  • Using Wordfence
  • Using Wordfence Firewall
  • Have “Block fake google crawlers immediately” checked
  • Site or pages on your site not indexed, deindexed or sporadic indexing
  • New pages not indexing easily or sporadic indexing

NOTE: We also had the Yoast SEO plugin saying “Google can’t index your site” this is related to this Wordfence issue as Onpage.org uses bots that Wordfence blocks as fake Google Bots.

Wrap up

Hopefully this helps you solve or prevent a potentially disastrous problem with your site deindexing, pages deindexing or rankings dropping significantly.

Please share this post on your social media channels, tweet it, FB share it and more!

Leave a Comment