{"id":10241,"date":"2019-07-04T12:31:59","date_gmt":"2019-07-04T12:31:59","guid":{"rendered":"\/blog\/?p=10241"},"modified":"2022-11-24T09:54:38","modified_gmt":"2022-11-24T09:54:38","slug":"google-has-canceled-support-for-robots-txt-noindex","status":"publish","type":"post","link":"https:\/\/www.ezrankings.com\/blog\/google-has-canceled-support-for-robots-txt-noindex\/","title":{"rendered":"Google Has Canceled Support For Robots.Txt Noindex"},"content":{"rendered":"

\"Google<\/p>\n

Google has announced that GoogleBot will no longer follow a Robots.txt directive related to indexing. Publishers relying on the robots.txt noindex directive have until September 1, 2019 to remove it and begin using an alternative suggested by Google.<\/p>\n

For anyone new to SEO<\/a>, these might sound like a bunch of foreign terms. However, it is of importance to note. The Googlebot or Google\u2019s crawler is essentially a bot that goes through or crawls through pages and adds them to the index, which is Google\u2019s database. Based on this indexation, Google can rank different websites.<\/p>\n

The Robot.txt directive is essentially a command given by a website that tells the Googlebot which pages on your site to go and which pages on your site to avoid. Normally it was used to optimize a website\u2019s crawl-ability, or the ability for a crawlbot not to bump into any problems while going through a website.<\/p>\n

Why Google is Cancelling it<\/b><\/h2>\n

Google does not consider the noindex Robot.txt an official directive. They used to be in support of it, however, it did not work in 8% of the cases. It was not a fool-proof directive. They have officially unsupported noindex, crawl delay and no-follow directives within robot.txt files. They have told websites that have it to remove their directives till September 1st, 2019. However, Google establishes that they have been asking websites not to rely on it for many years<\/u><\/span>.<\/p>\n

In essence, it can be summed up that while open-sourcing parser library, the team at Google analyzed robots.txt rules and its usage. In particular, they focused on rules unsupported by internet draft, such as nofollow, crawl-delay, and noindex. Since the rules were undocumented by Google, naturally, the usage related Googlebot is quite low. Digging further, their team noticed that the usage was contradicted by several other rules in all, but 0.001% of robots.txt files on the internet. The mistakes and errors hurt websites’ presence in SERPs in ways that preferably webmasters didn\u2019t want.<\/p>\n

Google\u2019s official announcement in the morning of July 2nd<\/sup> said that were bidding adieu to and unsupported and undocumented rules in robots.txt. Those relying on these set of rules should learn about the available options posted by Google in their blog post. <\/span><\/p>\n

In their official blog about it, this is what they had written, right before suggesting alternatives:<\/p>\n

\u201cTo maintain a working and healthy ecosystem and preparing for potential open-source releases in the future, we\u2019re retiring the codes that handle unpublished and unsupported rules (such as noindex) on September 1, 2019.\u201d <\/i><\/p>\n

They believe that sites actually hurt themselves more than they help themselves with these noindex directives, as stated by Gary Illyes<\/u><\/span><\/a>. But they have made it clear that they have thought this through<\/u><\/span>, especially since they have been skeptical about these directives for so many years. They do not expect disallowing the noindex robot.txt directive to hurt anyone\u2019s site profoundly<\/u><\/span>.<\/p>\n

Also Read:<\/strong><\/p>\n

Google\u2019s BroadCore Algorithm Update 2019<\/a><\/strong><\/p>\n

How to Rank Your Website for \u201cNear Me\u201d Searches<\/a><\/strong><\/p>\n

Alternatives Suggested by Google<\/b><\/h2>\n

Google did not want sites and companies to be rendered helpless with this change, so they gave a comprehensive list of things one could do otherwise. If you happen to be affected by this change, this is what Google posted with regards to alternatives:<\/p>\n