.Google's John Mueller answered an inquiry about why Google marks webpages that are disallowed coming from creeping through robots.txt and why the it is actually safe to dismiss the related Browse Console files about those crawls.Robot Traffic To Concern Parameter URLs.The person inquiring the concern recorded that robots were creating web links to non-existent query parameter Links (? q= xyz) to web pages along with noindex meta tags that are also blocked in robots.txt. What prompted the question is actually that Google is crawling the web links to those pages, obtaining obstructed through robots.txt (without watching a noindex robots meta tag) at that point acquiring shown up in Google.com Look Console as "Indexed, though shut out through robots.txt.".The person inquired the adhering to inquiry:." But here is actually the huge concern: why will Google mark webpages when they can not even view the content? What's the perk during that?".Google.com's John Mueller affirmed that if they can not creep the page they can't view the noindex meta tag. He also creates an intriguing reference of the internet site: search operator, suggesting to disregard the results since the "common" customers won't find those outcomes.He wrote:." Yes, you are actually right: if we can't crawl the page, our team can't observe the noindex. That mentioned, if our company can't crawl the web pages, after that there's certainly not a lot for our company to mark. So while you may observe several of those webpages with a targeted internet site:- question, the common consumer won't observe them, so I would not bother it. Noindex is also alright (without robots.txt disallow), it simply means the Links will wind up being crept (and find yourself in the Look Console file for crawled/not indexed-- neither of these standings lead to concerns to the rest of the internet site). The important part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the limitations in operation the Website: hunt accelerated hunt operator for analysis reasons. Among those explanations is considering that it is actually certainly not connected to the routine search index, it is actually a different factor entirely.Google.com's John Mueller discussed the website search driver in 2021:." The brief response is actually that a website: question is certainly not indicated to become comprehensive, nor utilized for diagnostics functions.An internet site concern is a details kind of hunt that restricts the results to a specific web site. It's basically just words site, a bowel, and then the website's domain name.This query limits the results to a specific site. It's not meant to become a thorough collection of all the pages coming from that web site.".2. Noindex tag without using a robots.txt is actually great for these kinds of scenarios where a bot is actually connecting to non-existent pages that are actually acquiring found out through Googlebot.3. URLs with the noindex tag will certainly produce a "crawled/not recorded" entry in Search Console which those will not have an unfavorable impact on the remainder of the site.Read the concern and answer on LinkedIn:.Why would Google index webpages when they can not even observe the content?Included Image by Shutterstock/Krakenimages. com.