.Google's John Mueller answered a question concerning why Google marks webpages that are actually disallowed from crawling through robots.txt and also why the it is actually risk-free to overlook the associated Look Console documents about those creeps.Robot Visitor Traffic To Concern Specification URLs.The individual asking the inquiry documented that bots were actually developing links to non-existent inquiry guideline Links (? q= xyz) to pages along with noindex meta tags that are actually also blocked in robots.txt. What triggered the question is that Google is actually crawling the web links to those pages, obtaining blocked out through robots.txt (without noticing a noindex robots meta tag) then receiving shown up in Google Search Console as "Indexed, though blocked by robots.txt.".The individual inquired the adhering to concern:." Yet below's the major concern: why would Google index pages when they can not even find the web content? What's the perk because?".Google.com's John Mueller validated that if they can't creep the webpage they can't observe the noindex meta tag. He also produces an intriguing acknowledgment of the website: search driver, advising to overlook the results since the "typical" customers will not view those outcomes.He wrote:." Yes, you're proper: if our team can't creep the webpage, we can't see the noindex. That claimed, if our experts can not creep the web pages, after that there's certainly not a whole lot for our company to mark. Thus while you may find some of those web pages with a targeted website:- concern, the common customer won't find all of them, so I wouldn't fuss over it. Noindex is actually likewise great (without robots.txt disallow), it simply indicates the URLs will certainly end up being actually crept (and find yourself in the Browse Console record for crawled/not catalogued-- neither of these standings result in issues to the rest of the web site). The integral part is actually that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the limitations in using the Web site: search accelerated search operator for analysis explanations. Some of those reasons is actually because it's not hooked up to the regular search index, it is actually a separate trait entirely.Google's John Mueller discussed the site search operator in 2021:." The brief answer is actually that a web site: inquiry is certainly not indicated to be complete, nor used for diagnostics reasons.A website concern is actually a particular kind of hunt that limits the outcomes to a specific site. It is actually generally just the word web site, a digestive tract, and then the internet site's domain.This query confines the results to a details web site. It is actually not suggested to be an extensive compilation of all the webpages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is actually great for these type of scenarios where a crawler is actually connecting to non-existent web pages that are actually obtaining uncovered by Googlebot.3. URLs along with the noindex tag will definitely create a "crawled/not listed" item in Explore Console and also those will not have a damaging result on the rest of the web site.Go through the concern and also answer on LinkedIn:.Why would certainly Google index pages when they can't even observe the content?Included Graphic by Shutterstock/Krakenimages. com.