Why Is My Site Not Indexed? Status Codes, robots, UA

#First work out which "not found" you have

There are at least three situations behind "we are not in search results": never indexed, indexed but not ranking, and indexable but failing to crawl. Discussing them together only produces unhelpful conclusions about content quality.

The quickest separation is the site: operator plus one or two keywords.

#Line one: what the status code says

The status code decides how a crawler treats the page, and it outweighs content quality by a wide margin. Excellent content served with a 503 does not get indexed.

The codes worth keeping straight:

• 200: fine - but a 200 that returns an empty shell with the content in JavaScript gives a crawler nothing
• 301: permanent redirect. Right for a site restructure, hard to undo once wrong
• 404: not found. Use it when you are unsure whether the page will return
• 410: gone, permanently. Removes a URL from the index faster than 404
• 5xx: server failure. Sustained 5xx makes crawlers reduce their rate, and recovery takes time after the fix

One confusion worth flagging in the HTTP status code reference: 301 and 302 may flip a POST to GET, so API redirects need 307 or 308. And 502 with 504 usually comes from a reverse proxy - look at the application behind it first, not at the proxy.

#Line two: is robots or noindex in the way?

robots.txt is dangerous precisely because it only affects crawlers - your own visit looks perfectly normal, so a mistake produces no warning at all, just a flat index count. The robots.txt checker both parses the rule groups and tests any URL against a chosen crawler identity.

The matching rules are finer than most people assume:

• Longest match wins: with Disallow: /admin/ and Allow: /admin/public/ together, the longer Allow takes effect
• Equal length: Allow wins, which is Google's convention
• Wildcards and end anchors: * matches any run of characters and a trailing $ anchors the end, so /*.pdf$ blocks only URLs ending in .pdf
• Empty value means no restriction: Disallow: with nothing after it allows everything - it does not block everything
• No rule matched means allowed, which is the default

Check three things in order: does User-agent: * cover the crawler you care about, is there a Disallow: /, and is the Sitemap line an absolute URL. Those three cover most self-inflicted blocks.

While you are here, check the meta tags. The robots value produced by the meta tag generator is fixed at index, follow, so pages you want excluded need manual editing. It also measures title and description by pixel width, which is why Chinese titles truncate around 30 characters and English around 60.

#Line three: is it really a crawler?

A User-Agent is whatever the client sends, so "the UA says Googlebot" is not evidence of anything. The only real verification is checking whether the source IP belongs to that search engine's published ranges.

When reading results from the UA parser, accept the premise first: it suits analytics and adaptation, never authorisation or risk control. Several results that look like bugs are simply correct:

• Windows 11 shows as 10/11: the UA still says Windows NT 10.0, because Microsoft froze the version token
• iPad shows as macOS desktop: since iPadOS 13, iPads request the desktop site by default
• Chrome's UA also contains Safari: Chromium keeps Safari/537.36 for compatibility, so matching must check Edg/, OPR/ and Chrome/ before Safari/

Crawler detection looks for bot, spider, crawler and slurp, and recognises the major engines. Just remember: anything claiming to be Googlebot can be any machine at all.

#Is the domain itself healthy?

An expiring domain, an odd domain status, or a very new registration stalls indexing in ways the website itself cannot reveal. The Whois lookup shows four things: registrar, creation date, expiry date, and domain status.

Two of them matter most:

• Expiry: the tool computes days remaining. Letting it lapse takes down the site, business email and every third-party verification at once
• Status: pendingDelete or redemptionPeriod means the deletion or redemption process has started and the window is short

One factor people forget: a new domain has a trust period, and search engines will not crawl it heavily right away. The useful work during that time is clean structure, solid internal linking and a submitted sitemap, not constant content rewrites.

#Which tool to use

Work through the lines in the order above: status code, then robots and noindex, then UA, then the domain. The first three explain most cases of zero indexing, while domain problems tend to show up as an entire site vanishing at once.

This complements what to check before launching a site - that page covers prevention, this one covers diagnosis after launch.

#Frequently asked questions

#I submitted a sitemap and still nothing is indexed. Why?

A sitemap announces URLs; it does not demand indexing. When submission changes nothing, one of the three lines is still failing: wrong status codes, a robots or noindex block, or content that duplicates other pages heavily. The sitemap itself must also be reachable, valid XML, and consistent with the canonical URLs.

#Should I use 404 or 410?

If the page might come back, use 404 - it means "not found right now" and leaves room. If the content is permanently gone and will never return, use 410, which removes the URL from the index faster. Never return 200 with "this page does not exist", because that gets an empty page indexed as real content.

#It is indexed but does not rank. What now?

Indexing and ranking are separate. Being indexed only means the engine knows the URL exists; ranking depends on competition, relevance to the query and overall site quality. At this point the work is titles, body copy and internal links, not more technical digging.

#How long does recovery take after fixing robots.txt?

Once the file is corrected, the next crawl picks up the new rules, but re-fetching the pages that were blocked takes time and depends on your site's crawl priority. To speed it up, resubmit the sitemap and request a fresh crawl of key pages from the search console.

#How can I confirm a visitor really is Googlebot?

Compare the source IP in your access logs against the search engine's published range list, or run a reverse DNS lookup to check the hostname. The UA string alone cannot tell you anything, since anyone can send a request with Googlebot in the header - a very common trick in scraping traffic.