VellumUp
PricingBlogIntegrationsComponents
Sign inRegister
Back to tools

Robots.txt Tester

Fetch and read any site's robots.txt, check whether a specific URL is blocked for a given crawler, and catch issues like a missing file or an accidental site-wide block.

About this tool

This tool fetches the robots.txt file at the root of the address you enter, and reads its user-agent groups, Allow and Disallow rules, crawl-delay values, and listed sitemaps. If you also enter a path, it checks that exact path against the file using the same rule-matching a real crawler uses: the most specific matching user-agent group applies, and within that group the longest matching pattern wins, with Allow beating Disallow on an exact tie. This mirrors the documented behavior of major search engine crawlers, not a simplified approximation. Only the robots.txt file itself is read, so it does not check whether the rest of the site actually matches what the file describes.

Two real, easy to miss details worth knowing when writing or debugging a robots.txt file: paths in Allow and Disallow rules are case sensitive, so a rule written as Disallow: /Private/ does nothing to block a real folder at /private/, they are treated as two different paths entirely. And Google specifically only processes the first 500 KiB of a robots.txt file, anything past that limit is silently ignored with no warning shown anywhere, so an unusually large file (generated automatically by some platforms with very long, repetitive rule lists) can have its later rules quietly dropped.

Explore more free tools

Meta Tags Checker

Check a page's title, description, canonical URL, Open Graph, and Twitter Card tags for issues.

Hreflang Tag Checker

Check a page's hreflang tags for missing self-reference, duplicate codes, and non-reciprocal alternates.

Redirect Checker

Follow a URL's full redirect chain hop by hop, with status codes and warnings for loops and insecure hops.

Frequently asked questions

It tells crawlers which parts of a site they are allowed or not allowed to request. It is a voluntary convention that well-behaved crawlers, including every major search engine, follow, but it is not a security or access control mechanism: a disallowed page can still be reached directly by anyone with the URL, and a determined bad actor can simply ignore the file.
Disallow only stops a crawler from requesting the page's content. If other pages link to a disallowed URL, a search engine can still index the URL itself, sometimes showing it in results with no description, because it was never allowed to read the page to generate one. Keeping a page out of search results reliably usually requires a noindex tag instead of, or alongside, a Disallow rule. Disallow and noindex answer two different questions: Disallow controls whether a page gets crawled at all, noindex controls whether a page that has been crawled is allowed to appear in search results, and using only one of them when you actually need the other is one of the most common real mistakes people make with robots.txt.
When more than one Allow or Disallow rule in the same user-agent group matches a path, the rule with the longest, most specific pattern wins, regardless of which order the rules appear in the file. This is why a broad Disallow: / can be paired with a more specific Allow: /public/ to reopen just that one section, and why rule order in the file does not itself determine the outcome.
A robots.txt file can define different rules for different crawlers by name, such as Googlebot or Bingbot, alongside a general User-agent: * group that applies to everyone else. A path can be allowed for one named crawler and blocked for all others at the same time, so the same URL can have a different answer depending on which crawler is asking.
Not necessarily. When no robots.txt file exists, the standard, documented behavior is that crawlers treat the entire site as allowed. It is worth having one anyway mainly to be explicit about that, and to list sitemap locations, rather than because a missing file itself causes harm.
No. The address is used only to fetch robots.txt for that one check, and nothing is saved. Only public web addresses can be checked: requests to private or internal network addresses are refused.
Yes. Field names like User-agent, Allow, and Disallow are not case sensitive, but the path patterns themselves are: Disallow: /Private/ and a real folder at /private/ are treated as two entirely different paths, so a rule written with the wrong casing simply will not match. This is a common, easy to miss cause of a rule that looks correct but does not actually block what it was meant to.
Yes. Google specifically states it only processes the first 500 KiB of a robots.txt file, and anything beyond that limit is silently ignored with no error or warning shown anywhere. This rarely matters for a hand-written file, but can be a real issue for a very large, automatically generated rule list, where rules near the end of the file may quietly have no effect at all.

Want SEO and GEO content
in your brand voice, automatically?

Try it risk-free — a content plan, keyword research, and up to 5 articles on us.

Say hello

support@vellumup.com

Follow us

Product

PricingFree ToolsShopify App StoreWix App MarketWordPress Pluginnpx vellumup-init

Company

BlogIntegrationsComponentsAffiliates

Integrations

DocumentationWordPressShopifyWebflowWixWebhooksNext.js

Legal

Terms of ServicePrivacy PolicyAccessibility
VellumUp

© 2026 VellumUp. All rights reserved.