Robots.txt Validator

Robots.txt Validator takes a site address, requests /robots.txt from that origin, and then answers the question you actually came with: may this crawler fetch this path? The Crawler box chooses the group — the longest user-agent token that prefixes the name you typed, with the wildcard group as the fallback — and the URL or path box supplies the target. The verdict names the winning line by number, lists every Allow and Disallow in that group with a yes-or-no beside each, and reports the group's Crawl-delay when one is set. Matching follows Google's specification: * and $ are honoured, the longest matching pattern wins, Allow beats Disallow on a tie, and a Disallow with no value after the colon allows everything.

Because it is a validator and not only a fetcher, it also reads the file for the mistakes that make a rule silently do nothing: a misspelt directive, an Allow or Disallow written above the first User-agent line so that no crawler claims it, a pattern beginning with neither / nor *, a relative Sitemap address, a $ placed anywhere but the end, an unencoded space in a path, a Crawl-delay that is not a number, and Noindex:, which Google has never obeyed. Those come back as a list of errors and warnings beneath the verdict, together with the number of groups in the file and every Sitemap line declared anywhere in it.

How the fetch is made decides which file you get. Only the scheme, host and port of what you type are used — the path is thrown away and the request goes to the root — so https://example.com and http://www.example.com are two separate files that can quietly drift apart. The request identifies itself with OmniKitly's own user agent, follows up to five redirects with every hop re-checked before it is opened, and stops reading at 512 KB. The work happens on the processing service, so the address, the crawler name and the path you are testing all travel there over an encrypted connection, and the job record and the JSON report it produced are deleted together once the short retention window passes.

Three limits are worth knowing before you lean on an answer. The verdict is computed from the first 20,000 characters the fetch keeps, so a robots.txt longer than that is matched against a truncated copy and the decision can come out wrong on a rule below the cut — the size shown in the report is the size of that excerpt, not of the served file. Any response that is not a plain-text 200 skips the verdict altogether and shows the raw JSON report instead, which is honest but is not an allow-or-block answer. And the input box is labelled as though it also accepted pasted rules, with a Load sample button that fills it with some; that path does not work, because whatever sits in the box is sent as an address to fetch and pasted rules come back as an invalid URL.

Secure server processing. This operation requires temporary server processing.