Robots.txt Generator

robots.txt is a plain-text file served from the root of a domain (always at /robots.txt) that tells crawlers which paths they may fetch. It is organized into groups: each User-agent line names a bot, and the Allow/Disallow lines below it list path prefixes. A crawler obeys the most specific group that matches its name and falls back to the * group otherwise. The standard was proposed in 1994 and formalized as RFC 9309 in 2022.

Crucially, robots.txt is advisory, not enforcement. Well-behaved crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot) respect it, but nothing technically prevents a bot from ignoring it. It is also not an indexing control: a page blocked in robots.txt can still appear in Google results if other sites link to it — Google just cannot read its content. To keep a page out of search results, use a noindex meta tag or authentication instead, and note that noindex only works when the page is not blocked from crawling.

Blocking AI crawlers is a genuine tradeoff. Disallowing bots like GPTBot (OpenAI), CCBot (Common Crawl), anthropic-ai, and Google-Extended (Gemini training) opts your content out of model training datasets — but it can also reduce your visibility in AI assistants and AI-powered search answers that increasingly drive referral traffic. Blocking Google-Extended does not affect Googlebot or your normal search rankings; the two are deliberately separate. There is no universally right answer: publishers protecting paid content often block, while sites competing for discovery often allow.