Build a valid robots.txt with presets, custom rules, AI-crawler blocking and your sitemap line.
robots.txt is a plain text file at the root of a domain that tells automated crawlers which paths they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309, and its structure is simple: groups that start with one or more User-agent lines followed by Disallow and Allow rules, plus standalone Sitemap lines that apply to the whole file. Paths are matched by prefix, * stands for any sequence of characters and $ anchors the end of a URL, so /*.pdf$ blocks every PDF. An empty "Disallow:" is the canonical way of saying "crawl anything".
This generator writes that syntax for you and cleans up the input on the way: every path gets a leading slash if you forgot it, lines starting with # are kept as comments, an empty rule list becomes the canonical allow-everything group, and the crawl delay is only written when it is a positive number. The AI crawler option appends a separate group listing GPTBot, ChatGPT-User, OAI-SearchBot, CCBot, anthropic-ai, ClaudeBot, Claude-Web, Google-Extended, PerplexityBot, Applebot-Extended, Bytespider, Amazonbot, Meta-ExternalAgent and cohere-ai under a single Disallow: /, which is how you opt out of AI training and answer-engine crawling without touching Googlebot. The Sitemap line is validated as an absolute URL, because relative sitemap paths are ignored by search engines.
Two honest limits are worth repeating. First, robots.txt is a request, not a wall: well-behaved crawlers obey it, scrapers ignore it, and anything you list there is publicly readable — never use it to hide sensitive paths. Second, disallowing a URL stops crawling, not indexing; a blocked page that other sites link to can still appear in results without a description, so use a noindex meta tag on the page itself when you want it out of the index. The file is generated entirely in your browser and downloaded locally — nothing about your site is uploaded here.
User-agent:
In the root of the domain, reachable at https://yoursite.com/robots.txt. Crawlers only look there — a file in a subfolder is ignored, and each subdomain needs its own.
No. It stops the page from being crawled, but a blocked URL can still be indexed from external links. To keep a page out of results, allow crawling and add a noindex meta tag to the page.
Some do and some do not. OpenAI, Google and Anthropic document their user-agents and honour the file; many scrapers ignore it entirely. Blocking is a clear signal, not enforcement.
No — Googlebot ignores the directive and adjusts its own rate, though Bing and Yandex do read it. Use Search Console if you need to slow Google down.
No. Blocking CSS or JS files prevents search engines from rendering your pages as users see them, which can hurt rankings, and the sitemap must stay reachable.
No. The file is assembled by JavaScript on your device and saved straight to your downloads folder.
Vai.la turns any URL into a short link with click statistics, QR Code and your own biolink.
Vai.la is not responsible for how the tools are used or for decisions made based on their results.