Tools / robots.txt Generator
robots.txt Generator
Free robots.txt generator. Build user-agent groups with allow and disallow rules, add a sitemap, block AI crawlers and copy the file.
Save this as robots.txt in your site root, so it answers at /robots.txt. It guides crawling, not indexing — use a noindex meta tag to keep a page out of search results.
Frequently asked questions
Does robots.txt keep a page out of Google?
No. It asks crawlers not to fetch a URL, but a blocked page can still be indexed from links pointing at it. To keep a page out of results, let it be crawled and serve a noindex meta tag instead.
Where does the file go?
In the site root, so it answers at https://yourdomain.com/robots.txt. Crawlers do not look anywhere else, and a robots.txt in a subfolder is ignored.
How do I block AI crawlers?
Use the AI crawlers button. It adds groups for GPTBot, CCBot, ClaudeBot, PerplexityBot and Google-Extended, each disallowing the whole site. Google-Extended controls AI training only and does not affect Search.
Is Crawl-delay respected?
Bing and Yandex honor it. Google ignores it — use the crawl rate settings in Search Console instead.
Why do I need a group for *?
Crawlers only follow the most specific group that matches their name. Any bot you did not name explicitly falls back to the * group, and without one it gets no rules at all.
Let's build something you're proud of
I help businesses launch, optimize and maintain their online presence, from code to SEO.
Let's talkRelated tools
Image Resizer and Compressor
Free online image resizer and compressor. Resize photos, convert to WebP or JPEG and shrink file size in your browser, with no upload.
Cron Expression Explainer
Free cron expression explainer. Translate a five-field crontab line into plain English and preview the next scheduled runs in your browser.
htaccess Redirect Generator
Free .htaccess redirect generator for Apache and cPanel. Build 301 and 302 mod_rewrite rules, force HTTPS and pick a canonical www host.
CSS Gradient and Shadow Generator
Free CSS gradient and box-shadow generator. Build linear or radial gradients and layered shadows with a live preview, then copy the CSS.
Color Contrast Checker
Free WCAG color contrast checker. Test text against background colors, see the contrast ratio and AA/AAA results, and get an accessible color suggestion.
Regex Tester
Free online regular expression tester. Try JavaScript regex patterns with flags, see match indexes and groups, and preview replacements in your browser.
CSV ↔ JSON Converter
Free online CSV to JSON and JSON to CSV converter. Parse quoted fields, choose comma, semicolon or tab delimiters and convert locally in your browser.
UUID / ULID Generator
Free online UUID and ULID generator. Create one or many UUIDv4 or ULID values in your browser with optional uppercase output and no signup.
Text Diff
Free online text diff and compare tool. Compare two texts line by line with added, removed and unchanged highlighting. Runs in your browser with no upload.
Unix Timestamp Converter
Free online Unix timestamp converter. Convert epoch seconds or milliseconds to local and UTC dates, or turn a date into a Unix timestamp in your browser.
Case Converter
Free online case converter. Switch text between lowercase, UPPERCASE, Title Case, snake_case, kebab-case, camelCase and PascalCase in your browser.
QR Code Generator
Free online QR code generator. Create a PNG QR code from a URL or text in your browser, then download it. No signup and no upload.
Duplicate Detector
Free online duplicate line finder. Detect repeated lines, remove duplicates, or count frequencies in lists and logs. Runs in your browser with no upload.
Image to Base64 Converter
Free online image to Base64 and Base64 to image converter. Convert PNG, JPEG, WebP, or GIF in your browser with no upload.
Password Generator
Free online password generator. Create strong random passwords with custom length, letters, numbers, symbols and optional ambiguous-character exclusion.
Random Number Generator, Picker & Shuffler
Free online random number generator, list picker and shuffler. Generate a random integer, pick one item from a list, or shuffle lines.
URL Encoder and Decoder
Encode and decode URL strings online. Free percent-encoding tool for query parameters, slugs and full URLs.
Base64 Encoder and Decoder
Encode and decode Base64 strings online. Free Base64 converter that supports Unicode text, API tokens and data URLs.
Meta Tag Generator
Generate SEO meta tags, Open Graph and Twitter Card markup for your pages. Free online meta tag builder with live preview.
JSON Formatter and Validator
Format, minify and validate JSON online. Free JSON beautifier with syntax error messages for debugging APIs and config files.
Pixels to Ems Conversion
Convert pixels to ems and ems to pixels based on base font size. Free px to em calculator for responsive CSS.
Table Generator
Generate HTML table code from rows and columns. Free online table generator with copy-to-clipboard for web projects.
HEX to RGB to HSL Conversion
Convert colors between HEX, RGB and HSL formats instantly. Free online color converter for CSS and design work.
CSS Minifier and Beautifier
Minify CSS to reduce file size or beautify CSS for readable formatting. Free online CSS tool for developers.
HTML Minifier and Beautifier
Minify or beautify HTML online. Remove whitespace and comments to reduce file size, or format code for readability.