Your Website Is Hiding From ChatGPT

A surprising number of robots.txt files block AI crawlers by accident — copy-pasted boilerplate from an old template, or an overcautious “block all bots” rule added for a completely different reason years ago. If that's true for your site, you're not losing ground in AI answers — you were never in the running to begin with.

The crawlers that matter right now

How to actually check your site

1. Open yoursite.com/robots.txt directly in a browser.
2. Search for each crawler name above — an explicit Disallow: / under any of them blocks that platform entirely.
3. Check whether an llms.txt file exists at the root — a growing, informal convention that gives AI systems a map of what's safe and useful to read.
4. If unsure, test with a live URL-fetch tool to confirm the page actually returns content to that agent, not just that robots.txt allows it.

What “allowed” actually controls

Being blocked doesn't lower your ranking anywhere — it simply makes your business invisible to anything that model answers. It's a binary exclusion, not a penalty you can partially recover from with better content.

Read this with healthy skepticism

• Being crawlable doesn't guarantee being cited — it's a necessary condition, not a sufficient one.
• Allowing every crawler isn't automatically right for every business; if you have real reasons to block AI training (not just inherited defaults), that's a legitimate choice to make deliberately, not by accident.