AI agents
What is GPTBot, and why is it visiting your website?
GPTBot is the web crawler operated by OpenAI. It visits public websites and reads their text so OpenAI can train and ground its models, including ChatGPT. GPTBot identifies itself with the user-agent token "GPTBot" and respects robots.txt directives, so you can allow or block it per path.
See your site's llms.txt in seconds — free
Generate my llms.txtWhat does GPTBot do with your pages?
GPTBot fetches your public pages and extracts their text. That text can inform how ChatGPT answers questions about your business — what you sell, your policies, your prices. If your pages are hard to parse, GPTBot may capture an incomplete or wrong picture, and answers about you suffer.
Should you allow or block GPTBot?
Blocking GPTBot in robots.txt removes your site from OpenAI’s training and grounding. Most businesses that want to be findable in the ChatGPT era allow it. If you allow it, an llms.txt file gives GPTBot a clean, accurate summary to read first — so it understands you correctly.
FAQ
Does GPTBot respect robots.txt?
Yes. GPTBot checks robots.txt and honors Disallow rules for its user-agent. To block it, add a rule for User-agent: GPTBot. To welcome it, allow access and serve an llms.txt so it reads an accurate summary of your site.