AI agents
What is Bytespider, and is it crawling your website?
Bytespider is the web crawler operated by ByteDance, the company behind TikTok. It reads public web pages, reportedly to support ByteDance’s AI products. It identifies itself as "Bytespider". Site owners who want to control ByteDance’s access can allow or disallow it in robots.txt.
See your site's llms.txt in seconds — free
Generate my llms.txtHow do you control Bytespider?
Add a robots.txt rule for User-agent: Bytespider to disallow paths you don’t want it reading. Reports on Bytespider’s robots.txt compliance have been mixed, so AgentFront also logs its actual visits — giving you evidence of what it fetched, not just what you asked.
FAQ
How can I see if Bytespider visited my site?
AgentFront classifies Bytespider by its user-agent and records each visit — which page, when — so you can confirm its behavior against your robots.txt rules instead of guessing.