Babel42Bot
You saw us in your logs
Babel42Bot is Babel42's crawler. It fetched your pages because somebody asked us to audit that site. Our audit tells a site owner whether AI assistants can read their pages, so the first thing it has to do is try to read them, exactly the way one of those assistants would.
We do not crawl continuously, we do not build an index, and nothing we read is used to train a model. We measure the pages, produce a report for the person who asked, and keep the numbers rather than your content.
The details you came for
- User agent
- Babel42Bot/1.0 (+https://babel42.io/bot)
- robots.txt name
- babel42bot
- Requests at once
- Three per site, at most
- Purpose
- One-off audits requested by a person, not continuous indexing
- Pages per audit
- Up to 10 for the free audit, up to 1,000 for the deep audit
- JavaScript
- Never executed. We read the HTML your server sends
- Content use
- Measured, scored and discarded. Never used to train anything
How to stop us
Add this to your robots.txt. We obey it, and a blocked site is reported to whoever asked for the audit as exactly that: a site that asked us not to look.
User-agent: babel42bot Disallow: /
To keep us out of one part of the site only, name that path instead of the slash. If you would rather we left entirely and stayed gone, write to support@babel42.io and we will block the domain at our end as well.
If this is your site
You can run the audit yourself, free, and see everything we saw. It checks your robots.txt against 14 AI crawlers and reports which ones your rules turn away, what your pages look like without JavaScript, and whether each section of a page can be quoted on its own.
Run the free auditFurther reading
- How the named crawlers differ from each other and which ones decide whether you can be quoted at all.
- What a crawler is actually refused by usually a firewall rule nobody remembers switching on, not robots.txt.
- What happens to a fetched page next retrieval-augmented generation, explained without the maths.