NeOn crawler

NeOn is the documentation crawler of Zeta-Reticuli.ai Inc., a Delaware corporation. It reads the official, public documentation of the technologies and public bodies our AI agents work with, so that the agents answer from the current source instead of from memory.

How to recognise it

Every request carries this User-Agent:

NeOn/1.0 (Zeta-Reticuli.ai NeOn documentation crawler; +https://neon.zeta-reticuli.ai/bot)

Its robots.txt token is neon. It runs on Cloudflare Containers, so requests come from Cloudflare's network. It never disguises its origin and never uses residential or rotating proxies.

What it fetches

• Public pages, PDFs and XML of documentation sites that are rendered on the server and have no public source repository we could read instead.
• Only the URLs a site lists in the sitemap named in its robots.txt, and links found on those pages within the same host.
• Plain HTTP GET requests only. It does not run JavaScript, submit forms, sign in, or search.
• Only successful (HTTP 200) responses are kept.

How it behaves

• It reads robots.txt first and obeys it exactly (RFC 9309), including every Disallow rule and any Crawl-delay.
• One connection per host at a time, and at least one second between requests to the same host.
• A pilot of about 1,000 URLs comes first; a full crawl of a site is planned and costed before it starts.

How to stop it

Add this to your robots.txt and NeOn will stop fetching your site:

User-agent: neon
Disallow: /

Contact

Questions, complaints or a request to slow down or stop: hello@zeta-reticuli.ai. Please include the host name and, if you can, the time of the requests.