Enruta

Docs Reference

EnrutaDiscovery crawler

LiveUpdated

View as Markdown

EnrutaDiscovery reads the public pages of online stores to check how ready they are for AI shopping assistants: structured data, page size, robots.txt and crawler access. Enruta, Inc. runs it.

How it identifies itself

Every request carries this user agent:

Mozilla/5.0 (compatible; EnrutaDiscovery/0.1; +https://enruta.ai/bot)

Every request is also signed with Web Bot Auth, HTTP message signatures (RFC 9421) made with an Ed25519 key. The Signature-Input and Signature headers cover the request’s authority and the Signature-Agent header, whose value is "https://enruta.ai", and carry the tag web-bot-auth.

The public keys are in the key directory at https://enruta.ai/.well-known/http-message-signatures-directory. The directory response is signed with the same keys, so a verifier can check both.

What it does

  • Reads only public pages: the home page, product pages, robots.txt, sitemaps and the feeds a store publishes.
  • Obeys robots.txt (RFC 9309) for the token EnrutaDiscovery and for *, including a Crawl-delay of up to 10 seconds.
  • Sends at most one request a second to a store, and at most 40 requests to a store in one check.
  • Never submits forms, adds to a cart, signs in or starts a checkout.
  • Expected volume: under 10,000 requests a week across all stores.

With a store owner’s written consent, a check can also request one product page under the user agent of each of the main AI crawlers, to show the owner which of them the store turns away. Those requests are signed as EnrutaDiscovery too.

How to block it

Add this to robots.txt. The crawler stops reading the store, and the check reports that it could not read the pages.

User-agent: EnrutaDiscovery
Disallow: /

Contact

Questions, or a request to stop: hello@enruta.ai.