# EnrutaDiscovery crawler

The crawler behind Enruta's AI shopping checks: what it reads, how it identifies itself, and how to verify or block it.

Status: Live · Updated: 2026-10-02
Source: https://enruta.ai/docs/crawler

EnrutaDiscovery reads the public pages of online stores to check how ready they are for AI shopping assistants: structured data, page size, robots.txt and crawler access. Enruta, Inc. runs it.

## How it identifies itself

Every request carries this user agent:

```text
Mozilla/5.0 (compatible; EnrutaDiscovery/0.1; +https://enruta.ai/bot)
```

Every request is also signed with Web Bot Auth, HTTP message signatures ([RFC 9421](https://www.rfc-editor.org/rfc/rfc9421)) made with an Ed25519 key. The `Signature-Input` and `Signature` headers cover the request's authority and the `Signature-Agent` header, whose value is `"https://enruta.ai"`, and carry the tag `web-bot-auth`.

The public keys are in the key directory at [https://enruta.ai/.well-known/http-message-signatures-directory](https://enruta.ai/.well-known/http-message-signatures-directory). The directory response is signed with the same keys, so a verifier can check both.

## What it does

- Reads only public pages: the home page, product pages, robots.txt, sitemaps and the feeds a store publishes.
- Obeys robots.txt ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)) for the token `EnrutaDiscovery` and for `*`, including a Crawl-delay of up to 10 seconds.
- Sends at most one request a second to a store, and at most 40 requests to a store in one check.
- Never submits forms, adds to a cart, signs in or starts a checkout.
- Expected volume: under 10,000 requests a week across all stores.

With a store owner's written consent, a check can also request one product page under the user agent of each of the main AI crawlers, to show the owner which of them the store turns away. Those requests are signed as EnrutaDiscovery too.

## How to block it

Add this to robots.txt. The crawler stops reading the store, and the check reports that it could not read the pages.

```text
User-agent: EnrutaDiscovery
Disallow: /
```

## Contact

Questions, or a request to stop: hello@enruta.ai.