Our crawler

What it is

Excuto tests websites for its customers. An audit is one automated test run of one website. The test browser sends a user agent string of this form:

ExcutoBot/1.0 (+https://excuto.app/crawler)

The product token is ExcutoBot. The browser sends this string on every page request. The robots.txt request sends it too.

What it does

The bot loads your pages in Chromium, like a visitor. It runs the scripts of the page. For each page it does these things:

• It resizes the window to several screen sizes, from 280 to 2560 pixels wide.
• It scrolls the page to the bottom and past the end.
• It turns on dark mode, reduced motion and forced colors, and it widens the text spacing.
• It reads the layout and the accessibility of the page.
• It scans the scripts and the JSON answers of your site for exposed secret keys.

The bot follows links on the same origin only, from the home page, up to 3 links deep. A redirect to another site ends the visit. Like any browser, the bot also loads the files that a page uses from other hosts, such as scripts and images.

When the bot finds a possible problem, it replays the steps. It records a screenshot and a video of a problem that repeats. During an audit, it also keeps one live picture of the current page. It deletes that picture when the audit ends.

We do not blur screenshots or video. When model review is on, we can send the visible text of a page and screenshots to an AI model provider. See Subprocessors. Staff approve every finding before a customer sees it.

How fast it goes

The bot visits one page at a time. It waits 1 second between two page visits. A passive audit visits at most 25 pages and has a time budget of 12 minutes. An active audit visits at most 40 pages and has a time budget of 25 minutes.

Some steps load a page again. The bot replays up to 20 possible problems per audit, 2 times each, with up to 3 at the same time. Model review also loads some pages again, up to 3 at the same time.

If a page answers with status 429, the bot waits as the Retry-After header says, for at most 30 seconds. Then it tries once more. If the page answers 429 again, the bot skips the page.

One organization can start at most 20 audits each day, except batch jobs that we start.

robots.txt

In a passive audit, the bot reads /robots.txt once, before it visits any page. It obeys the file for every link that it visits. The group for ExcutoBot wins over the group for *. The longest matching rule wins, and allow wins a tie.

• If the file answers with status 2xx, the bot obeys it.
• If the answer is 3xx or 4xx, the bot treats the file as missing and allows all pages.
• If the answer is 5xx, or no answer comes, the bot disallows all pages.

An active audit does not read robots.txt. Before an active audit starts, the site owner proves ownership with a DNS TXT record. See Active audits below.

What it never does in passive mode

• It does not click, type or press keys.
• It does not submit a form.
• It does not log in. The bot holds no login for any site.
• It sends no request except GET and HEAD. The browser aborts every other method.
• It does not reach private networks, localhost, or ports other than 80 and 443.

Any customer can order a passive audit of a public website. The page itself can still send GET requests when it loads, as in every browser.

Active audits

An active audit runs only after the site owner proves control of the host. The owner adds a DNS TXT record named _qa-verify. followed by the host name, and our service checks it. An active audit does more than a passive audit:

• It fills fields with fixed test values, for example qa.user@example.com, and with long or unusual text.
• It submits forms, and it clicks buttons and links. This can create test records on your site.
• It presses the Tab key, slows the network, and goes offline for a moment.
• It performs at most 60 form and click actions in its first pass. The Tab key, input and repeated-click tests above add more.

It never fills password, file or hidden fields. It skips a field that looks like a card field by its autocomplete value, name or id. It does not click a control whose name holds logout, delete, remove, unsubscribe or close account.

IP addresses

The bot connects from these IP addresses: {{CRAWLER_IPS}}

Anyone can copy a user agent string. When you filter traffic, match the IP addresses as well.

How to block it

To block the bot in a passive audit, add this to your robots.txt file:

User-agent: ExcutoBot
Disallow: /

To block only part of your site, name the path:

User-agent: ExcutoBot
Disallow: /private/

robots.txt does not stop an active audit. If you did not authorize one, email us. For a firewall, refuse requests that carry the token ExcutoBot in the user agent, and refuse the IP addresses above.

Ask us to stop or to delete data

We keep findings, screenshots, videos and error records about your site for the customer who ordered the audit. We have no self-service tool to erase them yet.

• To ask us to stop auditing your site, email {{CONTACT_EMAIL}}.
• To ask us to delete data about your site, email {{PRIVACY_EMAIL}}.

Give the host name of your site in your email. We answer within {{REQUEST_RESPONSE_DAYS}} days.

Who runs it

{{LEGAL_ENTITY}} runs the crawler. Our address is {{LEGAL_ADDRESS}}.