Skip to main content
IlityBot is the crawler Ility uses to visit your site. Every crawl, scan, and health-check request Ility makes comes from IlityBot. If your site blocks bots, allow IlityBot so Ility can find and scan your pages.

How IlityBot identifies itself

Every request carries a user agent that includes this token:
The full user agent is a desktop Chrome string with the IlityBot token at the end, for example:
The Chrome version can change. Match on IlityBot rather than the whole string.

robots.txt

IlityBot reads your site’s robots.txt and follows it.
  • Which group applies. A group that names IlityBot takes priority over the * group. Without one, IlityBot follows the * group. The match ignores case and any /1.0 version.
  • Crawl-delay. IlityBot honors Crawl-delay from the group that applies. It always waits at least 1 second between requests. A delay longer than 10 seconds is capped at 10 seconds.
For example, this file keeps IlityBot out of /admin/ and slows it to one request every 2 seconds, without changing the rules for other bots:
robots.txt
A group that names IlityBot replaces the * group entirely. In the example above, IlityBot doesn’t follow Disallow: /private/. Repeat any * rules you want IlityBot to follow in its own group.

Verify requests with Web Bot Auth

Anyone can copy a user agent. To prove a request really comes from Ility, IlityBot signs its plain fetches with Web Bot Auth, which uses HTTP Message Signatures.
  • Signed: crawl requests and the fetches for robots.txt and sitemaps. Each carries Signature, Signature-Input, and Signature-Agent headers.
  • Not signed: requests made while a page is rendered in a browser during a scan. Allow these by user agent.
The Signature-Agent header names the origin that publishes IlityBot’s public key. The key directory lives at this path on that origin:
Bot protection services that support Web Bot Auth, such as Cloudflare, can fetch this directory and verify signatures for you.

Allow IlityBot through your firewall

If your site sits behind a firewall or bot protection, Ility’s requests may be challenged or blocked. Pages then fail with rate limiting or HTTP errors. See When a page fails. To let IlityBot through:
1

Add an allow rule for the user agent

In your firewall, WAF, or bot protection, create a rule that allows or skips challenges for requests whose user agent contains IlityBot.
2

Verify signatures where you can

If your provider supports Web Bot Auth, turn on signature verification so it only trusts signed IlityBot requests. Keep the user agent rule too, because browser-rendered scan requests aren’t signed.
3

Check rate limits

If your provider rate limits by request count, exempt IlityBot or raise the limit. To slow IlityBot down instead, set Crawl-delay in robots.txt.

Cloudflare

Create a WAF custom rule that matches requests where the user agent contains IlityBot, and set its action to skip the protections that challenge it. Cloudflare supports Web Bot Auth, so it can also verify IlityBot’s signed requests. See Cloudflare’s documentation on WAF custom rules for the exact steps.

Other providers

Akamai, Vercel, Fastly, and similar services each have their own way to allow traffic. Look for a setting that allows requests by user agent, or that verifies bots with Web Bot Auth, and apply it to IlityBot. If Ility still can’t reach your pages, contact us.