How IlityBot identifies itself
Every request carries a user agent that includes this token:IlityBot rather than the whole string.
robots.txt
IlityBot reads your site’srobots.txt and follows it.
- Which group applies. A group that names
IlityBottakes priority over the*group. Without one, IlityBot follows the*group. The match ignores case and any/1.0version. - Crawl-delay. IlityBot honors
Crawl-delayfrom the group that applies. It always waits at least 1 second between requests. A delay longer than 10 seconds is capped at 10 seconds.
/admin/ and slows it to one request every 2 seconds, without changing the rules for other bots:
robots.txt
A group that names IlityBot replaces the
* group entirely. In the example above, IlityBot doesn’t follow Disallow: /private/. Repeat any * rules you want IlityBot to follow in its own group.Verify requests with Web Bot Auth
Anyone can copy a user agent. To prove a request really comes from Ility, IlityBot signs its plain fetches with Web Bot Auth, which uses HTTP Message Signatures.- Signed: crawl requests and the fetches for
robots.txtand sitemaps. Each carriesSignature,Signature-Input, andSignature-Agentheaders. - Not signed: requests made while a page is rendered in a browser during a scan. Allow these by user agent.
Signature-Agent header names the origin that publishes IlityBot’s public key. The key directory lives at this path on that origin:
Allow IlityBot through your firewall
If your site sits behind a firewall or bot protection, Ility’s requests may be challenged or blocked. Pages then fail with rate limiting or HTTP errors. See When a page fails. To let IlityBot through:1
Add an allow rule for the user agent
In your firewall, WAF, or bot protection, create a rule that allows or skips challenges for requests whose user agent contains
IlityBot.2
Verify signatures where you can
If your provider supports Web Bot Auth, turn on signature verification so it only trusts signed IlityBot requests. Keep the user agent rule too, because browser-rendered scan requests aren’t signed.
3
Check rate limits
If your provider rate limits by request count, exempt IlityBot or raise the limit. To slow IlityBot down instead, set
Crawl-delay in robots.txt.Cloudflare
Create a WAF custom rule that matches requests where the user agent containsIlityBot, and set its action to skip the protections that challenge it. Cloudflare supports Web Bot Auth, so it can also verify IlityBot’s signed requests. See Cloudflare’s documentation on WAF custom rules for the exact steps.
Other providers
Akamai, Vercel, Fastly, and similar services each have their own way to allow traffic. Look for a setting that allows requests by user agent, or that verifies bots with Web Bot Auth, and apply it toIlityBot.
If Ility still can’t reach your pages, contact us.