SiteVisorSiteVisor
For webmasters

SiteVisorBot — our web crawler

If you see SiteVisorBot in your server logs, that is us. This page explains what the bot does, how to confirm a request really comes from us, and how to control its behaviour.

How the bot identifies itself

A single User-Agent string is used by every one of our tools. We do not use any other, and we never identify as somebody else.

Bot nameSiteVisorBot
User-Agent stringSiteVisorBot/1.0 (+https://sitevisor.cloud/bot)
Match patternSiteVisorBot

This substring is enough to write a rule in robots.txt or in your firewall.

What the bot does

SiteVisorBot does not build a search index and does not harvest content for later use. Every crawl is started by a person — a site owner or a specialist auditing a site with our tools.

  • Technical site audits: response codes, headings, canonicals, duplicates, broken links.
  • Parsing sitemap.xml and checking that the URLs listed in it are reachable.
  • Validating robots.txt and how its rules apply to specific URLs.
  • Auditing a site for AI-agent readiness and parsing structured data.
  • Downloading a single user-supplied file by URL — a PDF to convert, for example.

Every crawl is finite: each run has a page limit and a time budget. We do not run continuous background crawling.

How to verify the bot is really ours

Anyone can forge a User-Agent string, so verify the address the request came from, not the header.

Reverse DNS

All of our crawling nodes have PTR records under the zone below. The check runs both ways: IP → hostname, then hostname → IP. If both directions match and the hostname ends with our suffix, the request is ours.

*.bot.sitevisor.cloud

1. Resolve the hostname for the IP in your log:

host 198.51.100.10

2. Confirm the hostname resolves back to the same IP:

host crawler-1.bot.sitevisor.cloud

IP list

The IP list is published separately — if your firewall needs it, contact us and we will send the current set of addresses.

How to control the bot

We read robots.txt before every request. Changes take effect within an hour — that is how long we cache the file.

Block the bot from the whole site

User-agent: SiteVisorBot
Disallow: /

Block a single section

User-agent: SiteVisorBot
Disallow: /private/
Disallow: /cart

Slow the crawl down

We honour Crawl-delay: it sets the pause between requests in seconds and can only ever slow us down, never speed us up. When a delay is set, the bot stops sending parallel requests.

User-agent: SiteVisorBot
Crawl-delay: 5

Throttle the bot temporarily

The bot treats a 429 or 503 response as a request to back off. This is handy when your server is under peak load right now and you would rather not edit robots.txt.

What we commit to

  • We read robots.txt and honour Allow, Disallow and Crawl-delay.
  • We never identify as another bot and never disguise ourselves as a regular browser.
  • We do not try to work around blocks, captchas or IP restrictions.
  • We keep a moderate rate and slow down when the server starts returning errors.
  • We do not collect personal data and do not enter areas behind authentication.

Contact us

If the bot is creating load, behaving differently from what this page describes, or you simply want it to stop visiting your site — email us. We read and act on it.

support@sitevisor.cloud