Citevera
Run free audit
Scanner policy

Citevera-Scanner bot policy

What our scanner does, how it identifies itself, and how to allowlist it through common WAFs.

Citevera-Scanner is a user-initiated bot. It only fetches a site when a customer (or a public scan via scan.citevera.com) asks us to audit that site. We do not continuously crawl the open web.

What we fetch

  • /robots.txt
  • /sitemap.xml (and any sitemaps referenced from robots.txt)
  • /llms.txt and /llms-full.txt if present
  • The homepage plus up to 9 additional internal pages selected from the sitemap

Per-page: one request, 10-second timeout, 5MB cap. No crawl spanning hosts, no repeated polling. A full audit makes fewer than 15 HTTP requests to the target origin.

How we identify ourselves

Every request carries this User-Agent:

bash
Citevera-Scanner/1.0 (+https://citevera.com/bot)

The UA string links back to this page so WAF operators who see us in their logs can verify we are a known scanner.

We respect robots.txt

If your robots.txt disallows Citevera-Scanner or*, we will not crawl HTML. We still read robots.txt, sitemap.xml, and llms.txt (public discovery files) so the audit can still report on crawlability even when page content is off-limits.

To block us explicitly:

bash
User-agent: Citevera-Scanner
Disallow: /

How to allowlist Citevera-Scanner

Cloudflare

If our scanner trips a Cloudflare challenge, create a WAF rule that skips the Managed Challenge for our UA. In the Cloudflare dashboard: Security > WAF > Create rule.

bash
# Field         Operator             Value
User Agent      contains             Citevera-Scanner

# Action
Skip > All Managed Rules

More restrictive alternative (recommended): constrain the skip rule to requests coming from our outbound IP. Contact support@citevera.comand we will share the IP block currently in use.

AWS WAF

Add a rule with a String Match Condition on the User-Agent header containing Citevera-Scanner, then set the rule action to Allow and place it above any challenge or block rules.

WordPress firewall plugins (Wordfence, Sucuri)

Add Citevera-Scanner/1.0 to the plugin's User-Agent allowlist. Most firewall plugins have a dedicated "Allowed bots" or "Whitelisted User Agents" field under their firewall settings.

What we do NOT do

  • Follow links outside the origin you asked us to scan
  • Submit forms, click buttons, or trigger any interaction
  • Re-crawl on a schedule unless you explicitly subscribe to a paid audit tier with scheduled re-scans
  • Retry aggressively - one fetch per page, fail closed
  • Ignore robots.txt. If you disallow us there, we stop.

Questions

Email support@citevera.com. We answer scanner-policy questions within one business day and will share our current outbound IP range on request.