Some crawlers are thieves in disguise.

Executive summary

A User-Agent is a label, not an identity. Of 12,948 requests carrying recognised AI-crawler names, 10,064 — 77.7% — could not be tied to the claimed provider by available network evidence.

Unverified does not automatically mean malicious. The defensible floor is the 4,355 requests whose targets were themselves sensitive: environment files, credentials, keys, configuration, runtime interfaces, source repositories or deployment files.

What the requests targeted

The largest category was environment-variable discovery. These files commonly contain database URLs, API tokens and service credentials when a system is misconfigured.

Categories are mutually exclusive. Counts cover only unverified AI-labelled requests to explicitly sensitive target patterns.

The exact paths they requested

These are the most frequently requested sensitive paths. They show why the activity is classified as attempted credential discovery rather than ordinary content crawling.

Paths are public request targets, not evidence that the corresponding files existed. Representative live checks returned the public homepage fallback.

Which crawler identities were used

The names below are claimed identities, not attribution. A User-Agent string can be copied by any HTTP client.

Where the unverified traffic came from

7,049 of the 10,064 unverified requests — 70.0% — came from 29 source IP identities on generic Google Cloud infrastructure. Their reverse DNS names ended in .googleusercontent.com.

Network ownership establishes the hosting environment, not the operator. Cloud providers are shared infrastructure and are not the attacker by default. Cloudflare-origin traffic can also include proxy or synthetic traffic and remains unverified unless provider identity evidence matches.

Did anything leak?

The endpoints returned HTTP 200 because the static site fallback served the homepage for unknown paths. The response bodies had the same page title and canonical URL. Their only difference was a per-request Cloudflare challenge identifier.

This matters for interpretation: HTTP 200 in a crawler log does not prove that the requested secret existed or was retrieved.

What leaders should do

• Verify identityCombine User-Agent claims with provider IP ranges, reverse DNS and network ownership evidence.
• Separate analyticsReport verified AI crawlers, unverified claims and security probes as different traffic classes.
• Protect without disappearingReturn real 404/403 responses for sensitive paths while keeping legitimate AI crawler access intentional.

The strategic point: blocking every AI-labelled request can reduce legitimate discovery and citation opportunity. Trusting every AI-labelled request creates security exposure. Identity verification separates the two.

Method and definitions

The study covers requests to the apex host verisai.eu during a 90-day observation window ending 8 September 2026. The bot registry first identified recognised AI crawler User-Agents. VerisAI then evaluated provider identity using the available combination of Cloudflare bot signals, published provider CIDR ranges, reverse DNS, ASN ownership and Cloudflare Radar evidence.

Key definitions

• AI-labelled: the User-Agent matched a recognised AI crawler in the VerisAI registry.
• Provider verified: available network evidence matched the claimed provider with medium or high confidence.
• Unverified: the provider identity could not be confirmed. This is not, by itself, proof of malicious intent.
• Sensitive-target attempt: an unverified request targeted a path associated with credentials, secrets, configuration, source/deployment material or runtime/API inspection.
• Source IP identity: a unique source IP counted through an irreversible hash. Raw IP addresses are not published.

Limitations

• This is a field study of one domain, not an estimate of prevalence across the EU or the wider web.
• One IP may represent multiple systems, and one system may use multiple IPs.
• The study measures requests and observable network evidence; it does not identify human operators.
• The phrase “thieves in disguise” is editorial shorthand for credential-seeking requests using impersonated crawler identities. The measured claim is attempted discovery, not proven theft.
• The remaining 5,709 unverified requests targeted public or other paths and are not automatically classified as malicious.

Use and cite the evidence

Aggregated results are published under CC BY 4.0. Raw IP addresses, request identifiers and full User-Agent strings are withheld.

• JSON dataset
  Metadata, findings and tables
• CSV target classes
  Five target categories
• CSV requested paths
  Top sensitive paths
• CSV networks
  Observed network owners
• Methodology
  Definitions and limitations
• SQL queries
  Reproducible aggregations

Cite as

VerisAI (2026). EU AI Readiness Observatory, Threat Note 01: Some crawlers are thieves in disguise. 90-day observation ending 8 September 2026. https://verisai.eu/observatory/threat-note-01-ai-crawler-impersonation