Practical Lens 69: A crawler name is not proof of identity

Claimed identity

The crawler name supplied in the HTTP User-Agent, such as GPTBot, OAI-SearchBot or ClaudeBot. Any HTTP client can copy this text.

Verified identity

A requester whose claimed provider is supported by a trusted verified-bot signal, provider-published network ranges, confirmed DNS evidence or a valid request signature.

Why this happens

User-Agent strings are simple to copy, while many dashboards and rules classify bots by name alone. A request can therefore inherit a trusted label without inheriting the provider’s network, DNS or cryptographic identity.

What this usually indicates

• A firewall allowlist matches only the User-Agent.
• Analytics report named bots without a verification field.
• Unverified and verified crawler traffic is combined.
• Sensitive-path requests are counted as ordinary crawling.
• Exceptions are granted by crawler name across every route.
• No owner reviews changes to provider verification evidence.

What to verify (evidence-only)

1. Inventory trusted crawler rules. Find every allow, bypass and analytics rule based on a bot name.
2. Add independent evidence. Use a trusted verified-bot signal, current provider IP ranges, confirmed DNS or request signatures where supported.
3. Separate confidence classes. Record verified, unverified and inconclusive requests separately.
4. Classify the target. Treat credential, configuration, source and administrative paths as security events, not content discovery.
5. Retest exceptions. Confirm that legitimate verified crawlers can reach intended public pages without receiving access to sensitive routes.

Terminal check: find unsafe crawler decisions

Open Terminal in the folder containing crawler-identity-check.csv, paste the command and press Enter. The expected result is every unverified request that is allowed or sensitive-path request that is not blocked.

python3 -c 'import csv,sys; r=list(csv.DictReader(open(sys.argv[1],encoding="utf-8-sig"))); bad=[x for x in r if (x["verification_result"].strip().lower()!="verified" and x["action"].strip().lower()=="allow") or (x["path_class"].strip().lower()=="sensitive" and x["action"].strip().lower()!="block")]; [print("{} | {} | {} | owner: {}".format(x["claimed_bot"],x["verification_result"],x["action"],x["owner"])) for x in bad]; print("Unsafe crawler decisions: {}/{}".format(len(bad),len(r)))' crawler-identity-check.csv

Business meaning: every listed row can create unmeasured security exposure or false AI-traffic reporting.

PowerShell check: find unsafe crawler decisions

Open PowerShell in the folder containing the same CSV, paste the commands and press Enter. The expected result is the same list and count.

$rows = Import-Csv .\crawler-identity-check.csv
$issues = $rows | Where-Object { ($_.verification_result -ne 'verified' -and $_.action -eq 'allow') -or ($_.path_class -eq 'sensitive' -and $_.action -ne 'block') }
$issues | Format-Table claimed_bot,verification_result,path_class,action,owner -AutoSize
'Unsafe crawler decisions: {0}/{1}' -f $issues.Count,$rows.Count

Business meaning: review the identity evidence and route policy before keeping any listed exception.