Practical Lens 57
Practical Lens 57: Robots policy is now a business decision
A crawler rule is no longer only a website setting. It can create a deliberate trade-off between content protection and the opportunity for AI systems to access public evidence. That trade-off needs a named owner, not an accidental configuration.
What this lens means
Robots.txt is a public file that gives crawl instructions by user agent. It is not a universal enforcement mechanism and it does not predict an AI answer. Its business value is simpler: it records part of the company's access choice. When that choice is made without shared ownership, visibility goals and content-protection goals can pull in opposite directions.
The decision
For each relevant crawler, decide whether access is allowed, limited or disallowed, and state the reason in business language.
The control
Keep the approved decision and the live robots.txt file comparable. A rule that cannot be reconciled with a decision register is a governance gap.
Key terms
- Access policy
- The approved position on which automated agents may request which public content, and why.
- Decision register
- A short record that makes ownership, rationale, approval and review date auditable.
- Implementation check
- A comparison between the approved rule and what the public website currently returns.
- Exception
- A different treatment for a crawler or path. It must be explicit; otherwise it is merely unexplained drift.
Why this happens
Different teams see different parts of the risk. Marketing may want discoverability. Technology may reduce load or exposure. Legal may consider intellectual-property, contractual or privacy obligations. Each perspective can be reasonable, but a robots file cannot resolve the conflict by itself. It only exposes the result of whichever decision reached production.
What this usually indicates
- No executive has been assigned accountability for public AI access decisions.
- Website changes are deployed without a recorded business rationale.
- The robots file contains inherited directives whose purpose is no longer known.
- Teams measure AI visibility but cannot explain the access choices that may affect it.
These are governance observations, not proof that any specific AI system has used or ignored a page.
What to verify (evidence-only)
- Name the accountable owner. One executive owns the business decision; marketing, technology and legal provide recorded input.
- Make a one-page access register. Include crawler, allowed/limited/disallowed decision, public paths affected, rationale, approver, implementation date and next review date.
- Read the live file. Fetch
/robots.txtfrom the production domain. Do not rely on a staging copy, a CMS setting or memory. - Reconcile exceptions. Every relevant live directive must map to a register entry. Every register entry must have an implementation outcome.
- Log the decision. Retain the approved version and an evidence capture of the live file. Repeat after a platform, legal or positioning change.
Access register template
Track each crawler decision as a comparable record. A rule with no register entry, or a register entry with no live match, is the governance gap to close first.
crawler,decision,business_rationale,accountable_owner,live_check
[user_agent],allow / limit / disallow,[visibility or protection rationale],[role],pass / investigate
[user_agent],allow / limit / disallow,[visibility or protection rationale],[role],pass / investigateTerminal check example
Replace the example domain, run the command, then compare every displayed directive with the approved register. The expected result is a readable copy of the live robots file and any lines that name the agents you chose to govern.
SITE="https://www.example.com"
curl -fsSL "$SITE/robots.txt" -o robots.txt
sed -n '1,220p' robots.txt
grep -Ein '^(user-agent|allow|disallow|sitemap):|GPTBot|ChatGPT-User|ClaudeBot|Google-Extended|PerplexityBot' robots.txt || trueThis does not prove AI visibility. It proves what the public website currently instructs crawler clients to do. A missing or unexpected rule is a decision-control issue to resolve with the policy owner.
PowerShell check example
Run this against the production domain. The expected result is the live file saved as robots.txt and the relevant directives shown for review.
$site = 'https://www.example.com'
$robots = Invoke-WebRequest -Uri "$site/robots.txt" -UseBasicParsing
$robots.Content | Set-Content -Path robots.txt -Encoding utf8
$robots.Content -split "`n" |
Select-String -Pattern '^(User-agent|Allow|Disallow|Sitemap):|GPTBot|ChatGPT-User|ClaudeBot|Google-Extended|PerplexityBot' -CaseSensitive:$falseRetain this output with the decision record. It lets leadership distinguish an intentional policy from a configuration that has simply persisted.
Decision rule for CXOs
Do not approve a blanket "allow" or "block" purely as a technical preference. Approve the smallest policy that states the business objective, the content scope, the accountable owner and the review trigger. If those four points cannot be stated, defer the change until they can.
Frequently Asked Questions
Does robots.txt decide whether an AI system will mention a company?
No. A robots.txt rule is an access instruction for crawlers that choose to follow it. It is not evidence that a system will or will not use, cite or recommend a company.
Who should own AI crawler policy?
The decision should have one accountable business owner and recorded input from marketing, technology and legal. The web team implements and verifies the approved rule.
What is the first control to put in place?
Maintain a short access register: crawler, decision, business rationale, owner, approver, implementation date and next review date. Compare it with the live robots.txt file after every change.
Should every AI crawler be treated the same?
Not necessarily. The appropriate rule depends on the company's objectives, content type, contractual commitments and risk assessment. The key is that differences are deliberate and documented.
How often should the policy be reviewed?
Review it when business objectives, content rights, technology or crawler rules change, and set a regular governance review so the live configuration does not become an undocumented exception.
Run a free AI Readiness baseline, then compare the finding with this diagnostic framework.