How to Verify an AI Crawler Is Who It Says It Is

The article says an AI crawler’s user-agent string is not proof of identity and can be spoofed.

It compares three verification methods: reverse DNS plus forward lookup, operator-published IP ranges, and Web Bot Auth signatures.

The last option is promising but not yet widely documented or adopted by crawler operators.

The issue matters because sites use crawler identity to control indexing, access, and bot-defense exemptions.