The AI crawler status in Search AI helps you verify whether AI search engines such as ChatGPT, Gemini, and Perplexity can access and use your website content.
The report checks your website's HTTP accessibility and robots.txt configuration, allowing you to quickly identify issues that may prevent AI crawlers from reading your website or using your content to generate responses. If issues are detected, Search AI provides recommendations to help resolve them and improve your AI visibility.
Access the AI Crawler Status
Follow these steps to access the Website Audit Report:
Log in to your Birdeye account and navigate to Search AI from the left navigation rail.
Expand Reports and select AI crawler status.
Add a Website
Before Search AI can analyze your website, you must add the website URL.
Step 1
Click + Add URL at the top right of the page.
Step 2
Enter your website URL and click Add.
Search AI scans the website and generates an audit report.
Review HTTP Results
The HTTP Results section checks whether AI search engines can access your website through standard HTTP requests.
For each website URL, you'll see the status for supported AI platforms, including:
ChatGPT
Gemini
Perplexity
For every platform, Search AI verifies:
Responses – Whether the AI platform can access your website content and generate responses using it.
Training – Whether the website content is available for AI model training (where applicable).
Status indicators
Green indicates the crawler successfully accessed the website.
Red indicates the crawler could not access the website or the content is restricted
Review robots.txt Results
The robots.txt Results section verifies whether your website's robots.txt file allows AI crawlers to access your content.
Search AI checks the robots.txt configuration for supported AI platforms and displays whether access is allowed or blocked.
If an AI crawler is blocked in robots.txt, it may not be able to crawl or index your website content.
Resolve Website Accessibility Issues
If Search AI detects issues with HTTP accessibility or robots.txt configuration, the report highlights the affected AI platforms using red status indicators.
You can use these insights to identify accessibility issues and update your website configuration so AI crawlers can access your content successfully.
Search AI also surfaces recommendations in the Actions section to help resolve detected issues and improve your website's visibility across AI search engines.
Understanding the Report
The Website Audit Report helps you answer questions such as:
Can ChatGPT access my website?
Is my website blocked by robots.txt?
Which AI search engines can read my content?
Why isn't my website appearing in AI-generated responses?
What configuration issues should I fix to improve AI visibility?
FAQs
Which AI search engines are supported?
The Website Audit Report currently checks website accessibility for supported AI platforms, such as ChatGPT, Gemini, and Perplexity.
What do the green and red status indicators mean?
Green indicates the AI crawler successfully accessed your website.
Red indicates the crawler could not access your website, or the content is blocked.
What's the difference between HTTP Results and robots.txt Results?
HTTP Results verify whether AI crawlers can access your website through HTTP requests.
robots.txt Results verify whether your website's robots.txt configuration allows or blocks AI crawlers.
What should I do if a crawler is blocked?
Review the reported issue and update your website configuration accordingly. If recommendations are available, use the guidance provided in the Actions section to resolve accessibility issues and improve AI visibility.