Real Estate AEO: Can AI Search Read Your Website?

Open your agent profile or a useful local guide while signed out. Check that its main answer, business identity and links are visible. A login page, a blank shell or an error cannot substitute for the public article you intended to publish.

Ask your developer to confirm the final HTTP response and any robots meta or X-Robots-Tag directive. A successful page load in your browser is useful evidence, but the response an automated crawler receives can differ. Record the exact URL, date and response.

Open the plain-text robots.txt file at the root of the actual public domain. Identify the groups that match search crawlers and whether their rules permit the page you are testing. Check that the sitemap points to the canonical public site.

Do not paste a generic allow-all file over an existing policy. Preserve restrictions on dashboards, private previews and internal endpoints. A robots.txt restriction manages crawling; it is not authentication and should never be used to protect private information.

OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for possible model-training use. These controls are independent. ChatGPT-User supports user-directed visits and is not the control for search inclusion.

Anthropic identifies Claude-SearchBot for search, Claude-User for user-directed retrieval and ClaudeBot for potential model-training collection. Perplexity identifies PerplexityBot for search and Perplexity-User for user-directed visits. Its crawler documentation says user-requested retrieval generally ignores robots.txt.

Google uses Googlebot controls for Search, including AI features. Google-Extended governs certain Gemini training and grounding uses and does not control inclusion or rankings in Google Search.

A site may allow a crawler in robots.txt while its firewall serves a challenge or blocks the request. Perplexity and OpenAI publish network information for verifying their crawlers. Ask your host to inspect legitimate requests using current provider documentation and available logs.

A request with a copied bot name is not proof of a genuine provider visit. Do not disable security protections simply to make a test pass. Have the technical owner propose a narrow change if verified search requests are blocked, and review its effect before applying it.

Do not invent a universal AI crawler list or assume all assistants retrieve the web in the same way. xAI’s web-search documentation describes a search tool; it is not a public guarantee that an individual real estate site will appear in every Grok answer.

Our recommended check is to record whether a test actually used web search and which URLs it cited. Keep Grok observations separate from Google Search and Gemini observations. An API search experiment does not establish how every consumer app session behaves.

Our recommended worksheet records the canonical URL, page response, visible main content, indexing directives, matching robots group, verified firewall evidence and the next action. Use “not tested” when evidence is missing rather than marking a platform as passed.

Then check actual indexing and repeated search-enabled answers separately. Eligibility is a prerequisite to investigate, not evidence that a page is ranking. Review the foundational AI SEO guide for the wider plan, or discuss an access issue through our real estate AEO page.