How AI Search Reads Your Website

Short answer: an AI search engine sends a crawler to fetch your page, reads the HTML it gets back, and keeps the text it can parse. Google renders JavaScript in a second pass; Google's own documentation notes that not all bots can. Pages that deliver facts as server-rendered text, allow the named AI crawlers in robots.txt, and state price, location and contact on the page are the ones that get quoted.

Sources are documentation from Google, OpenAI, Perplexity, Cloudflare and Wix, checked 17 October 2026.

What does an AI crawler see on my page?

Four steps, the same for every engine:

1. Request the URL. If robots.txt disallows the crawler's user agent, most stop here.
2. Receive HTML. What the server sends is what the bot has. Text in that HTML is read. Text inserted later by JavaScript may or may not be.
3. Parse the structure: title, headings, paragraphs, lists, tables, links, structured data.
4. Index the page or quote a passage from it.

Google documents three phases, crawling, rendering and indexing, with a render queue where pages can wait "for several seconds or longer" before a headless Chromium runs the JavaScript (Google Search Central). The same page states: "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript."

Neither OpenAI's nor Perplexity's crawler documentation says their bots render JavaScript (OpenAI, Perplexity). Treat the raw HTML as the only thing you can rely on.

Check it yourself:

• View the page source in your browser and search for your price or phone number. If it is not in the source, a non-rendering bot does not see it.
• In Chrome DevTools, disable JavaScript and reload. What remains is what a plain crawler reads.
• Run the URL through Search Console's URL Inspection and compare the rendered HTML with the source.

Can AI read sites built on Wix or heavy JavaScript?

It depends on how the platform serves the page.

Google's AI guide states Google "is able to process content within JavaScript as long as it isn't blocked" (Google Search Central). That covers AI Overviews and AI Mode. It does not cover ChatGPT search or Perplexity, whose documentation is silent on rendering.

Which robots.txt rules affect AI search?

Each engine names its crawler and states what blocking it does. Blocking a training bot does not remove you from answers; blocking a search bot does.

Sources: Google common crawlers, OpenAI crawlers, Perplexity crawlers.

Two settings to check beyond robots.txt:

• CDN bot rules. Cloudflare announced on 1 July 2025 that new domains on its network block AI crawlers by default (Cloudflare). Allow the search bots you want.
• Snippet tags. Google states that nosnippet, data-nosnippet, max-snippet and noindex limit what appears in Search, including its AI features (Google Search Central).

Decision for most SMEs: allow Googlebot, OAI-SearchBot and PerplexityBot. Decide on GPTBot and Google-Extended separately, since those are training questions and do not change whether you are cited.

How should service pages be structured for AI answers?

Engines quote passages, not pages. Structure each service page so a passage can stand on its own.

• H1 names the service and the place: "Corporate secretarial services in Singapore".
• The first paragraph answers the buying questions: what, for whom, price range, timeline.
• H2s are the questions buyers ask, each answered in its first two sentences.
• One table for anything compared: packages, tiers, timelines.
• Lists for inclusions and steps.
• A dated "last checked" line and a named firm or author.
• Structured data that repeats the visible facts. Google states structured data should match the visible text and that no special schema is needed for AI features.

Avoid hero text set as an image, prices only in a downloadable brochure, and tabs or accordions on platforms that load the hidden text only on click. Check the source to confirm the text is there.

Which business facts should every page state?

AI answers to "who", "where" and "how much" are only as good as the text the engine finds. Put these facts on every page, in the footer if nowhere else, and mirror them in LocalBusiness or Organization markup. Google lists name and address as required and telephone, opening hours, price range, coordinates and URL as recommended for LocalBusiness (Google Search Central).

• Registered name and UEN, or the Hong Kong business registration number.
• Address and service area.
• Phone or WhatsApp number and email.
• Opening hours.
• Price range or starting price per service.
• Languages served.
• Year founded, licences and accreditations.

Keep the same facts on Google Business Profile, LinkedIn and directories. Engines cross-check sources, and a mismatch between your site and a directory is a common reason an answer gives the wrong phone number.

What cense does

cense builds in Next.js with server-rendered pages, so every fact is in the HTML a crawler receives, and sets robots rules and security headers per bot at build time. Hosting and crawl setup are described on website infrastructure, and the search side on SEO and AI search.