AI search visibility in the UAE: a one-afternoon crawl-layer audit

Digital marketing

Ask ChatGPT, Gemini or Perplexity to name a supplier, a clinic or a logistics partner in the UAE, and the answer is drawn from the handful of sites the assistant can actually reach and parse. Most teams treat visibility in AI answers as a content and reputation problem. For a surprising share of UAE websites it is a plumbing problem first: a legacy "block AI bots" rule in robots.txt, a CDN or WAF rule that 403s anything with "bot" in the user agent, or a client-side rendered catalogue that hands crawlers an empty HTML shell. The audit below takes an afternoon, needs no budget, and ends with a one-page checklist you can forward to whoever controls your CDN.

Our earlier article on being recommended by ChatGPT and Gemini covered the content and reputation side: what to publish, how to earn citations, how to be quotable. This one covers the layer beneath it. A crawler that cannot fetch your page, cannot read its HTML, or cannot resolve your entity data will never cite you, however good the copy is. Fix the crawl layer first, because it is cheap, verifiable and permanent.

The crawlers that matter and what each one does

"AI traffic" and "AI crawlers" are not one thing. Each major operator runs separate bots for separate jobs: one crawls for model training, one crawls to build a search or answer index, and one fetches a single page live when a user asks a question. The controls are independent. OpenAI states this explicitly in its publisher FAQ: blocking GPTBot (training) has no bearing on whether OAI-SearchBot (ChatGPT search) can index you, and vice versa. The same split exists at Anthropic and Perplexity.

OpenAI's bot documentation confirms the operational details you need: OAI-SearchBot and GPTBot are configured independently; robots.txt changes can take up to 24 hours to affect search behaviour; and because ChatGPT-User fetches are initiated by a user, "robots.txt rules may not apply", so a robots.txt allow does not guarantee a live fetch will go through, and a disallow will not reliably stop one. Google's crawler overview draws the same control distinction: Google-Extended is a product token for AI training consent, while Googlebot is the token whose blocking actually removes you from Search.

Two consequences for UAE marketing teams. First, "we block AI bots" is not a single decision; it is at least four decisions per provider, and the default plugins usually make all of them at once. Second, if your WAF is doing the blocking, your robots.txt settings are irrelevant.

Step 1: read your own robots.txt like a crawler

The most common failure in this layer is a blanket rule added in 2023 or 2024, often by a privacy plugin or on someone's advice to "stop AI stealing our content". Fetch the file and read it as a crawler would:

curl -s https://www.example.ae/robots.txt

Look for these patterns:

# 1. The blanket block (blocks every crawler with no more specific group)
User-agent: *
Disallow: /

# 2. The partial block (blocks training, and everyone assumes it blocks search too)
User-agent: GPTBot
Disallow: /

# 3. The path mistake (Arabic pages blocked while English is open)
User-agent: *
Disallow: /ar/

Remember robots.txt group semantics: each crawler matches the group with the most specific user-agent token, so a * group does not apply to a crawler that has its own group, and a * group of Disallow: / still blocks every bot you forgot to list, including OAI-SearchBot and PerplexityBot. If you want to be cited in AI search but not used for model training, that is expressible in one file:

# Be in ChatGPT search and live fetches, stay out of training
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

# Everyone else: open the site, keep private paths closed
User-agent: *
Allow: /
Disallow: /cart
Disallow: /account
Disallow: /checkout

Sitemap: https://www.example.ae/sitemap-index.xml

Deploy it, then wait up to 24 hours before concluding anything from ChatGPT's behaviour, which is OpenAI's documented propagation window for robots.txt changes. As a rule of thumb, re-audit the file whenever a plugin, a consent tool or a CDN feature "with AI protection" is installed, because several of them rewrite robots.txt silently.

Step 2: check what the CDN and WAF do to bots

This is where the hidden blocking lives. A clean robots.txt means nothing if the edge returns 403 or a JavaScript challenge to anything identifying itself as a crawler. Perplexity's bot guidance describes the reverse problem (sites wanting to let PerplexityBot through Cloudflare or AWS WAF), and its pattern applies to any provider: a Cloudflare custom WAF rule allowing a crawler by source IP range plus user-agent string, or an AWS WAF IP set combined with a string match on the user agent. Perplexity also warns that WAF configuration changes take time to propagate.

Test from outside your network, once per crawler identity:

for ua in \
  "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
do
  curl -s -o /dev/null -w "%{http_code} $ua\n" \
    -A "$ua" https://www.example.ae/en/catalogue/
done

Anything other than 200 needs an explanation. The three failure modes to look for:

• A block rule matched on a substring. A rule intended for GPTBot written as "user agent contains bot" also kills Googlebot, OAI-SearchBot and Bingbot. Ask your platform team for the actual rule expression, not a summary.
• Bot management features that rate-limit or challenge unverified crawlers. If the CDN vendor's "AI scraper protection" toggle is on, the named AI crawlers may be challenged unless their published IP ranges are allowlisted and your WAF trusts that list.
• Stale IP lists. All the major providers publish verified ranges as JSON (openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, openai.com/adsbot.json, perplexity.com/perplexitybot.json, plus Anthropic's published crawler ranges). If someone pasted those ranges into the WAF once and never again, some will be wrong by now. Fetch them on a schedule and reconcile.

Finally, ask for a 30-day report of blocks by user agent from the CDN or WAF console. If OAI-SearchBot or Googlebot appear in that report, you are paying to be invisible, and that report is the evidence you need for the ticket in the last section.

Step 3: check what the crawler actually receives

Assume nothing about JavaScript rendering. Googlebot renders, but a fair share of answer-engine and training crawlers fetch HTML and parse what is there. If your catalogue, branch or service pages are client-side rendered, the HTML payload is an empty shell and you should treat those pages as invisible to any crawler that does not execute your JavaScript.

Compare raw HTML against the rendered page:

# How much real content is in the raw HTML?
curl -s https://www.example.ae/ar/catalogue/pumps/ | wc -c
curl -s https://www.example.ae/ar/catalogue/pumps/ | grep -c "ضخ"

# And what does the rendered DOM contain? (check in the browser devtools Network tab)

If the raw HTML returns 3 KB of shell and the product names only appear after hydration, you have three options in ascending order of effort: server-side rendering for those routes, static generation at build time, or pre-rendering specific templates for crawlers. Do it for the pages you want cited first: the top 20 service or category pages by commercial value, then the rest.

The regional trap to check explicitly: the Arabic-only PDF brochure. A common pattern in the region is a UAE site where the entire Arabic product or service story lives in a PDF that is scanned images or outlined fonts with no extractable text, linked from a thin HTML page. A text-based crawler gets nothing from it. Publish the Arabic content as HTML (a page per brochure topic is enough), keep the PDF as a download, and make sure the Arabic text in the PDF is selectable. Note also that Google documents crawl limits by file type in its common crawler documentation, so oversized image-heavy catalogue pages can be truncated mid-parse.

Step 4: entity data, languages and llms.txt

Once a crawler can read the page, it needs to know who you are. For a UAE organisation, the entity layer has a regional specificity that generic schema tutorials miss: the Arabic and English names must agree with each other and with your trade licence name.

Mark up the homepage and contact page with Organization or LocalBusiness (see schema.org/LocalBusiness), including both names:

{
  "@context": "https://schema.org",
  "@type": "LocalBusiness",
  "name": "Al Rayan Industrial Supplies LLC",
  "alternateName": "الريان للمواد الصناعية",
  "url": "https://www.example.ae",
  "telephone": "+971-4-000-0000",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "Warehouse 12, Street 8, Al Quoz Industrial Area 3",
    "addressLocality": "Dubai",
    "addressRegion": "Dubai",
    "addressCountry": "AE"
  },
  "areaServed": { "@type": "Country", "name": "United Arab Emirates" },
  "sameAs": [
    "https://www.linkedin.com/company/example",
    "https://maps.app.goo.gl/example"
  ],
  "openingHoursSpecification": [{
    "@type": "OpeningHoursSpecification",
    "dayOfWeek": ["Monday","Tuesday","Wednesday","Thursday"],
    "opens": "08:00", "closes": "18:00"
  }]
}

Checklist for the entity layer in the Gulf context:

• Name consistency. name and alternateName should match the trade name on your DED or free zone licence, because that is what third-party data providers and Google Business Profile already reconcile against. A site that says "Al Rayan Group", a licence that says "Al Rayan Industrial Supplies LLC" and an Arabic name that says something else is three entities to a machine.
• NAP consistency. Name, address and phone identical in the site footer, the schema, and every Google Business Profile listing and directory entry you control. Use the +971 country code format in the schema.
• sameAs. Link the official profiles you actually maintain. Two good links beat eight stale ones.
• FAQ content. Mark up genuine question and answer pairs with schema.org/FAQPage on service pages. Do not expect FAQ rich results in Google for them; the value here is that assistants quote clean Q and A pairs directly.
• hreflang. Annotate ar-AE and en-AE properly, with return tags on every language version and an x-default:

<link rel="alternate" hreflang="ar-AE" href="https://www.example.ae/ar/pumps/" />
<link rel="alternate" hreflang="en-AE" href="https://www.example.ae/en/pumps/" />
<link rel="alternate" hreflang="x-default" href="https://www.example.ae/en/pumps/" />

Then llms.txt. The llms.txt proposal asks sites to publish a markdown file at the root that lists the pages worth reading, with brief descriptions and, ideally, markdown versions of key pages. It is a proposal, not a standard, and it is not an access control: it does not grant or deny crawling, and it is not a substitute for robots.txt. Treat it as a 20-minute favour to any assistant that looks for one:

# Al Rayan Industrial Supplies

> Industrial MRO distributor in the UAE: catalogue, stock and delivery from Dubai, Sharjah and Abu Dhabi. Prices in AED, pages in Arabic and English.

## Catalogue
- [Pumps and compressors](/en/catalogue/pumps.md): centrifugal and diaphragm pumps, 200+ SKUs in stock.
- [Filtration and consumables](/en/catalogue/filtration.md)

## Company
- [Branches, hours and delivery areas](/en/branches.md)
- [Request a quotation](/en/contact.md)

Its practical value is that your URL map becomes unambiguous in a single fetch, and it costs almost nothing to publish.

Step 5: measure whether the traffic is worth chasing

You cannot manage this inside an AI assistant's own analytics, so you measure it in GA4. The problem: GA4 has no default AI channel, so ChatGPT referrals land in "Referral" (or "Unassigned", because ChatGPT appends utm_source=chatgpt.com without a medium, as confirmed in the publisher FAQ). Build a custom channel group instead. In GA4: Admin, Data Display, Channel Groups, copy the default group, add a channel named "AI assistants" and move it above "Referral", because GA4 matches channel rules top to bottom. A working source rule:

Source matches regex:
^(chatgpt\.com|chat\.openai\.com|chat\.gpt\.com|openai\.com|perplexity\.ai|
pplx\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai|anthropic\.com|
deepseek\.com|grok\.com|x\.ai|you\.com|edgeservices\.bing\.com)$

Add a second rule in the same channel for Medium matches regex ^(ai|assistant|referral)$ combined with Source matching the same pattern, so sessions tagged utm_source=chatgpt.com with no medium stop landing in Unassigned. The wider regex, covering 30-plus assistants, and the rule-order detail are documented in this GA4 AI traffic tracking guide.

Then interpret the numbers honestly:

• You will undercount. One analysis cited in that guide estimates that only a minority of AI-referred sessions are measurable in analytics: app-to-browser handoffs drop the referrer, and copied links show as Direct or Organic. Add one survey question to your lead form ("How did you hear about us?") and compare.
• Judge on conversion, not volume. AI-referred sessions are usually a small share of total traffic, so session count tells you almost nothing. Track enquiry, quotation-request and qualified-call rates from the "AI assistants" channel against your site median, and weigh lead quality (company size, intent, whether sales accepts the lead) rather than visit count.
• Crawler hits are not analytics. GA4 filters known bots, so the crawl data you need for this audit comes from CDN, WAF and server logs. Keep the two datasets separate.

The one-page audit and the fixes to forward

Copy this into whatever ticketing system your platform team uses. Everything in steps 1 to 3 is verifiable in an afternoon.

AI SEARCH CRAWL AUDIT                    site: www.example.ae   date: ____
1. robots.txt
   [ ] Fetch /robots.txt. Any "Disallow: /" against * or an AI bot? ________
   [ ] OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot: Allow
   [ ] GPTBot / ClaudeBot: decide explicitly (training consent: yes / no)
   [ ] /ar/ paths not disallowed; sitemap declared
2. CDN and WAF
   [ ] curl each crawler UA against a key URL: all return 200?
   [ ] No block rule matching a substring ("bot") instead of named bots
   [ ] Published IP ranges (gptbot.json, searchbot.json, chatgpt-user.json,
       adsbot.json, perplexitybot.json, Anthropic's crawler ranges)
       allowlisted and refreshed on a schedule
   [ ] 30-day WAF report of blocks by user agent requested
3. Content delivery
   [ ] Raw HTML of top 20 target pages contains the real copy (curl | grep)
   [ ] Arabic content published as HTML, not only in PDF brochures
   [ ] hreflang ar-AE / en-AE with return tags and x-default
4. Entity and navigation
   [ ] Organization/LocalBusiness JSON-LD with name + Arabic alternateName
   [ ] NAP matches trade licence, Google Business Profile and directories
   [ ] FAQPage on service pages; llms.txt at the root
5. Measurement
   [ ] GA4 custom channel "AI assistants" live (regex + utm_source=chatgpt.com)
   [ ] Lead-form source survey question live
   [ ] Baseline conversion rate recorded for comparison in 90 days

Priority order for fixes, in case you cannot do everything at once:

The ticket to whoever controls the CDN can be one paragraph: "Allow OAI-SearchBot, ChatGPT-User, Googlebot, Claude-SearchBot and PerplexityBot through bot management and WAF rules by verified IP range and user agent, refresh those ranges weekly from the providers' published JSON endpoints, and send us 30 days of block counts by user agent. Do not apply any rule that matches on the substring bot."

What to do next

Run the audit this week, before anyone writes another word of content. The sequence matters: robots.txt and WAF first (same day, zero cost, and everything else is wasted effort while a blanket block is live), rendering and entity data next, llms.txt whenever, GA4 measurement from day one so that you have a baseline when the fixes land. Then let the content and reputation work from the earlier article do its job, on a site that crawlers can actually read.

If you need to decide what AI search visibility is worth spending on for your organisation, where it sits against paid and owned channels, and what to fix first, that judgement call is what our AI strategy and readiness work is for. The wider marketing and platform programme that follows sits under digital transformation. Our own team in the UAE does the consulting, the deployment and the support here. Either way, the audit above is yours: run it, and you will know within a day whether the assistants can read you at all.

https://www.azrty.com/blog/ai-search-visibility-in-the-uae-a-one-afternoon-crawl-layer-audit

<a href="https://www.azrty.com/blog/ai-search-visibility-in-the-uae-a-one-afternoon-crawl-layer-audit">AI search visibility in the UAE: a one-afternoon crawl-layer audit</a> (Azrty)