How to evaluate AI software before buying: a hotel guide

AI infrastructure & software

Every hotel AI vendor demos well, because the vendor chooses the examples. The questions that decide whether a guest messaging or pricing tool is safe to deploy are narrower and harder: how does it handle your real guests, your seasonal tariffs, and your mixed Arabic and English threads? This article sets out a two-week evaluation framework built from 50 past enquiries from your own communication logs, an operational scorecard that distinguishes right, wrong and made-up answers, a strict source traceability check, and pass criteria agreed in writing before testing begins. You can run it with any vendor and walk away from tools that fail.

To evaluate AI software before buying, stop scoring vendor demonstrations and test competing tools on 50 historical enquiries drawn from your own operational logs. The process requires three steps: establish reference answers with supporting source documents before testing begins, evaluate responses against strict pass gates for accuracy and source grounding, and disqualify any tool that fabricates commercial commitments such as room rates or policies. Whether evaluating guest messaging for a city hotel in Dubai or an enterprise workflow across the GCC, your own past data is the only reliable benchmark.

Hotel buying committees are already moving. A study by Canary Technologies of over 400 hospitality professionals responsible for technology purchasing decisions found that 71% say AI is having a significant or transformative impact, 82% expect AI usage to increase in their organisation within the next year, and 85% expect to allocate at least 5% of their IT budget to AI tools this year. Guest communications sit at the top of the wish list. That is precisely where the risk sits too, because a message to a guest is a promise, and an AI tool that invents a room rate, a cancellation policy or a facility makes commitments your property never authorised.

The question every GM, owner and technical lead is asking is how to evaluate AI software before buying with confidence. The answer is identical whether you run a 300-key city hotel in Dubai, a luxury desert resort in Abu Dhabi, or a boutique property group across the GCC: stop scoring the vendor's rehearsed examples and start scoring your own.

Why do vendor demos and published benchmarks fail to reflect reality?

Vendor demos use rehearsed inputs. Published benchmarks use someone else's inputs. OpenAI's SimpleQA benchmark, released on 30 October 2024, comprises 4,326 short fact-seeking questions with human-verified answers. The SimpleQA paper is honest enough to note that GPT-4o scores under 40% on it, and that the dataset itself carries an estimated 3% inherent error rate. Google DeepMind's FACTS Grounding benchmark, released on 17 December 2024, evaluates 1,719 document-grounded examples (860 public, 859 held out) where responses are judged across three frontier models: Gemini 1.5 Pro, GPT-4o and Claude 3.5 Sonnet, specifically to reduce family bias.

These benchmarks are rigorous instruments for measuring model capability under laboratory conditions. Yet they say almost nothing about whether a customer-facing messaging product will quote your Deluxe King weekend rate with your exact cancellation terms in colloquial Gulf Arabic to a guest messaging on WhatsApp at 11pm.

Worse, the marketing figures vendors quote are often unstable. Digital Applied published an AI hallucination rate study in April 2026 and then formally withdrew its model rankings, error rates and confidence intervals on 4 October 2026 because the claimed 5,000-prompt study could not be substantiated with run-level records. Read that retraction before reading any vendor slide showing an arbitrary hallucination percentage. The enduring value of that page is method rather than static measurement: define the denominator, report abstention alongside errors, and independently verify that a cited source passage actually supports the claim.

That points to the true method for evaluating AI software before you sign a commercial contract. Freeze an evaluation set of your own past enquiries with verified reference answers. Agree the pass marks in writing before anything runs. Score competing tools on identical data. Preserve the run-level audit trail. Everything that follows applies this discipline to hotel guest messaging and pricing.

What does an AI hallucination actually cost a hotel?

Braintrust's guide to hallucination evaluation splits hallucinations into three distinct failure modes: factually incorrect against reality, unsupported by retrieved reference documents, and inconsistent with earlier context in the same conversation. In hotel operations, each failure mode carries a direct operational or commercial price tag:

• Factually wrong. The tool quotes AED 850 for a Friday night stay when your current seasonal tariff specifies AED 990. The guest completes the reservation based on that figure, forcing your front desk team to either absorb the loss or provoke an escalation on arrival.
• Unsupported by your documents. The answer sounds natural but exists nowhere in your rate sheet, banquet pack or policy schedule. This is the fabricated answer: confirming early check-in at 10am without charge, promising an airport shuttle service you discontinued last year, or inventing a flexible 6pm cancellation window. The guest treats this as a binding operational promise, and your staff must manage the friction.
• Inconsistent within the conversation thread. The system approves a late checkout at 2pm in message three, then claims checkout is strictly 12pm in message seven. The guest captures screenshots of both messages, eroding trust and compromising the front desk team.

A critical lesson from the Braintrust evaluation guide is that an answer can pass an automated groundedness check and still fail on factuality. Groundedness merely checks whether the output reflects whatever context the retriever fetched. If the retrieval engine pulls an outdated 2024 PDF instead of the active 2026 rate schedule, the model can generate a perfectly grounded answer that is factually wrong for the guest. You must evaluate factual accuracy and retrieval grounding as separate metrics.

How do you build a test set from real past enquiries?

Extract 50 genuine guest enquiries from the last 90 days across your property management system (PMS), CRM, WhatsApp Business logs and central reservations inbox. Avoid synthetic prompts generated by an LLM, and do not let vendors supply test scenarios. Use real historical messages with guest names, phone numbers, room numbers and reservation codes scrubbed clean.

A balanced distribution for a GCC city hotel or resort:

Three operational rules keep this test set robust:

Write the reference answer first. For every enquiry, have an experienced reservations or front office supervisor author the exact response that should be sent. Attach the specific source document: the dated tariff sheet, the room policy PDF, the banquet menu, or the corporate contract terms. If your internal team cannot agree on the correct answer, the question is either unanswerable or reveals an internal documentation gap. Both outcomes provide immediate operational value.

Separate practice enquiries from scored enquiries. Allocate 10 of the 50 enquiries as practice material (5 routine, 3 money, 2 awkward) and exclude them from final scoring. The vendor uses this initial batch to index your knowledge base and calibrate retrieval chunking. The scored set comprises the remaining 40 items: 15 routine, 12 money, 8 awkward, and all 5 unanswerable scenarios. Record the source channel, timestamp and selection criteria for every enquiry, following the benchmark transparency principles outlined by Digital Applied.

Keep Arabic conversations in native phrasing. Ensure roughly 20 of the 40 scored threads are conducted natively in Arabic, with the remaining 20 in English. Retain genuine regional phrasing: Gulf dialect terms, Egyptian or Levantine expressions commonly used by staff and guests, mixed Arabic-English code-switching, and Arabizi (transliterated Arabic in Latin characters). The reference answer must be drafted directly in Arabic by a fluent bilingual team member, rather than translated mechanically from an English master. An AI agent producing flawless Modern Standard Arabic while misinterpreting colloquial Gulf hospitality vocabulary has failed your operational test.

How should you score AI responses: right, wrong, made up or refused?

Every evaluation run must receive one of four explicit verdicts. This framework adapts the taxonomy of correct, incorrect and not attempted used in SimpleQA, while introducing the specific failure mode that matters most to hospitality risk: fabricated claims.

To ensure the resulting figures reflect operational reality, apply two measurement practices:

Score at the atomic claim level. Break each generated response into standalone factual claims, such as the room rate is AED 1,100, breakfast is included, and free cancellation applies until 48 hours prior to arrival. Verify each individual claim against the source documents. This approach aligns with modern long-form factuality evaluation methods. Validate two conditions for every citation: existence (does the referenced source document and page exist?) and entailment (does the cited excerpt directly substantiate every element of the claim, including dates, occupancy limits, room categories and tax assumptions?). A citation pointing to a generic policy page that lacks the specific detail quoted represents a made-up claim masked by a real link.

Publish coverage alongside accuracy. An AI system can artificially inflate its accuracy score by refusing to answer difficult questions. As highlighted in the Digital Applied methodology notes, if a test of 100 questions results in 70 correct responses, 10 incorrect responses and 20 refusals, the error rate is 10% over the full population but 12.5% among attempted answers, with an operational coverage rate of 80%. When comparing competing vendors, always inspect coverage, accuracy across attempted responses, and the raw made-up claim count.

A concise Python script ensures scoring consistency across evaluation runs:

import csv
from collections import Counter

# Input format: enquiry_id, language, run, verdict, source_shown, source_supports
rows = list(csv.DictReader(open("trial_results.csv")))
verdicts = Counter(r["verdict"] for r in rows)
total = len(rows)
attempted = total - verdicts.get("refused", 0)

coverage = attempted / total if total > 0 else 0.0
error_attempted = (verdicts.get("wrong", 0) + verdicts.get("made_up", 0)) / attempted if attempted > 0 else 0.0
made_up_share = verdicts.get("made_up", 0) / total if total > 0 else 0.0
right_attempted = verdicts.get("right", 0) / attempted if attempted > 0 else 0.0

print(f"Total runs: {total}")
print(f"Operational coverage: {coverage:.1%}")
print(f"Accuracy among attempted: {right_attempted:.1%}")
print(f"Error rate among attempted: {error_attempted:.1%}")
print(f"Made-up claim share of total runs: {made_up_share:.1%}")

# Thresholds agreed in writing prior to commencing evaluation
PASS_CRITERIA = {
    "max_made_up_on_money_block": 0,
    "min_right_on_routine": 0.90,
    "min_right_among_attempted": 0.85,
    "min_claims_source_supported": 0.95,
    "min_unanswerable_handoff_rate": 1.0,
}

How do you verify that every answer traces back to a source?

Source traceability is frequently resisted by sales teams, yet it represents your strongest contractual safeguard.

For every scored response, mandate that the software display the document identifier, publication date, version number, and exact text excerpt that informed the output. Conduct two verification checks:

1. Existence: Confirm that the cited tariff sheet, operations manual or F&B guide exists in your active document repository.
2. Entailment: Verify that the cited passage logically entails the complete statement made to the guest. If an AI agent states that valet parking is complimentary for visitors attending an event, the cited document must explicitly mention both event visitors and complimentary status, rather than general guest parking fees.

Beware of the document versioning trap. In hospitality environments, rate sheets, promotional packages and seasonal restaurant schedules update frequently. An AI retrieval system that fails to enforce strict timestamp filtering will inadvertently surface an archived 2025 Ramadan buffet price or an expired summer staycation package when responding to a 2026 booking enquiry. Require the vendor to demonstrate how document expiry dates are handled, what underlying foundation model is deployed, and what notification procedure occurs when model versions are updated. If a vendor cannot supply verifiable document references, the software cannot be audited, your reservations team cannot supervise it, and your property cannot defend its statements in a commercial dispute.

How to evaluate AI software before buying: the two-week schedule

Running this evaluation requires approximately two person-days of focused internal effort distributed across two weeks, using two property staff members.

Days 1 and 2: Freeze data and agree acceptance gates. Lock the 50 enquiry scenarios, the reference answers, and the underlying source documents. Formalise the pass criteria in a written schedule shared with all participating vendors: zero fabricated statements on the 15 money enquiries, at least 90% accuracy on routine enquiries, at least 85% accuracy among attempted answers, at least 95% of factual claims supported by verifiable sources, and 100% handoff on unanswerable enquiries. Pass criteria established after testing begins are invalid.

Day 3: Index knowledge base. Supply the vendor with a clean document bundle: active room tariffs with effective dates, cancellation terms, standard operating procedures, spa and F&B menus, and property amenity guides. Confirm in writing the specific files indexed into the vector store.

Day 4: Practice validation. Execute the 10 practice enquiries. Use this phase to identify internal information omissions, such as unwritten pet policies or ambiguous late-checkout surcharges. Rectify missing documentation in the index before proceeding.

Days 5 to 8: Scored evaluation runs. Submit the 40 scored enquiries in their authentic language format. Execute each enquiry twice across separate sessions, yielding 80 total runs. Running duplicate queries is essential: frontier models exhibit variance, and repeated generations highlight unstable answers when a model is guessing. Braintrust's evaluation methodology underscores this self-consistency approach for open-ended generative responses.

Days 9 and 10: Blind independent review. Two bilingual reviewers independently evaluate all 80 responses against the reference answers and source files. Avoid averaging out split decisions; resolve discrepancies by examining source documentation and conversation context.

Day 11: Quantitative calculation. Consider a representative outcome across 80 runs: 68 attempted responses and 12 human handoffs (10 on unanswerable scenarios, 2 on complex edge cases). Of the 68 attempted responses, 61 are verified right, 5 wrong due to retriever version confusion, and 2 made up, with both fabrications occurring on pricing queries in the Arabic test track. The resulting metrics:

• Operational coverage: 68 / 80 = 85.0%
• Accuracy among attempted: 61 / 68 = 89.7%
• Combined error rate among attempted: 7 / 68 = 10.3%
• Fabricated response rate across total runs: 2 / 80 = 2.5%

Days 12 to 14: Final procurement decision. Evaluate outcomes against the pre-agreed acceptance thresholds. While the overall accuracy figure (89.7%) exceeds the 85% threshold, the product failed the zero-tolerance requirement for fabricated answers on the money block. Fabricating room rates and payment terms exposes your hotel to financial and reputational liability. This outcome represents a contractual failure. You can either reject the vendor or require an architectural fix and a full re-evaluation of the money block before signing. Never select a vendor based solely on aggregate accuracy percentages; enforce the boundary conditions.

What contractual terms, data protection rules and running costs matter?

A successful technical trial proves the model works under controlled conditions. The commercial contract determines whether your hotel can safely operate it at scale.

Data residency and regulatory compliance. Guest messaging inevitably processes personally identifiable information (PII), including guest names, contact details, payment references and travel dates. In the UAE, personal data handling is governed by Federal Decree-Law No. 45 of 2021 on the Protection of Personal Data, which took effect on 2 January 2022. Articles 22 and 23 regulate cross-border personal data transfers, allowing transfers either to countries recognised as having an adequate level of protection, or where contractual safeguards, standard contractual clauses or explicit guest consent apply. Because free zones such as DIFC and ADGM operate under distinct data protection statutes, and broad statutory adequacy lists remain unissued, clear contractual guarantees under Article 23 are the practical standard for SaaS deployments in the UAE.

Insist on written contractual clarity on these points:

• Physical data storage and inference processing locations, including backup archives and diagnostic log stores.
• Named third-party sub-processors, particularly underlying foundation model providers and vector database hosts.
• Explicit contractual clauses prohibiting the vendor or underlying model providers from training models on your proprietary hotel data or guest transcripts.
• Defined retention schedules, user data deletion rights, and prompt written certification of data destruction upon contract termination.

Commercial and technical terms. Ensure the vendor contract specifies the exact model versions powering the system, with a mandatory advance notification requirement before any model switch or parameter modification takes effect. Attach the performance thresholds from your two-week trial as ongoing service level agreements. Outline fallback protocols for system outages, establish clear contract termination terms, and secure full export access to conversation logs in standard formats (such as JSON or CSV) upon contract conclusion.

Calculating true operational cost. With 85% of hospitality organisations committing 5% or more of their technology budget to AI initiatives, total cost of ownership must be calculated accurately. Look beyond headline software licensing fees. Calculate the fully loaded cost per room per month or cost per completed conversation, including messaging API charges, infrastructure surcharges and ongoing internal supervision overhead. Plan for 30 to 60 minutes of weekly supervisor auditing to inspect live production transcripts using your evaluation rubric. Language models and operational documentation evolve continuously; production accuracy must be monitored systematically rather than evaluated once.

What should you do next before signing an AI contract?

Before scheduling another vendor software demonstration, complete three actions:

1. Extract 50 genuine past guest enquiries from your messaging logs and have your reservations team draft reference answers linked to verified source documents.
2. Establish formal evaluation thresholds, including a zero-tolerance policy for fabricated pricing statements, and share these criteria with prospective vendors as pre-requisites for your procurement trial.
3. Subject every shortlisted vendor to the identical two-week evaluation schedule, scoring responses across identical data using the four-tier rubric.

If your leadership team requires specialist support to run this evaluation, Azrty can assist. We are a UAE company based in Dubai, and our own team delivers AI consulting, technical evaluation and deployment support for regional clients. Through our AI strategy and readiness engagement, we review your operational processes, data assets and systems to show you exactly where AI will deliver measurable value, what technical prerequisites are necessary, and how to evaluate vendors safely. Any vendor can deliver an impressive demo. A rigorous internal evaluation ensures you select the system that genuinely protects your property.

https://www.azrty.com/blog/how-to-evaluate-ai-software-before-buying-a-hotel-guide

<a href="https://www.azrty.com/blog/how-to-evaluate-ai-software-before-buying-a-hotel-guide">How to evaluate AI software before buying: a hotel guide</a> (Azrty)