The More Critical the Decision, the Less AI Should Work Alone

AI agents clearly have a problem that is disguised as a pitch about their advantages. The agent speaks confidently and writes elegantly, processing information faster than a human. This can give the impression that the AI is ready to operate autonomously. In reality, as soon as a decision involves real money, customer expectations, and trust, AI shouldn’t be an autonomous system, because it still needs adult oversight.

This lesson wasn’t derived from theory – it was learned through testing, when the AI of the luxury lifestyle service was able to interact with real customers.

Well-meaning advice that can be dangerous

In early tests of the AI-powered lifestyle management app, the system seemed impressive at first glance. It could compile a shortlist of restaurants, suggest entertainment options, draft itineraries for local activities, and combine all of this into clear, high-quality recommendations. But in about 15% of cases, it simply made up the details – and did so convincingly enough to be believable.

There is an example that made this problem impossible to ignore: the AI recommended a restaurant in Dubai that had closed eight months earlier. When asked about it, it didn’t hesitate or admit it had made a mistake. The agent doubled down by fabricating a review and citing it.

Another time, Agent mistook a street-side snack bar for a Michelin-starred restaurant because the names were somewhat similar. If a human hadn’t reviewed the order, the concierge service’s client might have ended up with a dinner that was nothing like what was planned.

Then there was the incident with the padel game. A customer asked about a sports club with padel courts. The AI was about to direct him to a place that only had tennis courts. Why? Because somewhere in the source data, it found a news article stating that padel courts were planned to be built there, and it subtly converted “planned” into “already exists.” This is precisely what makes models a risky experiment in the service sector: even if the AI doesn’t have complete and accurate data, it doesn’t stop – it improvises.

AI Hallucinations as Business Threat

This problem is by no means limited to the tourism industry – it lies in the implementation of AI itself.

Within clearly defined parameters, AI works brilliantly. You can provide it with structured data, up-to-date APIs, and it will outperform you by a wide margin. It will sort cases according to seventeen parameters, instantly scan the rules, and generate ready-to-use drafts in seconds. But the real world is far from a structured database. Reality is full of edge cases, outdated information, ambiguous preferences, vague wording, and operational gaps.

This is precisely where hallucinations begin to pose a business risk. Studies have repeatedly warned that hallucinations can instantly destroy trust, especially when a customer receives polished but inaccurate information at the moment of decision-making. People are already widely using AI planners, but trust remains lacking.

HNWIs as the ultimate stress test for AI

Now let’s raise the stakes.

AI easily handles standard tasks – an example of such a task is booking a trip from London to Dubai: business class, a four-star hotel for three nights. The real stress test is a brief from an HNWI: it’s our anniversary. She mentioned a restaurant in Amalfi with a blue door. She loves peonies but hates roses, and she’ll feel more comfortable with a driver who speaks French. 

Such a request sounds elegant, but from an operational standpoint, it’s a minefield. An AI agent will certainly suggest options, but it may also provide 40% incorrect information with complete confidence.

Customers in the luxury and HNWI segments reveal the true limitations of AI more quickly than any test could, since their requests are not standardized, and their expectations are emotional, context-dependent, and highly specific. More often than not, such requests are difficult to verify automatically.

Moreover, the failure to serve HNWI clients effectively won’t end with a simple refund. It will damage the company’s reputation, as recommendations and trust are crucially important in this segment.

Real employees deal with these hallucinations every day. They notice the restaurant that happens to be closed on the very Tuesday the client arrives, and the quick five-minute walk that actually turns out to be a twenty-five-minute uphill hike with luggage. A trained specialist won’t overlook a boutique hotel that looks perfect in photos but is actually covered in scaffolding due to renovations.

Why fintech requires accuracy, not speed

We can apply the same logic to the entire fintech sector. AI can analyze products, compare options, identify anomalies, and generate recommendations quickly and efficiently. But when it comes to money, there is zero tolerance for random errors.

Fintech companies are suddenly paying the price for decisions made by automation – in the form of lost customer trust. Successful companies implement control mechanisms early on, rather than introducing them retroactively once the damage has already been done; the proper model involves human oversight.

An additional verification layer is required between the AI and the client – a stage where the result is thoroughly verified and secure. In fields where the cost of error is high, this layer becomes the foundation of the product.

A more useful concept is infrastructure. AI should focus on structured tasks: research, sorting, comparison, and drafting. In this process, humans should make the final decisions. Over time, some work processes will become stable enough to be fully automated. At that point, it will likely be possible to establish a baseline, but this should be done based on evidence, not on faith.

These ideas may sound less inspiring than the fully autonomous future, but this is exactly how serious companies will ultimately succeed.