I asked AI to rank popular travel blogs for their authenticity, here's what I learned

Hi Ian,

Thank you very much for your thoughtful comments and for taking the time to replicate the experiment yourself.

Your observations are very accurate and they actually match my later experiences when I continued testing how different AI systems behave in similar scenarios. I ran additional experiments using the paid Gemini model in several modes (Fast, Thinking, and Deep Search), and the results confirmed many of the limitations you pointed out.

However, in the article I deliberately focused on the simplest possible model: the free AI integrated directly into Google Search.

This choice was intentional. Most internet users (roughly 95%) rely on answers generated directly inside the search engine interface and never use advanced paid language models for deeper analysis. Because of that, my goal was not to obtain the most objective or technically rigorous ranking possible, but rather to observe what kind of answer a typical user actually receives today.

In other words, I wanted to understand how the “default AI layer” of the web currently interprets signals of first-hand travel experience — including all its weaknesses and constraints.

I also hoped that publishing the methodology would encourage others to repeat the experiment across different models, browsers, prompts, and geographic locations, because these factors clearly influence the results. Your comparison using Claude is exactly the kind of independent verification I was hoping to see, so thank you for doing that.

From my additional testing it appears that even advanced paid models often do not directly analyse the full content of the evaluated websites in real time. Instead, they rely heavily on training data, previously indexed material, and already existing signals available across the web. In many cases they synthesise patterns rather than performing a full live evaluation of each source.

Interestingly, even when explicitly instructed to read and analyse a complete website, models frequently respond based on partial representations rather than a comprehensive review of the actual page content. This suggests that what we are observing is not a traditional audit process, but a probabilistic reconstruction based on prior knowledge signals.

For this reason I fully agree with your comment regarding visual authenticity scoring. Without an agent capable of actively browsing websites and analysing images directly, such evaluation cannot be treated as literal verification.

At the same time, I think this limitation is part of what makes the experiment interesting. The goal was not to measure objective authenticity itself, but to observe how current public AI systems interpret authenticity signals when asked this type of question.

And since these systems increasingly function as a first layer of discovery for readers, understanding their behaviour may already be relevant for travel publishers today.

Finally, regarding the computational aspect — I suspect another practical constraint is efficiency. A truly deep, page-by-page analytical assessment across multiple travel blogs would require significantly more processing time and resources than what real-time conversational AI systems are currently designed to provide at scale. Most users expect answers instantly, not after twenty minutes of structured crawling and evaluation, so models optimise for speed and plausibility rather than exhaustive verification.

Thanks again for your valuable feedback. It definitely helps refine the direction for a possible second version of the methodology.