AI Brand Mentions are Unreliable: How Can Visibility be Quantified? 

Recently, every marketer has been wondering the same thing: can we measure brand visibility in AI responses in the same way we track rankings in search? Until recently, the answer to that question hasn’t had any solid data behind it.

But now, thanks to a new investigation from SparkToro, we’ve been given a much needed deeper insight into AI brand mentions and whether or not AI visibility tracking tools are worth your time and money. 

So, let’s take a look at the study, its findings, and what it means for you. 

The Study

Rand Fishkin from SparkToro ran an experiment with the hypothesis that because AI tools produce such randomised lists of brand and product recommendations, and user prompts vary so significantly, attempting to track rankings or visibility is pointless. 

To test the theory, Fishkin, along with Patrick O’Donnell (co-founder of Gumshoe AI), asked 600 volunteers to run 12 different prompts through three popular AI tools: ChatGPT, Claude, and Google's AI, then organised the responses. 

The Results 

The reality of AI recommendations is that it's incredibly rare for the same set of brands to appear in the same order twice. The probability of getting identical lists from ChaptGPT or Google’s AI was under 1% and seeing the same order was even rarer. Claude did offer marginally better results, but inconsistency was still very much prevalent. 

Overall, AIs don’t give a consistent list of brand or product recommendations, and each response was unique in three key ways: 

• List content - which brands appeared varied wildly from prompt to prompt 
• Order - even when the same brand did show, it rarely appeared in the same place each time 
• List length - the number of recommendations was inconsistent from one run to the next 

‍

‍

These systems don’t generate responses like traditional search results. Instead, they predict responses based on patterns learned from data, not by following fixed rules or sorting information in a set way. This means tracking visibility across AI outputs is unreliable. 

That being said, Fishkin’s hypothesis wasn’t entirely correct. 

The study did find that if you run the same prompt enough times, you can get a sense of which brands have a stronger underlying association in the model’s data. So, how often something appears may matter more than where it appears, and it is possible to get a rough measure of visibility when you look at visibility percentage. 

What Does This Mean For Marketers? 

Traditional rank tracking doesn’t apply here. The level of variation undermines most forms of AI visibility tracking and doesn’t provide meaningful insight. Treating AI visibility like traditional ranking is misleading, especially since people ask the same question in many different ways, and even a slight variation in wording can produce different results. 

So, what should you be doing? 

• Evaluating presence - instead of tracking where your brand appears, track how often it appears across many different prompts. Frequency will tell you more than rank 
• Using multiple data sources - AI visibility alone isn’t enough. You need to combine it with traditional brand metrics, competitor analysis, and real customer research to get a clear picture
• Accounting for prompt variety - results change depending on how a user asks a question. You need to be gathering data from multiple prompt variations to gain any real insight 

AI Visibility Does Have Value When Measured Correctly 

AI recommendations aren’t reliable so treating them like traditional search engine results is a wasted effort that can lead to poor conclusions. However, this doesn’t mean AI visibility has absolutely no value, it just means you need to be measuring it in a different way. Concentrate on how often your brand appears and not where it sits on a list to get a better idea of AI visibility. 

If you’re concerned about your brand’s AI visibility, then don’t hesitate to get in touch with us today.