Google Has No Solution For Misleading AI Overviews

AI Overviews are one of Google’s newest features. They were announced at I/O 2024 alongside a long list of other AI-powered updates. Now, AI Overviews are making headlines for all the wrong reasons.

We recently wrote a couple of blog posts about this new feature, examining how AI Overviews work and assessing their impact on the quality of search. However, things are continuing to move at a rapid pace.

Over the last few days, we’ve seen dozens of headlines highlighting the dubious, misleading and sometimes dangerous answers being produced by AI Overviews. 

Here’s a recap of recent events, the backlash and the issues surrounding AI Overviews.

Sundar Pichai and the Big AI Overviews Problem

As discussed in our previous blog posts, there is ongoing concern surrounding AI Overviews, their impact on search and their ability to provide accurate answers. However, this is not a view that Google CEO, Sundar Pichai, seems to share.

In a recent interview with The Verge, Pichai was asked about the impact of AI Overviews. He responded by saying “I remain optimistic” and added:

“People are responding very positively to AI Overviews. It’s one of the most positive changes I’ve seen in Search based on metrics. But people do jump off on it. And when you give context around it, they actually jump off it. It actually helps them understand, and so they engage with content underneath, too.”

He also discussed “hallucinations” and called them an “unsolved problem.” According to Pichai, these hallucinations are an “inherent feature” of Large Language Models (LLMs) and it’s this that “makes these models very creative.”

Pichai explained:

“It’s incredibly creative. But LLMs aren’t necessarily the best approach to always get at factuality… There are still times it’s going to get it wrong, but I don’t think I would look at that and underestimate how useful it can be at the same time. I think that would be the wrong way to think about it.”

AI Overviews Continue To Spiral

Despite Pichai’s positive stance, several users have shared examples of AI Overviews providing questionable answers. These range from failures to understand the nuance and context of content, such as taking satirical articles at face value, to providing potentially harmful and unsafe answers.

There’s one particular answer that’s gone viral: Google’s suggestion of adding glue to pizza. An X user shared a screenshot of the search ‘cheese not sticking to pizza’ and the response from AI Overview. This included the sentence ‘You can also add about ⅛ cup of non-toxic glue to the sauce to give it more tackiness.’

There have also been other false answers, from calling pythons mammals to finding no African countries starting with the letter K. 

Google responded to these instances, calling them “isolated examples.” The spokesperson also said:

“The vast majority of AI Overviews provide high quality information, with links to dig deeper on the web. Many of the examples we’ve seen have been uncommon queries, and we’ve also seen examples that were doctored or that we couldn’t reproduce. We conducted extensive testing before launching this new experience, and as with other features we’ve launched in Search, we appreciate the feedback. We’re taking swift action where appropriate under our content policies, and using these examples to develop broader improvements to our systems, some of which have already started to roll out.”

However, it’s not just AI Overviews that are posing a problem. According to posts on X, some of these responses are also being produced within featured snippets.

Ongoing Tracking and Monitoring Issues

In addition to false answers, the data from AI Overviews is difficult to track. This is because the feature is only appearing for users who are signed in to their Google Account. Even then, Google states that this ‘experimental generative AI feature’ is only available for ‘some’ searches. 

Why is this a problem? Firstly, it makes it more difficult for third-party monitoring tools to secure reliable, widespread data. Secondly, it means these complaints are only coming from a small fraction of Google users. When the rollout is fully complete, it’s likely we’ll see a lot more complaints as an increasing number people will start to see incorrect results.

The good news? Google is using feedback to make changes to their system. Additionally, AI Overviews are yet to roll out to all users. This means Google still has time to fix these issues, prevent the floodgates from opening and limit their impact on search.

Do you have questions about AI Overviews? Or concerns about their impact on search? Then speak to our SEO experts today. We can provide insights on Google Search and help you navigate this new AI-powered age.