August 2, 2024
As a direct result of this move, Google is now the only mainstream search engine that provides search results from Reddit.
In updating its robots.txt file to prohibit most search engines and AI crawlers from accessing its content, it appears as though Reddit is sending a message: if rivals such as Bing or DuckDuckGo would like to show Reddit posts in their search results, they’ll need to pay for the privilege, just like Google does.
The news that Reddit was ramping up its crackdown on web crawlers was first reported by 404 Media, who were among the first to notice that Reddit had started to block search engines from returning recent posts and comments in their results.
Practically speaking, this means that if you tag ‘site:reddit.com’ onto the end of your search query on Bing or any other search engine that isn’t Google, you’ll either see nothing at all, some outdated Reddit posts that are no longer relevant, or you’ll see a link that reads “We would like to show you a description here but the site won’t allow us.”
Initially, many thought that Reddit had made the move because of the aforementioned deal with Google. Interestingly however, Tim Rathschmidt, a spokesperson for Reddit, denied this. He told the Verge that the development was “not at all related to our recent partnership with Google”.
He added that “we have been in discussions with multiple search engines… We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”
So, at face value at least, it appears as though the move isn’t tied to any kind of exclusivity agreement that Reddit has with Google. However, it should be noted that Google’s deal with Reddit is reportedly worth $60 million annually. Added to this, Google has a ‘hidden gems algorithm’ that helped Reddit gain a lot of traffic. Resultantly, Reddit is currently dominating Google’s search results and Google has also tested providing ‘special treatment’ for Reddit. As a result, it’s safe to say that the relationship between the two companies is strong.
It’s worth noting here that Reddit currently has a strict policy against ‘scraping’ and web crawlers. After all, as we mentioned earlier, Reddit has tweaked its robots.txt file, which directs AI crawlers to which pages they can or can’t glean information from.
At the time, Reddit said it took this move because it was “a signal to those who don’t have an agreement with us that they shouldn’t be accessing Reddit data”, according to Ben Lee, Reddit’s chief legal officer.
Immediately following this, Microsoft spokesperson Caitlin Roulston said, “Microsoft respects the robots.txt standard and we honour the directions provided by websites that do not want content on their pages to be used with our generative AI models,” adding that Bing stopped crawling Reddit when the platform updated its robots.txt file on July 1st.
Of course, all websites rely on search engines to generate traffic. As a result, the news that Reddit is blocking a range of popular search engines from viewing its content is incredibly interesting.
After all, by stopping most search engines from scraping its pages with AI bots, Reddit is essentially walling off its content in order to keep it out of search results.
However, to an extent, the move is also unsurprising. After all, over the past year, Reddit has sought to protect its data in order to appease its investors and open up new sources of revenue.
For example, the company has also made its API more expensive for some third-party developers and it reportedly threatened that it would block Google’s crawlers if the company didn’t stop using the platform’s data to train its AI for free.
It’s also worth pointing out that such a move is timely, as many searchers have turned to platforms such as Reddit and Quora in recent months as chatbots have begun to fill the internet with questionable, unethical and at times dangerous content.
In a quest to avoid AI-generated results, many have started appending their searches with the term ‘Reddit’. Now, users who search on Bing cannot do this and will instead be forced back to the Google ecosystem.
Well, for now at least. Whether Microsoft/Bing strikes a deal with Reddit remains to be seen. Overall, we’d guess this is likely. After all, this move from Reddit only increases Google’s monopoly power, which will be of great interest to regulators. This was pointed out by Ori Zilbershtein over on X:
So, watch this space. If Reddit does strike another lucrative deal with Bing, it will be interesting to see if other platforms like Quora follow Reddit’s lead.
