When AdTech Misrepresents Their Own A/B Testing | R bloggers

Introduction: I took a break from blogging for about eight years and focused on children and related tasks. This year I’m going to try to start over.
My goal is to write about (advertising-related) research and how practitioners can apply the insights.

Today’s article comes from the “International Journal of Research in Marketing” and describes how major Ad Tech companies (Google / Meta) are miscommunicating their methods and potentially inflating the results of advertising A/B testing.

Takeaways for advertising practitioners?

1. Treat platform “A/B testing” with skepticism: If Google or Meta say they are going to do an A/B test, assume it is an observational analysis unless they specifically confirm proper randomization. Don’t rely on these results for high-stakes decisions.
2. Understand the limitations: Observational analysis can still be valuable, but you must understand its limitations. It can show correlations, not causation. Use it for insights, not final conclusions.
3. Push for better options: If you’re a large advertiser, push your platform representatives for truly experimental testing options. The technology exists, it’s just not offered.
4. Consider third-party solutions: Valid A/B testing may require you to use third-party tools or build your own testing infrastructure outside of the platform’s built-in tools.

Long version:

I recently came across an article by Bögershausen, Oertzen, and Bock (2025) titled “On the persistent mischaracterization of Google and Facebook A/B tests,” which reveals something important about how major advertising platforms represent their testing tools. The article shows that Google and Meta systematically misrepresented the nature of their A/B testing capabilities to researchers and advertisers.

Here’s the core problem: Google and Meta present their A/B testing tools as running clean, randomized, controlled experiments. They position these as true experimental designs where users are randomly assigned to different ad variations, allowing advertisers to compare performance in a scientifically sound way. But the reality is very different.

The authors analyzed what these platforms actually do and discovered that their “A/B tests” are not real experiments at all. Instead of random assignment, these tools typically use non-observational approaches that do not make good control groups. Users are not randomly assigned to treatment conditions; they only see different ads based on the targeting determined by the platform’s algorithm.

This matters because real A/B testing requires randomization to establish causal inference. Without proper randomization, you can’t attribute performance differences to the ad variations themselves. There may be systematic differences in who sees what ad, confounding factors, or other biases. The platforms essentially sell observational analyzes dressed up as experimental design.

Why would the platforms do this? The article suggests that it is partly a communication problem; the tools are marketed as “A/B testing” because that’s a familiar, comforting term to advertisers, even if the actual methodology doesn’t match what that term means in experimental design. It’s also probably easier to sell “we run an A/B test” than “we show different ads to different people and analyze the results.”

The problem is serious for several reasons:

First, it misleads advertisers who think they are conducting scientifically sound experiments. When an advertiser sees Google or Meta say “run an A/B test to compare your ads,” they expect good randomization and causal inference. But they don’t understand that; they receive observational analyzes with all their limitations.

Second, it creates false confidence in marketing decisions. Advertisers make budget allocation and creative decisions based on these “A/B testing” results, believing they have experimental evidence when they do not. This can lead to suboptimal marketing strategies because the fundamental validity of the results is compromised.

Third, it creates a credibility problem for the platforms. When researchers investigate how these tools actually work and discover that they are not real experiments, it undermines trust in the platforms more broadly. The article notes that this misrepresentation has persisted for years at both companies, despite researchers pointing out the problems.

The authors say (and I completely agree) that Google and Meta are doing two things:

1. Communicate more accurately about what their A/B testing tools actually do. Be clear: these are observational analyzes and not randomized, controlled experiments. Stop using terminology that implies good experimental design when that is not what is happening.
2. Offer truly experimental options. Ideally, the platforms should provide advertisers and researchers with a way to conduct true randomized A/B testing, correctly assigning users to different ad variations in a truly random manner. This would provide the opportunity for valid causal inferences when necessary.

This is especially interesting because it is not about technical complexity; the platforms have the infrastructure to run good experiments (they do this internally all the time). It’s about transparency and offering advertisers the right tools.

The article reminds us that we need to be critical of what platforms tell us about their tools, even – or especially – when the terminology sounds familiar and reassuring. “A/B testing” has a specific meaning in experimental design, and marketing platforms should not co-opt that term for something fundamentally different.

Reference: Bögershausen, A., Oertzen, T. v., & Bock, K. (2025). On the continued mischaracterization of Google and Facebook A/B testing.

Note: AI-assisted content.

Related