While the article is from 2017, the relevance is high today.
Large adtech companies are coalescing and supporting a single browser framework that gives them a modicum of control around how adblockers are integrated into the extensions, while also funding directly or indirectly realistic browser competition through Firefox (sorry Brave and Vivaldi... still a ways to go market share-wise). The economist in me thinks there is formal and informal coordination to make this ad "market" exist (no one wants ads, really, they want search engines with relevant results) and there is exercise in market power to the detriment of businesses purchasing advertising (and waste to society writ large).
There's a lot more collusion going on here as well. I work in the ad measurement/ad tech industry; it is rife with misaligned incentives.
Ad measurement companies have no incentive to accurately report the incremental sales (i.e., that which would not have happened, had the ad not aired) of the ad they are measuring. Why not? Because their clients, the marketing function at the advertiser, don't get their bonuses paid based on the accuracy of the measurement firm's work product. They get their bonuses (and ultimately salary) based on the success of their marketing efforts. If a company like mine reports that an ad on TikTok for Dran-o didn't sell any Dran-o that wouldn't have been sold anyway, Dran-o's marketers will just find a different ad measurement company.
Furthermore, as a platform, TikTok has to prove to potential advertisers that TikTok is a good platform on which to place ads. So TikTok also has no incentive for accurate measurement.
So, what does a company like mine do? Juice the numbers. Because the company leadership is judged not on accuracy of measurement, but profit.
It's a lot like the bond rating agencies that lead, in part, to the 2008 housing crisis, getting paid by the people whose homework they were grading.
This is an entirely true story, btw. Apparently if you advertise Dran-o to kids on TikTok, you get something north of a $2.00 ROI. Completely ridiculous.
In another anecdote, we measure advertising on YouTube. Google would only let us implement changes to our measurement software if they "improved" the software. (Again, I don't mean accuracy. Neither did Google.)
How do you solve this? Well, you can start by letting the CFO at the advertiser own the ad measurement responsibility. The CFO actually has incentive to understand if the marketers are just pissing away money. The shareholders do too.
Perhaps there's also room for an industry watchdog, paid for by advertising platforms and advertisers, that certified the practices of measurement companies?
Anyway, shareholders are getting fleeced. All so the Valley can rake in beaucoup bucks at the expense of shareholders. With the added benefit of a layer of surveillance apparatus suffusing the internet.
Fun times!
There's actually a lot of scholarship on how advertising's effects are a lot lower than the ad measurement and advertising industry thinks they are. (Go figure that the people who profit from advertising think it works.) Freakonomics Radio episodes 440 and 441 are a good place to start, for those who are interested.
Your point about the numbers being fudged makes intuitive sense, and almost feels obvious, however...
I definitely remember asking my mom to buy liquid plumber when I was about 13 because I had a slow drain in a bathroom she never used. I only realized that slow=clogged from the commercials I saw.
> Your point about the numbers being fudged makes intuitive sense, and almost feels obvious, however...
I can confirm that point on a small scale; it's something I repeatedly mentioned on HN for years. I used to work literally next desk to a small social media marketing agency, that spun out of the company I was employed by. I knew the people running that agency, I got to see how they work, even read and commented on the reports they sent to customers.
It was as GP described, with a twist - the people working in said agency weren't exactly good with numbers. They understood, however, the idea that a graph going up and to the right is good. Despite the numbers having little connection to reality, they wrote reports that highlighted how successful their campaigns were. Those reports were read by customers equally proficient in mathematics, and similarly good at tracking sales on their own side. Bottomline: money changed hands, both sides were happy, the whole thing was still completely untethered from reality.
Hum... You are claiming the GP's claim that the reported numbers are higher than the real ones is false because you once on your lifetime got evidence that the real number is non-zero?
I think he's claiming that showing drano ads to kids could have a positive ROI based on lived experience. To be fair, while the parent post was compelling, it didn't have any "references" either.
You did not understand the original comment. They implied it without saying it in as many words. Here's how: they presented the fact that Drano apparently gets a $2 ROI showing ads on TikTok as evidence to back up their claim that the companies measuring the effectiveness of ads are making shit up. You can argue whether that's the intended communication, but clearly that's the communication I received (and probably not just me, hence the other person explaining that Drano ads for kids could have a positive ROI).
As a side note, I must say I feel this conversation took on a negative (and condescending?) tone.
Yes, it was definitely just a fun anecdote. Your example of advertising drano to kids made me laugh, and then I remembered being the reason we bought drain cleaner in my home, which made me laugh again, so I shared it.
You work in the ad measurement/ad tech industry, so this should be an easy question for you to answer.
When your solutions are "juicing the numbers" how are they addressing the miscorrelation between third party tracking tools for both impressions and clicks on the platforms, but also what would come through in analytics in the site or app?
Without once mentioning a pixel and how that all works, you claim that numbers are being inflated, how so? Are you talking about over-reporting, mostly caused by impression conversions?
If you are "getting a $2 ROI showing Dran-O ads to kids on TikTok" there have to be sales occurring in the first place in order for TikTok to claim credit for them.
Can you also go into more detail about what tracking solution for YouTube you needed to contact Google about? That is confusing to me.
I've been in the ad world for almost 8 years now. If a platform is over-reporting conversions I have other ways to audit that.
<edit> I should point out that we were measuring brick & mortar sales, not clickthroughs. So no cross site tracking involved. Maybe that's the easiest way to clear things up for you? </edit>
If you look at a distribution or histogram of ROIs from our thousands of studies, you will see that it approximates a gaussian distribution...right up until the left hand side hits $1.00, where it looks like the distribution hit a brick wall.
Same thing for a distribution of incremental lifts, except that the brick wall is right above 0% lift.
It's a discontinuity that shouldn't exist, like the Polish high school exit exams. See if you can guess what the minimum score to pass* was:
Except, unlike that distribution, there is virtually nothing to the left of the discontinuity at my company.
Without talking about the tech behind it, what I can say is that the fraud is not occurring on the impressions and tracking side. It's in the statistical tools that infer the incrementality of the purchasing behavior. Seriously, I urge you to listen to the Freakonomics episodes I mentioned.
*The minimum score was 30. This is pretty damning evidence that teachers were juicing the tests of kids who filed by one or two questions.
Thanks for the info. Doing brick and mortar sales attribution is a whole different ball-game.
Frankly the number of dots that have to be connected between different systems in order to make that stuff work is too difficult for me to trust the data. If I'm going to trust a black box, I'm not going to trust programmatic exchanges and attribution vendors. I'm going to trust platforms that have vertical integration at as many stages as possible.
> we were measuring brick & mortar sales, not clickthroughs
I think this is probably where you get the room to juice the numbers like this. If you were tracking online conversions, people would easily be able to compare with an incrementality study run by the provider (say Google) and see that you are lying/wrong.
Replying here, because I cant reply to your comment several levels down.
I can see how you would have that perspective as a third-party analyst of conversion.
It's actually easy to do if you are the exchange/sell-side provider of the advertising space. In your logic, where the advertiser running the incrementality study would have won the auction, just randomly remove their bid from some won auctions, have the second highest bidder win, and then track that user as the "control" for conversion purposes vs. users you did display the ad to. Voilà!
That's not observation. That's statistical inference. I am referring, strictly, to direct, empiricial observation. With any kind of statistical inference, you get into a pissing match about assumptions, model quality, etc... Is the "control" a good stand-in for the behavior of the "test", etc.
Incrementality is able to be estimated, but unobservable. And therein lies the potential for fraud.
Sure, if by unobservable you mean in the same way that we can't truly "observe" the population average height of humans, even if we can probably get a pretty darn good estimate.
You can observe the results of the experiment and then make statistical inference about the "true" incrementality. You don't even have to perform inference necessarily, you can just provide the direct results of your experiment to the advertiser and let them do the inference.
But I do fail to see how a third-party could possibly infer this measure, given that they have no capacity to do a true incrementality experiment on the platform.
Are you running the incrementality study, or is your provider (e.g. Google)? Because one of the points here is that providers also have an incentive to fudge the numbers - they're not in business of being accurate, they're in business of making it seem their services help you.
To riff off the average height example - you can get a pretty darn good estimate based on sampling and good statistical practice... unless the people doing the sampling have a reason to bias the measurement, or - if you're sampling on your own - people who control where you go and who you meet...
Oh that could definitely be the case, although I'll note that the larger actors have more to lose and are at more risk of being caught for engaging in fraud.
I think my usage of "you" was confusing because I work in sell-side advertising.
You nailed this. Tirole discusses this in his paper hierarchy on bueracracies. Your internal stakeholders aren't always aligned to the company, sometimes it is to a vendor.
> It is impossible to actually observe what would have happened, had there been no advertisement, because there in fact was an advertisement.
I can see how you would have that perspective as a third-party analyst of conversion.
It's actually easy to do if you are the exchange/sell-side provider of the advertising space. In your logic, where the advertiser requesting the incrementality study would have won the auction, just randomly remove their bid from some won auctions, have the second highest bidder win, and then track that user as the "control" for conversion purposes vs. users you did display the ad to. Voilà!
This is how Google in-house incrementality studies are run, for instance.
If you aren't the one supplying the ad space or the exchange (ie. Google or FB), then I don't believe you can actually measure incrementality on your end.
You just need the data on who does (and by extension, doesn't) see the ad, as well as the shopping behavior of those same people. Both of which can be had by someone who isn't supplying the ad space or the exchange.
No, you misquote me. I said that is is impossible to observe incrementality. It is rather trivial to estimate incrementality, statistically, but all statistical models produce estimates. Not observations.
How do you get the data if you're not a supplier? You contract for the supplier.
There's a book on this topic that explicitly draws the parallels to the housing market meltdown, "Subprime Attention Crisis." There is also a professor at Columbia, Kinshuk Jerath, who has written explicitly about these misaligned incentives.
Brave advocates should spend most of their energy going after Chrome and Safari users. If Brave can convert them better than Firefox can (which, sadly, seems to be quite a low bar), they'll overtake soon enough.
Attempting to convert Firefox users is not a long term strategy, and doesn't do as much or anything to create a better web.
I had been using Firefox for something like 15 years before I switched to Brave a few months ago. I was really skeptical of the crypto stuff Brave was doing and I still am, but at least they're innovating and trying different financing models. Meanwhile Firefox spent the last few years removing features that kept me using it over Chrome: rss, compact density, bookmark descriptions, etc. On top of that Mozilla gets 90% of their money from Google, and they still don't have a way to donate directly to Firefox, the only option is to donate to Mozilla and watch them spend most of the money on silly things like Pocket and other wasteful initiatives, while the people in charge just line their pockets.
20 years on FF here, I just switched recently for the exact same reasons as you. I got tired of their antics (not political, just goofy stuff like Pocket) and feature cuts. I tested every browser available, and came down to my preferences being Edge and Brave. Can't say I had an inkling of attraction to Vivaldi, Opera, or any of the rest.
I'm currently using Edge with uBlock Origin Lite. I can't tell the difference between uBO and uBOL. AdGuard MV3 also seems to work fine but I've used uBO so long that I'm sticking to it out of comfort.
That said, I keep Tor installed for as close to perfect privacy as possible. It feels academic though, as everything I do is pretty pedestrian and I generally avoid Google's services. I do prefer Edge to Brave, there's many features that I like here, but it's my backup plan if somehow Edge's adblocking doesn't work as well as it does today.
Even though I don't use it daily, I promote Brave to anyone that would normally be inclined to use Firefox. Otherwise, I push native browsers for all the inherent advantages they have. For me that's Safari on iOS and Edge on Windows. There's no doubt that built-in adblockers like Brave has is the future, they're just ahead of everyone in many ways. For me, Brave is what Firefox should be by now, in all regards.
Likewise. I loved Firefox and found their containers super useful. I've not been shy in sharing that their summer 2020 mobile rollout was so poor I moved away from it on both mobile and desktop (I often get flack from Firefox diehards for mentioning this though). Ah well. Chrome profiles for now, unfortunately.
I like Vivaldi on mobile with built in adblocking and some bit of control for text size (aging has not been kind to my eyes, mobile text is too small most of the time). Vivaldi hasn't made it pass evaluation stage on desktop for me as it often crashes when I try it (linux distros). (ed: tried it years before)
Browsers are going to continue to be a central part of our OS -- no stopping that. I hope the companies can be taken to task for colluding and that innovation is not stifled.
The last time I evaluated it in Fedora (maybe 3 years now?) it was crashing under moderate use, and haven't had time to check recently. Perhaps when some time clears up I should give it another go.
I edited my original comment, which give Brave kudos for their growth, because I wanted to acknowledge their fight against an oligarchic adtech market. Perhaps ironically, I removed the comment because I thought it could lead to a pro/anti Brave fight that I wasn't intending to ignite. (I'm not a Brave user due to their prior business model and feature rollout around crypto).
It would be nice to see innovation in this space but web standards implementation requirements may make that bar too high? I'm not a browser dev, just an armchair techie.
What are you referring to? Which "Large adtech companies" support a framework that gives them "control" over adblocking, and how? If you're referring to MV3, then I think you've blown things significantly out of proportion
I wouldn't call it "blown things significantly out of proportion", not unless you're okay with compromised ad blocking capabilities, which some people seem to be in the name of security which is oxymoronic.
I'm saying you're blowing things out of proportion by referring to companies, plural, and saying that any other ad tech company is at all associated with what Google is doing. Even if they stand to gain from it, they have absolutely no input into the process (it's not even an open web platform like other Chrome changes!). And as every single thread on the subject on HN has made excruciatingly clear, Google has nothing to gain from the MV3 changes ad-wise, since they serve all of their ads from the exact same domain they've always served them from, and they can be blocked with 1 (one) single filter rule. The limitations in MV3 primarily affect the sketchiest and newest adtech providers, exactly who Chrome is already seeking to block for user experience reasons and who Google Ads probably won't work with. There's no reason to believe that the MV3 filter changes are the result of any collusion or malicious intent inside of Google, unless for some reason they feel like taking revenue share AWAY from their ads business by helping companies they hate.
Large adtech companies are coalescing and supporting a single browser framework that gives them a modicum of control around how adblockers are integrated into the extensions, while also funding directly or indirectly realistic browser competition through Firefox (sorry Brave and Vivaldi... still a ways to go market share-wise). The economist in me thinks there is formal and informal coordination to make this ad "market" exist (no one wants ads, really, they want search engines with relevant results) and there is exercise in market power to the detriment of businesses purchasing advertising (and waste to society writ large).