Anatomy of an experiment: putting the 'super' in Super Thanks
Oct 06, 2022 – [[read-time]] minute read
Editor’s note by Neal Mohan, Chief Product Officer: A lot of thought goes into all the products, tools, and features we bring to creators and viewers on YouTube. We gather feedback, weigh different options, and think through how to execute at scale. Critically, before making it to the final product launch, we take a special approach to experiments that makes each and every one of these advances possible. In this installment of our innovation series, we give a behind-the-scenes look at the role experiments played in bringing Super Thanks to the platform.
Does anyone remember when we first launched “Viewer applause” back in 2019? Viewer applause — now renamed Super Thanks — is our fourth paid digital good and has given fans an early way to financially show support for their favorite creators directly from a watch page. It was an exciting moment for all of us who’d been focused on deepening the relationship between creators and their audiences, and giving creators yet another avenue to diversify how they make money on YouTube. But how did we get from that first, basic iteration of Viewer applause into the improved version of Super Thanks that we eventually launched to all creators in April of this year? For that, look no further than the experiments we ran to get there.
Why we conduct experiments
First, it’s helpful to understand the way we think about experiments. At the heart of our approach is to learn something new or figure out how to improve an existing experience. We test all kinds of enhancements, ranging from small tweaks that take relatively little time to bigger improvements that demand a much bigger window to run. And we launch experiments throughout the year across all our products and features in development—lasting for as long as it takes to get statistically significant data that meaningfully informs our product decisions.
For a lot of our features, we never stop running experiments as we continually look to make improvements that give a better experience to the entire YouTube community. That’s not just viewers and creators, but also advertisers, media partners, third-party developers, and the music industry.
How we run experiments
The way we approach experiments mirrors the scientific method in a few key ways. We start by coming up with a set of clear hypotheses we want to test. Then all the teams that work together to launch new features — product, user experience, research, and engineering — collaborate on designing the experiments or research studies that can give insight into these hypotheses. Our ultimate goal: to measure the impact of a proposed change and gain better user understanding so that we can shape a product for the better.
Behind all this is a focus on qualitative and quantitative data. Some feature improvements are better evaluated through qualitative assessments like user experience research sessions. But other features are gauged best through quantitative approaches like live traffic experiments.
For Super Thanks, we ran a variety of tests that ranged in length, methodology, and type of data. In particular, experiments helped evolve the design of the Super Thanks’ icon and the addition of more price point options for viewers. Here’s how.
Originally, the main icon for Viewer applause on a watch page was two hands clapping. The idea was simple. In real life, if you enjoy or appreciate a piece of entertainment, you clap. On YouTube, if you like a video? Click the LIKE button. Really liked it? Click on APPLAUD.
There was just one problem. For viewers, the icon didn’t convey that tapping the Viewer applause icon would let them tip their favorite creators. We needed to create an image that would better express the Super Thanks’ value proposition to viewers. So we designed a new icon featuring a dollar sign and a heart.
In a qualitative research session that YouTube conducted between June 18th and 20th of 2020, the majority of participants quickly understood the meaning of the new design. Viewers had a better expectation of the feature’s function, namely that fans could thank their favorite creators through a monetary tip. As one of our testers in the experiment explained, “Because it has a heart and a dollar sign, it looks like one of those things where you can send little bits of cash to say ‘I like this video.’”
To learn more quantitatively, in July of 2020, we launched an A/B experiment for the heart-dollar sign icon for Super Thanks. Results showed 20% fewer clicks than the clapping icon, and purchase conversions (i.e., completed sales) increased by over 50%. This data essentially served as a bridge to understanding users: If more viewers actually bought a Super Thanks via the heart-dollar sign icon, then they better understood it.
We also wondered if giving viewers more options would result in greater revenue. Initially, we only had one price point of $2, and heard from plenty of viewers that they wanted to customize amounts and give more. As we added a few more options, we wanted to develop a pricing strategy for Super Thanks that worked well for our audience.
Instead of being overwhelmed with more options, fans appreciated more pricing flexibility — and it led to Super Thanks generating more money for creators.
In an experiment we conducted in August and September of this year, we offered nine additional price points – ranging from $1 to $500 – to the existing price points of $2, $5, $10 and $50 USD and saw more than a 10% increase in Super Thanks revenue. Instead of being overwhelmed with more options, fans appreciated more pricing flexibility — and it led to Super Thanks generating more money for creators.
Experiments: the road to success
Feedback from experiments has been—and continues to be—a key driver in improvements to Super Thanks for both viewers and creators. As of July 2022, more than 500,000 channels have enabled Super Thanks. And the fan-funding feature continues to build momentum and help creators and artists earn more and diversify their revenue streams.
There will be more experiments to come. Another frequent buyer request is for Super Thanks comments to stand out even more and get more recognition —something we’re hoping to test next year. Creators also want to know when they’ve received a Super Thanks, so we’re aiming to deliver push notifications in the coming months. And Super Thanks is now available in even more countries, something that creators and fans around the world have asked for since we first launched. Earlier this month, we expanded the feature to 30 more countries, including Nigeria, Turkey, and Indonesia.
Experiments for Super Thanks have shown why we’ll continue to make testing the center of product development: They’re indispensable for raising the bar on technical excellence, and also a crucial window into the hearts and minds of our creators and viewers. We’ll continue to be super thankful for this as we continue to bring you the best video platform in the years ahead.