By Barbara Macdonald, Product Manager, Paid Digital Goods.
Editor’s note by Neal Mohan, Chief Product Officer: A lot of thought goes into all the products, tools, and features we bring to creators and viewers on YouTube. We gather feedback, weigh different options, and think through how to execute at scale. Critically, before making it to the final product launch, we take a special approach to experiments that makes each and every one of these advances possible. In this installment of our innovation series, we give a behind-the-scenes look at the role experiments played in bringing Super Thanks to the platform.
Does anyone remember when we first launched “Viewer applause” back in 2019? Viewer applause — now renamed Super Thanks — is our fourth paid digital good and has given fans an early way to financially show support for their favorite creators directly from a watch page. It was an exciting moment for all of us who’d been focused on deepening the relationship between creators and their audiences, and giving creators yet another avenue to diversify how they make money on YouTube. But how did we get from that first, basic iteration of Viewer applause into the improved version of Super Thanks that we eventually launched to all creators in April of this year? For that, look no further than the experiments we ran to get there.
Why we conduct experiments
First, it’s helpful to understand the way we think about experiments. At the heart of our approach is to learn something new or figure out how to improve an existing experience. We test all kinds of enhancements, ranging from small tweaks that take relatively little time to bigger improvements that demand a much bigger window to run. And we launch experiments throughout the year across all our products and features in development—lasting for as long as it takes to get statistically significant data that meaningfully informs our product decisions.
For a lot of our features, we never stop running experiments as we continually look to make improvements that give a better experience to the entire YouTube community. That’s not just viewers and creators, but also advertisers, media partners, third-party developers, and the music industry.
How we run experiments
The way we approach experiments mirrors the scientific method in a few key ways. We start by coming up with a set of clear hypotheses we want to test. Then all the teams that work together to launch new features — product, user experience, research, and engineering — collaborate on designing the experiments or research studies that can give insight into these hypotheses. Our ultimate goal: to measure the impact of a proposed change and gain better user understanding so that we can shape a product for the better.
Behind all this is a focus on qualitative and quantitative data. Some feature improvements are better evaluated through qualitative assessments like user experience research sessions. But other features are gauged best through quantitative approaches like live traffic experiments.
For Super Thanks, we ran a variety of tests that ranged in length, methodology, and type of data. In particular, experiments helped evolve the design of the Super Thanks’ icon and the addition of more price point options for viewers. Here’s how.
Originally, the main icon for Viewer applause on a watch page was two hands clapping. The idea was simple. In real life, if you enjoy or appreciate a piece of entertainment, you clap. On YouTube, if you like a video? Click the LIKE button. Really liked it? Click on APPLAUD.
…
Experiments: the road to success
Feedback from experiments has been—and continues to be—a key driver in improvements to Super Thanks for both viewers and creators. As of July 2022, more than 500,000 channels have enabled Super Thanks. And the fan-funding feature continues to build momentum and help creators and artists earn more and diversify their revenue streams.
There will be more experiments to come. Another frequent buyer request is for Super Thanks comments to stand out even more and get more recognition —something we’re hoping to test next year. Creators also want to know when they’ve received a Super Thanks, so we’re aiming to deliver push notifications in the coming months. And Super Thanks is now available in even more countries, something that creators and fans around the world have asked for since we first launched. Earlier this month, we expanded the feature to 30 more countries, including Nigeria, Turkey, and Indonesia.
Experiments for Super Thanks have shown why we’ll continue to make testing the center of product development: They’re indispensable for raising the bar on technical excellence, and also a crucial window into the hearts and minds of our creators and viewers. We’ll continue to be super thankful for this as we continue to bring you the best video platform in the years ahead.
Taken from: https://blog.youtube/inside-youtube/how-its-made-putting-super-in-super-thanks/