#Growth 8: How to Approach Experimentation: Bite-sized Changes vs. Big Swings

Media Thumbnail
  • 0.5
  • 1
  • 1.25
  • 1.5
  • 1.75
  • 2
This is a podcast episode titled, #Growth 8: How to Approach Experimentation: Bite-sized Changes vs. Big Swings. The summary for this episode is: Today on #Growth, Matt is talking all about experimentation. Specifically how to approach changes you might make to your site or product – bite-sized or incremental changes vs. going all out with big swings. Matt breaks this down with a real-world example you've probably faced yourself – updating your homepage. There are pros and cons to both and Matt lays it all out here.

Matt Bilotti: Hello, and welcome to another episode of# Growth. I am your host, Matt Bilotti, the guy with the beard, although by the time this episode airs, the beard will probably be gone, because I mostly grow it out as a winter coat type thing to keep me warm on my walks to the office from home, but as soon as it starts getting warmer, I buzz the whole thing off. So anyway, I digress. Today, I want to talk about two different ways to approach changes or experimentation, and the two different ways are bite- sized changes, or big swings. So let's take this in the frame of an example. So let's say you want to improve the conversion rate of your homepage. And so, you can make a bite- sized change and you could change the CTA on the page to say," Get started," whereas it used to say," Sign up for free." Right? So you can make that change, it's an incremental change, and at the end of the experiment, you'll really easily be able to point back to it and say," This number moved because we changed this copy," right? You can isolate perfectly what the thing was that changed. What your other option is, is to do what I call a big swing, so this is much more of an overhaul. So instead of changing the CTA on the homepage, you change basically everything, right? You change all the variables, you design a completely new page, you have a different headline, different CTAs, different colors, all that stuff, and at the end of it, it gets a little bit harder to isolate exactly what it was that made the difference. So, let's say you ran a test and it was 5% better, it's hard to say was it the combination of all those things, was it the headline that people connected with? However, you're far more likely to see a bigger change than when you make an incremental change, right? An incremental change or a bite- sized change is going to give you an incremental improvement, a big swing or a complete overhaul is way more likely to net you a better result. So I want to talk a little bit about one example that I have from here at Drift, and then I also want to talk about how you choose which one to do, and when to think about that. So we have a chat widget, if you've ever signed up for Drift or saw it, you probably have, because you're listening to Seeking Wisdom. And the chat, which is installed on our website, it has at the bottom a link that says," Both by Drift" or" Powered by Drift," and clicking on that would take you back to landing page on Drift's website to recommend that you sign up or chat with us, or check it out. So we had those pages going to drift.com/ poweredby, that was the URL. And so, we looked at it and said," This is actually one of our higher trafficked pages, and no one's done anything with it since we first built it," right? We first built the landing page two and a half years ago, we threw it up there, we grabbed some copy that we had from the homepage, and made a little bit more of a personalized type experience coming from a chat widget, and we put it there, and it was great. And we're looking at this and saying," Wow, conversion rate is in the low single digits. Is there any way that we can get this higher?" So we have two options, right? We can make bite- sized changes and change headline, change copy, that kind of thing, or we could take big swings. And we said," You know what? We're at a starting point, so let's take a big swing," right? A big swing is going to give us a new starting point, rather than saying that," Oh, we probably had some good stuff to begin with," or maybe we were looking at it and saying that the benchmark of this conversion rate is much lower than what we would expect, maybe that's something that you're doing when you're looking at this, then it's time to take a big swing, because you'll get a bigger result. So what we did on those pages, I'll tell you exactly what we did. We had a version where we removed as much friction as possible. So this used to be just a regular landing page, but we said," What do we want people to do on this page?" Right?" We want them to sign up for the product, because they just experienced the product, and what's the best way to do that?" Well, they show up to this page right now, and it's pretty easy to get distracted, so what if we removed the header? What if we removed the footer? What if we made it so you can't scroll on the page? And what if all that there was to do was to enter your email to sign up, and copy that headline that said," Want a chat widget like you just saw on blank. com?" That's not the exact copy that we used, but it's pretty similar. And we rolled that out as a big swing saying," Eh, if it doesn't work, whatever, we'll try something else and build off the other version that we had before." However, we actually wound up getting a three to four times in increase in the conversion rate, and that's because we took those big swings. And that to us was a moment where we said," Huh, now we have a new starting point and the conversion rates just jumped a ton. We probably don't need to take another big swing, because this thing exceeded all of our expectations, and now let's try some bite- sized changes." So then we took this new version and we tried changing the headline, and we tried changing the CTA, and we tried changing the colors, and the social proof that was on the right side of the page, right? Then we started doubling down into the bite- sized stuff. So when you're thinking about," Should I bundle a bunch of changes together, or should I make one single change to isolate the variable?", the important thing to think about is how far below where we want this number to be, are we right now? Right? So if you're trying to get a conversion rate to 10%, and right now it's at 2. 5%, you should probably take a bunch of big swings, right? You're not going to be able to just make bite- sized changes over and over. I mean, you can, it'll just take you a long time to make those changes over and over to go from 2. 5% to 3%, to 3. 7% to 4. 6%, right? There's going to be a lot of trial and error along the way. So when you're thinking about," What should I bundle here?" Look at the difference between where you are now to where you want to be, and if that is like a two to three X type change, or a very significant raise in a conversion rate, if you're at a 70% conversion rate and you're trying to get it to a 77% conversion rate, you should probably make some bigger changes and make big swings, because that's where you're going to find all the value of a new variable. Now, of course, you're going to exit the experiment and say," I don't know, we changed a bunch of stuff. What was it that worked for really well?" And that's when you can then back into," All right, let's now swap out the headline and see if that is it." But at the end of the day, most of the time, it kind of won't matter, because you have a new normal, you have a new baseline, and then you get to build off of that. So if you're trying to get incremental improvements, then make bite- sized changes, right? Those will get you there, without risking losing a huge drop- off rate. But if you're way below where you want something to be, then you just got to keep taking those big swings until something works, and realistically, you'll try seven different things, and maybe one of them will work. Right? The example that I just talked through on our Powered by pages with the no header, no footer, all that, that was one big swing in a bucket of five different big swings that we tried all at once, it just happened to be the one that worked really well, but it wound up being three times better because it was so different and such a big swing, that it wound up panning out. So that's what I got for today. Thank you so much for listening. If you have ideas, feedback, anything at all, my email is matt @ drift. com. I would absolutely love to hear from you and hear what I should be talking about next, and I will catch you on the next episode. Thanks so much. Bye.


Today on #Growth, Matt is talking all about experimentation. Specifically how to approach changes you might make to your site or product – bite-sized or incremental changes vs. going all out with big swings. Matt breaks this down with a real-world example you've probably faced yourself – updating your homepage. There are pros and cons to both and Matt lays it all out here.